Top Open-source intelligence (OSINT) Tools

Overview on Open-source intelligence (OSINT)

The first step in many penetration tests is to gather information about the organization via passive intelligence gathering methods. Passive methods are those that do not actively engage the target organization's systems, technology, defenses, people, or locations. The information gathered through this process is often called OSINT, or open source intelligence.

Through data mining, you can analyze large data sets to reveal patterns or hidden anomalies. An open-source framework that pentesters can use to aid in the data mining process is called the OSINT Framework. The OSINT Framework is a static web page focused on information gathering and provides web links and resources that can be used during the reconnaissance process. The OSINT Framework helps point users in the right direction to find useful intelligence from various public and paid resources.

There are several tools available at your disposal which can help you get alot of information which are publicly available on Internet using OSINT Tools such as Maltego, Shodan, Google Dorks, Recon-ng, Harvester, Mitaka etc.

Now I will try to cover some of the tools but you can also check these other articles written on other tools which are not covered in this article:

• How to perform in-browser OSINT using Mitaka
• Easy OSINT using infooze tool V 1.0 [With Examples]

 

1. Maltego CE: Relationship Enumeration

An interactive data mining software tool that can help users visualize and analyze relationships using publicly accessible data from the Internet is called Maltego.

Try to execute maltego from the terminal from the GUI console and if it isn't installed then the terminal will prompt you for installation:

sudo apt update
maltego

The version of Maltego we refer to in this chapter is Maltego CE, the community edition. More information about the versions of Maltego that are available is on the company website:

Before using Maltego, you will need to create an account and register for a Maltego license. Setup of Maltego is fairly straightforward. If you launch Maltego from the Kali main menu, you will see the splash screen appear as the application is loading.

To start mining some data, click the Maltego icon in the top left, and select New to start a new graph.

You can then click on Machines in the top menu to get the Machines toolbar to appear. Select Run Machine to choose an existing machine

For this exercise, choose Footprint L2 CE and select Next.

The Footprint L2 machine requires additional inputs. Enter in a domain name to use for the online investigation and then click Finish. Once the initial collection is complete, you should see some entities displayed on the graph

The Entity Palette describes each type of graphical icon and what it represents. Maltego allows you to toggle between layouts

. When you hover the mouse over the top of an entity, you will see the relationship definition and information sources listed in the Detailed View window. You can use this information to draw conclusions about the network configuration, such as subdomain information, IP address assignments, etc.

When you right-click on an entity on the graph, you can configure and run any number of applicable transforms to extend the search criteria and build upon established relationships

 

2. Google Dorks: Information Leakage Enumeration

Search engines are another place to seek OSINT about your targets. Dorks are queries that you can use to find data about your target. Most of us use search engines such as DuckDuckGo, Bing, and Google to locate information. What you might not know is that search engines, such as Google, can perform much more powerful searches than most people ever dream of. Google can translate documents, perform news searches, and do image searches. In addition, hackers and attackers can use it to do something that has been termed Google hacking.

Some search engines, such as Google, allow advanced query parameters to find information that a normal query might bury under thousands of results.

• Filetype: Directs Google to search only within the text of a particular type of file (for example, filetype:xls)
• Inurl: Directs Google to search only within the specified URL of a document (for example, inurl:search-text)
• Link: Directs Google to search within hyperlinks for a specific term (for example, link:www.domain.com)
• Intitle: Directs Google to search for a term within the title of a document (for example, intitle: “Index of.etc”)

Now that we have discussed some basic Google search techniques, let’s look at advanced Google hacking. Exploit Database hosts the Google Hacking Database, which is filled with dorks that work in Google.

Let’s see some examples how we can use Google dorks to find information!

• Search for all user with the name John Doe on Facebook using John Doe site: facebook.com
• Search for all articles with security in the URL and vulnerability as content using inurl:security intext:vulnerability
• Search for anything inside a specific site using site:www.golinuxcloud.com
• Search for all website with a specific extension and some string using site:*.at security

GHDB has the following search categories:

• Footholds
• Files containing usernames
• Sensitive directories
• Web server detection
• Vulnerable files
• Vulnerable servers
• Error messages
• Files containing juicy info
• Files containing passwords
• Sensitive online shopping info
• Network or vulnerability data
• Pages containing login portals
• Various online devices
• Advisories and vulnerabilities

GHDB is a community effort. Anyone can upload a new Google dork to perform these types of searches. Once you start playing with the dorks in GHDB, you will be surprised by the unbelievable things found through Google hacking. GHDB has made using Google dorks very easy, and there are other options as well.

 

3. Shodan: Device and Service Enumeration

Shodan is an organization that scans the Internet 24 hours a day, 365 days a year. The results of those scans are stored in a database that can be queried at shodan.io or by using an API. The main page provides a search box that can be used to examine content to find keywords or phrases. Advanced queries enable users to apply filters and boolean logic operators to drill down into the results. It works similarly to a typical web search engine such as Google, Yahoo!, or Bing.

Let's perform a quick search  for any data results that return with the keyword “telnet”

 As you can see, a substantial number of results were returned and categorized under Top Countries, Top Services, Top Organizations, Top Operating Systems, and Top Products. 

 One of the most useful features with Shodan is the ability to apply filters with the search criteria. If you wanted to find data with the keyword “telnet” along with any ports that have port 23 open and are located in the United States, we could apply the search phrase telnet port:23 country:"US" in the search box to narrow the search.

 

Below image shows a query performed to find network infrastructure devices that are running a broken protocol called Cisco Smart Install. Attackers have leveraged this protocol for years to compromise different infrastructures. Cisco removed this protocol from its systems many years ago. However, many people are still using it in devices connected to the public Internet.

Cisco has warned customers for many years about the misuse of this feature in an informational security advisory that can be accessed at Cisco Smart Install Protocol Misuse

Keep in mind that even though this is public information, you should not interact with any systems shown in Shodan results without permission from the owner. If the owner has a bug bounty program, you may get recognition and a reward for finding the affected system.

You can read more at another article written on the same topic https://www.golinuxcloud.com/open-source-intelligence-osint-tools/#2_Shodan

 

4. Recon-ng: Network and User Enumeration

Recon-ng is a modular framework, which makes it easy to develop and integrate new functionality. It is highly effective in social networking site enumeration because of its use of application programming interfaces (APIs) to gather information. It also includes a reporting feature that allows you to export data in different report formats.

Here is a table which the list of some of the commands which you will be using on recon-ng:

Now recon-ng is installed by default on Kali Linux so let's start by accessing the terminal:

As you can see on the console, by default there are no modules installed. You can use following command to get the list of available modules:

marketplace search

The letter D in the table header in the output dictates that the module has dependencies. The letter K indicates that an API key is needed in order to use the resources used in a particular module.

You can refresh the data about the available modules by using the marketplace refresh command

To install them, you may need to type marketplace install all at the Recon-ng prompt. After you have done this, the startup screen shows the total number of modules that are supported for each category. 

The Recon-ng module categories are:

• Recon modules  Reconnaissance modules.
• Disabled modules Modules that are disabled based on your system configuration. If you have none that are disabled, this may not show up.
• Reporting modules  Compile a report in various formats.
• Import modules  Import target listing using supported formats.
• Exploitation modules  Supported exploitation modules.
• Discovery modules  Informational discovery modules.

Now lets perform a practical example to find Point of Contact (POC) using the whois_pocs module.

The first thing to do is create a workspace to manage information collection. The workspace and information gathered during module execution will be stored in the database.

workspaces create example

To see a list of workspaces in the database, execute workspaces list:

 To run a simple WHOIS query and pull contact information for a domain, load the whois_pocs module:

modules load whois_pocs

Once the module is loaded, you can see more information about the module, including configurable parameters, by typing info. You can also show the current context variables by typing options while the module is loaded. This also lets you set values. In this case, set your search target:

options set SOURCE comptia.org

When you execute the module using the run command, the discovery process could take a few minutes,

Once the collection process has completed, the show <table name> command can be used to display the results stored in select tables within the database. For a list of all tables, type show with no arguments.

[recon-ng][example][whois_pocs] > show
Shows various framework items

Usage: show <companies|contacts|credentials|domains|hosts|leaks|locations|netblocks|ports|profiles|pushpins|repositories|vulnerabilities>

Let's list the contacts information

[recon-ng][example][whois_pocs] > show contacts

  +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | rowid | first_name | middle_name |       last_name       |           email           |     title     |         region        |    country    | phone | notes |   module   |
  +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | 1     |            |             | COMPTIA Administrator | administrator@comptia.org | Whois contact | Downers Grove, IL     | United States |       |       | whois_pocs |
  | 2     | Jeff       |             | Rosen                 | jrosen@comptia.org        | Whois contact | Downers Grove, IL     | United States |       |       | whois_pocs |
  | 3     | Randall    |             | Palm                  | rpalm@comptia.org         | Whois contact | Oakbrooke Terrace, IL | United States |       |       | whois_pocs |
  | 4     | Robert     |             | Rohrman               | rrohrman@comptia.org      | Whois contact | Downers Grove, IL     | United States |       |       | whois_pocs |
  +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[*] 4 rows returned

You can also add APIs to recon-ng to be able to use more set of modules for different type of queries. For example let me show you how to get API Key from Shodan.

First of all visit the official website of Shodan and create a free account using any available email address. Once you have signed up and activated your account by validating the email address, you can head over to your profile area and search for API Key section. Click on Show to view your key.

Copy the API key and head over to recon-ng terminal. Execute the following command along with the copied API key:

keys add shodan_api YOUR_SHODAN_API_KEY

You can verify if your key is added to the Recon-ng DB by using keys list:

Similarly you can add APIs from other servers to be able to search through their database and use more modules.

You can refer https://github.com/lanmaster53/recon-ng-marketplace/wiki/API-Keys on steps to acquire API KEY.

 

5. theHarvester: Email and subdomain enumeration

theHarvester is another excellent tool for pentesters to perform DNS recon. This tool can gather information from various web pages (such as search engines) in order to discover domains, subdomains, and e-mail addresses. 

theHarvester is installed by default in Kali Linux so let's start using it directly. The very basic and most used syntax is to perform a search for a specific domain using some data source.

theHarvester -d <domain-to-search> -b <data-source>

You can check the help section to get the list of supported data source.

  -b SOURCE, --source SOURCE
                        anubis, baidu, bevigil, binaryedge, bing, bingapi, bufferoverun, brave, censys, certspotter, criminalip, crtsh, dnsdumpster, duckduckgo, fullhunt, github-code, hackertarget,
                        hunter, hunterhow, intelx, netlas, onyphe, otx, pentesttools, projectdiscovery, rapiddns, rocketreach, securityTrails, sitedossier, subdomaincenter, subdomainfinderc99,
                        threatminer, tomba, urlscan, virustotal, yahoo, zoomeye

At the time of writing this article I was using 4.5.1 version wherein Google as a data source was removed. So we will use bing for our example, let's try to search for all information related to "comptia.org" using Bing

theHarvester -d comptia.org -b bing

As you can see we have found a list of email address and hosts related to comptia.org on Bing.

There are multiple options such as listed below which you can combine with theHarvester command to further refine and control the output. Here are some examples but you can always use theHarvester --help to get the complete list of supported options:

Set limits to your results.

theHarvester -d example.com -l 400 -b google

Save the result in HTML file by using -f option.

theHarvester -d example.com -b google -f results.html

You can also perform a DNS Brute Force search using --dns-brute-force as used below:

theHarvester --dns-brute -d comptia.org

This command will perform a DNS brute force search on the domain "comptia.org" to discover subdomains or hostnames by guessing them. The tool uses a list of common subdomain names and tries each one to see if it resolves to an IP address, thereby identifying potentially unlisted or obscure subdomains associated with the main domain.

esx1.comptia.org:
drupal.comptia.org:
zion.comptia.org:
cbf1.comptia.org:
vologda.comptia.org:
nissan.comptia.org:
www.portfolio.comptia.org:
moodle.comptia.org:
cs.comptia.org:
enet1.comptia.org:
interactive.comptia.org:
holiday.comptia.org:
culture.comptia.org:
...

Here now we see there are many sub domain names using comptia.org.