Directory Busting in Kali Linux

One of the first things that Pentesters do when attempting their thing is to create a site map or a directory tree of the website. You see, it’s not always obvious where certain pages are. If we weren’t told that certain web pages existed (like a login page), we wouldn’t even look for them. Directory busting is the process of discovering existing but hidden directories and files on a website. However, doing this manually can be tedious and laborious. As such, we must resort to existing tools that can do the job for us. This tutorial will learn the various ways one can make a directory bust a web page.

Wordlists

The pentester’s best friend is a good word list. But because we use them in so many different scenarios, we have different word lists for different scenarios. For example, you’d need a good password list when attempting to break a password. To date, rockyou is the list that most pentesters use for password cracking. Rockyou is a list of actual passwords obtained by breaking websites/companies; the point is that these passwords are real.

Similarly, when attempting to design the directory tree of a website, you need a good word list to help you find the appropriate branch of the tree. What exactly am I trying to say?? You’re going to need a super good password list for each scenario for directory busting. And thank the Lord, someone’s created one just for us; it’s called SecLists. Download it and keep it tucked away.

git clone https://github.com/danielmiessler/SecLists

Now that we have what we need, back to business – let’s design directory trees!

Package #1: Gobuster

Gobuster is a bruteforcing package. It can be used to bruteforce URLs (using dir), DNS subdomains (using dns), virtual host names on target web servers (using vhost), and open amazon s3 buckets (using s3).

Installing Gobuster

To install Gobuster (please note that you need > go 1.16.0):

For help with any of the commands, you can simply type:

Dir Search

Dir search is a specific type of search. To search for a site map or a URL search, you will need a word list which you can specify using the switch -w.

However, you can specify a lot of details while creating the site map:

Ex: gobuster dir -u https://mysite.com/path/to/folder -c ‘session=123456’ -t 50 -w common-files.txt -x .php,.html

You don’t have to specify everything, but you do need to specify at least the –url, and the –wordlist. Further information can be found at https://github.com/OJ/gobuster.

For example, let’s try this on linuxhint.com:

Ex: gobuster dir -u https://linuxhint.com -w /usr/share/dirb/wordlists/common.txt

You can play around with the word lists from SecLists and use various switches.

For example, let’s say I’m after all the pages ending with a php:

And mind you, this is the beginning; there are tons of other options that you can explore!!

Package #2: DirBuster

DirBuster, written in java, is used to bruteforce web application servers to find hidden files and folders. It works by sending GET requests and waiting for a response. It then notes the response code and the directory file. A response code of 200 means success.

You will find DirBuster GUI in the Launcher. Please note here that it’s not a CLI but rather a GUI!

To install it:

Once you start DirBuster, you will be asked for a host; you have to specify the full URL and the port.

• So, for example: https:linuxhint.com:443
• Select the SecList list that you want. Click on Browse and select your Word List.
• Under the extension section, fill out the extension that you’re after. For example, it can be php or html.

(Click on the “Go Faster” button if you’d like it to be quick.)

It will take some time for the report to be generated; however, when it’s finished, you can click on the “report” button; the latter will generate a report. On the next screen, you will have to specify the details of the report type, the location where you want to save it, and the name you want to call it, and then click on “Generate Report”. This will generate the report and save it.

The part that I like about this particular tool is the Results – Tree View. This will give you the website’s structure. Mind you; there will be limitations – your tree is only as good as your word list and the specifications that you give it.

I stopped the process after a minute or two, but this is what I got in that period:

Package #3: DirSearch

DirSearch is a CLI bruteforcer of web applications to find hidden files and folders.

To install it:

or

To use it:

Ex: python3 dirsearch.py -e php,html,js -u https://linuxhint.com

Or if you installed it using sudo apt install:

Ex: dirsearch -e php,html,js -u https://linuxhint.com –include-status=200

Please note here that –include-status=200 means that it should include only those directories and files that get a response code of 200.

As you might have noticed, LinuxHint.com is a WordPress site with a login page at wp-login.php. This is the kind of information that you’ll get by directory busting. But, as you can see, there are other pages too, some of which are attack vectors for Pentesters.

Package #4: DirB

DirB is another package just like gobuster, DirBuster, and DirSearch.

To install it:

To use it:

Ex: dirb https://linuxhint.com /usr/share/dirb/wordlists/common.txt -N 301

Here, it will use the wordlist called common.txt (but this is up to you); it will ignore response code 301.

For more switches and their functions, see the help page.

These are the major packages for directory busting, but there are a few more out there!

Directory busting is a technique used by all Pentesters (and the bad guys) to find out what directories and files can be found within a site of interest. It can be used to create a directory tree; it can be used to find hidden pages. There are many packages available for the job, amongst which gobuster, DirBuster, DirSearch, and DirB. In this tutorial, we reviewed these directory busting packages. The best among them is always gobuster since it offers a very large selection of switches to play around with, but the rest aren’t bad. So use them carefully and wisely to protect your sites.

Happy Coding!