The pentester’s best friend is a good word list. But because we use them in so many different scenarios, we have different word lists for different scenarios. For example, you’d need a good password list when attempting to break a password. To date, rockyou is the list that most pentesters use for password cracking. Rockyou is a list of actual passwords obtained by breaking websites/companies; the point is that these passwords are real.
Similarly, when attempting to design the directory tree of a website, you need a good word list to help you find the appropriate branch of the tree. What exactly am I trying to say?? You’re going to need a super good password list for each scenario for directory busting. And thank the Lord, someone’s created one just for us; it’s called SecLists. Download it and keep it tucked away.
Now that we have what we need, back to business – let’s design directory trees!
Package #1: Gobuster
Gobuster is a bruteforcing package. It can be used to bruteforce URLs (using dir), DNS subdomains (using dns), virtual host names on target web servers (using vhost), and open amazon s3 buckets (using s3).
To install Gobuster (please note that you need > go 1.16.0):
sudo apt install gobuster
For help with any of the commands, you can simply type:
Dir search is a specific type of search. To search for a site map or a URL search, you will need a word list which you can specify using the switch -w.
However, you can specify a lot of details while creating the site map:
Ex: gobuster dir -u https://mysite.com/path/to/folder -c ‘session=123456’ -t 50 -w common-files.txt -x .php,.html
-c, --cookies <cookies>
-t, --threads <int>
-w, --wordlist <word list>
-x, --extensions <extensions separated by commas>
You don’t have to specify everything, but you do need to specify at least the –url, and the –wordlist. Further information can be found at https://github.com/OJ/gobuster.
For example, let’s try this on linuxhint.com:
Ex: gobuster dir -u https://linuxhint.com -w /usr/share/dirb/wordlists/common.txt
You can play around with the word lists from SecLists and use various switches.
For example, let’s say I’m after all the pages ending with a php:
And mind you, this is the beginning; there are tons of other options that you can explore!!
Package #2: DirBuster
DirBuster, written in java, is used to bruteforce web application servers to find hidden files and folders. It works by sending GET requests and waiting for a response. It then notes the response code and the directory file. A response code of 200 means success.
You will find DirBuster GUI in the Launcher. Please note here that it’s not a CLI but rather a GUI!
To install it:
Once you start DirBuster, you will be asked for a host; you have to specify the full URL and the port.
- So, for example: https:linuxhint.com:443
- Select the SecList list that you want. Click on Browse and select your Word List.
- Under the extension section, fill out the extension that you’re after. For example, it can be php or html.
(Click on the “Go Faster” button if you’d like it to be quick.)
It will take some time for the report to be generated; however, when it’s finished, you can click on the “report” button; the latter will generate a report. On the next screen, you will have to specify the details of the report type, the location where you want to save it, and the name you want to call it, and then click on “Generate Report”. This will generate the report and save it.
The part that I like about this particular tool is the Results – Tree View. This will give you the website’s structure. Mind you; there will be limitations – your tree is only as good as your word list and the specifications that you give it.
I stopped the process after a minute or two, but this is what I got in that period:
Package #3: DirSearch
DirSearch is a CLI bruteforcer of web applications to find hidden files and folders.
To install it:
pip3 install -r requirements.txt
python3 dirsearch.py -u <URL>> -e <EXTENSIONS>
To use it:
Ex: python3 dirsearch.py -e php,html,js -u https://linuxhint.com
Or if you installed it using sudo apt install:
Ex: dirsearch -e php,html,js -u https://linuxhint.com –include-status=200
Please note here that –include-status=200 means that it should include only those directories and files that get a response code of 200.
As you might have noticed, LinuxHint.com is a WordPress site with a login page at wp-login.php. This is the kind of information that you’ll get by directory busting. But, as you can see, there are other pages too, some of which are attack vectors for Pentesters.
Package #4: DirB
DirB is another package just like gobuster, DirBuster, and DirSearch.
To install it:
To use it:
Ex: dirb https://linuxhint.com /usr/share/dirb/wordlists/common.txt -N 301
Here, it will use the wordlist called common.txt (but this is up to you); it will ignore response code 301.
For more switches and their functions, see the help page.
These are the major packages for directory busting, but there are a few more out there!
Directory busting is a technique used by all Pentesters (and the bad guys) to find out what directories and files can be found within a site of interest. It can be used to create a directory tree; it can be used to find hidden pages. There are many packages available for the job, amongst which gobuster, DirBuster, DirSearch, and DirB. In this tutorial, we reviewed these directory busting packages. The best among them is always gobuster since it offers a very large selection of switches to play around with, but the rest aren’t bad. So use them carefully and wisely to protect your sites.