Web Content Discovery

The Hidden Content Problem

Web applications contain pages, files, and directories that are not linked from the main navigation. Administrative panels, debug endpoints, API documentation, and configuration files exist on the server but are hidden from casual browsing. Content discovery is the process of finding these hidden resources by systematically testing paths against the server.

How Directory Brute-Forcing Works

You provide a wordlist of common directory and file names. The tool appends each word to the target URL and sends a request. If the server returns a 200 (or other non-404 status), the path exists.

# Using ffuf
ffuf -u https://target.example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -t 50 -mc 200,204,301,302,307,401,403

# Using gobuster
gobuster dir -u https://target.example.com -w /usr/share/wordlists/dirb/common.txt -t 50

# Using feroxbuster (Rust-based, fast)
feroxbuster -u https://target.example.com -w /usr/share/wordlists/dirb/common.txt -t 50

Wordlist Selection

The quality of your content discovery depends on your wordlist. Common wordlists:

• dirb common.txt — 4,600 common directories (good starting point)
• dirbuster directory-list-2.3-medium.txt — 220,000 entries (comprehensive)
• raft-large-words.txt — Large wordlist from real-world data
• SecLists Discovery/Web-Content — Curated by security professionals

# SecLists comprehensive scan
ffuf -u https://target.example.com/FUZZ \
  -w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-big.txt \
  -t 100 -mc 200,204,301,302,307,401,403 -o results.json

Response Code Filtering

Not all responses indicate existing content. Filter by status code:

• 200 — Page exists and is accessible
• 301/302 — Redirect (may indicate existing content that redirects to login)
• 401/403 — Authentication required or forbidden (content exists but access is restricted)
• 404 — Not found (filter these out)

# Only show interesting status codes
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -mc 200,301,302,401,403

File Extension Search

Search for specific file types that may expose sensitive information:

# Search for backup files
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -x .bak,.old,.save,.swp,.txt,.sql

# Search for configuration files
ffuf -u https://target.example.com/FUZZ -w config-wordlist.txt -x .conf,.config,.ini,.env

Recursive Discovery

Once you find a directory, scan inside it recursively:

# Recursive scan with gobuster
gobuster dir -u https://target.example.com -w wordlist.txt -r -recursion-depth 3

# ffuf with recursion
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -recursion -recursion-depth 3

This discovers nested paths like /admin/panel/users that would not be found in a single-level scan.

Why Backup Files Matter

Developers often leave backup files on production servers. These files contain source code, configuration, or database dumps that should never be publicly accessible. Finding a backup file is often the fastest path to compromising an application.

Common Backup File Patterns

# Backup file extensions
.bak .old .save .swp .swo .tmp .copy .orig .backup

# Common backup filenames
index.php.bak
index.php~
index.php.old
config.php.bak
database.sql
backup.zip
site.tar.gz
web.config.bak
.env.bak

# Editor swap files (vim, emacs)
.index.php.swp
.index.php.swo
.config.php.swp

Searching for Backup Files

# Search for backup files with ffuf
ffuf -u https://target.example.com/FUZZ -w filenames.txt -x .bak,.old,.save,.swp,.tmp

# Using gobuster with specific extensions
gobuster dir -u https://target.example.com -w wordlist.txt -x .bak,.old,.sql,.zip

Source Code Leaks

Source code leaks reveal the application's internal logic, database credentials, and potential vulnerabilities:

# Git repository exposed
curl -s https://target.example.com/.git/config
curl -s https://target.example.com/.git/HEAD

# SVN repository exposed
curl -s https://target.example.com/.svn/entries

# Environment file
curl -s https://target.example.com/.env

# Configuration files
curl -s https://target.example.com/config.php.bak
curl -s https://target.example.com/web.config

Downloading and Analyzing Leaked Files

When you find a backup file, download and analyze it:

# Download the file
curl -s -o config.php.bak https://target.example.com/config.php.bak

# Analyze for credentials
grep -i 'password\|secret\|key\|token' config.php.bak

# Check for database credentials
grep -i 'db_host\|db_user\|db_pass\|database' config.php.bak

Practical Example: Finding a Database Backup

You are testing target.example.com. You run:

ffuf -u https://target.example.com/FUZZ -w wordlist.txt -x .sql,.bak,.zip -mc 200,301,302

Results show:

• /backup.sql — 200 OK, 2.3 MB
• /db_backup.zip — 200 OK, 1.8 MB
• /config.php.bak — 200 OK, 4 KB

You download config.php.bak and find:

<?php
define('DB_HOST', 'localhost');
define('DB_USER', 'admin');
define('DB_PASS', 'Sup3rS3cr3t!');
define('DB_NAME', 'production');
?>

You now have database credentials. You download backup.sql and find the entire user database with hashed passwords. This is a critical finding.

Preventing Backup File Exposure

From a defensive perspective, backup files should:

• Never be stored in web-accessible directories
• Be stored outside the document root
• Be protected by access controls
• Be deleted after use
• Be excluded from version control (.gitignore)

The Wildcard 404 Problem

Some web servers are configured to return a 200 status code (or a custom 404 page) for any URL, even non-existent ones. This is called a wildcard 404 or soft 404. It makes content discovery nearly impossible because every path appears to exist.

Detecting Wildcard 404s

Before starting content discovery, test with a random path:

# Request a path that should not exist
curl -s -o /dev/null -w "%{http_code}" https://target.example.com/this-path-should-not-exist-12345

# If it returns 200, you have a wildcard 404 problem

Filtering by Response Size

When wildcard 404s return 200, filter by response size. Real pages have different sizes than the wildcard 404 page:

# Get the size of a wildcard 404 response
WILDCARD_SIZE=$(curl -s https://target.example.com/random12345 | wc -c)

# Filter results that differ from wildcard size
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -fs $WILDCARD_SIZE

Filtering by Response Content

Some wildcard 404 pages contain specific text. Filter by content:

# Filter responses containing "Page not found"
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -fr "Page not found"

# Filter by regex pattern
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -fr "404|not found|error"

Using Calibration

Calibrate your content discovery tool against the target:

# feroxbuster auto-calibrates by testing random paths
feroxbuster -u https://target.example.com -w wordlist.txt --auto-calibration

# ffuf with calibration
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -ac

Advanced Filtering Techniques

Time-based filtering: Some applications respond faster to non-existent paths:

# Filter by response time
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -ft 100  # filter responses under 100ms

Content length ranges:

# Only show responses with content length between 1000 and 50000
ffuf -u https://target.example.com/FUZZ -w wordlist.txt -fl 1000,50000

Practical Example: Overcoming Wildcard 404s

You scan target.example.com and every path returns 200. You:

1. Detect the problem: curl https://target.example.com/random12345 returns 200 with 1,234 bytes
2. Calibrate: Run feroxbuster -u https://target.example.com -w wordlist.txt --auto-calibration
3. Filter: The tool automatically filters responses matching the wildcard 404 signature
4. Discover: You find /admin (4,567 bytes), /api/docs (8,901 bytes), and /backup.zip (2,345,678 bytes) — all different from the 1,234-byte wildcard response

Content Discovery Best Practices

• Always calibrate before starting — detect wildcard 404s first
• Use multiple wordlists — different wordlists find different content
• Scan recursively — nested directories hide valuable content
• Try multiple extensions — .php, .asp, .jsp, .bak, .old
• Vary user-agents — some content is only served to specific browsers
• Check for case sensitivity — Linux servers are case-sensitive; /Admin ≠ /admin