Vulnerability Overview
robots.txt is a publicly readable file that instructs search engine crawlers which paths to skip. Paths listed under Disallow are not restricted from human visitors — they are fully accessible and, by being listed, are actively advertised to anyone who checks the file.
Exploitation
Visiting /robots.txt revealed:
Disallow: /cc6b1.htmlNavigating to /cc6b1.html directly returned the flag.
Key Insight:
robots.txt is a voluntary standard for search engine behavior — it has no enforcement mechanism. It is one of the first files a security researcher or attacker checks on any engagement because developers consistently use it to hide paths they consider sensitive. If a resource requires restricted access, implement authentication — do not rely on its absence from search index results.