# All robots will spider the whole domain but the folders listed below User-agent: * Disallow: /cgi-bin/ Disallow: /de.backup/ Disallow: /LCK/ Disallow: /old_stuff/ Disallow: /previous version/ Disallow: /test/