# robots.txt for all virtual hosts under twp control. bdp, 2003-Sep-24. # Intent is to keep everybody from crawling our site. Some crawlers may ignore this request though. # This will have to be modified in the event we ever index our own site # # 19-May-2005, bdp, allow ARM to index bcr.arm.gov # 19-Jul-2005, bdp, modif to allow ARM googlebot # 30-May-2007, bdp, temp allow dev.google.arm.gov # allow ARM's crawler to index this site User-agent: arm-crawler Disallow: User-agent: dev.google.arm.gov Disallow: # disallow everything else User-agent: * Disallow: /