# Tells Scanning Robots Where They Are And Are Not Welcome # # User-agent: can also specify by name; "*" is for everyone # Disallow: disallow if this matches first part of requested path # # For now disallow all we can modify this as needed to allow certain crawlers. # # Site Description: # Central clearing house for anthropological resources and web services. # Site provides an anthropological resource database that covers many # categories. We also provide FREE and PAID web services. # Last updated: 04/24/02 # Known Bad Bots: User-agent: * Disallow: /.procmailrc Disallow: /.mysql_history Disallow: /cgibin/admin/ Disallow: /cgibin/career/ Disallow: /errors/ Disallow: /images/ Disallow: /logs/ Disallow: /mail/ Disallow: /mbox/ Disallow: /madagascar/images/ Disallow: /madagascar/plugins/ Disallow: /members/bin/ Disallow: /members/images/ Disallow: /members/plugins/ Disallow: /plugins/ Disallow: /vlib/bin/ Disallow: /vlib/images/ Disallow: /vlib/plugins/ Disallow: /bin/ User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Wiredon Disallow: / User-agent: SkullyAgent Disallow: / User-agent: INFOMINE Disallow: / # Disallow other Bots: