The Official GoogleBlog published a post regarding the Robots Exclusion Protocol (REP). REP allows you even more flexibility now. take a look at these highlights:

  • CONTENT=”unavailable_after:” - a great way of letting the Search engines know when your content expires / is not relevant anymore
  • X-Robots-Tag: noarchive, nosnippet - do you have pages that you don’t want to be archived? not shown in the sippets?
  • CONTENT=”NOFOLLOW” - don’t let the Search engines follow the links

Read these three posts if you want to know more:

  1. Controlling how search engines access and index your website
  2. The Robots Exclusion Protocol
  3. Robots Exclusion Protocol: now with even more flexibility

admin :: Jul.30.2007 :: SEO, google, robots.txt :: No Comments »