Google, Yahoo, MSN, and Ask have got together and announced a new robots.txt feature, sitemap auto-discovery.

“The new open-format autodiscovery allows webmasters to specify the location of their sitemaps within their robots.txt file, eliminating the need to submit sitemaps to each search engine separately“.

What are site-maps?

A sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site. More information here. Formatting guidelines are here.

What is the robots.txt specification for a sitemap?

Sitemap: <sitemap_location>