Search engines team up for site crawling

By Shaun Nichols on Nov 20, 2006 10:05 AM
Filed under Software

Google, MSN and Yahoo adopt standard to speed up search engine indexing.

Yahoo and Microsoft have partnered with Google to support the Sitemaps 0.90 protocol.

Sitemaps allow webmasters to craft an XML file containing information about recently updated pages. When a search engine's indexing program spiders the site, the Sitemap file informs it which pages require indexing.

The protocol was introduced in June 2005 by Google, which has published the technology under a Creative Commons licence.

The indexing system will now be used by Windows Live Search and Yahoo Search in addition to Google Search.

"The launch of Sitemaps is significant because it allows a single, easy way for websites to provide content and metadata to search engines," said Tim Mayer, senior director of product management at Yahoo Search.

Google said that the Sitemap system saves bandwidth for site owners by letting spiders request fewer pages. It also streamlines the indexing process, making updated content available on the search engines sooner.

The three companies have launched a website that explains the Sitemaps protocol and how to generate the Sitemap XML files.

Webmasters can either hand-code the file themselves or use a Sitemap-generator from either Google or a third-party developer.