Helping search engines index your content
Google has launched SiteMaps to help webmasters make sure that Google –and other search engines– can find and crawl their content. The best source of details so far is Danny Sullivan’s interview with Shiva Shivakumar in yesterday’s SearchEngineWatch. Shivakumar is the engineering director and technical lead for Google Sitemaps. Excerpt (quoting Shivakumar): ‘[How does this work?] Webmasters create XML files containing the URLs they want crawled, along with optional hints about the URLs such as things like when the page last changed, and the rate of change. They host the Sitemap on their server and tell us where it is. We provide an open-source tool called Sitemap Generator to assist in this process. Eventually, we are hoping webservers will natively support the protocol so there are no extra steps for webmasters. When a Sitemap changes, we support auto-notifying us so we can pick up the newest version….At this early stage, we cannot guarantee that we’ll crawl or index all your URLs. But as we understand the data better, we hope to get more of the data into our crawl and indices….[Is this free?] Absolutely. Also, this is an open protocol [under a Creative Commons Attribution/Share-Alike license]. We are hoping all webservers and search engines adopt this protocol and benefit from the increased collaboration.’
(PS: OA repositories and journals should definitely try this.)
