XML Sitemap Submission via the Robots.Txt File
In March 2008, the "Sitemaps Protocol" was expanded to include the "AutoDiscovery of Sitemaps" using robots.txt to let the SEs supporting the protocol know where to find your Sitemaps. We subsequently also announced the all client support for Sitemap cross-submissions using Google Webmaster Tools.
The days of paid submission has ended.
Today, as a standard service, we have married the two, allowing for Sitemap cross-submissions using robots.txt. This concept presents a common path for your site's submission to Google, Yahoo, Bing! (MSN) and Ask. Now, most of the second tier SEs are following suit as well.
This is how it's done!
Sitemap Submission Via Robots.Txt File
Add the following dark code to your robots.txt file to automatically submit your site to all participating SEs.
Robots.txt File Coding Example
# Robots.txt file created by 10/03/2010
# For domain: http://www.catanich.com
#
# All other robots will spider the domain
User-agent: *
Disallow: /_common/
Disallow: /_private/
Disallow: /_ScriptLibrary/
Disallow: /aspnet_client/
Disallow: /_client/
Sitemap: http://www.catanich.com/sitemap.xml
Sitemap: http://www.catanich.com/sitemap.xml.gz
Sitemap: http://www.catanich.com/urllist.txt
Sitemap: http://www.catanich.com/urllist.txt.gz
Design Notes
Any blank line in the robots.txt file will cause an error. No blank lines.
