Internet Marketing Services
|
SEO Tip - Sitemap Submission Via Robots.Txt File
In March 2008, the Sitemaps protocol was expanded to include the autodiscovery
of Sitemaps using robots.txt to let the SEs supporting the
protocol know about your Sitemaps. We subsequently also announced support for
Sitemap cross-submissions using Google Webmaster Tools, making it possible to
submit Sitemaps for multiple hosts on a single dedicated host. So it was only
time before we took the next logical step of marrying the two and allowing
Sitemap cross-submissions using robots.txt. And today we're doing just that.
Sitemap Submission Via Robots.Txt File
Add the following code to your robots.txt file to automatically submit
your site to all participating SEs.
Robots.txt File Coding Example
# Robots.txt file created by 10/03/2008
# For domain: http://www.catanich.com
#
# All other robots will spider the domain
User-agent: *
Disallow: http://www.catanich.com/sitemap.xml
Disallow: http://www.catanich.com/sitemap.xml.gz
Disallow: /Tex-Mex-Recipes%5C_vti_cnf/
Disallow: /Tex-Mex-Recipes\_vti_cnf/
Disallow: /_common/
Disallow: /_private/
Disallow: /_ScriptLibrary/
Disallow: /aspnet_client/
Disallow: /_client/
Sitemap: http://www.catanich.com/site-map.xml
Sitemap: http://www.catanich.com/site-map.xml.gz
Sitemap: http://www.catanich.com/urllist.txt
Sitemap: http://www.catanich.com/urllist.txt.gz
Note
Any blank line in the robots.txt file will cause an error. No blank lines.
|