In case you happen to be a webmaster then you will always like your website to be seen in search results. Your website and its various web pages must be crawled and indexed by search engine bots in order to be shown in search results.
The code of your website consists of two different files known as: Robots.txt and Sitemap
Robots.txt is located in your site’s root directory which contains simple text in it.
This file tells the search engine robots what to crawl and what not to crawl. This file also consists of commands that suggest which search engine robots are allowed to crawl and which are not.
So, what happens exactly when search bots look for robots.txt file in a website. Therefore, it is important to have a robots.txt file in the initial place. A default robots.txt allows all search robots to crawl all pages on your site.
One more important information that Robots.txt contain is information about Sitemap.
What is a sitemap? A Sitemap is an XML file containing a list of all web pages in your site. It may also have the additional information of each URL in the form of Meta data. Taking the help of a sitemap search engine bots explore, crawl and index all the webpages in a site.
Are Robots.txt and sitemaps related?
It was in the year 2006 when, Yahoo, Microsoft and Google collectively supported standardized protocol of submitting webpages to as site through the channel of sitemaps. You can submit your sitemaps with the help of Google webmaster tools and Bing webmaster tools.
What is auto-discovery of sitemaps?
The system of finding the sitemap via robots.txt is termed as auto-discovery of sitemaps. It means that it was perfectly fine if you did not submit the sitemap to individual search engines. They are going to get the sitemap’s location from robots.txt file initially.
How To Create Robots.txt File With Sitemap Location?
1: By Locating Your Sitemap URL
In case a third party has developed your website then you should check initially whether they have provided your site with a sitemap or not. The URL to the sitemap of your site usually looks like this: http://www.example.com/sitemap.xml
In place of ‘example’ type this URL in our browser.
2: By Locating Your Robots.txt File
Just type domain.com/robots.txt in order to check whether your site has a robots.txt.
In case you don’t have a robots.txt file then you can create one file and this can be added to the root directory of web server.