When XSitePro publishes your Web site it uploads a special file called ‘robots.txt’. This file communicates, to passing Search Engine Spiders, a list of those pages on your site that you are happy for Search Engines to index, and those you are not.
Now you might be thinking “why wouldn’t I want all my pages indexed?”
The simple reason is that search engines are not interested in your functional pages (things like Thank-you pages, success pages, download pages for example). In fact, these pages can actually detract from your site’s overall performance in the search engine – they’re effectively diluting the impact of your quality content-led pages.
Similarly, you may not want certain pages on your site indexed. For example, if you are offering a free PDF download as a ‘thank you’ for visitors filling out a name-grab form, you would not want that ‘Free PDF download’ page indexed as this would inevitably draw traffic directly (circumventing your name-grab form in the process).
So we need a way of indicating whether or not individual pages can be indexed or not – and that’s just what this Robots tab allows us to do. Unlike the ‘Robots’ button on the Other tab (which gives a one-stop overview of all your site’s pages) this Robots tab, in the advanced page settings window, gives control at an individual page level.
To load this tab, within the Advanced Page Settings window, click the “Robots” tab.
There are three check-boxes in the screen, and they have the following properties:
Allow robots to visit this page (robots.txt) – This checkbox, if checked, adds the current page to the list of pages within the Robots.txt file, so that search engine spiders can ‘see’ your page. By default this is checked – uncheck only if you do not want visiting search engine spiders to ‘see’ this page.
Index this page (robots meta tag) – This checkbox controls whether or not the current page is assigned a meta tag (in the HTML code) that tells the visiting search engine spider to index this Web page. Unchecking this box will specify that you do not want this page indexed.
Why have both ‘robots.txt’ and a ‘robots meta tag’?
If a search engine finds a page that (although not listed for indexing, in the robots.txt file) is linked to from another page on your site then you need a way of informing at a page level, that you do not wish this page to be indexed. This is what your ‘robots meta tag’ achieves.
Follow links on this page (robots meta tag) – As well as specifying whether or not the current page should be indexed, the robots meta tag can also inform the visiting search engine spider whether or not the links, contained on this Web page, should be followed. That is to say, there may be times when you do not want the search engines continually trawling pages linked to or from this current page.
To inform search engine robots that all of the links on this page should be followed, check the checkbox. To prevent links being followed, simply uncheck the checkbox.
One example of such a situation where you would want to prevent links being followed might be if you were developing a new site and (while the pages are live) you may not wish to have the pages indexed and/or links followed.
When using this Robots tab, any settings you change will show through into the ‘global’ robots module, launched from the Other Tab. You therefore have multiple ways of editing your pages’ robots attributes.
If you do not wish to make any changes on any of the other advanced page settings tabs, for this page, you should click the button to save your changes and return to the Page Settings screen in XSitePro.
Note: Until you click this button (even if you navigate to other tabs in the advanced settings window) your changes will not have been committed.