Crawling and indexing: what actually blocks you

The order to work in

Technical work has a natural order and doing it out of order wastes time. Discovery before crawling, crawling before indexing, indexing before anything to do with quality. A page that is not discovered cannot be blocked by a robots file, and a page blocked by a robots file will not be helped by better writing.

The page indexing report in Search Console gives you the whole picture in one screen, grouped by reason. Read it once a month and act on the categories rather than the totals. A site with a large number of excluded pages is not necessarily unhealthy; the question is always which pages and why.

Nothing links to it

The commonest fault and the least discussed. Pages added through a booking system, pages built as landing pages for a campaign, pages that used to be in a menu and were quietly removed. Each is still live and reachable by anybody with the address, and invisible to everybody else.

The fix is structural rather than technical: every page should have a parent, per the site plan in Module 09, Information architecture: the site plan, and should be linked from it. Module 23, Internal linking as structure deals with internal linking as a discipline. In the meantime, the practical test is to browse your own site from the homepage and try to reach every page. Anything you cannot reach is a page you have accidentally hidden.

robots.txt and what belongs in it

The robots file tells crawlers which paths they may fetch. For a clinic site it should be short and it should be dull. A workable one allows everything, disallows the small number of paths that genuinely should not be crawled, and names the sitemap.

Two mistakes to avoid. First, blocking CSS or JavaScript directories, which prevents the crawler from rendering the page as a user sees it and can make a perfectly good page look broken. Second, using robots.txt to keep a page out of the index, which does not work: a blocked page can still be indexed from links elsewhere, and because it cannot be fetched, any noindex directive on it is never seen. Google's documentation on robots.txt is explicit about this and it is the single most common misunderstanding in the whole area.

The staging noindex

A new site is built on a staging address, which is quite properly set to noindex so that the half-finished version does not appear in results. Then it goes live and somebody forgets to remove the directive. The site is perfect, the writing is excellent, and nothing at all happens for months.

This is the most damaging single fault in this module and it is entirely invisible from the front of the site. It is also the first thing to check on any site that launched recently and has produced nothing. The check takes ten seconds: view the source of the homepage and search for the word noindex, or use URL inspection, which states it plainly.

Any time a clinic tells you their new site has produced nothing at all, check for a noindex before believing any other explanation.

Canonicals pointing elsewhere

A canonical tag tells a search engine which version of a page is the one to index. Used correctly it resolves duplication. Used carelessly it tells the engine to ignore the page entirely.

Two failures recur on clinic sites. A template that puts the same canonical on every page, usually the homepage, which asks the engine to discard the whole site. And a canonical carried over from a duplicated page, so that a new treatment page declares itself a copy of the one it was duplicated from. Both are template faults rather than page faults, which means finding one usually means finding several.

Rendering

If the content of your pages is inserted by JavaScript, a crawler has to run that JavaScript to see it. Usually it can. Sometimes it cannot, or the script depends on something that fails, or the content only loads after an interaction such as a click on a tab.

The last is the practical one for clinics: treatment details, price tables and FAQ answers hidden behind accordions or tabs are generally fine if the text is in the page and merely hidden by CSS, and generally invisible if the text is fetched only when the tab is clicked. The distinction is not visible to a person and is decisive for a crawler.

The test is the rendered HTML in URL inspection, not the browser. If your price table is not in the rendered HTML, it does not exist as far as search is concerned.

Sitemaps that tell the truth

A sitemap is a list of the pages you want indexed. It is not a ranking device and it does not guarantee indexing. What it does is help discovery, particularly on a small site with few incoming links, which describes nearly every clinic.

Three rules. Include only pages you want indexed, which means no redirected URLs, no noindexed pages and no error pages. Keep the last-modified dates truthful, because a sitemap where every page claims to have changed today is a sitemap nobody will trust. And regenerate it when the site changes rather than maintaining it by hand, which nobody does reliably.

Submit it once in Search Console and then leave it alone. Resubmitting weekly does nothing, and the temptation to do so is usually a symptom of the impatience described in Module 04, How search works, in the detail a clinic needs.