The Pages Google Never Sees: A Guide to Fast Indexing

There is a category of SEO problem that content cannot solve and that almost no report surfaces: pages that are never discovered. They are not penalised, they are not low quality — they have simply never been visited by a crawler. They sit there, published and invisible.

On Milan projects the phenomenon hits three kinds of site hardest: catalogues that add references every week, publications and company blogs posting frequently, and sites with modern architecture where much of the navigation is built in JavaScript. In all three the gap between publishing and discovery can exceed three weeks — and in some niches three weeks is the entire useful window.

How a company notices it has this problem

The symptoms are recognisable and are almost always mistaken for something else.

First: the number of pages receiving impressions is far below the number of pages published. A catalogue of 3,000 references where only 900 URLs have had a single impression in ninety days almost certainly has a problem upstream.

Second: new content takes weeks to appear while old pages keep ranking normally. That is the signal that the site is being visited but the new pages are not being reached — typically because the sitemap regenerates rarely, or because nothing internal links to the new material.

Third: pages reachable only through filters, internal search or progressive loading never appear at all. If a URL only exists after the user clicks a filter, for a crawler that URL may not exist at all.

What the indexing module actually does

The mechanism is deliberately simple: you submit URLs — one at a time, in bulk, or as a sitemap file — and the platform brings the crawlers to those addresses. The log then shows the verified visits: which bot, which URL, which response code, at what time.

The third row is the one that surprises people who have never thought about it: an article that mentions you but that no crawler has ever visited transmits no signal at all. Verifying it is the only way to know whether a visibility campaign produced anything beyond an invoice.

The two-week control test

Before changing anything in the infrastructure, measure the real size of the problem with a simple experiment.

If the submitted group is clearly ahead, the bottleneck is discovery and the infrastructure is what to fix. If both groups are equally stuck and the bots nevertheless returned 200, the problem is not discovery: it is the perceived quality of the pages, which is an entirely different piece of work.

Do not confuse the two levels

Seeing a 200 in the log means the bot arrived and received the page. If after three weeks that page still receives no impressions, more submissions are pointless: the ball has passed to the content, the internal structure and the authority of the site.

The most common structural causes, and how they are fixed

The sitemap is not really up to date

Many systems regenerate the sitemap on a fixed schedule, or include only part of the catalogue for performance reasons. The check is trivial: open the sitemap and look for the last page you published. If it is not there, the rest of the reasoning is moot.

The new content is linked from nowhere

A page reachable only from the sitemap has a hard life. It needs at least one link from a page the crawlers visit often: the home page, the main category, the blog index. In catalogues this is the number one cause of orphan pages.

The navigation is built in JavaScript

If filters and pagination produce URLs only after an interaction, those URLs have to be reachable statically as well. This is not an aesthetic question: it is the difference between a catalogue of 3,000 pages and a catalogue of 300.

Too much is published, all at once

Loading two thousand pages in one day rarely ends well. Publication spread across weeks, with internal links looked after, is absorbed far better.

A weekly workflow that works

For an editorial team or an e-commerce department publishing regularly, the most effective process is also the dullest, and it fits into fifteen minutes a week.

On Friday, collect the URLs published in the previous five days and submit them in bulk. The following Monday, open the log and check which received a visit. Those that did not are checked by hand: is there an internal link? are they in the sitemap? do they respond correctly to a direct request? Nine times out of ten the answer is in the second question.

At the end of the month, count how many published pages received impressions within fourteen days. It is a metric nobody uses and that belongs in every editorial report, because it measures the health of the publishing machine rather than the skill of the writer.

Migrations and redesigns: the most delicate moment

The case where indexing decides between a successful migration and a lost quarter is the site change. New platform, new URL structure, new theme: all legitimate operations, all capable of removing half the organic traffic if handled badly.

The sequence we recommend is always the same. Before the migration, export the complete list of URLs that receive impressions: that list, not the sitemap, is the asset to protect. Then prepare the one-to-one redirect map for the pages with traffic, avoiding the shortcut of a mass redirect to the home page or to a generic category.

On switchover day, submit the full new sitemap and watch the visit log. In the first seventy-two hours the useful information is not ranking — which will be unstable regardless — but which sections of the new site the crawlers are actually reaching. If an entire area receives no visits, the problem shows up immediately and is corrected in a day, instead of being discovered three weeks later from the revenue figures.

In the following two weeks, compare the list of URLs that used to receive impressions with those receiving them now. The absences are the repair list, in order of traffic lost. It is an inelegant method and a very effective one.

How to explain the problem to the technical team

Marketing often struggles to get these changes through, because they sound abstract. What works is bringing three concrete things instead of concepts.

The quick checklist

In most of the sites we analyse at least two of those answers are negative — and they are almost always worth more than any copy-optimisation work scheduled for the quarter.

The particular case of events and seasons

Milan runs on calendars: trade fairs, themed weeks, presentations, sale periods. Pages tied to an event have a very narrow window of usefulness, and publishing three days ahead almost certainly means publishing too late.

The practical rule we apply: the event page goes live at least six weeks early, with whatever data is already available, and is updated as things firm up. That gives it time to be discovered, evaluated and ranked before demand spikes. Submitting it for indexing at publication removes the most uncertain part of the process, but it does not replace the head start.

How much publishing frequency matters

A recurring question is whether publishing frequency affects discovery speed. What we observe across projects is that sites with a regular rhythm — even a modest one, two or three pages a week — are visited more consistently than those publishing twenty pages in a day and then nothing for two months.

The practical consequence is that, for the same resources, it is better to spread production than to concentrate it. It holds for catalogues and for company blogs alike: regularity builds a visiting habit that then converts into faster discovery when something important goes out, such as a new collection or a page tied to a fixed deadline.

The hidden cost of invisible pages

There is an economic angle that rarely enters technical discussions. Every product page or article published has a cost: photography, specification, copy, review. In the mid-sized catalogues we analyse, the full cost of a well-made product page sits between twenty and sixty euro.

If four hundred pages of a catalogue have never been reached by a crawler, the arithmetic is quick: several thousand euro of work already paid for that produces no organic return. Presented in that form — not as a technical problem but as capital tied up — the argument gets management attention far faster than any chart about positions.

Frequently asked questions