What this guide covers
  • The three stages a search engine runs — crawling, indexing, ranking — and why confusing them causes most diagnostic mistakes.
  • How keywords are chosen from intent rather than from volume, and how they become pages.
  • Site architecture, on-page work and internal linking, in the order that pays.
  • What has genuinely changed with generative answers, and what has not: semalt.com/authorize.

Search engine optimisation is sold as a set of tricks, discussed as if it changed fundamentally every few months, and blamed whenever traffic falls. The reality is duller and more useful: search engines are trying to answer questions well, and SEO is the work of being the best available answer while remaining findable.

This guide covers the mechanism — what search engines do, what they weigh, and what the work consists of. It is deliberately foundational; if what you need is prioritisation and sequence, that is a separate article on this blog.

What a search engine actually does

Three distinct stages, and confusing them causes most diagnostic mistakes.

1
Crawling
A program follows links and requests pages, on a finite budget per site. Pages that are unreachable, blocked, or reachable only through a form or a script are not crawled, and nothing downstream can happen.
2
Indexing
The crawled page is understood and stored — or not, because it duplicates another page, has too little substance, or was judged not worth keeping. “It is in Google” and “Google has fetched it” are different claims.
3
Ranking
For a given query, indexed pages are ordered. This is the stage everyone talks about and the last one that matters: a page that is not indexed cannot rank, however good it is.
i
When traffic disappears, work backwards through the three stages
The cause is far more often stage one or two than a mysterious ranking penalty. Checking indexation first costs an hour; assuming a penalty costs a quarter.

What ranking actually weighs

Nobody outside the engines knows the full recipe, and anyone claiming a precise list is guessing. The broad categories, though, are well established and stable.

What is weighedThe question behind itWhere it matters most
RelevanceDoes the page address the need behind the query, not merely contain the words?Everywhere. Someone typing “how much does X cost” wants a number, not a quotation form
Quality and evidenceIs it substantial, accurate, and written by someone with a reason to be believed?Highest where the stakes are — health, money, legal exposure; lowest for trivia
AuthorityDo other credible sites reference this one, editorially?Bought links carry the risk without the upside
Experience of the pageSpeed, mobile usability, stability, content reachable without dismissing overlaysRarely makes a bad page rank; routinely stops a good one
ContextWhich searcher, in which place, in which language, on which device?There is no single ranking, which is why one average number hides the truth

Keywords: chosen from intent, not from volume

The most common way a keyword plan fails is being built from a volume column. A term with high volume and the wrong intent brings visitors who leave; a narrow term with buying intent brings a customer.

The useful distinction is not short versus long, but what the searcher is trying to do.

  • Informational. “How X works”. Worth pages, but supporting pages: they feed the commercial ones rather than replacing them.
  • Comparative. “X versus Y”, “best X for Z”. This is where most sites are absent, and where the decision is actually made.
  • Transactional. “X price”, “X near me”, “buy X”. Fewest searches, most revenue per search.
“Ten visitors comparing suppliers beat a thousand reading definitions. Volume is a vanity column until intent is attached to it.”
Why keyword plans built on volume disappoint

The practical test costs a minute and settles most arguments: search the query and look at what already ranks. If the first page is entirely buying pages, an article will not break in however good it is. If it is entirely guides, a product page will not. Match the format the results are already showing, or choose a different query.

From keywords to a structure

A list of keywords is not a plan. What turns it into one is grouping: every group of queries that a single page could satisfy becomes one page, and the groups that need different pages become different pages.

1
Group by the answer, not by the word
If two queries would be satisfied by the same page, they belong together. If they need different pages, splitting them is what prevents the two pages competing with each other.
2
Give each group one page, and one URL
Same page, same address, from now on. Publishing “guide”, “complete guide” and “strategies guide” as three pages splits one intent three ways and usually ranks none of them.
3
Connect the groups that belong together
Informational pages link to the commercial page they support; the commercial page links back. Content that only links to the home page wastes the connection.
4
Leave the rest out
Groups you have no genuine claim on stay off the plan, in writing, so nobody reopens the argument in six weeks.

Architecture: how depth and links distribute weight

Two structural properties do most of the work.

Depth. How many clicks from the home page to the page that has to rank. If a competitor’s equivalent page is one click from their home page and yours is four, your internal linking is the problem, not your copy.

Coherence of address. One page, one URL, one canonical, with every other form redirected. This single fix resolves a surprising share of “my pages compete with each other” problems, and it costs an afternoon.

!
Faceted navigation is where catalogues quietly break
Filter combinations that nobody searches for should not be indexable. Left open, they inflate the catalogue with near-identical weak pages that steal visibility from each other and consume the crawl budget the real pages needed.

On-page: what to write and what to stop writing

The title is a promise, not a label. A page with high impressions and a low click rate does not have a ranking problem — it has a promise problem, and rewriting the title fixes it within days. It is the cheapest intervention in SEO and the most frequently skipped.

The answer comes before the preamble. A section that opens with the number and then explains it is more useful to a hurried reader and more citable to an answer engine than one that reasons for three paragraphs first.

Headings describe the content, not the keyword. A heading that is the question a reader would ask, followed by a paragraph that answers it, does more for both audiences than a heading stuffed with a phrase.

Structured data matches what the page says. Mark up what is genuinely there — and nothing else. Inventing a claim in the markup is a liability, not an optimisation.

The on-page pass, in order of payback
  • ✓Titles and descriptions on pages that already earn impressions but few clicks
  • ✓One page per thing you sell, in the words customers use rather than internal vocabulary
  • ✓The answer near the top, with the reasoning below it
  • ✓Internal links from informational pages to the commercial page they support
  • ✓Structured data that states only what the page states

Authority: slow, partly outside your control, and decisive

Links remain a significant signal, but their value comes from being editorially given. This is the area where the gap between a small site and an established one is widest, and the only honest way to close it is slowly: genuine mentions from suppliers, associations, trade publications and partners who have a reason to cite you.

The market in cheap guest posts and directory packages exists precisely because this part is slow. The upside is negligible and the downside is durable.

What has genuinely changed, and what has not

Two real shifts are worth understanding.

Search engines have become far better at meaning than at matching. Repeating a phrase fifteen times stopped working long ago; covering a topic properly, including the questions that surround it, works well.

AI-generated answers increasingly sit above the results, which changes the shape of some traffic — particularly simple factual queries, where the answer is given without a click. What survives that is content offering something a summary cannot: judgement, specific experience, current local knowledge, evidence.

✓
What has not changed is the more important half
Be findable, be genuinely useful, be credible. Every technique that has stopped working was an attempt to appear to do those things without doing them.

The mistakes that account for most failures

  • Optimising for a keyword nobody uses. Check the words customers actually type before building a page around a phrase from your brochure.
  • Several pages for one intent. They compete with each other and none of them wins.
  • Confusing crawling with indexing. Months spent on ranking work while the pages were never indexed.
  • Judging by rankings alone. Position is a diagnostic; enquiries are the result.
  • Redesigning without redirects. The single fastest way to discard years of accumulated visibility.

How to tell whether it is working

Check impressions and clicks per query rather than total sessions; a rise in impressions with flat clicks is a title problem you can fix today. Track a small set of commercially meaningful queries rather than hundreds. Above all, count contacts. Everything else in this guide exists to produce them.

Frequently asked questions

How long before SEO work shows results?

Title and description fixes on pages already earning impressions can move within days. New pages typically need weeks to settle. Competitive commercial terms move over quarters, and authority over longer than that.

How many keywords should we track?

A small set that is commercially meaningful — twenty to forty for most sites — rather than hundreds. A long list stops being read, and an unread list changes no decisions.

Is technical SEO or content more important?

Neither, in the abstract. Technical work removes ceilings and content creates growth: whichever one is currently blocking you is the important one, and that is a question of diagnosis rather than preference.

Do we need to publish constantly?

No. One genuinely specific page a month, sustained, builds an asset base; four thin pages a month for three months and then silence produces maintenance and nothing else.

Measure the three stages on your own site

What is crawled, what is actually indexed, and where each page stands per query and per market.

Open the Semalt dashboard