Google Search works in three stages: it finds pages (crawling), works out what they are and stores them (indexing), and picks what to show for each query (serving and ranking). Almost every SEO problem you will meet is a failure at one of these stages, so knowing which one saves a lot of guessing. This guide follows Google’s own documentation, and corrects a few popular myths I repeated in the first version of this post, such as LSI keywords and social shares as ranking signals.
The Three Stages at a Glance
| Stage | What happens | What can go wrong |
|---|---|---|
| Crawling | Googlebot discovers URLs and downloads their content | Blocked by robots.txt, no links pointing to the page, slow or erroring server |
| Indexing | Google analyzes the page, picks a canonical version and stores it | Noindex tag, duplicate content, low quality, content that only appears if JavaScript fails to render |
| Serving | Google chooses and orders results for a specific query, location, language and device | Page is indexed but not relevant or useful enough to rank |
Google is explicit that none of this is guaranteed: even a page that meets the basic requirements may not be crawled, indexed or shown.
Stage 1: Crawling
Crawling is discovery plus download. Google mainly finds pages by following links from pages it already knows, and also uses sitemaps. A program called Googlebot decides which sites to visit, how often and how many pages to fetch, and it slows down when your server struggles, for example by returning HTTP 500 errors.
Two corrections to the old version of this post. Crawling is not a choice between “breadth-first” and “depth-first” strategies that you design yourself; that describes how someone might write their own crawler, not how you control Googlebot. And you do not need a crawl plan: you need to make pages reachable.
What helps Google crawl your site
- Links. Every page you want found should be linked from another page, with descriptive link text. Links from other sites are a key discovery path too (SEO Starter Guide).
- A sitemap. A list of the URLs you care about. It is helpful but optional.
- A healthy server. Fast, stable responses let Googlebot fetch more.
- robots.txt that does not block what you want indexed. Pages blocked there may be skipped, and so is anything behind a login.
Googlebot also runs JavaScript using a recent version of Chrome, so most client-rendered content can be seen. But if important text exists only after scripts run, rendering has to succeed first, so critical content is safer in the HTML itself. Make sure Google can access the same CSS and JavaScript a browser can; the URL Inspection tool in Search Console shows what Google sees. Page weight matters here too: see whether minifying resources helps SEO for measured numbers.
Stage 2: Indexing
After crawling, Google analyzes the page: its text, key tags such as the title and headings, images and videos. It groups duplicate or near-duplicate pages into a cluster and chooses one canonical version to represent them. The result is stored in the index, a very large database spread across thousands of computers.
The old post said indexing is the process of “retrieval and ranking”. That is wrong: retrieval and ranking happen at search time, in the serving stage. Indexing is the preparation done beforehand.
Why a crawled page may not be indexed
- A
noindexrule in a robots meta tag or header. - It duplicates another page, and Google picked a different canonical. Use redirects or
rel="canonical"deliberately; www vs non-www is a classic case of accidental duplicates. - Google judged the content low value. Search Console’s page indexing report shows statuses such as “Crawled – currently not indexed”, which usually means the page was fetched but not chosen.
- The content could not be rendered.
To check, search site:yourdomain.com to see whether your site is indexed at all, then use Search Console’s URL Inspection tool for a specific page.
Stage 3: Serving and Ranking
When someone searches, Google selects from its index the pages it judges most relevant and useful, adjusting for location, language and device. Google says ranking is automated and based on hundreds of factors, not a fixed list of 200. Its ranking systems guide names the systems involved. A few that matter for understanding the big picture:
| System | What it does |
|---|---|
| PageRank | Uses how pages link to each other to judge what they are about and how useful they may be; still part of the core ranking systems |
| Neural matching | Matches concepts in a query to concepts in pages |
| BERT | Interprets how word combinations express meaning and intent |
| RankBrain | Relates words to concepts so relevant pages can rank without exact keyword matches |
| Passage ranking | Evaluates individual sections of a page for relevance |
| Freshness, original content, reviews, site diversity | Systems for specific situations, such as recent events or product reviews |
| SpamBrain | Detects content and behavior that violate the spam policies |
Older systems are now folded in. Panda (2015) and Penguin (2016) became part of the core ranking systems, the helpful content system joined the core in March 2024, and Google lists Hummingbird (2013) as a retired system. They are history, not tools to optimize for in 2026. The guide does not discuss Pigeon, so I removed the claim about it.
Myths Worth Dropping
| Claim | What Google says or shows |
|---|---|
| “Google uses LSI keywords” | Latent semantic indexing is an older technique. It does not appear in Google’s list of ranking systems, which instead names neural matching, BERT and RankBrain for understanding meaning. |
| “Social shares are a ranking factor” | Social sharing is not among the ranking systems Google documents. Shares can bring visitors and links, which is a separate effect. |
| “The meta keywords tag helps” | Google Search does not use it. |
| “Keywords in the domain or URL are major” | They have little effect beyond showing in breadcrumbs. |
| “There is an ideal word count” | Content length has no ideal target, and word count alone does not affect ranking. |
| “Duplicate content gets a penalty” | No penalty as such, but duplicates waste crawl resources and force Google to choose a canonical. |
| “PageRank is everything” (or dead) | It is one of many signals and still in use. |
Those entries come from Google’s SEO Starter Guide. The same guide says E-E-A-T is not itself a ranking factor; it describes qualities Google’s systems try to reward through other signals.
Where AI Overviews and AI Mode Fit
Google says AI Overviews and AI Mode use the same foundations as the rest of Search: to appear as a supporting link, a page must be indexed and eligible to show with a snippet, and there are no additional requirements. Google also says you do not need special AI text files, extra markup or special structured data. These features can run several related searches (query fan-out), so they may link to a wider variety of pages. Their traffic is counted in the Web search type of Search Console’s Performance report.
Spam: What Gets Pages Demoted
Google’s spam policies list practices such as cloaking, doorway pages, hidden text, keyword stuffing, link spam, scaled content abuse, scraping, site reputation abuse and thin affiliate pages. If a tactic only makes sense because it fools a ranking system, assume it is on the list.
What You Can Do With This
- Make sure every important page is linked internally and listed in a sitemap.
- Check that robots.txt and any CDN rules let Googlebot fetch pages and the CSS and JavaScript they need.
- Keep key content in text in the HTML, with unique, accurate titles and descriptive link text.
- Use canonical URLs and redirects to avoid accidental duplicates.
- Use Search Console: the page indexing report to find stages where pages drop out, and URL Inspection to see how Google fetched and rendered a page.
- Write for the person searching. Relevance comes first; technical SEO only removes obstacles.
Using Google Search More Effectively
Google’s own help page on refining web searches documents these techniques:
| Technique | Example | Effect |
|---|---|---|
| Quotes | "how search works" | Match the exact words or phrase |
| Minus sign | jaguar speed -car | Exclude a word |
| site: | site:developers.google.com crawling | Limit results to one site or domain |
| before: and after: | core update after:2024-01-01 | Filter by last-updated date |
| filetype: | filetype:pdf web vitals | Find a specific file type |
I dropped the Wonder Wheel, Timeline and Related Searches tips from the old version; they are not documented in Google’s current help pages, so I could not confirm they still exist.
Conclusion
Search is crawl, index, serve. Fix problems at the earliest stage they appear: a page that cannot be crawled cannot be indexed, and a page that is not indexed cannot rank. Beyond that, ignore folklore about hidden formulas and focus on pages that are reachable, clear and genuinely useful for a specific query.
FAQs
How does Google Search work in simple terms?
Google crawls pages by following links and sitemaps, indexes what it finds by analyzing content and choosing canonical versions, then serves the pages it judges most relevant and useful for each query, adjusted for location, language and device.
What is the difference between crawling and indexing?
Crawling is discovering and downloading a page. Indexing is analyzing it and storing it in Google’s index. A page can be crawled but not indexed, for example because it is a duplicate or judged low value.
Does Google use LSI keywords?
No. Latent semantic indexing is not among the ranking systems Google documents. Google names systems such as neural matching, BERT and RankBrain for understanding meaning, so write naturally and cover the topic rather than hunting for LSI terms.
How many ranking factors does Google use?
Google says hundreds of factors, not a fixed list. The old figure of 200 is a persistent myth from years ago.
Do I need anything special to appear in AI Overviews or AI Mode?
No. Google says a page must be indexed and eligible to appear in Search with a snippet, and there are no additional requirements or special markup.
How can I check whether my page is indexed?
Search for the exact URL with the site: operator, or use the URL Inspection tool in Google Search Console, which shows whether the page is in the index and how Google fetched and rendered it.
