Chapter 1: How Search Engines Actually Work
1.1 What a search engine actually is
10 entries- 1 Search Engine A software system that matches a typed question against an index of hundreds of billions of pages and returns the best answers in a few hundred milliseconds.
- 2 SERP (Search Engine Results Page) The page returned after a query: a mix of paid ads, organic links, maps, video carousels and, increasingly, an AI-written summary sitting above everything else.
- 3 Query The exact string, typed or spoken, that a person hands to the search box.
- 4 Organic Results The unpaid listings a page earns through relevance, authority and technical health: the classic ten blue links.
- 5 Paid Search (PPC) Auction-based placements bought by advertisers, normally labelled "Sponsored" or "Ad".
- 6 CTR (Click-Through Rate) The share of people who saw your result and clicked it: clicks divided by impressions.
- 7 Impressions Every time your page is shown in a result set, whether or not anyone clicks.
- 8 Ranking (Position) The position a page holds for a specific query in the organic results.
- 9 Traffic The number of visits or distinct users reaching your site over a period, added up across every channel they arrived from.
- 10 Conversion Any action a visitor completes that you have already decided is worth money to the business.
1.2 Crawl, index, rank: the three-step loop
10 entries- 11 Crawler (Spider, Bot) An automated program a search engine sends out to follow links and pull pages back, around the clock, without ever getting tired.
- 12 Googlebot The family name for Google's crawlers, chiefly the smartphone agent and the desktop agent that trails behind it.
- 13 Crawling The step where a bot follows a link, requests the URL and downloads the HTML, stylesheets, scripts and images that make up the page.
- 14 Indexing The stage where the engine parses what it downloaded, compares it against everything it already holds, and decides whether the page has earned a permanent place in the database.
- 15 Ranking (Serving) The instant a query arrives and the engine selects and orders the pages it believes answer it best.
- 16 Index (Database) The distributed database holding a compressed record of every page the engine has decided to keep.
- 17 Crawl Budget The amount of fetching a search engine is willing to spend on one site in a given window of time.
- 18 Crawl Frequency How often a bot comes back to a given site or section.
- 19 Rendering The step where the engine executes a page's JavaScript the way a browser would, and assembles the finished document a person would actually see.
- 20 Discoverability How easily a bot can find and understand a page without being told where it is.
1.3 How engines read pages and people
15 entries- 21 Relevance The degree to which a page's subject and substance match what the person typing actually wanted.
- 22 Authority The engine's estimate of how far a site or page can be trusted on one specific subject.
- 23 Trustworthiness A judgement about whether the site is real, safe and not out to defraud anyone. It is the load-bearing element of E-E-A-T.
- 24 Search Intent The task a person is actually trying to finish when they type a query, which is often not the task the words literally describe.
- 25 Four Types of Search Intent Navigational, informational, commercial investigation and transactional. Nearly every query falls into one of the four.
- 26 Semantic Search Retrieval that works from meaning, context and synonymy rather than from literal string matching.
- 27 Entity A distinct thing or concept in the engine's knowledge graph carrying its own identity: a person, a place, a company, a brand, a film.
- 28 Knowledge Graph A very large database of entities and the relationships between them, maintained by Google.
- 29 Knowledge Panel The card on a results page that gathers an image, a description, social profiles, stock price or opening hours for one entity.
- 30 Featured Snippet (Position Zero) The direct answer box above the organic results, rendered as a paragraph, a list or a table.
- 31 People Also Ask The expandable question list embedded mid-results, which spawns further questions each time one is opened.
- 32 Rich Results Organic listings enhanced with stars, prices, availability, cooking time or an expandable FAQ, drawn from structured data.
- 33 Sitelinks The two to six shortcut links to key sections that appear beneath a brand result.
- 34 Personalization The adjustment of results based on where the searcher is, what they have searched before, what language they use and which device they hold.
- 35 Zero-Click Search A search that ends on the results page, because the answer arrived there through an AI summary, a featured snippet or a knowledge panel, and no site was visited.
1.4 Schools of SEO thinking and the quality bar
10 entries- 36 SEO (Search Engine Optimization) The systematic work of improving a site's internal and external signals so it ranks higher in unpaid results and earns the traffic that follows.
- 37 White Hat SEO Work that follows the published guidance and earns position by improving what users actually get.
- 38 Black Hat SEO Deliberately exploiting gaps in the algorithm with deception, to take position faster than honest work allows.
- 39 Gray Hat SEO Tactics in the gap between what is permitted and what is clearly manipulative, exploiting rules that were never written down.
- 40 On-Page SEO Everything done inside your own site, where you hold total control, to make pages understandable and worth ranking.
- 41 Off-Page SEO Work on other people's platforms to build the reputation and authority a site cannot grant itself.
- 42 Technical SEO Fixing architecture, code and server configuration so the engine can fetch, render and index the site without obstruction.
- 43 SEO-Friendly A site designed from the beginning around how engines fetch and interpret pages, rather than retrofitted after launch.
- 44 Ranking Factors The signals the algorithm weighs when ordering pages, numbering in the hundreds.
- 45 E-E-A-T The four dimensions human raters use to judge content quality under Google's guidelines: experience, expertise, authoritativeness and trust.
Chapter SOP and interview questions
The four-stage diagnostic, in the order failures actually happen
Most “SEO problems” are a stage problem wearing a costume. Before you touch a title tag, find out which stage is broken. Work down this list in order — never skip ahead.
- Is the URL discoverable? Check that it is linked from somewhere the crawler already visits, appears in the XML sitemap, and is not orphaned. If the answer is no, nothing downstream matters.
- Is it crawlable? Fetch the URL with
curl -Iand confirm a 200. Then fetch it with a Googlebot user agent. Then check robots.txt with the URL Inspection tool’s live test — do not trust your own reading of the file. - Is it renderable? Compare the raw HTML to the rendered DOM. If your primary copy, links or prices only exist after JavaScript runs, you are asking the renderer to do work that may be queued for days or skipped entirely.
- Is it indexable? Look for
noindex, canonical pointing elsewhere, or a duplicate that won the consolidation. The URL Inspection tool tells you the Google-selected canonical, which is the only one that counts. - Does it belong to a query? Search the exact title in quotes plus
site:yourdomain.com. If the page does not surface, the page is not associated with that intent — that is a content and internal-linking problem, not a technical one. - Is it competitive for that query? Rank tracking tells you position; the SERP tells you why. Read the top five results and ask what they have that you do not: format, freshness, entity coverage, brand.
- Is the click worth it? Impressions without clicks is a title and snippet problem. High position with low CTR means the result is being read and rejected.
- Does the click do anything? Sessions that bounce in three seconds are usually an intent mismatch — you ranked for a question and delivered a sales page.
Run this on five URLs per week for a quarter and you will have a real map of where the site loses traffic, instead of a list of tickets copied from an audit tool.
Building the model on a real site
The fastest way to learn how the engine behaves is to watch one page change. Pick a mid-traffic URL, log its status code, canonical, indexation state, position and clicks. Then make one change — a new internal link, a title rewrite, a redirect — and watch the same five numbers for four weeks. Repeat. After a dozen cycles you stop treating rank as a mystical event and start treating it as the output of a pipeline you can inspect.
Interview questions and answer frameworks
Q1: Walk me through what happens between publishing a page and it appearing in search results.
Framework: Four stages, and name the failure mode at each one. Discovery — the crawler has to find a link, from the sitemap, an internal page, or an external source. Failure: the page is orphaned. Crawling — it fetches the URL and gets a status code; failure: 404, 5xx, robots block, crawl budget spent elsewhere. Rendering and indexing — the HTML is parsed, JavaScript may be rendered, content is extracted, canonical is resolved, and the page is stored in the index. Failure: client-side-only content,
noindex, duplicate consolidation to the wrong URL. Ranking and serving — the query is matched against indexed documents, results are assembled with ads, maps and AI answers, and the layout is personalised. Failure: the page is indexed but associated with the wrong intent. Saying this in order signals that you debug in order, which is the actual skill.
Q2: A page is indexed but gets no traffic. What are the possible causes, ranked by likelihood?
Framework: Rank by frequency, not by what sounds clever. First, intent mismatch — the page answers a different question than the one people type. Second, position — being indexed at position 40 produces almost no clicks; check rank before blaming the title. Third, snippet — position 8 with a 1% CTR usually means the title and meta are being ignored or the query is answered directly in the SERP. Fourth, cannibalisation — another URL on the same site wins the query and your page is the supporting actor. Fifth, seasonality or a market change. Then say how you would confirm each: GSC query report for intent, rank tracking for position, CTR versus expected CTR for snippet quality, the Pages tab for cannibalisation.
Q3: Explain the difference between crawling, indexing and ranking to someone outside SEO.
Framework: Use a library. Crawling is the van that drives around picking up new books. Indexing is the catalogue that records what each book says, so it can be found without reshelving the library. Ranking is the librarian deciding which five books to put on the desk when you ask a specific question. Then make the practical point: a book can be in the building (crawled) and not in the catalogue (not indexed), and a book can be in the catalogue and still never reach the desk (ranked too low). This is why “we submitted the sitemap” is not an answer to “why is our page not ranking”.
Q4: Why does Google sometimes ignore a canonical tag?
Framework: Because canonical is a hint, not an instruction. The engine weighs the tag against other evidence: internal links, sitemaps, redirects, content similarity, and historical signals. If five internal links use a different URL, or the “canonical” version is slower and thinner, the engine may choose the other one. The reliable fix is to make the signals agree — canonical tag, internal links, sitemap entry, hreflang and redirect all pointing at the same URL — rather than adding a stronger tag. Mention that you can see the chosen canonical in URL Inspection, and that this is the number to argue with, not your own tag.