Topic
Index & Infrastructure
The plumbing, and the reason a perfect page can still be invisible. Before any quality model sees a document it has to be crawled, tiered into an index, scored by the serving stack and assembled into a result. Every step here is a place where a page simply does not arrive.
01Systems4
- 20BSegIndexer / TiersThe SegIndexer decides how quickly and how expensively Google indexes a page. Google does not index all pages equally fast or equally cheaply. The SegIndexer assigns each page to a 'tier' (level) of the index, from expensive, lightning-fast servers for important pages to slower, cheaper storage for rarely needed pages. A high tier rank means the page is quickly available when queries arrive.
- 21OCrawl-BudgetThe crawl budget sets how often and how regularly Google visits and updates a website's pages. Crawling means Google 'reading in' the content. Two factors determine the budget: capacity (how fast and error-free does the server respond?) and demand (how important does Google find these URLs?). For a dynamic weather site this is highly relevant: if Google doesn't crawl a page in time, the weather data in search results is outdated.
- 22BMustang / SuperRoot etc.Mustang, Ascorer, SuperRoot and Twiddler measure nothing themselves. They form Google's technical scoring infrastructure: they calculate and coordinate the ranking but do not judge content quality. Twiddler also contains 'host-crowding', a rule that prevents a single domain from dominating too many search results at once. These systems are the 'negative entry' of the matrix: there is nothing to optimize here.
- 23B/ORankBrain / DeepRank / FastSearchThese systems measure the semantic fit between a search query and a page: whether the meaning matches, not just the keywords. They measure relevance, not content quality. RankBrain was Google's first AI system for this purpose. DeepRank is a deeper language understanding layer. FastSearch accelerates this estimate for faster responses.
02Deep Dives4
- RankBrain and DeepRank: How AI Changed Google's Understanding of QueriesFrom keyword matching to semantic understanding — the AI systems that interpret what users mean, not what they type
- Crawl-Budget: How Google Decides How Often to Visit Your SiteThe officially documented system that determines crawl frequency — and why it matters for large sites with thousands of URLs
- SegIndexer: How Google Tiers Its IndexGoogle does not index all pages equally. The SegIndexer decides which tier (level) of the index a page lands in, from lightning-fast to slow storage.
- Mustang, Ascorer, SuperRoot: Google's Technical Scoring InfrastructureThe systems that calculate and coordinate the ranking. They do not judge content quality, but they determine which scores matter and how the scores combine.
03Leak fields4
- B/C
scaledSelectionTierRankRank used to select the index tier. Values run from 0 to 32767, the 16-bit maximum.SegIndexer - O
crawl capacity × demandThe product of two factors: what the server can handle and how much Google wants to fetch the URLs.Crawl-Budget - B/O
RankEmbedBERTBERT-based embedding (a numeric representation of text) for deeper query understanding.RankBrain / DeepRank - B/C
Mustang / Ascorer / SuperRootInfrastructure for scoring and serving, and a compositor. It evaluates nothing.Mustang
04Theses0
No thesis touches this topic yet.