Topic
Index & Infrastructure
The plumbing, and the reason a perfect page can still be invisible. Before any quality model sees a document it has to be crawled, tiered into an index, scored by the serving stack and assembled into a result. Every step here is a place where a page simply does not arrive.
01Systems4
- 20BSegIndexer / TiersGoogle doesn't index all pages equally quickly or cheaply. The SegIndexer decides which 'tier' (level) of the index a page lands in — from expensive, lightning-fast servers (for important pages) to slower, cheaper storage (for rarely needed pages). A high tier rank means: the page is quickly available when queries arrive.
- 21OCrawl-BudgetHow often and how regularly Google visits and updates a website's pages (crawling — Google 'reading in' the content). The budget is determined by two factors: capacity (how fast and error-free does the server respond?) and demand (how important does Google find these URLs?). For a dynamic weather site this is highly relevant: if Google doesn't crawl a page in time, the weather data in search results is outdated.
- 22BMustang / SuperRoot etc.Mustang, Ascorer, SuperRoot and Twiddler form Google's technical scoring infrastructure — they calculate and coordinate the ranking but do not themselves judge content quality. Twiddler also contains 'host-crowding': a rule that prevents a single domain from dominating too many search results at once. These systems are the 'negative entry' of the matrix — there is nothing to optimize here.
- 23B/ORankBrain / DeepRank / FastSearchThese systems measure the semantic fit between a search query and a page — whether the meaning matches, not just the keywords. RankBrain was Google's first AI system for this purpose; DeepRank is a deeper language understanding layer; FastSearch accelerates this estimate for faster responses. The key point: these systems measure relevance, not content quality.
02Deep Dives4
- RankBrain and DeepRank: How AI Changed Google's Understanding of QueriesFrom keyword matching to semantic understanding — the AI systems that interpret what users mean, not what they type
- Crawl-Budget: How Google Decides How Often to Visit Your SiteThe officially documented system that determines crawl frequency — and why it matters for large sites with thousands of URLs
- SegIndexer: How Google Tiers Its IndexNot all pages are indexed equally — the SegIndexer decides which tier (level) of the index a page lands in, from lightning-fast to slow storage
- Mustang, Ascorer, SuperRoot: Google's Technical Scoring InfrastructureThe systems that calculate and coordinate the ranking — they don't judge content quality, but they determine which scores matter and how they combine
03Leak fields4
- B/C
scaledSelectionTierRankRank for index-tier selection (0–32767 = 16-bit maximum).SegIndexer - O
crawl capacity × demandWhat the server can handle × how much Google wants to fetch the URLs.Crawl-Budget - B/O
RankEmbedBERTBERT-based embedding for deeper query understanding.RankBrain / DeepRank - B/C
Mustang / Ascorer / SuperRootScoring/serving infrastructure and compositor — evaluates nothing.Mustang
04Theses0
No thesis touches this topic yet.