Indexing

Indexing is the process of reading a site's content and storing it in a structure optimised for fast retrieval at query time.

An index trades write-time work for read-time speed. Rather than scanning every document when a query arrives, the engine precomputes a lookup — classically an inverted index mapping each token to the documents containing it, and in semantic systems a vector index mapping embeddings to their nearest neighbours.

Content reaches the index either by crawling (following links and parsing HTML, like a search engine) or by syncing (pulling structured records from a database, CMS or commerce API). Syncing is more accurate because it sees fields the rendered page flattens away; crawling is more general because it needs no integration.

Freshness is the recurring operational cost. A price change, a sold-out variant or an edited policy must reach the index quickly or search starts lying to visitors. Most real-world search failures in production are not ranking failures — they are staleness failures.