Indexing a website means collecting its content, normalising it into records, and storing those records for fast retrieval. You either crawl rendered pages or sync structured data from the source system. Syncing is more accurate and needs integration; crawling is more general and loses structure.
| Crawling | Syncing | |
|---|---|---|
| Setup | Point at a URL | Integrate per source |
| Data quality | Flattened HTML | Full structured fields |
| Freshness | Next crawl | Near-instant via webhooks |
| Best for | Content sites, docs | Commerce, apps, databases |
A stable ID, a title, searchable body text, a type, a URL, structured attributes for filtering, and freshness or availability state. For semantic retrieval, an embedding of the text. Records missing the attribute fields cannot be filtered on, regardless of how good the engine is.
Long pages should be split into passage-sized chunks with a little overlap, each retaining a reference to the parent page. Whole-document embeddings blur meaning across sections and retrieve badly for specific questions.
Webhooks for changes, a periodic full reconciliation to catch drift, and a visible index-age metric. Most production search complaints trace to stale records rather than to ranking.