Week 5 built the part that records changes. Week 6 built the part that acts on them: the poller, plus the operations layer around it.
The poller (!1244). This is the consumer side of incremental indexing. It reads the outbox, rebuilds each affected artifact with the Week 4 document builder, and pushes the result to the search engine. That closes the loop the outbox opened: an approved edit now reaches search in a couple of seconds instead of waiting up to a day for the nightly rebuild. Most of the effort went into the parts that have to be exactly right. Only one poller runs at a time, held by a database lock, so updates apply in order. Failed pushes back off and retry, and a watchdog recovers any job a crashed worker leaves behind. Cascades needed the most care: renaming something like an author or a collection can touch hundreds of thousands of artifacts, so those are expanded and processed in batches, so a large cascade never blocks a normal edit.
The ops layer (!1245). The poller works, but running it for real needs a few more pieces. It adds a health command (is the queue keeping up, or is a job stuck?), a cleanup command that prunes old finished rows, and a reconciliation command that acts as a backstop: it compares the live index against a fresh rebuild and repairs anything that has drifted, catching the rare change a capture site might miss. It also runs the poller as a long-running container that restarts itself if it crashes.
With both parts in, incremental indexing is complete: the outbox records a change, the poller reindexes the affected artifacts within seconds, and the ops layer keeps it running and catches anything that slips through.
| # | Day | Date | A short description of the work done |
|---|---|---|---|
| 1 | Monday | 2026/06/29 | Planned the poller: the claim, build, and push flow, cascade handling, the single-poller lock, and retries |
| 2 | Tuesday | 2026/06/30 | Built the search indexer and the claim layer |
| 3 | Wednesday | 2026/07/01 | Built the cascade resolver and the processor |
| 4 | Thursday | 2026/07/02 | Wrote the poller’s tests and the end-to-end harness, and opened the poller merge request (!1244) |
| 5 | Friday | 2026/07/03 | Built the ops health, cleanup, and reconciliation commands |
| 6 | Saturday | 2026/07/04 | Built the reconciliation drift check and the poller container |
| 7 | Sunday | 2026/07/05 | Tested the whole pipeline end to end and opened the ops merge request (!1245) |