Summary

Week 10 built the last piece of visibility the system was missing: a way to ask “is the email pipeline actually healthy right now?” without checking Redis, the worker, and Postfix one by one. This landed as bin/cake email_health, on the gsoc/feature/email-health-check branch, tracked in MR !1267. Before this, the first sign of trouble was a user reporting a missing email there was no single place to check the pipeline’s state.

What was built

The command runs six independent checks:

• Redis connectivity : PING/PONG handshake, plus confirms the cdli:queue:email key itself is reachable.
• Worker heartbeat : checks the email:worker:heartbeat timestamp against twice the configured loop interval, so it fails if the key is missing, malformed, or stale.
• Queue backlog : counts pending/processing jobs against a configurable threshold.
• Postfix reachability : a raw TCP connect to the configured host:port, independent of whether mail is currently queued.
• Recent failure rate : counts jobs failed in the last hour against its own threshold.
• Mail config validity : confirms the Postfix env vars are set and the port is valid, before anything even tries to connect.

Redis, heartbeat, Postfix, and config failures are treated as hard failures (exit 1, suitable for alerting or a container health check). Backlog and failure rate breaches are warnings (exit 0) worth surfacing, not necessarily worth paging someone over. All the thresholds are configurable via env vars, so dev and production can use different limits.

Testing meant faking both external dependencies rather than taking down real infrastructure mid test. FakeRedisForHealth and TestableEmailHealthCommand let me trigger each specific failure path deterministically, covering every pass/fail/warn combination. I also ran the command manually against the dev stack every check reports healthy under normal conditions, except the heartbeat check, which fails until the worker actually writes that key. That integration is still pending on the worker side.