Work / Exceptions and follow-through
Know when something needs attention—and who has acknowledged it.
An alert is only useful if a specific person knows about it and acts. A notification nobody acknowledges is an exception nobody owns.
- Setting
- Remote off-grid property with water storage
- Inputs
- Two tank-level sensors; scheduled camera snapshots of a water gauge
- Channels
- Phone calls, email, a Discord channel
- Status
- Live systems
The workflow
The property depends on stored water, and nobody is there to look at the tanks. Two level sensors report when storage drops to half and to a quarter. The failure to avoid is not the sensor tripping. It is the alert landing somewhere nobody is looking, and no one knowing whether anyone has acted.
A second need is routine visibility: a regular picture of a water-level gauge so the trend is visible without driving out.
What I built: tank alerts
- Two thresholds with different urgency. The half-full alert is a warning that something upstream may be leaking. The quarter-full alert is critical and the call language escalates.
- Each alert goes out on independent channels at once: a Discord message to me, an email to a family member, and phone calls to two people at the same time.
- The call asks the recipient to press 1 or say yes. That acknowledgement is recorded, announced in Discord, and stops the repeats.
- If nobody acknowledges, the alert fires again every 15 minutes.
- Calls only happen between 8am and 11pm Pacific. Outside that window the email and Discord message still go, so nobody is woken for a warning-level event but the alert is not lost.
- Each channel is sent independently, so a failure in one cannot block the others.
What I built: the camera
- Scheduled snapshots, with a rolling set of recent images plus one per day and one per month kept as an archive.
- A dashboard shows all three views and has a capture-now button, rate limited so it cannot be hammered.
- A health check runs every two hours and posts to Discord if the newest snapshot is more than eight hours old. I added it after a camera outage of about three days went unnoticed.
A design decision worth explaining
The camera sits on a home network that the cloud cannot reach. My first attempts pushed from the cloud into that network through a VM and later a VPN sidecar, and they kept failing in ways that were hard to diagnose.
I reversed the direction. A small program on a PC at the property holds one outbound connection to the database. The cloud function writes a capture command; the database pushes it down that connection; the PC takes the photo and uploads it. No inbound ports, no VPN. If two PCs are listening, a transaction lets the first one claim the command so there is no duplicate capture.
The lesson I took from it: when pushing into a private network is unreliable, have the device on the inside ask for work.
What stays manual
- Someone has to answer the call. The system escalates and repeats; it cannot fix a tank.
- I do not claim every path in the design is running. The sensor vendor's direct push integration needs a larger device count than this setup has, so it is not deployed. Alerts arrive through the vendor's alert emails and scheduled sensor checks instead.
- SensorA tank sensor reports a low levelAutomatic
- CheckSystem checks whether this alert is new, already acknowledged, or due a repeatAutomatic
- NotifyDiscord, email, and two phone calls go out together (calls only 8am to 11pm Pacific)Automatic
- AcknowledgeA recipient presses 1 or says yes on the callPerson
- RecordAcknowledgement is stored and announced; repeats stopAutomatic
- RepeatIf nobody answered, the alert fires again in 15 minutesAutomatic