Server monitoring and on-call alerting that people read

Setting up monitoring starts with a conversation rather than with software: we go through your services and agree, in writing, on what counts as an incident and what is ordinary working noise. A queue that grows for ten minutes inside the backup window is noise; the same queue growing at noon is an incident. Until that line is drawn, any monitoring system turns into a siren nobody hears by the second week.

Then I build the checks. Availability is measured from more than one place and includes a look at the content of the page — a server with an empty database returns a perfectly healthy response code. Free space on every disk and the rate at which it disappears, processor and memory load, the depth of background queues, expiry dates for certificates and domain renewals, and confirmation that a backup not only started but finished and can actually be restored. Every check carries a severity, and severity decides the channel: a line on a dashboard, a chat message, or a phone call at three in the morning.

Around the checks comes the human part: a duty roster, an escalation path for the case when the first person does not pick up, a status panel for management written in the language of the business instead of a wall of graphs, and a scheduled review of thresholds. Three things I state up front. An alert without an instruction is useless — each one comes with a note saying what to check first and whom to call if that does not help. Thresholds are tuned so that notifications never arrive in a stream: a list in which half the triggers are false stops being read at all. Access to the servers is personal and named; a single shared password for the whole team is not used.

What the work includes:
— a review of your services and an agreed list of what counts as an incident;
— availability and content checks, disk space, load and queue monitoring;
— certificate and domain expiry tracking, verification that backups succeed;
— alert routing by severity into different channels;
— a duty roster with shifts and a documented escalation ladder;
— a status panel for management and a short periodic report;
— a written instruction for every type of alert;
— a threshold review after the first month, based on collected statistics.