Infrastructure monitoring rollout

Our infrastructure has quietly outgrown the point where a person can keep it in their head. We run a fleet of servers and services, and today we usually learn something is broken from a customer complaint rather than from our own tools. Nobody has a single screen that shows whether everything is healthy, and when an incident does happen at night there is no agreed way to reach the right person. We want to stop firefighting blind and start seeing problems before our users do.

The task is to deploy a monitoring system across our environment end to end. We need metrics collected from hosts and key services, dashboards that make the current state obvious at a glance, and alerts that fire on real failures without drowning the team in noise. Alerts should reach an on-call rotation so it is always clear who is responsible right now, and quiet hours and escalation should be handled sensibly. We would rather have fewer, trustworthy alerts than a wall of warnings everyone learns to ignore.

Please tell us which tools you would choose and why, how you would keep the setup maintainable as we add servers, and how you would tune thresholds so an alert always means action is needed. Documentation and a short handover so our team can run it afterwards are part of the job.