Skip to main content
Infrastructure

Server monitoring

Collect less noise and more evidence somebody can act on.

The server dashboard is green, but customers cannot complete checkout. Monitoring has measured a machine while missing the service the business needs.

Useful monitoring begins with customer outcomes, connects them to application and infrastructure evidence, and sends a small number of actionable alerts to people able to respond. Metrics, logs and traces are tools inside that system. Buying an observability platform does not define what matters or who acts at 2am.

We design the signal and response together. An alert without an owner or a runbook is future notification fatigue.

Monitoring begins with the service promise

Before choosing agents or dashboards, we describe what users depend on: the public site returns useful content, sign-in succeeds, an order reaches fulfilment, or a scheduled import finishes before staff arrive.

Those promises lead to service-level indicators. Availability may be the proportion of valid requests completed successfully. Latency should reflect the routes customers use, often at a percentile that exposes the slow tail. A background job may be measured by freshness rather than HTTP status.

The target needs a time window and an agreed tolerance for failure. Google’s SRE guidance on service levels connects indicators, objectives and error budgets so product reliability can be managed rather than described as “always on.”

Not every internal component receives its own customer target. We monitor dependencies deeply enough to explain the service promise and prioritise what would consume it.

External checks prove a customer route from outside

A process can be running while DNS is wrong, a certificate has expired or the application returns a friendly error page with status 200. An external check crosses more of the route a customer takes.

The check should validate a useful response rather than a port alone. For a public site that may mean status, certificate and expected content. For a transaction it can exercise a safe synthetic account and confirm the resulting state without creating real charges or stock movement.

Checks from more than one location help distinguish a regional path from a complete outage. Frequency follows risk; a five-minute check can detect a failure after almost five minutes, before diagnosis and recovery begin.

Maintenance windows are recorded rather than muting all alerts by hand. A planned deployment can still break a service outside the expected component.

Metrics should expose traffic, errors, latency and pressure

Google’s monitoring distributed systems chapter describes latency, traffic, errors and saturation as four useful signals. They are a starting frame, not a compulsory four-panel dashboard.

Request rate and error ratio show service demand and failure. Latency needs a distribution rather than an average. Saturation is the resource approaching a limit: worker capacity, database connections, disk, memory or queue age.

Infrastructure metrics matter when they predict or explain customer impact. CPU at 90 per cent can be healthy batch work; a queue whose oldest job keeps ageing is direct evidence of a delayed outcome. We alert on the relationship rather than on every graph movement.

Prometheus’s instrumentation guidance recommends useful online-service and batch-job signals and warns about high-cardinality labels. A customer ID or raw URL in a metric can make storage cost and query performance unmanageable.

Logs, metrics and traces answer different diagnostic questions

Metrics show patterns and support efficient alerting. Logs record discrete events with context. Traces connect work as one request crosses services and queues. Collecting all three without correlation leaves an operator searching separate systems during an incident.

We generate a request or trace identifier at the boundary, pass it through application calls and include it in structured logs. OpenTelemetry provides vendor-neutral tracing, metrics and logging concepts that can reduce instrumentation dependence on one backend.

Logs should record decisions and failure context, not secrets or complete personal records. OWASP’s logging guidance covers security events, sanitisation and data that should normally be excluded.

Retention follows diagnostic, regulatory and cost needs. Keeping debug logs forever is expensive and risky; deleting every event after seven days may make a monthly failure impossible to investigate.

An alert is a request for human action

An alert should state what service is affected, the evidence, urgency, owner and first diagnostic route. If no action is possible, the signal belongs on a dashboard or in a report rather than waking somebody.

We distinguish symptoms from causes. A customer transaction failing is a high-value symptom. High CPU may explain it, but paging separately for both can create noise. Routing and grouping prevent one incident from producing dozens of independent notifications.

Thresholds need duration and recovery behaviour to avoid flapping. Multi-window burn-rate alerts can detect rapid consumption of an error budget as well as a slower sustained problem without making a single failed request an emergency.

Escalation is tested. The existence of an email integration does not prove somebody receives, understands and can access production from it. Ownership also covers holidays and supplier boundaries.

Dashboards support diagnosis, not reassurance

A wall of green graphs can conceal missing telemetry or a customer path nobody instrumented. Each dashboard should have an audience and a question: current service health, release comparison, capacity planning or incident investigation.

We show rates and percentiles with deployment annotations and clear units. A graph beginning at an arbitrary non-zero axis can turn normal variation into drama. A total counter without its rate can hide when the behaviour occurred.

Dashboards need links from alerts and runbooks. The operator should arrive at the relevant service, time range and release rather than reconstructing the context manually.

Regular review removes panels and alerts nobody uses. Monitoring is production code with maintenance and data cost, not an archive of everything an agent can emit.

Incidents should improve the monitoring model

During an incident, preserve a timeline of customer effect, alerts, deployments, decisions and recovery. Afterward, compare the first customer impact with the first useful signal. That gap tells us what monitoring missed.

A review should lead to proportionate controls: a new service indicator, a clearer runbook, better deployment marker or removal of a noisy alert. Adding more telemetry by default can make the next incident harder.

We also test expected failures before customers do. Stop a worker in staging, fill a controlled filesystem, expire a test credential or simulate a dependency timeout. Confirm that the signal appears and the documented response works.

Capacity trends belong here too. An alert at 95 per cent disk offers little purchasing time. Forecasting growth and setting an earlier ticket threshold turns monitoring into planning.

Questions to answer before buying monitoring

  • “Which customer promises need protection?” Derive signals from service outcomes.
  • “Which external check crosses the real route?” Validate content and state, not a port.
  • “Which metric predicts failure or explains impact?” Avoid collecting by default.
  • “How are requests correlated across logs, traces and jobs?” Preserve context.
  • “Who receives each alert and what can they do?” Test escalation and access.
  • “How long must evidence remain, and at what cost?” Set retention by data class.
  • “Which controlled failure will prove the monitoring?” Exercise the response.

When monitoring is ready to support production

The system is credible when it detects a failed customer journey, connects that failure to application and infrastructure evidence, and reaches an accountable responder. A small service may achieve this with external checks, structured logs and a few provider metrics.

Complex systems justify traces, richer service-level reporting and dedicated on-call tooling when those controls reduce diagnosis time. Scale of monitoring should follow scale of failure modes.

When observability spend is producing noise

Do not ingest every log and metric because storage is easy to enable. High-cardinality data, duplicate agents and unrestricted debug output can create a large bill without one better decision.

Avoid alerts based only on generic server thresholds when no customer or capacity action follows. Review pages that are repeatedly ignored; the problem is in the alert definition, ownership or service design.

And resist a new platform when the current gap is no service owner or runbook. Tool migration cannot supply operational responsibility.

Alexander De Sousa, Founder of Digital Royalty
Developer’s verdict

Monitor customer promises and page only on signals somebody can act on.

I start with the transaction or outcome the business owes its users, then work back to metrics, logs and traces. I would rather have ten owned alerts than a thousand generic thresholds nobody trusts. If we have not exercised a failure and received the expected signal, I do not consider the monitoring proved.

Alexander De Sousa · Founder, Digital Royalty · LinkedIn

Decision evidence

What makes production monitoring actionable rather than decorative

Customer checks define impact. Correlated telemetry explains it, while alert policy, access and runbooks turn evidence into a response.

The monitoring commitment in six decisions

Decision unit
A customer-visible service promise with a measurable indicator, objective, response owner and acceptable failure budget.
Customer effect
Earlier detection and shorter diagnosis when the system observes real journeys; little benefit when it only reports machine activity.
Adoption cost
Low for external checks and basic metrics; increasing with application instrumentation, tracing, retention, on-call tooling and specialist operation.
Ongoing owner
Service owners responsible for indicators, alert quality, instrumentation, retention, runbooks, escalation tests and improvement after incidents.
Failure mode
Alert fatigue hides real impact, telemetry cannot be correlated, or storage cost grows without supporting a customer or operating decision.
Proof required
A controlled failure triggers the right alert, reaches the right person and can be followed through correlated evidence to recovery.

Monitoring routes to match to service risk

  • Managed external uptime checks

    Right when: A small public service first needs independent availability, certificate and response validation without operating a monitoring stack.

    Watch for: A home-page check does not cover signed-in transactions, background freshness or the internal evidence needed for diagnosis.

  • Cloud-provider native monitoring

    Right when: Most infrastructure lives with one provider and native resource metrics, logs and alarms give the shortest route to useful coverage.

    Watch for: Application journeys, cross-provider dependencies, cardinality and retention costs still need design.

  • OpenTelemetry with chosen backends

    Right when: Several services need consistent traces, metrics and logs without binding all instrumentation to one storage vendor.

    Watch for: Running collectors, controlling volume and designing useful spans remain work; an open protocol is not a finished operating system.

  • Commercial observability platform

    Right when: Faster setup, integrated correlation and managed on-call workflows repay licence and data costs for a material service estate.

    Watch for: Model ingestion and retention at production volume and preserve a credible data and instrumentation exit route.

Research behind this monitoring position

Get started

Tell us what you need

A few quick questions, then a straight answer from a real person — usually within a few hours.

Tell us what you're working on

Whether it's a new site, a platform, or a process that shouldn't be manual any more — we'll tell you honestly if we can help.

Start your project or email hello@digitalroyalty.co.uk