Observability: logs, healthcheck, graceful shutdown

Make your service observable and cleanly stoppable: JSON logs, liveness/readiness probes, and SIGTERM handling.

9 min · 3 questions

In production, a log is meant for a machine (Loki, Datadog, CloudWatch), not a human: emit one JSON object per line (one line = one event) with a level, an ISO timestamp, and structured fields - never free-form interpolations that are impossible to query. A little-known detail: writing to stdout is synchronous to a file or a terminal, but asynchronous to a pipe. So a chatty logger can block the event loop depending on the destination - that's the whole point of libraries like pino, which serialize fast and can offload the writing.

function log(level, message, fields = {}) {
  const entry = {
    level,
    message,
    time: new Date().toISOString(),
    pid: process.pid,
    ...fields,
  };
  process.stdout.write(JSON.stringify(entry) + "\n");
}

log("info", "server started", { port: 3000 });
log("error", "db unreachable", { retryInMs: 5000 });

Distinguish two probes. Liveness answers "should the process be restarted?": it must stay trivial, with no external dependency. Readiness answers "can I receive traffic?": it may check the database, and above all switch to 503 during shutdown, so the load balancer removes the instance before it goes down. Confusing the two causes healthy processes to be restarted the moment a dependency hiccups.

import { createServer } from "node:http";

let ready = true;

const server = createServer((req, res) => {
  if (req.url === "/healthz") {
    res.writeHead(ready ? 200 : 503);
    res.end(ready ? "ok" : "draining");
    return;
  }
  res.end("hello");
});

server.listen(3000);

function shutdown() {
  ready = false;
  server.closeIdleConnections();
  server.close(() => process.exit(0));
  setTimeout(() => process.exit(1), 10_000).unref();
}

process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);

The sequence of a graceful shutdown: the orchestrator sends SIGTERM; you flip readiness to 503; server.close() refuses new connections and waits for in-flight requests to finish; closeIdleConnections() closes idle keep-alive connections that would otherwise hold close() indefinitely. The unref()-marked timer acts as a safety net: it kills the process after 10 seconds, without on its own preventing the process from exiting earlier.

process.on("uncaughtException", (err) => {
  const entry = {
    level: "fatal",
    message: err.message,
    stack: err.stack,
    time: new Date().toISOString(),
  };
  process.stderr.write(JSON.stringify(entry) + "\n");
  process.exit(1);
});

process.on("unhandledRejection", (reason) => {
  throw reason;
});

Knowledge check

Make sure you remember the key points of this lesson.

1. What is the difference between liveness and readiness probes?
   • None: they are two names for the same test
   • Liveness checks that the process is running; readiness that it can accept traffic, dependencies included
   • Readiness restarts the container, liveness removes it from the load balancer
2. What does server.close() do exactly?
   • It immediately cuts all open connections
   • It kills the process after a fixed 30-second delay
   • It refuses new connections and waits for existing ones to finish
3. What should an uncaughtException handler do?
   • Log the error, then terminate the process
   • Ignore the error: Node recovers on its own
   • Automatically retry the failed operation