Systemd, Processes, & Zero-Downtime Deploys

The line between a hobby project and production is often the line between node server.js & in a tmux session and a properly supervised service. The first crashes on the next reboot, the next OOM, the next SIGHUP, and you find out from a customer. The second restarts itself, logs to a structured place, runs as a non-privileged user, and survives whatever the host throws at it. Today is about that line, and about the next-most-important question: how do you replace the running version with a new one without anyone noticing?

What systemd Actually Does

systemd is the init system on every modern Linux distro that matters. Process 1, the parent of everything else, the one that keeps services running and reports when they stop. Older mental models map it to upstart, sysvinit, or supervisord; for our purposes it's a single tool with three relevant capabilities:

• Process supervision: run a service, restart it on failure, run it on boot.
• Dependency ordering: "start nginx after the network is up," "start the worker after Postgres is reachable."
• Log aggregation: stdout/stderr go into the journal, queryable with journalctl.

The unit of deployment is a unit file. A unit file is a small INI-style config that says: how to start the service, how to stop it, when to restart it, who owns it, what it depends on. Here's the smallest production-shaped one you'd write.

[Unit]
Description=Acme web app
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=acme
Group=acme
WorkingDirectory=/opt/acme/current
EnvironmentFile=/opt/acme/shared/env
ExecStart=/usr/bin/node /opt/acme/current/server.js
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
TimeoutStopSec=30s
Restart=always
RestartSec=5s

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/acme/shared /var/log/acme
PrivateTmp=true
LockPersonality=true
MemoryDenyWriteExecute=true

# Resource limits
LimitNOFILE=65535
MemoryMax=1G
CPUQuota=200%

[Install]
WantedBy=multi-user.target

That's a lot of stanzas. Let's read them in groups.

Type, dependencies, and lifecycle

• Type=simple — the most common; ExecStart is the main process, alive as long as it runs. Other types: forking (daemon double-forks itself; rare in modern apps), notify (the app sends sd_notify when ready; clean), oneshot (run-and-exit, e.g. a migration).
• After=network-online.target — start after the network is up.
• ExecStart — the actual command. Always use absolute paths; systemd doesn't read your shell PATH.
• Restart=always + RestartSec=5s — die, wait 5s, come back. on-failure is gentler if you want to leave a manually-stopped service stopped.
• TimeoutStopSec=30s — give the app this long to shut down gracefully on stop, then SIGKILL it.

User, working directory, environment

• User=acme — never run as root unless you must. Create a dedicated user: sudo adduser --system --no-create-home --group acme.
• WorkingDirectory — the cwd of the process. Useful for relative paths in your code (don't rely on them, but they happen).
• EnvironmentFile — load env vars from a file. The right home for secrets and config; mode 600, owned by the service user.

Hardening — free for the asking

systemd has a long list of sandboxing options that cost nothing and prevent classes of compromise.

• NoNewPrivileges=true — process can't gain new caps via setuid binaries.
• ProtectSystem=strict — most of the filesystem is read-only to this service. Specify ReadWritePaths for the directories it actually needs to write.
• ProtectHome=true — /home hidden.
• PrivateTmp=true — its /tmp is namespaced, not shared with other processes.
• MemoryDenyWriteExecute, LockPersonality, RestrictNamespaces, SystemCallFilter — increasingly aggressive, increasingly likely to break a particular language runtime. Enable progressively, test thoroughly.

Day-to-Day Operation

sudo systemctl daemon-reload                  # after editing a unit file
sudo systemctl enable --now acme.service      # start now, also start on boot
sudo systemctl status acme.service             # current state + last lines of log
sudo systemctl restart acme.service
sudo systemctl reload  acme.service             # if app supports SIGHUP
sudo systemctl stop    acme.service
sudo systemctl is-active acme.service          # scriptable; "active" or non-zero exit
sudo systemctl is-failed acme.service

sudo journalctl -u acme.service -n 200 --no-pager
sudo journalctl -u acme.service -f              # follow live
sudo journalctl -u acme.service --since "10 min ago" -p err
sudo journalctl -u acme.service --since today --grep 'panic|exception'

Two things to internalise:

1. daemon-reload is required after any edit to a unit file. Without it, systemd serves the old version.
2. journalctl is the universal log tool: per-unit (-u), priority (-p err), since/until, follow (-f), grep (--grep), JSON output (-o json) for tooling.

Graceful Shutdown — From the App Side

systemd's contract on stop is: SIGTERM, wait TimeoutStopSec, SIGKILL. Your app's job is to react to SIGTERM by:

1. Stop accepting new connections (close the listening socket).
2. Finish in-flight requests (the existing connections drain).
3. Close downstream connections (DB pool, queue clients).
4. Exit 0.

If you don't do this, every restart drops in-flight requests and clients see 502s. Most languages have a one-liner.

const http = require('http');
const server = http.createServer(app).listen(3000);

async function shutdown(signal) {
  console.log(`Received ${signal}, draining…`);
  server.close(async () => {
    await db.end();           // close pg pool
    await queue.disconnect();
    process.exit(0);
  });
  // Hard deadline — don't hang past systemd's TimeoutStopSec
  setTimeout(() => {
    console.error('Drain timeout, forcing exit');
    process.exit(1);
  }, 25000).unref();
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT',  () => shutdown('SIGINT'));

Set the app's hard timeout (the setTimeout here, 25s) to be under systemd's TimeoutStopSec (30s) so the app exits cleanly before systemd reaches for SIGKILL.

Environment Files & Secrets

Twelve-factor says "config in env vars." Where do those vars come from?

• EnvironmentFile in the unit. Plaintext on disk; mode 600, owned by the service user.
• systemd credentials — newer pattern; loads secrets from disk into a tmpfs accessible only to the service. LoadCredential=stripe.key:/etc/credentials/stripe.key.
• External secret manager (Vault, AWS Secrets Manager, Doppler) — the app fetches at startup using its IAM role. The most flexible; adds a startup-time dependency.

For a single-box EC2 deployment, an EnvironmentFile populated by your deploy script is fine. For anything multi-host, push toward Secrets Manager — rotation, audit, no plaintext on disk.

The Deploy Question — Four Strategies

Given a healthy running service, how do you replace its code with the next version? Four patterns trade availability for simplicity.

1. Recreate

Stop the old, deploy, start the new. Downtime equals startup time — usually 5–60 seconds. Fine for internal tools, dev environments, batch workers. Maintenance pages soften the impact for user-facing services.

2. Rolling

With multiple instances behind a load balancer: replace them one at a time, draining each before the swap. No downtime if your app does graceful shutdown. The default in Kubernetes Deployments, ECS services, ALB target groups. Trade-off: old and new versions run simultaneously — your code must tolerate that (DB schema compatibility, contract back/forward compat).

3. Blue-Green

Run two complete environments. "Blue" is live; "green" is the next version, fully provisioned but receiving no traffic. Validate green, then flip the load balancer (or DNS) to point at green. Roll back in seconds by flipping back. Costs 2× infra during the cutover, but the rollback story is unbeatable.

4. Canary

Send a small fraction (1%, 5%) of live traffic to the new version. Measure error rates, latency, business metrics. If healthy, ramp up; if not, roll back having affected only a sliver. Implementations: weighted DNS (slow), weighted upstreams in NGINX/Envoy/Istio (fast), feature flags inside the app (most flexible).

The Atomic Symlink Swap

The classical Capistrano-style deploy on a single box looks like this:

/opt/acme/
├─ releases/
│  ├─ 20260502-101355/      ← previous release
│  ├─ 20260503-094502/      ← new release just unpacked
│  └─ 20260503-152008/      ← (older still kept for rollback)
├─ shared/
│  ├─ env
│  └─ logs/
└─ current → releases/20260503-094502/   ← symlink, the live version

Deploy = unpack the new release into releases/<timestamp>, run migrations, then atomically swap the current symlink:

set -euo pipefail
NEW=/opt/acme/releases/$(date -u +%Y%m%d-%H%M%S)

# 1. Build & ship the artifact
rsync -a --delete ./build/ acme@host:$NEW/
ssh acme@host "cd $NEW && npm ci --omit=dev"

# 2. Run migrations (idempotent, backward-compatible — see warning)
ssh acme@host "cd $NEW && npm run migrate"

# 3. Atomic swap: ln -nfs is non-atomic; use mv on a temp link to be truly atomic
ssh acme@host "ln -nfs $NEW /opt/acme/current.new && mv -Tf /opt/acme/current.new /opt/acme/current"

# 4. Reload the service so it picks up the new symlink target
ssh acme@host "sudo systemctl reload acme.service"
# (or restart, if the app can't SIGHUP-reload its code)

# 5. Prune old releases — keep last 5 for rollback
ssh acme@host "ls -1dt /opt/acme/releases/* | tail -n +6 | xargs -r rm -rf"

Rollback is one symlink swap to a previous timestamp + reload. Cheap, fast, requires no rebuild.

Connecting Strategy + Symlink + systemd

The pieces fit together for a single-box rolling-style deploy on one EC2:

1. Run two systemd services on different ports: acme@blue.service on 3000, acme@green.service on 3001 (systemd template units make this clean).
2. NGINX upstream pool has both, marked equal weight. Both versions serve traffic when both are running.
3. Deploy procedure:
   1. Drop the new release symlink for blue.
   2. Reload acme@blue.service — it picks up the new code.
   3. Wait for its /healthz to come back green.
   4. Repeat for green.
4. If a deploy fails, the second instance is still on the old code; revert the symlink, reload, no traffic was harmed.

For multi-box, the load balancer (ALB or your own NGINX in front) does the rolling automatically — drain a target, deploy, return it to service, repeat.

Templated Units — Many Instances of One App

systemd has a built-in template syntax: a unit file with @ in the name accepts an instance argument. acme@.service with %i in its body becomes acme@blue, acme@green, acme@worker-1, etc.

[Unit]
Description=Acme web app — instance %i

[Service]
Type=simple
User=acme
EnvironmentFile=/opt/acme/shared/env.%i
ExecStart=/usr/bin/node /opt/acme/current/server.js
Restart=always

[Install]
WantedBy=multi-user.target

# Manage like:
# sudo systemctl enable --now acme@blue.service
# sudo systemctl enable --now acme@green.service

Each instance reads its own env.blue or env.green file, listens on its assigned port (set via env var), and is supervised independently.

Timers — The Modern Cron

systemd timers replace cron for scheduled work, with the advantages: full systemd context (logs, failure handling, dependencies, sandboxing), human-readable schedule, and easy ad-hoc trigger.

# /etc/systemd/system/backup.service
[Unit]
Description=Daily database backup

[Service]
Type=oneshot
User=acme
ExecStart=/opt/acme/scripts/backup.sh

# /etc/systemd/system/backup.timer
[Unit]
Description=Run daily database backup

[Timer]
OnCalendar=*-*-* 03:30:00       # every day at 03:30 local
RandomizedDelaySec=15m            # avoid stampede across many hosts
Persistent=true                    # if the host was off at 03:30, run on next boot

[Install]
WantedBy=timers.target

# Enable: sudo systemctl enable --now backup.timer
# List:   systemctl list-timers --all
# Run now: sudo systemctl start backup.service

What Goes Wrong Mid-Deploy

1. Service won't start, no useful error. journalctl -u acme.service -n 200 first. systemd-analyze verify acme.service for syntax issues. strace as a last resort.
2. EnvironmentFile doesn't load. Mode 600 plus owner=service user; trailing newline; no shell expansion (FOO=$BAR is literal in EnvironmentFile, not expanded).
3. Migrations time out and the service starts on a half-migrated DB. Use a oneshot migration unit before the app: Before=acme.service with ExecStart=migrate. The app will fail to start if migrations fail.
4. Old releases pile up, fill disk. Prune in your deploy script (see above) or with a daily timer.
5. Symlink swap not atomic. ln -nfs deletes-then-creates with a tiny window; use ln + mv -T for true atomicity.
6. Health check returns 200 before app is ready. Watch out for frameworks that 200 on the listening port before the worker pool is warm. The health endpoint should explicitly check downstream readiness.
7. Forgot daemon-reload after editing the unit. systemd is using the old version. Always reload.

Rolling Back Fast

Speed matters more than pride. Three patterns:

• Symlink rollback. One ln -nfs <old-release> current + reload. Seconds.
• Blue-green flip. Repoint the load balancer at the old environment. Seconds.
• Canary abort. Set the canary weight to 0%. The new version is still running but receives no traffic. Seconds.

What you do not want during a rollback: a fresh git clone, npm install, build, push. "How fast can I roll back" is a property of your deploy system, not your skill at typing commands.