← All field notes
App Security

Zombie Container: Why Your Deploys Fail Silently and the Old Version Keeps Serving

The deploy says success, the site still runs old code, and a worker you thought was gone keeps processing jobs. Eight checks to find the zombie, with the commands.

Matthew TurleyNovember 5, 20266 min read

Your deploy log ends in a green tick. You open the site and the change is not there. Or the change is there, and an hour later a customer reports behavior from last week. Either way, something old is still running, and nothing told you the new version failed to take over.

This is the zombie container problem: an old container that should have been replaced or stopped, still serving traffic or still pulling jobs from a queue. It matters beyond annoyance. An old web container can keep serving a bundle that still contains a key you removed, and an old worker can keep processing payments with stale settings. Here are the checks, in the order that finds it fastest.

1. Ask the running app which version it is

Before touching Docker, make the app tell you what it is. Add a tiny endpoint that returns the commit your build was made from.

// app/api/version/route.ts
export function GET() {
  return Response.json({ sha: process.env.GIT_SHA ?? 'unknown', built: process.env.BUILD_TIME ?? 'unknown' })
}

Pass GIT_SHA in as a build argument in CI. Now compare curl https://yourapp.com/api/version with git rev-parse HEAD. If they differ, you have an old version serving, and you have proof, not a hunch.

2. List what is actually running, with creation times

docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.CreatedAt}}\t{{.Status}}'

Look for three things: a container created before your deploy, a name you do not recognize (an old service name from before a rename), and a status like Restarting or unhealthy. A new container that keeps restarting is the cause of "deploy succeeded, nothing changed" in most setups.

3. Read why the new container did not take over

docker ps -a --filter status=exited --filter status=restarting
docker logs --tail 100 <new-container-name>

The new container's last 100 lines usually name the problem: a missing environment variable, a port already in use, a migration that failed, a build that needs a different Node version. Your platform kept the old container up while the new one crashed. That is a reasonable safety behavior, and it only becomes a silent failure when nothing reports it.

4. Check who owns the port

If the new container says the address is already in use, an old process holds it.

ss -ltnp | grep ':3000'

If it shows a process outside Docker, or an old container you thought was removed, stop that one. With Docker Compose, a renamed service leaves the old container behind as an orphan, and docker compose up -d --remove-orphans clears it.

5. Make sure your settings change actually reached the container

A restart does not apply a changed environment. Environment variables are fixed when the container is created, so docker restart keeps the old values. Recreate it instead:

docker compose up -d --force-recreate <service>

This is a classic cause of "I rotated the key and the app still uses the old one". If you rotated a leaked secret and did not recreate the container, the exposed key may still be live in the running process.

6. Look for the zombie worker

The dangerous zombie is not the web container. It is the background worker, the cron runner or the queue consumer. Two workers reading the same queue means a job can be picked up by the old one, with old code and old settings, and your fix appears to work only half the time. Count the consumers:

docker ps --filter name=worker --format '{{.Names}} {{.Image}} {{.CreatedAt}}'

You expect one per worker service, all created after the deploy. If a payment or email job is processed by an old worker, that can look like an order that "randomly" fails.

7. Check that the image tag is really new

Deploying :latest with a cached image is a quiet failure: the deploy pulls nothing new because the tag name did not change.

docker inspect --format '{{.Image}}' <container-name>
docker images --digests | head

Compare the image digest in the container with the one you pushed. Deploy by commit SHA tag, not by latest, so a new build always means a new tag.

8. Make a failed deploy loud

The root fix is that a failed rollout must announce itself. Three small changes:

  • A real health check. A check that only returns 200 for the home page passes while the database connection is broken. Have the health endpoint touch the database, and set the container's health check to call it.
  • Fail the pipeline when the deploy step fails. Look for || true or ignored exit codes in your deploy script. They turn a failed deploy into a green tick.
  • An outside probe after every deploy. A job that calls /api/version and fails the pipeline when the SHA does not match the commit just deployed. This is the one that turns a silent failure into a red build.

What the free check and the $47 scan can and cannot tell you

An outside scan cannot see your containers, your logs or your queue, so it cannot find a zombie. What it can do is look at what the old version is serving the public. If you removed an exposed key or a debug route in the latest commit, run the free Leak Check after the deploy: if the key is still in the live bundle, the old version is still answering.

The $47 Complete Scan does the same from further out: exposed config such as .env, /.git and source maps, security headers, auth endpoints and anonymous table access. Run it after a deploy to confirm that what you thought you fixed is not still visible from the internet. It does not replace the version endpoint above.

Next step

Add the /api/version endpoint today and compare it with your latest commit. That one comparison tells you whether you have a zombie right now. Then run the free Leak Check on the live URL to confirm what a stranger sees after your last deploy. If deploys keep surprising you, the underlying gap is in the path from commit to running container, which is the subject of the post on why you cannot deploy the app you built and the MVP-to-production gap.

Related: Stripe webhook works in test, fails in production.

Worried your app has holes like these?

Drop your email and we will run a free teardown of your stack and send back the leaks we find, no pitch.

M
Matthew Turley, Continuum

Fractional CTO and embedded technical partner. 20+ years shipping production software.

Run a free Leak Check →
Zombie Container Failing Deploys Silently: 8-Step Checklist