Docker's image age is somebody else's clock

· 7 min read
Contents

The CREATED column in docker image ls looks like an age, and on a machine that pulls its images, it is somebody else's. Created is stamped when the image is built, upstream, and travels inside it. A postgres tag pulled yesterday can show two months.

On my workstation, a disk cleanup I ran removed images older than 30 days, and it read their age from Created. Of the 93 images removed in that pass, 71 had been pulled within the previous 30 days, and 27 of the removed images were referenced by project files.

111 GB of build cache and 247 orphan volumes#

The starting point, before anything was deleted: 111 GB of build cache, 80 GB of images (300 of them, 3 used by a container), and 16 GB of volumes, 247 of them attached to nothing. The machine uses Docker's containerd image store, so the image data sat under /var/lib/containerd, not /var/lib/docker.

The first pass went after build cache, 6.6 GB of dangling images, the unattached anonymous volumes, a benchmark stack's named Postgres volumes and mock images, and "old" public images. The last item is where the clock mattered.

Created is when someone else built it#

docker image inspect carries two timestamps that could each pass for an age. Created is, in the API's words, when "the image was created", and it travels inside the image: the same value on every machine that pulls it. Metadata.LastTagTime is "local to the daemon, and not part of the image itself", the last time this daemon tagged the image.

Not a command from the cleanup, just the two fields side by side:

docker image inspect -f '{{.Created}}  {{.Metadata.LastTagTime}}' <image>

On the containerd image store a pull updates the image record, and LastTagTime reports the newest update, so it works as the local arrival time. On the older overlay2 store it doesn't: per the moby source, only tag, build, commit and import set it, and an image that was only ever pulled carries a zero date. Check which store you have before trusting either field.

Even the right field is only an approximation. It says when an image arrived, not when anything last ran it.

Counting the damage from saved inspect JSON#

The inspect JSON had been saved before the removal, so the damage could be counted after the fact instead of guessed. Reading LastTagTime from it gave the 71; checking the removed images against project files gave the 27.

Those 27 came back by their exact previous digest: docker pull repo@sha256:…, then docker tag back to the old name. A plain docker pull repo:tag would have fetched whatever the tag points at today; the digest gets each project the image it had before. One pull failed transiently and went through on retry.

The 27 now carry the day of the restore as their pull time. Their real pull dates survive only in the saved JSON. And a digest alone doesn't prove an image can be pulled back: on the containerd image store, even locally built images get one.

Docker keeps no last-used date#

Docker has no record of when an image was last used. docker events returns only the last 256 events, held in memory, so it can't speak for last month. Usage history exists only if you start logging it before you need it.

247 volumes with no owner#

Anonymous volumes are worse. Each one carries a single label, com.docker.volume.anonymous, and once its container is gone nothing on the volume says which project made it.

So I looked inside instead: mount many volumes read-only into one Alpine container and check for marker files, PG_VERSION for Postgres, WiredTiger for MongoDB, mnesia for RabbitMQ, dump.rdb for Redis. 75 of the orphans held throwaway MongoDB instances, each with nothing but system collections and about 200 MB of preallocated journal.

They came from one benchmark script. It ended with docker compose down, without -v, and its EXIT trap ran the same command, so every run left its anonymous volumes behind. Images often declare their data directories as volumes (the official mongo image has VOLUME /data/db /data/configdb), and a container that starts without a mount for them gets anonymous ones nobody wrote into the compose file. The fix is an order of operations:

docker compose rm -fsv
docker compose down

rm -fsv stops and removes the stack's containers together with their anonymous volumes and leaves named volumes alone. The order matters: after down the containers are already gone, rm finds nothing to attach -v to, and every volume stays. down -v is no shortcut either, since it removes the named volumes too. The two-volume test and the trap version are in Stop docker compose from orphaning anonymous volumes.

Six more anonymous volumes held Postgres data directories nobody could place. strings on the catalog file listed their database names, and each database came to about 7.5 MB, which is what an empty one weighs. Test databases, never filled.

A stack that started mid-cleanup#

During the cleanup, a benchmark stack from another session came up. Docker refused to delete anything it was using, which is a safety net the daemon provides for free. It was still a reminder to run docker ps again before each destructive step, not just at the start.

A weekly tidy that only deletes what comes back#

What came out of it is a user-level systemd timer for Sunday 04:00, with Persistent=true so a missed run catches up the next time the timer starts, and a wait of up to ten minutes for Docker to come up. It removes only things that regenerate:

  • stopped containers created more than 30 days ago, except those matching a protect pattern for projects under test;
  • dangling images;
  • unattached anonymous volumes created more than 14 days ago, minus a keep-list;
  • build cache above 40 GB, least recently used first, via --max-used-space.

Tagged images it never deletes. It reports the ones pulled 60 or more days ago, not started in 60 days according to its own event log, and not referenced by any compose file, Dockerfile or Testcontainers code. A companion service appends every container start to a JSONL file, which gives the tidy the last-used date Docker doesn't keep.

Before the first Sunday run, a review of the weekly tidy's rules pointed out what its container rule would do: delete three stopped MQTT watcher containers, and the UPS telemetry they had captured with them. The telemetry was saved first, 8,245 lines of it, and then the containers went. Their command lines also held a password in plain text: anything in a container's command line is readable by anyone who can run docker inspect.

By its own rules, the tidy would have been right. Stopped, created more than 30 days ago, matching no protect pattern: those containers were exactly what it was built to remove. Nothing the rule looked at (state, creation date, protect pattern) said they held anything worth keeping.

Filed under: docker, containers, linux

Sources#