Watchtower woke up, looked for updates and went back to sleep - 476 times without achieving anything.
Six months without updates - and a library broken by one
Automatic updates are usually argued as if there were one right answer. There is not. There are two real risks, they pull in opposite directions, and the difference between them is what should decide how you set things up.
FIG. 01
The standstill
The failure that never reported itself
The setup mounted a registry credentials file into the container. The path did not exist on the host - and when Docker is asked to mount something that is not there, it creates it as a directory. So Watchtower found an empty folder where it expected a file, and every single image pull failed.
The container was running. The health check was green. It simply did nothing. That is the dangerous kind of failure: not the one that knocks something over, but the one that looks like everything is fine. Had it crashed in March, I would have known in March.
The result of those 476 runs. Every one of them ended with scanned=0 and updated=0.
From the failure appearing to it being found. By accident, while reviewing something else.
Recovered from a database backup the program had left in a subfolder. Without it, the structure would have been gone.
The arithmetic: six months without security updates on a reverse proxy, a home automation system, a network controller and twenty other services - several of them reachable from the internet.
FIG. 02
The counter-example
And then the opposite problem
The same day I found a photo library of twenty-four thousand images that had been frozen since August the year before. It had broken during an upgrade: the software had jumped from version 1.127 to 2.x, and along the way the developers had switched database engines. Skip the intermediate steps and the new server cannot read the old database.
It started up, found nothing it could understand, and effectively began again from scratch. The images were still on disk, and they were recovered - but only because the program had left a database backup in a subfolder nobody had looked in.
The jump
- v1.127
- v2.x
- database unreadable
- empty library
The path that works
- v1.127
- intermediate release
- engine migrated
- v2.x
- data intact
So on the same day, in the same setup: one service broken by missing updates, and one service broken by an update.
FIG. 03
The trade-off
Both things are true
The two risks are not the same size, and they do not behave alike. That is where the answer is.
The invisible risk
Not updating means known vulnerabilities in software exposed to the internet. The risk grows quietly, and you will not notice it on your own.
The visible risk
Updating automatically means a container can come up overnight in a version that cannot read its own data. The risk is sudden - but you find out immediately.
The difference that decides
The first risk is invisible and growing. The second is visible and immediate. Visible problems are far cheaper than invisible ones, so the answer is not to pick a side, but to make the visible risk manageable.
FIG. 04
The policy
Four choices make the difference
What separates "Watchtower is on" from "Watchtower is under control" is not the tool. It is four decisions about how it is used.
Opt in, not opt out
Without --label-enable, Watchtower updates everything. With it, only containers you have explicitly labelled. That reverses the burden of proof: forgetfulness leads to too little being updated, rather than something important being swept along.
A time when somebody is awake
The default is an interval counted from when the container started, which in practice means an arbitrary hour. Saturday morning is a better choice: if something breaks you know within minutes and have the whole day to roll back. Mind the timezone - a server running UTC reads "10 o'clock" differently than you do.
Something that tells you
An automatic update without monitoring is a bet that nothing goes wrong. A health check per machine, reporting in every five minutes, is the investment that makes the automation defensible. Without it, this is not automation but hope.
Pin what must not move
Databases cannot downgrade their own data files. Your proxy and your VPN are precisely the things that give you access to fix anything. And software with a migration path has to go through the intermediate releases in order. None of those three belong in the automatic pool.
FIG. 05
The pattern
This is not about Watchtower
Both failures had the same shape: the system looked healthy while it was not working. Watchtower ran and updated nothing. The photo library ran and was missing most of its contents. Neither sent a signal.
During the same cleanup I removed twenty-three orphaned certificates, one of which had been failing its renewal daily for months. Not because they took up space - but because an error that is always there makes you blind to the next one.
That is the real point. Automatic updates are not the goal. The goal is being able to tell the difference between "it works" and "it looks like it works" - and arranging things so that difference shows up within minutes rather than months. Watchtower with those four choices is a step towards that. Watchtower switched on without them is just one more thing that can fail in silence.
FIG. 06
Next
More to read
Translation without a token bill
PlantGeekz ships in 16 languages. Some is written with a paid model, the rest translated on our own GPU. Why the cheap model is not a shortcut when the text is full of botanical names.
Read the article
Plant recognition on our own GPU
How PlantGeekz puts a name on a plant photo with BioCLIP 2 on our own GPU. What it can do, what it cannot, and why keeping it at home is an advantage.
Read the article
Large data volumes
What happens to a system when the tables grow from a hundred thousand rows to twenty million. What it costs not to have the data model under control, and what I do about it.
Read the article
Got something that needs building properly?
Write a few lines about what you are dealing with. I will come back with what I think, including if the answer is that someone else would do it better.