I recently switched to using Docker volumes for logs., to reduce installation steps and coupling to infrastructure. I created a single shared volume by stack.
However this implied that logs from stopped containers were never removed… because logs rotation was not doing its work anymore.
I then understood better why 12 factors app practices recommends to only log to STDOUT, and let Docker handle with log storage and removal. And I decide to adapt.
- Stop JSON logging on files
- Replace human readable logging on STDOUT by json logging
- Change filebeat configuration to container mode
At the same time, I benefited from this change in many ways:
- Traefik, metricbeat and filebeat logs are also captured
- Elasticsearch and kibana logs are captured on dev
- All logs have been switched to JSON
- Filebeat allow enriching logs with Docker metadata, allowing to know easily the origin of the log
- Filebeat processors and scripting allow reshaping the logs to have a common format for all sources 🙂 Thanks Elastic devs!
It is all in place and deployed! More observability than ever.