Logging Docker using syslog-ng

Event logging is a central source of information for IT operations and security. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or forwards them for further analysis. The same feature set comes handy also in a containerized environment. Containerization, and Docker in particular, changed the way we distribute and run applications. Containers provide isolated environments, which make it possible to run applications with conflicting dependencies on the same host. There are even dedicated container hosts, like Atomic Host, which do not allow you to install any applications on the host directly, only in containers. You can install your central syslog-ng server in a container and enjoy all benefits of containerization. You can use syslog-ng also for collecting Docker logs. Docker already provides many drivers for logging, even for central log collection. On the other hand remote logging drivers arrive with a minimalist feature set and you are not able to use the “docker logs” command any more. To have the best of both worlds, you can use the journald logging driver in Docker, and use syslog-ng to read Docker logs from journald and forward log messages to your central log server or other destinations. There are many software that log to files or pipes instead of their stdout, the place where Docker expects them. Fortunately by using Docker volumes you can share data among containers and syslog-ng can collect these logs as well. The use of the wildcard-file source gives you additional flexibility. These look like three separate use cases, but you can freely combine any of these and utilize many more syslog-ng features, like message parsing and enrichment or Big Data destinations. I help you to get started with any of these by giving a quick introduction to the configuration of syslog-ng and showing example Docker command lines as well. By the end of my talk you should be ready to deploy syslog-ng in Docker and create a simple syslog-ng configuration.