Logging and program output are woven into the entire Linux system fabric. In Linux land, logs serve as a vital source of information about security incidents, application activities, and system events. Effectively managing your logs is crucial for troubleshooting and ensuring compliance. This article explores the importance of logging and the main types of logs before covering some helpful command line tools and other tips to help you manage Linux logs.
The importance of logging
Logging provides a high-level overview of the applications and processes your system runs automatically in the background. These logs provide an audit trail to discover what a system is doing or not doing and the successes and errors generated during those activities. Many applications in the Linux ecosystem produce output that is readable by humans and, if that output is redirected to a log file, automatically parsed by tools included on a Linux system. Though you don’t necessarily have to go into a log to do this, retrieving historical information without that info existing in a log would be difficult, if not impossible.
It’s important to recognize one of the significant benefits this facilitates in the workplace. By seamlessly integrating the user interface with the underlying system architecture, the feedback loop is shortened during the development phase and beyond. It then generates valuable output that can be transformed into actionable insights throughout your application lifecycle.
Types of logs
- System logs: capture information about the operating system, such as startup messages, hardware events, and kernel activities.
- Application logs: record specific software-related events, such as maintenance messages, errors, warnings, and user activities.
- Security logs: keep track of security-related events, including unauthorized access, regular logins, and firewall activities.
Any of these can then serve as input or trigger to apps or scripts — including integration with most serious enterprise monitoring software.
Syslog and journald log formats
Log files can be stored in different formats, two of the most common being journalized (centralized) system logs and plaintext logs.
Traditionally, the syslog format is stored in plaintext logs, which are easily read and analyzed. This format has advantages and disadvantages, as does journald, which stores logs in a binary format that are readable by the journalctl command.
Most modern Linux distributions tend to utilize systemd’s journald logs for at least their core system apps. Systemd journal logs provide advantages such as centralized logging facilities, real-time log monitoring of structured and indexed log data, and automatic log encryption. By comparison, traditional Linux plaintext logs’ arguably less restrictive nature might seem riskier at first glance. Don’t be fooled, though; plaintext logs still have many uses.
Most or all of the functions built into systemd, such as remote logging systems, centralized monitoring, log rotation, and backups, are also offered by other standard Linux packages. One disadvantage of journald is its comparatively “heavy” monolithic nature compared to the more granular Linux command-line tools used to interact with syslog files, like grep, awk, or tail. By comparison, when you use journald, you get all of its features installed and running: storage, encryption, centralized logging, log rotation, and more — whether or not you need it.
Feature comparison: syslog vs journald logging
| Feature | syslog | journald |
| Storage Format | Plain text files | Binary format with structured data |
| Compatibility | Widely supported by various log management tools | Requires systemd integration for full functionality |
| Log Rotation | Manual configuration of log rotation | Automatic rotation and compression of log files |
| Centralized Management | Relies on external log servers and tools | Built-in support for centralized logging |
| Filtering and Parsing | Advanced text filtering and parsing functionality from the standard Linux toolchain | Advanced filtering and querying using journalctl |
| Storage Efficiency | May consume more disk space due to plain text format | More efficient storage due to binary and indexed format |
| Real-time Monitoring | Requires external tools or tailing log files | Built-in real-time log monitoring with journalctl |
| Reliability | Vulnerable to log file corruption or loss in case of crashes | Atomic writes and resilient journal file structure |
| Encryption | External encryption mechanisms may be required | Supports encryption of log data |
Filtering and parsing log data: useful examples
Journald examples:
Example: journalctl -p err
Explanation: Filters the journal logs to display only entries with “err” severity level, which represents errors. This allows you to focus on error messages and quickly identify any critical issues in the system logs.
Example: journalctl _SYSTEMD_UNIT=nginx.service
Explanation: Filters the journal logs to display entries related to the nginx.service. Helps you isolate logs specific to e.g. the Nginx web server, making it easier to troubleshoot issues or monitor its activities.
Example: journalctl –since “2023-07-21” –until “2023-07-22”
Explanation: Filters the journal logs to display entries between the specified dates, in this case, from July 21, 2023, to July 22, 2023. Lets you focus on log entries within a specific time range, helping with debugging or investigating events that occurred during that period.
Syslog examples:
Example: grep -i “warning” syslog.log | grep -v “deprecated”
Explanation: Performs a two-step filtering process. Firstly uses grep with the -i option to do a case-sensitive search for lines matching the string “warning” in syslog.log, then pipes the output to another grep command with the -v option, excluding lines matching the string “deprecated”. This lets you filter out warning messages specifically marked as deprecated.
Example: grep ‘string1|string2[|string3]’ logfile.log
Explanation: Effectively an OR search using grep. Using single quotes and escaping the pipe symbol in between enables you to search for multiple strings in a single grep command.
Other logging tool tips
While the standard Linux tools are employed for the system’s own benefit, it is also useful to employ these tools elsewhere:
- logrotate: Useful in your app development phase to help manage your debugging output logs
- tail -f: the “tail” command prints the last 10 lines of a file before exiting. The “-f” switch “follows” the log file, watches for new log file entries, and updates as new lines are added.
- multitail: Displays multiple log files in tiled panes in real-time so you can monitor them simultaneously.
In this article, we discussed various aspects of logging’s significance, compared different types of logs, and touched on less usual use cases for logging in your own work. We also explored a number of command examples and other tools that contribute to effective Linux log management. Use Linux logs to track information about Linux operating systems, applications, and systems and better manage your Linux endpoints. Learn more about NinjaOne’s remote monitoring and endpoint management software, and how it can help you more easily manage your distributed Linux devices.
