Understanding Linux Logs: Overview with Examples

Understanding Linux Logs blog image

Logging and program output are woven into the entire Linux system fabric. In Linux land, logs serve as a vital source of information about security incidents, application activities, and system events. Effectively managing your logs is crucial for troubleshooting and ensuring compliance. This article explores the importance of logging and the main types of logs before covering some helpful command line tools and other tips to help you manage Linux logs.

The importance of logging

Logging provides a high-level overview of the applications and processes your system runs automatically in the background. These logs provide an audit trail to discover what a system is doing or not doing and the successes and errors generated during those activities. Many applications in the Linux ecosystem produce output that is readable by humans and, if that output is redirected to a log file, automatically parsed by tools included on a Linux system. Though you don’t necessarily have to go into a log to do this, retrieving historical information without that info existing in a log would be difficult, if not impossible. 

It’s important to recognize one of the significant benefits this facilitates in the workplace. By seamlessly integrating the user interface with the underlying system architecture, the feedback loop is shortened during the development phase and beyond. It then generates valuable output that can be transformed into actionable insights throughout your application lifecycle.

Types of logs

  • System logs: capture information about the operating system, such as startup messages, hardware events, and kernel activities. 
  • Application logs: record specific software-related events, such as maintenance messages, errors, warnings, and user activities.
  • Security logs: keep track of security-related events, including unauthorized access, regular logins, and firewall activities. 

Any of these can then serve as input or trigger to apps or scripts — including integration with most serious enterprise monitoring software.

Syslog and journald log formats

Log files can be stored in different formats, two of the most common being journalized (centralized) system logs and plaintext logs. 

Traditionally, the syslog format is stored in plaintext logs, which are easily read and analyzed. This format has advantages and disadvantages, as does journald, which stores logs in a binary format that are readable by the journalctl command. 

Most modern Linux distributions tend to utilize systemd’s journald logs for at least their core system apps. Systemd journal logs provide advantages such as centralized logging facilities, real-time log monitoring of structured and indexed log data, and automatic log encryption. By comparison, traditional Linux plaintext logs’ arguably less restrictive nature might seem riskier at first glance. Don’t be fooled, though; plaintext logs still have many uses.

Most or all of the functions built into systemd, such as remote logging systems, centralized monitoring, log rotation, and backups, are also offered by other standard Linux packages. One disadvantage of journald is its comparatively “heavy” monolithic nature compared to the more granular Linux command-line tools used to interact with syslog files, like grep, awk, or tail. By comparison, when you use journald, you get all of its features installed and running: storage, encryption, centralized logging, log rotation, and more — whether or not you need it.

Feature comparison: syslog vs journald logging

Feature syslog journald
Storage Format                                                                                  Plain text files Binary format with structured data
Compatibility Widely supported by various log management tools Requires systemd integration for full functionality
Log Rotation Manual configuration of log rotation Automatic rotation and compression of log files
Centralized Management Relies on external log servers and tools Built-in support for centralized logging
Filtering and Parsing Advanced text filtering and parsing functionality from the standard Linux toolchain Advanced filtering and querying using journalctl
Storage Efficiency May consume more disk space due to plain text format More efficient storage due to binary and indexed format
Real-time Monitoring Requires external tools or tailing log files Built-in real-time log monitoring with journalctl
Reliability Vulnerable to log file corruption or loss in case of crashes Atomic writes and resilient journal file structure
Encryption External encryption mechanisms may be required Supports encryption of log data

Filtering and parsing log data: useful examples

Journald examples:

Example: journalctl -p err

Explanation: Filters the journal logs to display only entries with “err” severity level, which represents errors. This allows you to focus on error messages and quickly identify any critical issues in the system logs.

Example: journalctl _SYSTEMD_UNIT=nginx.service

Explanation: Filters the journal logs to display entries related to the nginx.service. Helps you isolate logs specific to e.g. the Nginx web server, making it easier to troubleshoot issues or monitor its activities.

Example: journalctl –since “2023-07-21” –until “2023-07-22”

Explanation: Filters the journal logs to display entries between the specified dates, in this case, from July 21, 2023, to July 22, 2023. Lets you focus on log entries within a specific time range, helping with debugging or investigating events that occurred during that period.

Syslog examples:

Example: grep -i “warning” syslog.log | grep -v “deprecated”

Explanation: Performs a two-step filtering process. Firstly uses grep with the -i option to do a case-sensitive search for lines matching the string “warning” in syslog.log, then pipes the output to another grep command with the -v option, excluding lines matching the string “deprecated”. This lets you filter out warning messages specifically marked as deprecated.

Example: grep ‘string1|string2[|string3]’ logfile.log

Explanation: Effectively an OR search using grep. Using single quotes and escaping the pipe symbol in between enables you to search for multiple strings in a single grep command.

Other logging tool tips

While the standard Linux tools are employed for the system’s own benefit, it is also useful to employ these tools elsewhere:

  • logrotate: Useful in your app development phase to help manage your debugging output logs
  • tail -f: the “tail” command prints the last 10 lines of a file before exiting. The “-f” switch “follows” the log file, watches for new log file entries, and updates as new lines are added.
  • multitail: Displays multiple log files in tiled panes in real-time so you can monitor them simultaneously.

In this article, we discussed various aspects of logging’s significance, compared different types of logs, and touched on less usual use cases for logging in your own work. We also explored a number of command examples and other tools that contribute to effective Linux log managementUse Linux logs to track information about Linux operating systems, applications, and systems and better manage your Linux endpoints. Learn more about NinjaOne’s remote monitoring and endpoint management software, and how it can help you more easily manage your distributed Linux devices.

Next Steps

The fundamentals of device security are critical to your overall security posture. NinjaOne makes it easy to patch, harden, secure, and backup all their devices centrally, remotely, and at scale.

You might also like

Ready to become an IT Ninja?

Learn how NinjaOne can help you simplify IT operations.

Watch Demo×
×

See NinjaOne in action!

By submitting this form, I accept NinjaOne's privacy policy.

NinjaOne Terms & Conditions

By clicking the “I Accept” button below, you indicate your acceptance of the following legal terms as well as our Terms of Use:

  • Ownership Rights: NinjaOne owns and will continue to own all right, title, and interest in and to the script (including the copyright). NinjaOne is giving you a limited license to use the script in accordance with these legal terms.
  • Use Limitation: You may only use the script for your legitimate personal or internal business purposes, and you may not share the script with another party.
  • Republication Prohibition: Under no circumstances are you permitted to re-publish the script in any script library belonging to or under the control of any other software provider.
  • Warranty Disclaimer: The script is provided “as is” and “as available”, without warranty of any kind. NinjaOne makes no promise or guarantee that the script will be free from defects or that it will meet your specific needs or expectations.
  • Assumption of Risk: Your use of the script is at your own risk. You acknowledge that there are certain inherent risks in using the script, and you understand and assume each of those risks.
  • Waiver and Release: You will not hold NinjaOne responsible for any adverse or unintended consequences resulting from your use of the script, and you waive any legal or equitable rights or remedies you may have against NinjaOne relating to your use of the script.
  • EULA: If you are a NinjaOne customer, your use of the script is subject to the End User License Agreement applicable to you (EULA).