Rotating Logs With Logrotate in Linux

By Syed Wahaj

Introduction

When running applications or services in a Linux environment, they often generate log files to record important information and events. Over time, these log files can grow significantly, consuming valuable disk space. Additionally, large log files make it difficult to navigate and analyze the logs efficiently.

Log rotation is a mechanism used to manage log files effectively. It involves automatically archiving or compressing old log files and creating new ones, keeping the log directory organized and preventing logs from becoming unmanageable.

One popular tool for log rotation in Linux systems is logrotate. In this article, we will explore what logrotate is, how it works, and how to set it up for rotating logs on your Linux system.

What is Logrotate?

logrotate is a system utility in Linux that automates the process of rotating log files. It reads its configuration file to determine which log files to rotate, when to rotate them, and what actions to take during rotation. The configuration file for logrotate is typically located at /etc/logrotate.conf, and it can include additional configuration files from the /etc/logrotate.d/ directory.

Understanding Log Rotation Strategies

Before we delve into the usage of logrotate, let’s understand the common log rotation strategies:

Rotate by Time: Logs are rotated based on a specified time interval, such as daily, weekly, or monthly.
Rotate by Size: Logs are rotated when they reach a certain size limit.
Rotate by Number of Files: A fixed number of log files are kept, and older log files are deleted.
Compress Old Logs: Older log files are compressed to save disk space.

Installation

logrotate is usually pre-installed on many Linux distributions. However, if it’s not available on your system, you can install it using the package manager specific to your distribution:

On Debian/Ubuntu:

sudo apt-get update
sudo apt-get install logrotate

On Red Hat-based systems:

sudo yum install logrotate

Configuration

Let’s create a sample configuration for logrotate to rotate the logs of an imaginary application called “myapp.” We’ll assume that the application logs are stored in the directory /var/log/myapp/ and we want to rotate them daily, keeping the last 7 days’ worth of logs.

Create a configuration file for “myapp” logs:

sudo nano /etc/logrotate.d/myapp

Add the following configuration to the file:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

Let’s break down the configuration directives:

daily: The logs will be rotated daily.
rotate 7: Only 7 rotated log files will be kept, and older ones will be removed.
compress: Older log files will be compressed using gzip.
missingok: If the log file is missing, do not report an error.
notifempty: Do not rotate the log if it’s empty.
create 0644 <username> <groupname>: Create new log files with the specified permissions and ownership.

Replace <username> and <groupname> with the appropriate user and group that should own the new log files.

Testing the Configuration

Before letting logrotate run automatically, it’s a good idea to test the configuration to ensure it behaves as expected. You can do this using the following command:

sudo logrotate -d /etc/logrotate.d/myapp

The -d flag is used for debugging purposes. It will display the actions logrotate would take without actually performing them.

Enabling Automatic Log Rotation

Once you have verified that the configuration works correctly, it’s time to enable automatic log rotation. logrotate comes with a built-in scheduler that runs daily through a cron job.

To check the cron job configuration for logrotate, open the cron file:

sudo nano /etc/cron.daily/logrotate

The cron job should be set up to run logrotate daily and read its configuration from /etc/logrotate.conf.

By default, most Linux distributions include this cron job out of the box, so you likely won’t need to make any changes.

Manual Log Rotation

If you ever need to rotate logs manually, you can use the following command:

sudo logrotate -f /etc/logrotate.d/myapp

The -f flag forces logrotate to rotate logs immediately, regardless of the time-based rotation settings.

Handling Log Rotation for Different Applications

In the previous section, we focused on log rotation for a single application. However, in real-world scenarios, a Linux system might have multiple applications generating logs in different directories. To manage logs for various applications, you can create separate logrotate configuration files for each application.

For example, let’s assume you have another application called “mywebapp,” which generates logs in the directory /var/log/mywebapp/. You can create a separate configuration file for this application as follows:

sudo nano /etc/logrotate.d/mywebapp

Add the configuration for the “mywebapp” logs:

/var/log/mywebapp/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

Replace <username> and <groupname> with the appropriate user and group.

By organizing log rotation configurations for each application in separate files, you can easily manage and modify log rotation settings for individual applications without affecting others.

Pre and Post Rotation Hooks

logrotate allows you to execute custom scripts before and after log rotation. These hooks provide the flexibility to perform additional actions as needed during log rotation. For instance, you may want to restart the application after log rotation to start writing logs to a fresh file.

To define pre and post rotation hooks, you can add the following directives in your logrotate configuration:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
    prerotate
        # Custom script to execute before log rotation
        /path/to/pre_rotation_script.sh
    endscript
    postrotate
        # Custom script to execute after log rotation
        /path/to/post_rotation_script.sh
    endscript
}

Make sure to replace /path/to/pre_rotation_script.sh and /path/to/post_rotation_script.sh with the actual paths to your custom scripts.

Handling Log Files with Timestamps

Some applications generate log files with timestamps in their filenames. For example, a web server might generate log files named access.log.2023-07-26, indicating logs for a specific date.

To handle such log files, you can use wildcards in the logrotate configuration. For example:

/var/log/mywebapp/access.log.* {
    daily
    rotate 14
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

With this configuration, logrotate will rotate log files with any date suffix in their filenames.

Handling Application-specific Log Rotation Frequency

Not all applications require log rotation at the same frequency. Some applications might produce log files that need rotation more frequently than others.

To address this, you can create separate log rotation configurations for different applications with their specific rotation settings. For example:

# Log rotation for myapp (rotate daily)
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

# Log rotation for mywebapp (rotate weekly)
/var/log/mywebapp/*.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

In this example, “myapp” logs are rotated daily, while “mywebapp” logs are rotated weekly.

Handling Different Log File Extensions

Some applications generate logs with various file extensions. To rotate logs with different extensions, you can use wildcard patterns in the configuration. For example:

/var/log/myapp/*.log /var/log/myapp/*.txt {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0644 <username> <groupname>
}

With this configuration, logrotate will rotate both .log and .txt files in the specified directory.

Conclusion

logrotate is a powerful tool for managing log files in a Linux environment. By understanding its configuration options and strategies, you can effectively handle log rotation for different applications, set up pre and post rotation hooks for custom actions, and handle log files with varying filenames and extensions.

Regular log rotation ensures that log directories stay organized, disk space is used efficiently, and log files remain accessible and easy to analyze. As you manage multiple applications with different log generation rates, logrotate becomes an indispensable tool for maintaining a healthy logging infrastructure on your Linux system.