Linux Audit Logging

Linux Audit Logging: Core Concepts

When a security incident occurs, the first question is always "what happened?" Without audit logging, the answer is guesswork. The Linux audit subsystem (auditd) records syscall-level events — file access, command execution, authentication attempts, privilege changes — providing a detailed forensic trail that survives even if an attacker modifies application logs.

How auditd Works

auditd is a userspace daemon that receives event data from the kernel's audit subsystem. The kernel watches syscalls based on rules you define, and when a matching syscall occurs, it generates an audit record. auditd writes these records to disk (typically /var/log/audit/audit.log) and can forward them to a remote syslog server.

The key advantage over application logging is that audit records come from the kernel, not from the application. An attacker who compromises an application cannot easily suppress kernel-level audit records without also compromising the kernel itself.

Audit Rules: What to Watch

Audit rules fall into three categories:

File watches — Monitor access to specific files or directories:

# Watch /etc/passwd for any read, write, or attribute change
auditctl -w /etc/passwd -p wa -k identity_changes

# Watch /etc/shadow
auditctl -w /etc/shadow -p wa -k shadow_access

# Watch a directory recursively
auditctl -w /etc/nginx/ -p wa -k nginx_config

The -p flag controls which access types trigger an event: r (read), w (write), x (execute), a (attribute change). The -k flag assigns a key for searching.

System call rules — Monitor specific syscalls across the system:

# Log all executions of the 'execve' syscall (command execution)
auditctl -a always,exit -F arch=b64 -S execve -k command_execution

# Log all 'setuid'/'setgid' calls (privilege escalation)
auditctl -a always,exit -F arch=b64 -S setuid -S setgid -k privilege_escalation

# Log all 'open' calls on a specific file by a specific user
auditctl -a always,exit -F arch=b64 -S open -F path=/etc/shadow -F perm=r -k shadow_read

Immutable rules — Rules that cannot be removed without a reboot:

# Set the audit configuration to immutable (requires reboot to change)
auditctl -e 2

This prevents an attacker from disabling audit logging after gaining root access.

Reading Audit Logs

Use ausearch to query audit records:

# Search by key
ausearch -k identity_changes

# Search for events from a specific user
ausearch -ua 1000

# Search for events in a time range
ausearch -ts today -k command_execution

# Search for a specific syscall
ausearch -sc execve

aureport generates summary reports:

# Report on authentication events
aureport --auth --summary

# Report on executed commands
aureport --execve --summary

# Report on file access
aureport --file --summary

Where It Fits

auditd provides the forensic trail, but it does not prevent attacks. It works alongside preventive controls (firewalls, SSH hardening) and detective controls (intrusion detection systems, log analysis). Study nftables first to understand network-level controls, then use auditd to record what happens at the system level.

Linux Audit Logging: Practice

Audit logging is only useful if you can find the relevant events when something goes wrong. The exercises below build from basic file watching to comprehensive syscall monitoring.

Exercise 1: Watch Critical System Files

Monitor the files that attackers commonly target for persistence and privilege escalation:

# Watch authentication files
auditctl -w /etc/passwd -p wa -k identity_changes
auditctl -w /etc/shadow -p wa -k shadow_access
auditctl -w /etc/group -p wa -k identity_changes

# Watch SSH configuration
auditctl -w /etc/ssh/sshd_config -p wa -k ssh_config_change

# Watch sudoers
auditctl -w /etc/sudoers -p wa -k sudoers_change
auditctl -w /etc/sudoers.d/ -p wa -k sudoers_change

# Watch cron directories (common persistence mechanism)
auditctl -w /etc/cron.d/ -p wa -k cron_changes
auditctl -w /etc/crontab -p wa -k cron_changes

Verify the rules are active:

auditctl -l

Test by modifying a watched file:

echo "test" >> /etc/crontab
ausearch -k cron_changes --start recent

You should see an audit record showing the write to /etc/crontab.

Exercise 2: Monitor Command Execution

Log every command executed on the system:

# Log all execve syscalls (64-bit)
auditctl -a always,exit -F arch=b64 -S execve -k command_execution

# Log all execve syscalls (32-bit, for compatibility)
auditctl -a always,exit -F arch=b32 -S execve -k command_execution

Search for commands run by a specific user:

ausearch -k command_execution -ua 1000 --interpret

The --interpret flag translates numeric IDs to human-readable names (UIDs, GIDs, syscall names).

Exercise 3: Detect Privilege Escalation

Monitor for privilege escalation attempts:

# Log setuid/setgid calls
auditctl -a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation

# Log sudo usage
auditctl -w /var/log/sudo.log -p wa -k sudo_usage

Generate a report of privilege escalation events:

ausearch -k privilege_escalation --interpret | less

Exercise 4: Make Rules Persistent

Rules set with auditctl are lost on reboot. To persist them:

# Save current rules
auditctl -l > /etc/audit/rules.d/audit.rules

# Or write rules directly to the rules directory
cat > /etc/audit/rules.d/monitoring.rules << 'EOF'
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k shadow_access
-w /etc/ssh/sshd_config -p wa -k ssh_config_change
-a always,exit -F arch=b64 -S execve -k command_execution
-a always,exit -F arch=b64 -S setuid -S setgid -k privilege_escalation
EOF

# Reload rules
augenrules --load

Check Your Understanding

• Why is kernel-level audit logging more trustworthy than application-level logging?
• What does -p wa on a file watch actually monitor?
• How does auditctl -e 2 protect audit logs from an attacker with root access?

Linux Audit Logging: Trade-offs and Interviews

Audit logging generates data — lots of it. Without proper planning, you will drown in log volume while missing the events that matter. Understanding the trade-offs is essential for designing an effective logging strategy.

Log Volume vs. Coverage

Logging every syscall on a busy server generates gigabytes of data per day. This creates storage costs, makes searching slow, and can overwhelm the auditd daemon itself.

Strategies to manage volume:

• Be selective with syscall rules: Instead of logging all execve calls, log only executions by specific users or from specific directories.
• Use rate limiting: auditd can be configured to drop events if the rate exceeds a threshold.
• Forward to a centralized SIEM: Offload storage and analysis to a dedicated log server or SIEM (Security Information and Event Management) system.
• Set log rotation: Configure auditd.conf to rotate logs based on size or time.

Performance Impact

Every audited syscall incurs a kernel-level overhead. On a high-traffic database server, auditing every file access can measurably impact performance. The general rule: audit on security-critical systems (SSH servers, web servers facing the internet) and be more selective on performance-critical systems (databases, high-frequency trading).

When Audit Logging Is the Wrong Tool

• Real-time alerting: auditd writes events to disk. If you need real-time alerting (e.g., page someone when /etc/shadow is accessed), you need a log shipper (Filebeat, Fluentd) that streams events to an alerting system.
• Application-level context: auditd records syscalls, not application logic. If you need to know which user ran which database query, you need application-level logging.
• Compliance reporting: While auditd provides raw data, compliance frameworks often require formatted reports. Use aureport or export to a SIEM for compliance reporting.

How This Comes Up in Interviews

• "How would you detect if an attacker modified /etc/passwd?" Set up a file watch with auditctl -w /etc/passwd -p wa -k identity_changes, then search with ausearch -k identity_changes. The audit record will show the user, timestamp, and the syscall that modified the file.
• "How do you prevent an attacker from disabling audit logging? Use auditctl -e 2 to set the configuration to immutable. This requires a reboot to change, so an attacker cannot disable logging without rebooting the system (which itself generates alerts).
• "How do you handle audit log volume on a busy server? Be selective with rules, forward to a centralized SIEM, configure log rotation, and use rate limiting. Focus on high-value targets (authentication files, sudoers, SSH config) rather than trying to log everything.