Anastasia Kuznetsova, Security Vision
This study was conducted as part of the agency story elaboration for a whole group of products.
Hello again! We continue our series of articles on data collection techniques (Past articles here and here). In this (third) article we have published a concise analysis of two collection technologies on Linux Endpoint based on our log management and event collection products.
Let's start, perhaps, with the most common and, if I may say so, even legacy-technology auditd or Linux Audit Daemon. This is a component of the Linux Auditing System user space responsible for collecting and saving audit log records to disk.
To understand how auditd works, you first need to understand how the Linux daemon works in general.
A daemon is a background process that is independent of the active user's interaction, i.e. a normal user cannot control the periodic execution of the daemon. It is started at Linux boot and the init process acts as its parent process. To start and stop the daemon, you must initially access the /etc/init.d scripts in the OS.
In Linux, a process daemon has the suffix d. The presence of this suffix means that the process is a system process (aka daemon), the absence of it means that the process is a user process.
The audit subsystem in Linux consists of two groups of components - in kernel space it is kauditd, and in user space it is auditd.
The auditing system is an important part of the operating system because it logs events that occur on the system. These logs can be crucial for monitoring security breaches or security incidents. However, auditd is not responsible for viewing the logs - this can be done with ausearch or aureport.
Although the classification of event logs may vary from organization to organization, they can be classified into the following types:
- System event logs (/var/log/syslog, /var/log/kern.log, /var/log/messages)
These logs are created by the operating system and contain information about system events such as startup and shutdown, system errors and warnings.
- Security event logs (/var/log/secure, /var/log/auth.log)
These logs track security-related events such as user logins, failed login attempts, and other security-related activities. You can use them to identify potential security breaches or unauthorized access to your system.
- Application event logs
These are generated by software applications and contain information about application events such as errors, warnings, and other important events.
- Server event logs (/var/log/access.log)
Generated by web servers, application servers, or databases. These logs contain information about server activity, such as server errors, uptime and downtime, performance, and security issues. This information is useful for system administrators to identify and resolve server-related problems.
- Network event logs
Generated by network devices such as routers, switches, firewalls, and load balancers. They contain information about network activity, including traffic volumes, network errors, and network connectivity problems, which helps network administrators find and fix network performance problems.
- Cloud service event logs
These are logs that are generated by cloud services such as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS) providers. These logs provide information about the health, performance, and security of cloud services and help administrators identify problems and fix them.
The event log should contain enough information to assist in troubleshooting or analyzing a problem or incident.
With auditing, you can implement logging for the following events:
- access to file system objects (ANOM_MK_EXEC; ANOM_EXEC; PATH);
- execution of system calls (EXECVE; MMAP; MQ_OPEN; SYSCALL);
- launching user commands (USER_CMD; SOCKADDR);
- user logins (USER_LOGIN; ANOM_LOGIN_FAILURES);
- actions with accounts and privileges (ADD_USER; ANOM_LOGIN_ACCT; ANOM_MOD_ACCT).
A full list of event types can be viewed here.
Although applications and databases associated with the product may implement various precautions to prevent these incidents, these measures may not be sufficient even for typical attacks such as reading and modifying system files (/etc/sudoers, /etc/passwd, /etc/shadow, /etc/crontab), running system commands/bash, running reverse shell, creating/removing users, adding processes to autoloader via init/cron/system, a remote shell running on the machine, or other threats coming from attackers who have access to the machine. Auditd was created to solve this problem.
Auditd is like a black box in an airplane: it allows a system administrator to log various system events such as commands executed, system calls, file access information, and network statistics. System administrators can use these logs to detect unusual activity and trace back to find out how the system was compromised. This post-analysis can help respond to the incident and improve security to avoid similar occurrences in the future.
The rules for auditd are added to the /etc/audit/rules.d/audit.rules file by default. These rules are read by the auditctl utility at system startup. It is recommended to use the same path to save the auditd rules to ensure uninterrupted access to the rules during reboots.
Auditd.conf is located in the /etc/audit/ directory and is the configuration file for auditd. It contains specific configuration information for the auditd daemon and includes several configuration lines, where each line has a keyword, an equals sign, and a corresponding configuration value. Some keywords available in the file are:
- log_file
- log_format
- flush
- freq
- num_logs
- max_ log_file
- max_log_ file_action
As mentioned above, auditd does not allow you to view logs. To view logs, you must use the ausearch utility. It is used to query the audit daemon logs for events based on various search criteria.
Here are some of the most common options used to query logs.
-p - search for events with the specified process ID.
-m - search for events with the specified message type.
-sv - search for events with the specified success rate.
-ua - search for an event using user id, effective user id, or login user id or auid.
-ts - search for events with timestamps equal to or later than the specified end time.
There is no consensus on how long a log should be kept. The length of retention depends on compliance requirements, the company's information security policy, storage capacity or external requirements such as GDPR, Bank of Russia standard and FSTEC orders.
It is generally considered good practice to retain logs for a minimum of 90 days and potentially longer. However, you should work with your organization's legal and compliance teams to determine the appropriate duration for your specific needs.
Where auditd cannot be used (e.g., due to its low performance), eBPF may be appropriate. Over the past few years, eBPF has become very popular in the Linux community and beyond.
Classic BPF allows a user program (e.g. tcpdump) to retrieve certain network packets based on some filter. In fact, the filtering is done inside the Linux kernel. This greatly improves performance, since only packets that pass the filtering need to be copied into user space. eBPF is in turn an evolutionary development of BPF, which is used to safely and efficiently extend kernel capabilities without having to modify the kernel source code or load kernel modules. For simplicity, we can draw an analogy with a kind of virtual machine that defines the environment in which programs are executed.
eBPF can be used anywhere in kernel space or user space and even directly on network cards. The ability to offload eBPF programs to a NIC is particularly interesting from a security point of view. This means that the kernel can only process packets after the eBPF program. The main use cases are tracking, surveillance, performance and security measurement tools. Many of the latest performance detection, monitoring and tracking tools are written using eBPF and are gaining widespread adoption in cloud environments.
eBPF programs are interesting from a security point of view because they have superpowers. And here's what they can do:
1. Connect to system calls and user-space function calls, such as manipulating user-space data structures;
2. Overwrite the return values of the system call;
3. Call the system() system call to create a new process;
4. Some eBPF programs can be run on hardware devices (e.g. network cards);
5. The eBPF attack program can be installed as rootkit;
6. To be used for escape from a container.
Here are some of the main applications of eBPF:
- Safety
eBPF enables organizations to build better security by providing the ability to monitor and filter system events in real time, enabling early detection and response to potential security threats. eBPF programs can be used to monitor network traffic, system calls, and other system events to provide a better view of potential security risks. They provide packet- and socket-level views of network activity. In addition, eBPF can be used to enforce security policies by blocking or modifying certain system events based on predefined rules.
- Monitoring and observability
BPF improves monitoring and observability by enabling dynamic and efficient tracking of various system events. eBPF programs can be used to track network packets, system calls, function calls, and other events in real time, providing deep visibility into system behavior. eBPF achieves this deep visibility while minimizing system overhead by executing programs securely within the kernel and providing a low-level interface to interact with system events. This allows for efficient and flexible control of system behavior, enabling organizations to better monitor their systems.
- Network
eBPF addresses the packet processing needs of networking solutions by providing efficient and flexible packet filtering, routing, and modification in the Linux kernel. eBPF programs can be used to process packets at various stages of the network stack, enabling a wide variety of applications including firewalls, load balancers, and network monitoring tools. The low overhead and dynamic loading capabilities of eBPF make it a powerful tool for creating productive and scalable networking solutions.
- Performance profiling and tracing
eBPF enables organizations to profile and track the runtime behavior of systems and applications by allowing the creation of custom tracing tools that can capture and analyze a wide range of system events. eBPF programs can be used to trace function calls and create complex data structures based on statistics without the need to export large amounts of data.
In addition, eBPF can be used to collect data for offline analysis, allowing for more complete and in-depth performance analysis.
eBPF allows you to create small programs that are run by the kernel in response to an event (trigger) when it passes a certain interception point (kernel trace points, function entry/exit points, system calls, etc.).
Each such program consists of two files: for execution in the kernel (..._kern.c) and from user space (..._user.c). The eBPF program defines a map (key-value store/map) and functions bound to a partition (section). When the kernel issues an event of a certain type (e.g. tracepoint), the mapped functions are executed. The maps provide data exchange between the kernel program and the user-space program. If you want to dive deep into this technology and its capabilities, I highly recommend the following book, just fill out the form and download.
Due to its architecture and operating model, eBPF has obvious advantages:
- An eBPF program will never loop or lock up. The programs themselves undergo built-in checks to ensure that they will be executed to completion. The verification process monitors such problems as: memory access outside the allocated buffer, unauthorized access to kernel functions, etc.
- It outperforms auditd by reducing the cost of system resources. Such programs run as their own machine code and only in response to some trigger event. Let's not forget that kernel-level operation avoids context switching.
- The eBPF program runs in a sandbox inside the kernel, thus providing the necessary isolation and reducing risks.
So, we took a look at the main event monitoring technologies on Linux. The situation is that, although both technologies are still encountered in the practice of collecting events, the market is gradually pulling the blanket over eBPF because of its simplicity and flexibility in configuring monitoring.
For other useful and interesting articles, visit us on Hubr and our blog on our website.
.jpg)
