Unveiling the Gaps: Linux EDR Telemetry in Focus

Introduction

Visibility serves as the cornerstone for understanding and mitigating modern threats. The EDR Telemetry Project was established as a comprehensive effort to evaluate the depth and breadth of telemetry provided by Endpoint Detection and Response (EDR) solutions for Windows systems. Its purpose has been to illuminate the gaps that often go unnoticed, providing an objective measure of whether defenders can depend on these tools to identify, analyze, and respond confidently to adversarial activity.

If you don’t have medium Premium, you can read this blog here: https://kostas-ts.medium.com/unveiling-the-gaps-linux-edr-telemetry-in-focus-1290a010ad1b?sk=2f3ced9dbd47c83acd9e6b8fe26af119

As the project moved forward, we noticed an important area where we could improve: the need to put more emphasis on Linux systems. Linux is no longer a niche platform; it is the backbone of modern IT. From cloud environments to enterprise workloads and critical infrastructure, Linux powers much of what we depend on daily.

The decision to expand the project to include Linux was necessary. With attackers increasingly targeting Linux environments with sophisticated threats, defenders must have the tools to monitor and respond effectively. This article delves into why Linux telemetry matters…

The Rising Tide of Linux Threats

For years, Linux enjoyed a reputation as a “safe” operating system. Its smaller market share compared to Windows, combined with its robust permission model, made it less attractive to attackers. That era is over.

• Attackers are now actively targeting Linux systems with specialized malware, such as the perfctl malware, which was designed to evade detection and compromise millions of servers (AquaSec, 2024).
• Ransomware families like Helldown have also been observed expanding their capabilities to encrypt Linux systems, a sign of the growing value attackers see in these environments (Sekoia, 2024).
• These threats often exploit the unique characteristics of Linux environments, such as exposed Docker services, misconfigured Apache Hadoop instances, or vulnerable Redis deployments (Cado Security, 2024). The ability to evade detection through stealthy techniques — like kernel-level exploitation — adds to the challenge.

This trend underscores the critical need for enhanced Linux telemetry. Without it, defenders face blind spots that attackers can easily exploit. The challenge lies in detecting and understanding these attacks promptly to respond effectively, highlighting the importance of a swift response.

Expanding the EDR Telemetry Project to Linux

Adding Linux to the EDR Telemetry Project wasn’t a simple extension of the Windows framework. It required a complete rethinking of how to approach telemetry testing. Linux environments operate differently, with processes, services, and user activities managed in ways unique to the platform.

For instance, while Windows has well-defined service management through the Service Control Manager (SCM), Linux services are often managed via systemd, cron, or init.d. Monitoring service creation or modification on Linux means tracking changes to specific directories, such as /etc/systemd/system or /var/spool/cron.

Similarly, capturing user account changes requires monitoring system calls rather than relying on high-level process activity, as one might on Windows. This nuanced approach ensures the testing reflects real-world Linux operations and avoids superficial comparisons with Windows systems.

Through collaboration with the community — on platforms like GitHub, discord conversations in the EDR Telemetry Server and during one-on-one discussions— the framework was refined to focus on meaningful events. This collaboration improved the testing methodology and highlighted the importance of tailoring solutions to Linux’s unique needs.

Methodology and Testing Framework

Creating a comprehensive framework for testing Linux telemetry required a focus on accuracy, transparency, and relevance. Below, we outline the guiding principles and provide a detailed explanation of the categories and events tested.

Guiding Principles for Testing

1. Default Configurations
   To ensure fairness, all EDR solutions were tested with their default configurations. This reflects how these tools are deployed in real-world scenarios, where defenders may not have the time or expertise to fine-tune every setting. Testing with default setups highlights out-of-the-box capabilities and identifies areas where additional configuration may be needed.
2. Syscall-Based Event Generation
   The testing script uses system calls (syscalls) to generate events directly. By bypassing high-level processes like command-line utilities, we ensure the results focus on actual telemetry capabilities rather than inferred activities.
   For example — user account creation events were generated using syscalls directly, bypassing the need to execute useradd or adduser commands.
3. Linux-Specific Context
   Linux environments operate differently from Windows, so the testing framework was tailored to address these differences. Categories and events were chosen to reflect real-world attacker behaviours and defender needs, ensuring the results were relevant and actionable.

Categories and Events Tested

The testing categories and events include:

Process Activity

• Process Creation: Tracks when a process is started.
• Process Termination: Detects when a process ends, crucial for identifying potential tampering.

File Manipulation

• File Creation, Modification, Deletion: Covers all aspects of file lifecycle, essential for detecting malicious files and forensic evidence tampering.

User Activity

• User Logon, Logoff, Failed Logons: Tracks all login-related activities to identify access anomalies.

Script Activity

• Script Content: Monitors the execution of scripts to detect malicious automation.

Network Activity

• Network Connections, DNS Queries, Socket Listening: Covers communication paths attackers rely on for exfiltration or lateral movement.

Scheduled Task Activity

• Task Creation and Modification: Tracks persistence mechanisms like cron jobs.

User Account Activity

• Creation, Modification, Deletion: Monitors unauthorized access methods.

Driver/Module Activity

• Kernel Drivers, eBPF Events: Focuses on kernel-level threats.

Access Activity

• Raw Reads, Process Hijacking: Detects access and tampering attempts at sensitive levels.

Service Activity

• Service Creation, Modification: Tracks persistence and service-based abuse.

EDR Operations

• Start/Stop Events: Tracks if the agent itself is tampered with.

Sysmon and Auditd: Comparison for Context

Although Sysmon and Auditd are not EDR solutions, we included them in the testing for comparison purposes. These tools provide a frame of reference to understand what telemetry can be achieved through open-source tools.

For Auditd, we used the configuration provided by Florian’s audit rules. These rules offer comprehensive coverage for generating the necessary telemetry to create detections. Many organizations rely on Auditd to collect system activity data and use it in combination with detection rules to secure their Linux environments.

By including Sysmon and Auditd, we aim to illustrate what is achievable with careful configuration and open-source tools, helping defenders gauge how commercial EDR solutions compare to freely available alternatives.

Exploring the Results

The results revealed significant gaps in Linux telemetry compared to Windows. Key areas of weakness included:

• Kernel-Level Monitoring: Few solutions detected kernel module loads or eBPF activity, critical for identifying advanced threats.
• User Activity: Visibility into user logoffs or failed logon attempts was minimal, leaving potential blind spots in session tracking.
• Service and Scheduled Tasks: Weak coverage in monitoring cron jobs and service creation, two common persistence techniques for attackers.
• Process Tampering: Few solutions effectively monitored process hijacking or memory manipulation.

These gaps highlight the need for vendors to address Linux-specific blind spots and improve their telemetry capabilities.

Scoring Methodology

To objectively evaluate the telemetry capabilities of each EDR solution, we implemented a scoring system based on weighted categories. Each telemetry event is assigned a score, reflecting its importance for detection and response. The scoring weights were determined based on the relevance and criticality of each event.

For example:

• Process Creation: 1 point (high importance)
• Process Termination: 0.5 points (moderate importance)
• File Modification: 1 point (high importance)
• Network Connection: 1 point (high importance)
• User Logoff: 0.4 points (lower importance)

Partial implementations or telemetry that requires additional configurations are scored accordingly. For instance:

• Yes: Full visibility (1 point)
• Partially: Partial visibility (0.5 points)
• Via Enabling Telemetry: Requires additional configuration (1 point)

This scoring methodology allows us to rank each EDR solution objectively based on its default capabilities. The final scores provide a clear comparison of how well each product covers the essential telemetry needed for detecting and responding to Linux-based threats.

The Value of Linux Telemetry

Telemetry is the foundation for detection and response. Without it, defenders are left with incomplete data, making it harder to investigate incidents or detect emerging threats. Linux systems play critical roles in enterprise and cloud environments, and achieving parity with Windows in telemetry is no longer optional.

By improving Linux telemetry, we can build stronger detection capabilities, reduce response times, and give defenders the tools they need to protect critical systems effectively.

Call to Action

The Linux EDR Telemetry Project results will soon be released, offering detailed insights into the state of Linux telemetry. However, addressing these gaps will take more than just identifying them — it will require collaboration from vendors, researchers, and end-users.

We invite you to contribute, share feedback, and test new ideas. Together, we can close these gaps and create a stronger security ecosystem for Linux environments.

Conclusion

Linux systems are too important to be left with blind spots. By expanding the EDR Telemetry Project to include Linux, we’ve taken a critical step toward improving visibility and closing gaps in protection. The findings may be sobering but are also a call to action. With the collective efforts of the security community, we can highlight these gaps and ensure Linux environments receive as much protection as Windows systems from EDR vendors.

Visit the EDR-Telemetry website to view results in detail:

EDR Telemetry Project - Linux

A Note of Thanks and Invitation to Support

A huge thank you to all the contributors, community members, and supporters who have helped shape and improve the EDR Telemetry Project. Your insights, feedback, and collaboration have been invaluable in making this project a success.

If you find this project useful and would like to support its ongoing development and expansion, please consider making a donation or sponsoring the project. It takes a significant amount of time and effort to maintain and grow this initiative. You can contribute by visiting our sponsorship page: Donate or Sponsor the Project.

Your support helps us continue to uncover critical visibility gaps, expand our testing, and ultimately improve security for everyone.