Real-Time Program-Specific Log Consolidation for Intrusion Detection

Advantages

  • Reduces logs 219× for efficient data storage
  • Performs real-time analysis with zero offline cost
  • Minimizes memory use through subject activity consolidation
  • Maintains high detection accuracy with low alert fatigue

Summary

Enterprises today are increasingly targeted by Advanced Persistent Threats (APTs), stealthy, long-term attacks that penetrate systems and remain hidden for extended periods. Provenance-based Intrusion Detection Systems (PIDSes) help uncover these attacks by analyzing system-level logs, but they struggle with massive log volumes, redundant data, and severe server-side memory overhead. Existing log reduction techniques offer limited improvements and typically rely on offline, centralized processing making them unsuitable for real-time, scalable deployment across enterprise networks.

Nano is a real-time, program-specific log consolidation framework that redefines scalable APT detection. It introduces two novel data structures: the profile hierarchy, which groups process by execution sequence to represent their origin, and the access network, which consolidates redundant activity into causal links between programs and resources. In extensive experiments using four DARPA Transparent Computing datasets, Nano demonstrated up to 219× log reductions, 623× faster graph analysis, and detection accuracy comparable to leading PIDSes. Unlike existing methods, Nano eliminates the need for offline processing, operates in real-time, reduces alert fatigue through event correlation, and minimizes memory usage—making it uniquely efficient, scalable, and operationally viable for enterprise-level threat detection.

An example to build the access network. (a) The provenance graph that includes operations with timestamps of three subjects. R=Read, E=Execute, W=Write, and C=Create. (b) The access network that causally connects each profile with vertices using access links, where these links are merged from operations of its associated subjects. Subjects nginx and pEja72mA are associated with profiles ρ1 and ρ2, respectively.

Desired Partnerships

  • License
  • Sponsored Research
  • Co-Development
Patent Information: