Development of a Security Information and Event Management (SIEM) System

Development of a Security Information and Event Management (SIEM) System
Abstract
Security Information and Event Management (SIEM) systems serve as the central nervous system of modern Security Operations Centers, providing centralized visibility through log aggregation, event correlation, and real-time alerting. However, commercial SIEM solutions remain prohibitively expensive for small organizations and educational institutions, while open-source alternatives often require significant operational expertise to deploy effectively. This project presents the design and development of a lightweight SIEM system built on a Python/Flask stack with Redis Streams for event transport and MongoDB for persistent storage. The system implements core SIEM capabilities including multi-source log ingestion, normalization through configurable parsing rules, correlation-based detection with sliding window analysis, and a real-time analyst dashboard. Detection logic supports threshold-based rules (brute force detection), sequence-based correlation (multi-stage attack chains), and MITRE ATT&CK technique mapping. A web-based interface provides alert triage, event search, and dashboard visualization. Experimental validation demonstrates the system's ability to aggregate heterogeneous log sources, execute correlation rules against realistic attack scenarios, and present actionable alerts for security analyst investigation.
Keywords: SIEM; Log Aggregation; Event Correlation; Threat Detection; Security Operations; Open-Source Security
________________________________________
Chapter 1: Introduction
1.1 Research Background
The modern enterprise security landscape presents a fundamental visibility challenge. Organizations operate across cloud services, on-premises servers, user endpoints, network infrastructure, and SaaS applications—each generating its own stream of security-relevant events. Security teams must monitor dozens to hundreds of systems, a task that manual operations cannot accomplish at scale. A SIEM system addresses this challenge by consolidating security events into a unified view, enabling detection in minutes rather than days.
NIST SP 800-92 provides foundational guidance for computer security log management, while the Australian Signals Directorate's ACSC Best Practices for Event Logging and Threat Detection—co-authored with 14 partner agencies from eight countries—establishes that organizations should "log the right data, in a consistent format, with reliable timestamps, retained long enough to catch threats that dwell for months, and centralized so a SIEM or XDR can actually detect them" . This advisory explicitly states that it can take up to 18 months to discover a cyber security incident, and some malware dwells on the network 70 to 200 days before causing overt harm .
However, commercial SIEM platforms such as Splunk and Microsoft Sentinel impose significant licensing costs that render them inaccessible to many organizations. Open-source alternatives exist—Wazuh, ELK Stack, OSSIM, MozDef, and others—but these tools introduce their own challenges. As noted in recent analysis, open-source SIEM tools "are powerful—but they require proper setup, tuning, and expertise to be effective" . Most SIEM failures do not occur at deployment; they emerge later when ingestion fails, logs stop normalizing cleanly, and alert noise undermines trust in the system .
1.2 Problem Statement
Organizations seeking to implement SIEM capabilities face several critical barriers:
Cost Barriers: Commercial SIEM licensing is prohibitive for small and medium organizations, educational institutions, and individual practitioners. Licensing models based on Events Per Second (EPS) or storage volume penalize comprehensive log collection.
Operational Complexity: Open-source SIEM deployments require substantial engineering effort to maintain data pipelines, tune detection rules, and manage storage infrastructure. Teams with limited engineering time cannot sustain the operational overhead .
Detection Quality: Out-of-the-box rule sets often lack the context needed to distinguish genuine threats from benign anomalies, resulting in alert fatigue. Many rules fail in production because the underlying data is inconsistent—correlation logic that references "src_ip" misses events where logs label the same field "source_address" .
Normalization Gaps: Every vendor formats logs differently. Without reliable normalization, correlation rules that span multiple sources break down. As one analysis states, "Skipping normalization is one of the fastest ways to undermine your detection logic. The rules will look correct on paper but silently miss the events they were designed to catch" .
1.3 Research Objectives
This project aims to design and implement a lightweight SIEM system that:
1. Implements multi-source log ingestion supporting common formats (syslog, JSON, CEF) through configurable collectors.
2. Provides log normalization capabilities that standardize field names and timestamps across heterogeneous sources.
3. Implements a correlation engine supporting threshold-based rules and ordered sequence detection within sliding time windows.
4. Delivers a real-time analyst dashboard with alert triage, event search, and detection rule management.
5. Maps detections to MITRE ATT&CK techniques for standardized threat classification.
1.4 Thesis Structure
This thesis comprises five chapters. Chapter 1 introduces research background and objectives; Chapter 2 reviews SIEM architecture, normalization techniques, and correlation methodologies; Chapter 3 presents system design; Chapter 4 describes implementation details; Chapter 5 concludes and discusses future work.
________________________________________
Chapter 2: Literature Review
2.1 SIEM Core Architecture
A modern SIEM solution comprises several interconnected components forming an end-to-end security data pipeline :
Data Collection Layer: Agents, collectors, or APIs gather logs and telemetry from endpoints, servers, network devices, cloud services, and SaaS applications. The goal is comprehensive visibility, not just perimeter monitoring .
Parsing and Normalization: Standardizes varied log formats into a consistent schema, enabling cross-platform correlation. Without normalization, correlation rules cannot reliably join events across sources .
Correlation and Analytics Engine: Applies detection rules, threat intelligence, and statistical models to identify suspicious patterns. This is the "heart of SIEM systems" where patterns indicating security issues are detected .
Storage and Data Lake: Houses historical logs for compliance reporting and deep investigations. Hot/warm data tiering balances query performance against storage cost .
Alerting and Dashboards: Surfaces prioritized incidents and visualizes trends for analysts. Effective platforms focus on risk-based prioritization rather than alert volume .
Integration Layer: Connects with SOAR platforms, ticketing systems, and third-party tools to automate response .