A Best Practices Guide to Security Logs

Security logs are digital forensic evidence. They record system events, user actions, and network traffic that security teams analyze to detect threats and maintain compliance. As applications move to distributed architectures, log volumes have grown exponentially, creating new challenges.

This guide addresses the technical problems when security logging scales beyond traditional approaches. You’ll learn to implement deduplication strategies, build fault-tolerant pipelines, correlate events across systems, and optimize performance under high throughput.

The techniques covered here apply to any logging infrastructure. We’ll focus on four core problems: managing data volume through deduplication and aggregation, maintaining log integrity for compliance, building fault-tolerant pipelines, and correlating events across systems to detect coordinated attacks.

Distributed applications generate logs from hundreds of sources, whether managed cloud services, microservices, or on-premises infrastructure. The flood of logs overwhelms storage systems. In many cases, duplicate entries waste resources and significantly degrade query performance.

Effective security logging is done in four ways:

1. Selective log collection, where only events that provide security value are stored and processed. Collecting everything creates noise that obscures real threats and inflates storage costs.

2. Distributed storage, where one avoids single points of failure by distributing logs across multiple nodes. Centralized logging creates bottlenecks and makes the whole system vulnerable.

3. Logs should be routed based on their source, severity, and content type. This intelligent routing optimizes processing performance and prevents low-priority logs from overwhelming critical security data.

4. Using search-optimized, index-based storage like Elasticsearch or Opensearch instead of flat files makes handling terabytes of data a breeze. 

High-throughput environments face three core problems:

1. Log duplication: Multiple log shippers or network retries create identical entries, wasting storage and degrading query performance.

2. Processing bottlenecks: Parsing and enriching logs in real-time becomes CPU-bound at scale. Systems cannot process every entry individually without falling behind ingestion rates.

3. Storage scaling: Traditional databases hit performance walls with time-series log data. Write-heavy workloads require storage strategies different from transactional systems.

Deduplication prevents storing identical log entries multiple times. The challenge is checking for duplicates without querying the entire database for each incoming log.

A Bloom filter is a space-efficient probabilistic data structure that tests set membership. It uses multiple hash functions to map elements to bit positions in a fixed-size array. When checking if an element exists, the filter examines specific bit positions; if any are unset, the element does not exist. If all bits are set, however, the element probably exists (false positives are possible, and false negatives are impossible).

For logging, each cluster node maintains a bloom filter representing recently processed logs. When a new log arrives:

1. Check the Bloom filter for the target storage node

2. If the filter says "definitely new," store the log directly

3. If the filter says "possibly duplicate," perform a targeted database lookup to confirm

Bloom filtering in a four-node cluster where “n” represents shared storage nodes and “S” represents incoming data streams.

This approach reduces database queries by 90-95% in typical scenarios while using minimal memory (a few megabytes can track millions of log entries).

Aggregation reduces log volumes by summarizing similar events instead of storing each individually. Rather than keeping 1000 failed login attempts from the same IP address, the system stores this, for example:

"IP 192.168.1.100: 1,000 failed logins between 14:00-14:05."

Onum's group-by-action enables this near real-time aggregation. Logs with fields like IP_Address, Request_Type, and Timestamp can be grouped by IP over time windows and count request types. This reduces storage requirements while preserving the behavioral patterns security teams need to detect attacks.

However, aggregation becomes expensive when processing requires reading complete log entries. Modern systems use metadata-first architecture: extract structured fields (IP, timestamp, status code) during ingestion and index them separately from the raw log content.

The metadata-first approach isolates metadata from raw log material, allowing for rapid, scalable queries for summarization and detection. Metadata is sent and indexed for search and aggregation, while full logs are managed separately for archiving and further analysis.

The raw logs are stored in cheap archive storage (cold storage) for forensic analysis and audits. The extracted metadata, typically 5-10% of the original size, handles real-time queries and aggregation. This architecture makes time-series analysis practical at the petabyte scale.

Compliance frameworks like GDPR, ISO 27001, and NIST 800-53 require organizations to prove that security logs are authentic and unaltered. This "chain of custody" ensures logs can serve as forensic evidence in legal proceedings or security investigations.

The technical challenge is detecting tampering without storing multiple copies of log data. Digital signatures solve this by creating cryptographic fingerprints that change if the log content is modified.

Here is a possible implementation workflow:

1. When a log file is written, compute a cryptographic hash (typically SHA-256) of the entire file contents. This hash serves as a unique fingerprint. Any change to the log data produces an entirely different hash value.

2. Sign the hash using a private key (RSA-2048 or ECDSA P-256 are common choices). The resulting digital signature proves the log's authenticity (from the private key holder) and integrity (the content hasn't changed since signing).

3. To verify log integrity later, recompute the hash of the current log file and check it against the stored signature using the corresponding public key. If verification fails, the log has been tampered with or corrupted.

This approach scales efficiently because signatures are minor (256-512 bytes) regardless of log file size, and verification is computationally fast. Organizations typically sign log files hourly or daily to balance security requirements with operational overhead.

For real-time streaming logs, systems can use Merkle trees to efficiently sign batches of log entries, providing duplicate tamper detection with lower computational cost.

Security logs are only valuable if they're consistently collected. A single failed component, like a crashed log shipper and overloaded parser, can create blind spots that attackers exploit. Any pipeline disruption degrades threat detection and compromises compliance in environments where logs stream from hundreds of sources.

Fault-tolerant logging systems continue operating when individual components fail through three core strategies:

Deploy multiple log collectors per source system. If the primary collector fails, secondary collectors automatically take over without data loss. Tools like Fluentd and Logstash support active-passive configurations where backup collectors monitor the primary's health and activate when needed.

Place durable message queues (Apache Kafka, RabbitMQ) between log sources and processing systems. Queues buffer logs during downstream failures and replay missed data when components recover. This prevents log loss during parser crashes or storage outages.

When external dependencies (enrichment APIs, storage backends) repeatedly fail, circuit breakers stop sending traffic and return cached responses or skip non-critical processing. This prevents cascading failures, where one slow component blocks the entire pipeline.

After multiple failures, the circuit breaker "trips," preventing further calls from reaching the failing service. Requests are blocked while the circuit remains open. After a recovery timeout, the circuit allows test requests to check if the service has recovered.

Circuit-breaker pattern

Best practice: Start with redundant collection, which prevents the most common failure point, like agent crashes. Add message queuing for high-volume environments where temporary processing delays are acceptable. Implement circuit breakers last, focusing on external dependencies that could block critical log processing.

Lateral movement from web servers to databases, credential stuffing across applications, and privilege escalation through different services; these are examples of sophisticated attacks that a system may face at some point. Individual logs show isolated events, but correlation reveals the attack sequence.

Security teams need two correlation approaches: rule-based detection for known attack patterns and behavioral analysis for novel threats.

Rule-based correlation uses deterministic logic to detect known suspicious behaviors. These rules define specific event sequences that indicate attacks, such as failed logins followed by successful access on different systems.

This approach works well for detecting brute-force attacks, account takeovers, and other well-understood threat patterns. When tuned properly, rules execute quickly and produce few false positives. The following pseudo-code shows how this could work:

for each failed_login event:
    if same_ip_address has failed_login on different_system within 30_seconds:
        trigger_alert("Potential lateral movement detected")
        
for each privilege_escalation event:
    if same_user had suspicious_file_access within 5_minutes:
        trigger_alert("Coordinated privilege abuse")

Behavioral analysis detects deviations from standard user patterns using statistical models. 

A Markov chain, for example, tracks typical user behavior transitions: login → email → logout, and flags unusual sequences. If a user typically accesses email and documents but suddenly attempts database administration, the model flags this transition as statistically abnormal.

normal_pattern = build_behavior_model(user_history)
current_sequence = [login, admin_access, database_query]

if probability(current_sequence) < anomaly_threshold:
    trigger_alert("Unusual behavior pattern detected")

Best practice: Rule-based systems provide immediate value with lower operational overhead, so start there for high-confidence detection of known threats. You can then add behavioral analysis for environments where insider threats or novel attack vectors are primary concerns. Behavioral models require training data and ongoing tuning, but catch previously unseen attack patterns.

Compliance frameworks require logs to serve as forensic evidence. Logging systems must be implemented with verifiable integrity and controlled access from the start.

Data retention policies must specify exact timeframes and deletion procedures. For example, GDPR requires personal data deletion within 30 days of request, while financial regulations often mandate 7-year retention. Implement automated retention by tagging logs with retention classes:

log_entry = {
    "timestamp": "2025-01-15T10:30:00Z",
    "user_id": "user123",
    "action": "login_attempt",
    "retention_class": "gdpr_personal_data",  // Auto-delete after 2 years
    "compliance_tags": ["pci_dss", "sox"]
}

To remove personally identifiable information (PII) before storage, logs should be sanitized, meaning user identifiers should be hashed while preserving correlation ability, and sensitive fields like credit card numbers or social security numbers should be redacted.

Implement role-based access control (RBAC) with least privilege. Security analysts access threat detection logs, but only compliance officers access audit logs containing user activity.

Log all access to the logging system itself:

• Who queried which logs and when?

• What search terms were used?

• Which log files were downloaded or exported?

Use TLS 1.3 for all log transmission between collectors and storage systems, and encrypt log storage using AES-256 with key rotation every 90 days. You should also store encryption keys separately from log data using hardware security modules (HSMs) or key management services.

The biggest mistake in security logging is trying to solve every problem simultaneously. Teams often deploy complex correlation engines and AI-driven analytics before establishing fundamental log integrity and reliable collection. This creates expensive systems that miss fundamental threats because the underlying data is incomplete or untrustworthy.

Build your logging infrastructure in this order: reliable collection first, volume management, and advanced analytics. A simple system that captures 100% of security events beats a sophisticated one that misses 20% due to pipeline failures. Onum can help you build resilient pipelines to process your data.

Focus on three metrics that matter: 

1. Collection completeness (are you missing logs?)

2. Query response time (can analysts investigate quickly?)

3. Storage efficiency (what's your cost per GB retained?). 

These metrics reveal whether your logging system improves security posture or generates operational overhead.