IDS and IPS with Signatures

The Difference Between Watching and Blocking

Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) both analyze network traffic for malicious activity, but they differ in what they do when they find something. This distinction determines where you deploy them, how you configure them, and what operational impact they have.

IDS: The Passive Watcher

An IDS is deployed out-of-band — it receives a copy of the traffic (via a SPAN port, network tap, or flow data) but does not sit in the direct traffic path. When it detects something malicious, it generates an alert. It does not block, drop, or modify any traffic.

Client ──────────────→ Server
              │
          [ SPAN port ]
              │
           [ IDS ]  ← receives copy of traffic, generates alerts

Advantages of IDS:

• No latency impact — traffic flows directly between client and server; the IDS only observes
• No single point of failure — if the IDS goes down, traffic is unaffected
• Safe to run in alert-only mode — no risk of blocking legitimate traffic
• Good for monitoring and investigation — captures full packet data for forensic analysis

Disadvantages of IDS:

• Cannot stop attacks in progress — detection without prevention means the attack completes before a human responds
• Alert fatigue — passive systems tend to generate more alerts because there is no risk of blocking legitimate traffic
• Response time depends on humans — an alert at 3 AM may not be seen until morning

IPS: The Inline Guardian

An IPS sits directly in the traffic path. All traffic flows through it, and it can drop packets, reset connections, or block source IPs in real time when it detects malicious activity.

Client ────→ [ IPS ] ────→ Server
              │
         (inspects and can block)

Advantages of IPS:

• Real-time blocking — stops attacks before they reach the target
• Automated response — no human intervention needed for known threats
• Reduced dwell time — attacks are stopped in milliseconds, not hours

Disadvantages of IPS:

• Latency impact — every packet must be inspected, adding processing delay
• Single point of failure — if the IPS goes down, traffic stops (unless bypassed)
• False positive risk — a bad rule can block legitimate business traffic
• Requires careful tuning — aggressive blocking without tuning causes outages

Deployment Strategy

The choice between IDS and IPS depends on the traffic criticality and the organization's risk tolerance:

• IPS at the perimeter — block known-bad traffic before it enters the network. Use conservative rules (high-confidence signatures only) to minimize false positive risk.
• IPS in front of critical assets — protect high-value servers (databases, domain controllers) with more aggressive rules.
• IDS for monitoring — deploy IDS sensors on internal segments to detect lateral movement and insider threats without risking business disruption.
• IDS for compliance — some regulations require monitoring and logging even if blocking is not implemented.

Host-Based vs Network-Based

Network-based IDS/IPS (NIDS/NIPS) inspects traffic at the network level. It sees all traffic passing through its deployment point and can detect attacks against multiple hosts. It is blind to encrypted traffic unless TLS decryption is performed.

Host-based IDS/IPS (HIDS/HIPS) runs on individual endpoints and monitors system calls, file integrity, and local network traffic. It can detect attacks that never hit the network (e.g., a local privilege escalation) and can inspect traffic after decryption on the host. OSSEC, Wazuh, and OSQuery are examples of HIDS tools.

How Signatures Work: Pattern Matching on Network Traffic

Signature-based detection matches network traffic against a database of known attack patterns. Each signature defines a specific pattern — a byte sequence, a protocol anomaly, or a behavioral indicator — that indicates malicious activity. When traffic matches a signature, the IDS/IPS generates an alert (IDS) or blocks the traffic (IPS).

The most widely used open-source signature format is Snort (and its successor Suricata). Snort rules have a consistent structure that is worth understanding in detail.

Snort Rule Anatomy

alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (msg:"SQL Injection attempt"; content:"union select"; nocase; sid:1000001; rev:1;)

Breaking this down:

Content Matching

The content keyword specifies a byte sequence to search for in the packet payload. This is the core of most signatures.

# Match the string "cmd.exe" in any HTTP request
content:"cmd.exe"; nocase;

# Match a specific byte sequence (hex)
content:"|90 90 90 E8 C0 FF FF FF|";

# Match at a specific offset from the start of the payload
content:"GET"; depth:4;  # only check first 4 bytes

# Match at a specific position relative to a previous content match
content:"Authorization"; content:"Basic"; distance:0; within:20;

Protocol-Aware Matching

Snort can inspect protocol headers and match on their fields:

# Match HTTP URI
content:"admin"; http_uri;

# Match HTTP header
content:"User-Agent"; http_header; content:"sqlmap"; http_header;

# Match on TCP flags (SYN flood detection)
flags:S;  # match SYN flag

# Match on TTL (detect traceroute or spoofed packets)
ttl:64;

Writing a Signature: Step by Step

Suppose you want to detect attempts to access a known web shell (/cmd.jsp) on your web server:

alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (
    msg:"WEB-ATTACKS cmd.jsp access attempt";
    flow:established,to_server;
    content:"GET"; http_method;
    content:"/cmd.jsp"; http_uri; nocase;
    metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop;
    reference:url,example.com/cmd.jsp;
    classtype:web-application-attack;
    sid:1000002;
    rev:1;
)

Key design decisions:

• flow:established,to_server — only match on established connections, reducing false positives from random scanning
• http_method and http_uri — use protocol-aware matching to search only in the HTTP method and URI, not the entire payload
• nocase — match case-insensitively because attackers use mixed case to evade detection
• classtype — categorize the alert for prioritization
• sid — unique signature ID (1000000+ for custom rules to avoid conflicts with VRT rules)

Tuning: Reducing False Positives

A signature that fires on legitimate traffic is worse than no signature — it creates alert fatigue and causes analysts to ignore real alerts. Tuning is the process of refining signatures to reduce false positives without creating false negatives.

Common tuning techniques:

1. Add flow control — flow:established,to_server ensures the signature only matches on legitimate connection states
2. Use protocol-aware matching — http_uri instead of raw content reduces matches in binary payloads
3. Add thresholding — threshold:type both, track by_src, count 10, seconds 60 only alert after 10 matches in 60 seconds
4. Suppress known-good sources — suppress rules silence alerts from trusted IPs (e.g., vulnerability scanners)
5. Use pass action for known-good — whitelist traffic that would otherwise match

# Tuned signature: only alert after 5 matches in 60 seconds from the same source
alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (
    msg:"Possible SQL injection - union select";
    flow:established,to_server;
    content:"union"; http_uri; nocase;
    content:"select"; http_uri; nocase;
    threshold:type both, track by_src, count 5, seconds 60;
    sid:1000003;
    rev:1;
)

Limitations of Signature-Based Detection

Signatures are essential but insufficient. Their limitations:

• Zero-day attacks — no signature exists for an attack that has never been seen before
• Polymorphic malware — attackers mutate payloads to evade static byte patterns
• Encrypted traffic — signatures cannot inspect TLS-encrypted payloads without decryption
• False sense of security — a clean IDS report does not mean the network is safe

Signature-based detection must be complemented with:

• Anomaly-based detection — baseline normal behavior and alert on deviations
• Behavioral analysis — detect attack patterns (e.g., lateral movement) rather than specific payloads
• Threat intelligence feeds — update signatures with IOCs from industry sources

Running an IDS/IPS in Production

Deploying an IDS/IPS is not a set-and-forget operation. It requires ongoing tuning, rule management, and alert triage to remain effective. A poorly maintained IDS/IPS generates so many false positives that analysts ignore it — or worse, it blocks legitimate traffic and causes outages.

Deployment Architecture

Network Taps and SPAN Ports

For IDS deployment, traffic must be copied to the sensor. A network tap is a hardware device that physically copies traffic from a fiber or copper link. A SPAN (Switched Port Analyzer) port on a switch mirrors traffic from one or more ports to the sensor. Taps are more reliable than SPAN ports because they do not drop packets under load, but they require physical installation.

Inline Deployment for IPS

IPS must be deployed inline — traffic flows through it. This requires careful planning:

• Fail-open vs fail-closed — if the IPS loses power or crashes, does traffic flow (fail-open) or stop (fail-closed)? Fail-open prioritizes availability; fail-closed prioritizes security.
• Bypass taps — physical bypass devices that automatically route traffic around the IPS if it fails
• High availability — deploy IPS pairs in active-passive or active-active mode to eliminate single points of failure

Rule Management

Sources of Signatures

• Snort VRT (Vulnerability Research Team) — commercial rules from Cisco Talos, updated daily
• Snort Community Rules — free rules maintained by the community
• Emerging Threats (ET) Open — free open-source rules from Proofpoint
• Suricata ET Pro — commercial rules with broader coverage
• Custom rules — written in-house for organization-specific threats

Rule Update Process

# Update Snort rules
sudo snort_update.sh

# Verify rules compile without errors
sudo snort -T -c /etc/snort/snort.conf

# Reload rules without dropping traffic
sudo kill -USR2 $(cat /var/run/snort.pid)

Rule updates should be tested in a staging environment before production deployment. A bad rule update can cause an IPS to block legitimate traffic or an IDS to flood the SIEM with false positives.

Alert Triage and Prioritization

Not all alerts are equal. A well-tuned IDS/IPS generates alerts that are prioritized by:

1. Signature classification — classtype indicates the attack category (web-application-attack, trojan-activity, attempted-admin)
2. Impact flag — metadata:impact_flag red indicates high-confidence, high-impact detection
3. Source reputation — alerts from known-malicious IPs are higher priority
4. Target criticality — alerts on critical assets (databases, domain controllers) are higher priority
5. Context — correlated with other alerts (e.g., a port scan followed by an exploit attempt)

# Example: prioritize alerts in a SIEM
index=snort sourcetype=snort_alerts
| eval priority=case(
    impact_flag="red" AND target_criticality="high", "critical",
    impact_flag="red", "high",
    classtype="web-application-attack", "medium",
    true(), "low"
)
| stats count by priority, src_ip, dest_ip, msg
| sort -count

Performance Tuning

IDS/IPS performance depends on:

• Rule count — more rules means more processing per packet. Remove unused rules.
• Hardware — sufficient CPU, memory, and network interface capacity
• Traffic volume — ensure the sensor can handle peak traffic without dropping packets
• Pattern matching engine — Suricata's multi-threaded architecture scales better than Snort's single-threaded model for high-traffic environments

# Monitor Suricata performance
sudo suricata --dump-config | grep -i threads

# Check for packet drops
sudo cat /var/log/suricata/stats.log | grep -i capture.kernel_drops

Integration with SOC Workflows

IDS/IPS alerts feed into the SOC's SIEM and case management system. The integration should:

1. Forward alerts to the SIEM in real time (via syslog, CEF, or LEEF format)
2. Enrich alerts with asset context (owner, criticality, location)
3. Correlate IDS/IPS alerts with other data sources (endpoint logs, authentication logs, threat intelligence)
4. Automate initial triage — suppress known-false positives, escalate high-confidence alerts
5. Track alert-to-incident conversion rate as a tuning metric

The goal is to make the IDS/IPS a signal generator that feeds the SOC's detection and response pipeline, not a standalone tool that generates noise.