Elastic SIEM and Dashboards

How Elastic Detection Rules Work

Elastic Security (part of the Elastic Stack) provides a detection engine that runs queries against indexed events and generates alerts when matches are found. Rules are the core detection content — they define what to look for, when to alert, and what actions to take.

Rule Structure

An Elastic detection rule has these components:

1. Query (KQL) — the search that identifies matching events
2. Schedule — how often the rule runs (e.g., every 5 minutes)
3. Alert condition — when to generate an alert (e.g., when number of matches > 0)
4. Severity — the alert's severity (low, medium, high, critical)
5. Actions — what happens when the alert fires (create case, send email, webhook)
6. Exceptions — conditions that suppress the alert for known-good activity

KQL: Kibana Query Language

KQL is the query language used in Elastic. It is simpler than SPL but serves the same purpose — filtering and searching events.

# Find failed Windows logons
event.code: "4625" and winlog.channel: "Security"

# Find PowerShell execution with encoded commands
event.code: "4688" and process.command_line: "*powershell*" and process.command_line: "*-enc*"

# Find network connections to known-bad IPs
destination.ip: "185.220.101.4" and event.category: "network"

# Find authentication events from a specific source
source.ip: "192.168.1.100" and event.category: "authentication"

KQL supports Boolean logic (and, or, not), wildcards (*), and range queries (>, <, >=, <=).

Writing a Detection Rule

Suppose you want to detect PowerShell execution with base64-encoded commands — a common attacker technique.

# KQL query for the rule
event.code: "4688" and process.command_line: "*powershell*" and process.command_line: "*-enc*"

Rule configuration:

• Schedule: every 5 minutes
• Alert condition: when number of matches > 0
• Severity: high
• MITRE ATT&CK: T1059.001 (Command and Scripting Interpreter: PowerShell)
• Actions: create a case, send email to SOC

When this rule runs, Elastic searches the last 5 minutes of events for process creation events (4688) where the command line contains both "powershell" and "-enc". If any matches are found, an alert is generated with the matching events attached.

Rule Types

Query Rules

The simplest rule type. Run a KQL query on a schedule and alert when matches are found. Suitable for most detection scenarios.

Threshold Rules

Alert when the number of matches exceeds a threshold within a time window. Useful for detecting brute force or scanning activity.

# Threshold rule: alert when more than 20 failed logons from the same source in 5 minutes
event.code: "4625"
# Threshold: group by source.ip, count > 20, within 5 minutes

Machine Learning Rules

Use Elastic's machine learning jobs to detect anomalies — unusual process execution, rare network connections, or abnormal authentication patterns. These rules do not require you to define the exact pattern; the ML model learns normal behavior and alerts on deviations.

Event Correlation Rules

Match sequences of events across multiple sources. For example: a failed logon followed by a successful logon from the same source within 5 minutes, or a process creation followed by a network connection to a known C2 IP.

# Event correlation: failed logon followed by successful logon
# Sequence: event.code:4625 followed by event.code:4624 within 5 minutes, same source.ip

Rule Exceptions

Exceptions suppress alerts for known-good activity. Without exceptions, a penetration test or authorized administrative activity will generate alerts that waste analyst time.

# Exception: suppress PowerShell alerts from the IT admin workstation
not host.name: "IT-ADMIN-01"

# Exception: suppress alerts during scheduled maintenance windows
not (@timestamp >= "2026-09-30T02:00:00" and @timestamp <= "2026-09-30T04:00:00")

Exceptions are added to the rule and evaluated before the alert is generated. If an event matches both the rule query and an exception, no alert is created.

Building Dashboards That Analysts Actually Use

A Kibana dashboard is a collection of saved visualizations displayed on a single screen. For a SOC, dashboards provide the shift analyst with an at-a-glance view of security health — are alerts spiking? Are there active incidents? Which assets are most targeted?

Dashboard Design Principles

1. Purpose-driven — every dashboard serves a specific audience and use case. A shift analyst dashboard is different from a threat hunting dashboard.
2. At-a-glance — the most important information should be visible without scrolling or clicking.
3. Actionable — every visualization should lead to an action. If a chart shows a spike, the analyst should be able to click through to the underlying events.
4. Current — dashboards should auto-refresh (e.g., every 1 minute) so they reflect the current state.

Essential SOC Dashboards

Shift Overview Dashboard

The first screen a Tier 1 analyst sees when starting a shift. It answers: what is happening right now?

• Open alerts by severity — bar chart showing critical, high, medium, low alerts
• Alerts over time — line chart showing alert volume per hour for the last 24 hours
• Top targeted assets — table showing the assets with the most alerts
• Active incidents — table showing incidents currently being investigated
• Data source health — indicator showing whether all log sources are reporting

Threat Hunting Dashboard

Used by Tier 2 and Tier 3 analysts for proactive investigation.

• Process execution timeline — histogram of process creations over time, filterable by parent process
• Network connections by destination — world map showing outbound connections
• Authentication patterns — stacked bar chart of successful vs failed logons by source IP
• PowerShell activity — table of PowerShell executions with command lines
• File integrity monitoring — alerts on critical file changes

Incident Response Dashboard

Activated during an active incident. Focuses on the affected assets and attack timeline.

• Affected assets — list of hosts involved in the incident
• Attack timeline — chronological visualization of related events
• Lateral movement map — network graph showing connections between affected hosts
• Containment status — indicator showing which hosts have been isolated

Building a Visualization

In Kibana, a visualization is created from a saved search. The process:

1. Create a search — define the KQL query that retrieves the relevant events
2. Choose a visualization type — bar chart, line chart, pie chart, data table, metric, etc.
3. Configure the visualization — select the X and Y axes, aggregation type, and filters
4. Save the visualization — give it a descriptive name
5. Add to a dashboard — place the visualization on the dashboard with other related visualizations

# Example: create a bar chart of alerts by severity
# Search: event.dataset: "elastic_security" and signal.status: "open"
# Visualization: bar chart
# X-axis: signal.severity (terms aggregation)
# Y-axis: count of events

Dashboard Filters and Drill-Down

Dashboards should support interactive filtering. An analyst who sees a spike in the "Alerts over time" chart should be able to click on the spike and see the underlying alerts. This is achieved through:

• Dashboard-level filters — apply a filter (e.g., severity=critical) to all visualizations on the dashboard
• Drill-down links — clicking a visualization element opens a detailed view of the underlying events
• Lens — Kibana's drag-and-drop visualization tool that makes it easy to create and modify visualizations

Dashboard Automation

Dashboards can be automated to support SOC workflows:

• Auto-refresh — dashboards refresh at a configurable interval (e.g., every 60 seconds)
• Scheduled reports — export a PDF or CSV of the dashboard on a schedule (e.g., daily summary to management)
• Alert integration — clicking an alert on the dashboard opens the case in the case management system
• Shared dashboards — dashboards can be shared with other analysts or embedded in external tools via iframe

One Schema to Rule All Logs

The Elastic Common Schema (ECS) is an open-source specification that defines field names for common security event attributes. When all data sources use ECS field names, you can write a single query that works across Windows logs, Linux logs, firewall logs, and cloud logs.

Why ECS Matters

Without ECS, every log source uses different field names for the same concept:

With ECS, all of these map to source.ip, user.name, event.action, and host.name. A single query works across all sources:

# ECS-normalized query: find all failed authentications
event.category: "authentication" and event.outcome: "failure"

This query works whether the event came from Windows, Linux, a firewall, or a cloud service — as long as the data is normalized to ECS.

ECS Field Categories

ECS defines fields in categories:

• Base — @timestamp, message, labels, tags
• Host — host.name, host.ip, host.os.name, host.risk.static_level
• Source/Destination — source.ip, source.port, destination.ip, destination.port
• User — user.name, user.id, user.domain, user.roles
• Event — event.action, event.category, event.outcome, event.severity, event.code
• Network — network.protocol, network.direction, network.bytes
• Process — process.name, process.command_line, process.parent.name, process.pid
• File — file.name, file.path, file.hash.sha256
• Threat — threat.technique.id, threat.tactic.name, threat.indicator.ip

Implementing ECS

Elastic Agent and Beats

Elastic's data collection tools (Elastic Agent, Filebeat, Winlogbeat) output ECS-formatted data by default. When you configure a new data source, the integration includes an ECS mapping that transforms the native log format to ECS fields.

# Winlogbeat ECS output (default)
winlogbeat.event_logs:
  - name: Security
    event_id: 4625
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

Custom Log Sources

For log sources that do not have a built-in ECS mapping, you create an ingest pipeline that transforms the raw log to ECS fields.

// Ingest pipeline: map custom fields to ECS
{
  "processors": [
    {
      "set": {
        "field": "source.ip",
        "value": "{{src_ip}}"
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "{{action}}"
      }
    },
    {
      "date": {
        "field": "timestamp",
        "target_field": "@timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss"]
      }
    }
  ]
}

Third-Party Integrations

Many security tools (CrowdStrike, Palo Alto, Okta, AWS) have Elastic integrations that output ECS-formatted data. When evaluating a new log source, check whether an ECS-compatible integration exists before building a custom parser.

ECS and Detection Rules

ECS enables detection rules that work across data sources. A rule for "detect impossible travel" can query authentication events from Windows, Linux, and VPN — all using the same ECS fields (source.geo.country_name, user.name, @timestamp). Without ECS, the rule would need separate queries for each source with different field names.

# ECS-based impossible travel detection
# Works across all authentication sources
event.category: "authentication" and event.outcome: "success"
| stats dc(source.geo.country_name) as countries by user.name
| where countries > 1

The ECS approach makes detection rules portable, maintainable, and consistent across the entire data landscape.