# Splunk + n8n Security Event Automation

Automated security event detection and response pipeline. Splunk monitors for high-risk events (sudo usage, privilege escalation, domain admin changes), and fires webhook alerts to n8n, which triages them by severity and generates formatted alert cards (Slack/email-ready).

Both Splunk and n8n run as Docker containers on the same host and communicate over a shared Docker bridge network (`security-lab`).

## 

## Architecture

```
┌──────────────────────┐         ┌──────────────────────┐
│  Event Generator     │  HEC    │      Splunk          │
│  (Python script)     │────────>│  - security index    │
│  runs from your Mac  │ :8088   │  - 3 saved searches  │
└──────────────────────┘ HTTPS   │  - 5-min cron alerts │
                                 └──────────┬───────────┘
                                            │ webhook POST
                                            │ http://n8n:5678/webhook/security-alert
                                            │ (container-to-container)
                                            v
                                 ┌──────────────────────┐
                                 │        n8n           │
                                 │  - Normalize & triage│
                                 │  - Build alert card  │
                                 │  (Slack / Email)     │
                                 └──────────────────────┘
```

## Prerequisites

- **Splunk** already deployed and running via `deploy_splunk.sh` from the `self-hosted lab` repo (handles container creation, HEC/SSL setup, and the `security-lab` Docker network)
- **n8n** container running on the same `security-lab` Docker network
- **Python 3** with the `requests` library (`pip install requests`)

## What Each File Does

| File | Purpose |
|---|---|
| `savedsearches.conf` | Three Splunk alert definitions that detect security events and POST to n8n |
| `deploy_splunk_config.sh` | Creates the `security` index, copies `savedsearches.conf` into Splunk, and reloads config via REST API |
| `n8n_workflow.json` | n8n workflow (deployed via API): receives alerts, triages by type, builds Slack/email alert cards |
| `security_event_generator.py` | Python script that generates fake security events and sends them to Splunk via HEC (HTTPS) |
| `test_pipeline.sh` | End-to-end validation: checks Splunk HEC, n8n health, webhook, and sends sample events |

## Configuration

All scripts default to the `deploy_splunk.sh` defaults and `192.168.10.184` as the host. Override via environment variables if your setup differs:

| Variable | Default | Used by |
|---|---|---|
| `SPLUNK_HOST` | `192.168.10.184` | `deploy_splunk_config.sh`, `security_event_generator.py`, `test_pipeline.sh` |
| `SPLUNK_PASS` | `YourSecurePassword123!` | `deploy_splunk_config.sh` |
| `SPLUNK_HEC_TOKEN` | `27095ac8-fcba-45e0-9645-e1ac61716c14` | `security_event_generator.py`, `test_pipeline.sh` |
| `N8N_HOST` | `192.168.10.184` | `test_pipeline.sh` |

Example overriding for a different host:
```bash
SPLUNK_HOST=10.0.0.5 python3 security_event_generator.py
```

> **Note:** HEC uses HTTPS with a self-signed certificate. All scripts disable SSL verification automatically.

## Step-by-Step Setup

### Step 1: Verify Splunk is Running

Splunk should already be deployed via `deploy_splunk.sh`. Confirm it is healthy:

```bash
# From the Docker host
docker inspect splunk --format='{{.State.Health.Status}}'
# Should print: healthy

# From any machine on the network
curl -sk https://192.168.10.184:8088/services/collector/health
# Should print: {"text":"HEC is healthy","code":17}
```

### Step 2: Deploy the Security Index and Alert Definitions

This creates the `security` index (separate from the `threat_intel` index created by `deploy_splunk.sh`), copies the saved search/alert configs into the running Splunk container, and reloads the config via the REST API (no container restart required):

```bash
bash deploy_splunk_config.sh
```

This installs three scheduled alerts that run every 5 minutes:

| Alert Name | Detects | Severity |
|---|---|---|
| Security - Sudo Usage Detected | Any `sudo` command execution | Medium |
| Security - New Admin Privileges Granted | Users added to sudo/wheel/adm groups | High |
| Security - User Added to Domain Admin Group | Additions to Domain/Enterprise/Schema Admins | Critical |

Each alert POSTs matching results to `http://n8n:5678/webhook/security-alert` (container-to-container via the `security-lab` Docker network).

### Step 3: Deploy the n8n Workflow

Deploy the workflow via the n8n API. Replace `<N8N_HOST>` and `<API_KEY>` with your values:

```bash
# Create the workflow
curl -s -X POST "http://<N8N_HOST>:5678/api/v1/workflows" \
  -H "X-N8N-API-KEY: <API_KEY>" \
  -H "Content-Type: application/json" \
  -d @n8n_workflow.json

# Activate it (replace <WORKFLOW_ID> with the id from the response above)
curl -s -X POST "http://<N8N_HOST>:5678/api/v1/workflows/<WORKFLOW_ID>/activate" \
  -H "X-N8N-API-KEY: <API_KEY>" \
  -H "Content-Type: application/json"
```

The workflow does the following when it receives a webhook POST at `/webhook/security-alert`:

1. **Normalize and Triage** -- extracts fields from the Splunk alert JSON, routes by event type (`sudo_usage`, `new_admin_privs`, `domain_admin_add`), and assigns risk level (LOW/HIGH/CRITICAL) with recommended remediation
2. **Build Alert Card** -- generates Slack Block Kit message and HTML email body
3. **Respond to Splunk** -- returns 200 OK immediately
4. **Log Alert** -- writes to n8n execution log

Each step is documented with sticky notes visible in the n8n canvas.

### Step 4: Generate Test Events

Send simulated security events to Splunk via HEC (HTTPS):

```bash
# Send one of each event type (sudo, admin privs, domain admin)
python3 security_event_generator.py

# Send a specific type
python3 security_event_generator.py --type sudo_usage
python3 security_event_generator.py --type new_admin_privs
python3 security_event_generator.py --type domain_admin_add

# Send multiple of each
python3 security_event_generator.py --count 5

# Continuous random events every 5 seconds (Ctrl+C to stop)
python3 security_event_generator.py --flood --interval 5
```

### Step 5: Verify the Pipeline

Run the full end-to-end test:

```bash
bash test_pipeline.sh
```

This checks:
1. Splunk HEC is responding (HTTPS)
2. n8n is healthy
3. The n8n webhook accepts a direct POST (confirms the workflow is active)
4. Events sent to Splunk HEC are accepted

Expected output:
```
[1/4] Checking Splunk HEC...
  ✓ Splunk HEC is healthy
[2/4] Checking n8n...
  ✓ n8n is healthy
[3/4] Testing n8n webhook directly...
  ✓ n8n webhook responded 200 OK
[4/4] Sending test events via Splunk HEC...
  [OK]  sudo_usage           | host=... | actor=...
  [OK]  new_admin_privs      | host=... | actor=...
  [OK]  domain_admin_add     | host=... | actor=...
```

#### Manual Verification

**In Splunk** (http://192.168.10.184:8000):

```spl
index=security | table _time host event_type actor command target_user severity
```

**In n8n** (http://192.168.10.184:5678):

Go to **Executions** (left sidebar) to see each webhook trigger. Click any execution to inspect the data at each node.

> **Note:** Splunk alerts run on a 5-minute cron schedule. After sending test events, you may need to wait up to 5 minutes for Splunk to fire the alert to n8n. To trigger immediately: Splunk UI > **Settings** > **Searches, Reports, and Alerts** > find the alert > **Run**.

## Connecting Real Notification Channels

The workflow generates ready-to-use payloads but does not ship with Slack/email nodes pre-configured. To add them:

### Slack

1. In n8n, add a **Slack** node after **Build Alert Card**
2. Configure with your Slack OAuth token or webhook URL
3. Use the expression `{{ $json.slack_message }}` as the message body (it contains Block Kit formatting)

### Email

1. In n8n, add an **Email (SMTP)** node after **Build Alert Card**
2. Configure your SMTP credentials
3. Use `{{ $json.email_html }}` as the HTML body

## Troubleshooting

### Splunk HEC returns non-200

```bash
# Check if HEC is enabled (must use HTTPS)
curl -sk https://192.168.10.184:8088/services/collector/health
# Re-run deploy_splunk.sh from the self-hosted lab repo to fix HEC
```

### HEC connection refused or empty reply

HEC uses HTTPS with a self-signed certificate. If you get `Empty reply from server`, you are connecting over HTTP instead of HTTPS. All scripts in this repo use HTTPS by default.

### n8n webhook returns 404

The workflow is not active. Open n8n UI and toggle the workflow ON, or re-activate via the API.

### Events in Splunk but no n8n executions

- Alerts run every 5 minutes. Wait or trigger manually from Splunk UI.
- Verify both containers are on the same Docker network: `docker network inspect security-lab`
- Verify the webhook URL in `savedsearches.conf` uses the n8n container name: `http://n8n:5678/webhook/security-alert`
- Check Splunk alert logs: **Settings** > **Searches, Reports, and Alerts** > click the alert > **View recent activity**

### Containers can't reach each other

```bash
# Confirm both are on the same network
docker network inspect security-lab --format '{{range .Containers}}{{.Name}} {{end}}'
# Should list both "splunk" and "n8n"
```

### n8n not starting

```bash
docker logs n8n

# Ensure port is free
lsof -i :5678
```

## Stopping and Cleaning Up

Splunk and n8n are managed separately. Use `deploy_splunk.sh destroy` for Splunk. For n8n, stop or remove it via Docker directly.
