Detection-as-Code Rule Tuning Workflow

Detection-as-Code Rule Tuning Workflow#

Overview#

This workflow demonstrates an automated approach to tuning detection rules using Detection-as-Code principles, integrating SOAR case feedback with version-controlled rule management and CI/CD deployment.

Architecture#

graph LR
    A[SOAR Cases] -->|Analyst Feedback| B[AI Agent]
    B -->|Read Cases| C[MCP SOAR Server]
    B -->|Find Rules| D[Local Rules Repo]
    B -->|Create PR| E[GitHub MCP Server]
    D -->|Update Logic| F[Rule Changes]
    F -->|Pull Request| G[GitHub]
    G -->|CI/CD Pipeline| H[SIEM Platform]
    H -->|Deploy| I[Production Rules]

Prerequisites#

Required Components#

  • SIEM Platform (Google SecOps, Chronicle, Splunk, etc.)

  • SOAR Platform with case management

  • Git Repository for detection rules

  • CI/CD Pipeline (GitHub Actions, GitLab CI)

  • MCP Servers:

    • SIEM MCP Server (search events, manage rules)

    • SOAR MCP Server (list cases, read comments)

    • GitHub MCP Server (create branches, PRs)

  • AI Agent (Gemini) with MCP integration

Configuration#

# .env configuration
SIEM_API_ENDPOINT=https://your-siem.example.com
SOAR_API_ENDPOINT=https://your-soar.example.com
GITHUB_TOKEN=your_github_token
AI_MODEL=gemini-2.5-pro-preview

Automated Workflow#

Step 1: Monitor SOAR Cases for Tuning Opportunities#

The AI agent continuously monitors closed SOAR cases for tuning signals:

# Example monitoring query
search_criteria = {
    "status": "closed",
    "root_cause": ["normal_behavior", "false_positive", "authorized_activity"],
    "has_analyst_comments": True,
    "time_range": "last_7_days"
}

Step 2: Extract Tuning Requirements from Case Comments#

When a case contains tuning instructions, the agent extracts:

  • Rule Name: Which detection rule triggered the alert

  • Tuning Type: Exclusion, threshold adjustment, logic refinement

  • Specific Conditions: Field values, hostnames, user accounts to exclude

Example analyst comment:

This case was a false positive. User jack.torrance is authorized to execute 
ScreenConnect in our environment. Rule should be tuned to exclude events 
where hostname != "desktop-7xl2kp3".

Step 3: Locate and Analyze Local Rule Files#

The agent searches the local rules repository:

rules/
├── network/
├── endpoint/
│   ├── rmm_tools_execution.yaml
│   └── suspicious_process.yaml
└── cloud/

Rule format example:

# rmm_tools_execution.yaml
id: rmm-tools-execution
name: "Remote Monitoring Management Tools Execution"
description: "Detects execution of RMM tools that could be abused"
severity: medium
logic:
  query: |
    event.type = "process" AND
    process.name IN ("ScreenConnect.exe", "TeamViewer.exe", "AnyDesk.exe")
tags:
  - attack.t1219
  - remote_access

Step 4: Generate Rule Modifications#

The agent applies the tuning based on analyst feedback:

# Updated rule with exclusion
logic:
  query: |
    event.type = "process" AND
    process.name IN ("ScreenConnect.exe", "TeamViewer.exe", "AnyDesk.exe") AND
    NOT (process.name = "ScreenConnect.exe" AND 
         user.name = "jack.torrance" AND 
         host.name = "desktop-7xl2kp3")

Step 5: Create Branch and Pull Request#

The agent uses Git operations to propose changes:

# Create feature branch
git checkout -b tune/rmm-tools-case-4232

# Commit changes
git add rules/endpoint/rmm_tools_execution.yaml
git commit -m "Tune RMM tools rule based on case #4232 feedback

- Added exclusion for authorized ScreenConnect usage
- User: jack.torrance on host: desktop-7xl2kp3
- Reduces false positives for legitimate IT operations"

# Push and create PR
git push origin tune/rmm-tools-case-4232

Pull request includes:

  • Link to original SOAR case

  • Analyst comment justification

  • Specific changes made

  • Expected impact on false positive rate

Step 6: Automated Validation and Deployment#

GitHub Actions workflow (deploy-rules.yml):

name: Deploy Detection Rules
on:
  pull_request:
    paths:
      - 'rules/**'
  push:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Validate Rule Syntax
        run: |
          yamllint rules/
          detection-cli validate rules/
      
      - name: Test Rule Logic
        run: |
          # Test against sample events
          detection-cli test --rule ${{ github.event.pull_request.changed_files }}
      
      - name: Check Historical Impact
        run: |
          # Query last 30 days to estimate impact
          detection-cli backtest --rule ${{ github.event.pull_request.changed_files }}

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to SIEM
        run: |
          detection-cli deploy --env production
          
      - name: Notify Team
        run: |
          slack-notify "#detections" "Rule updates deployed from PR #${{ github.event.pull_request.number }}"

Step 7: Final Reporting (Agent Execution Artifacts)#

Upon completion of the workflow, the agent must generate a summary report containing:

  1. Sequence Diagram: A Mermaid diagram visualizing the steps taken.

  2. Execution Metadata: Date, time, and estimated cost (tokens/runtime).

  3. Outcome Summary: Link to PR and summary of changes.

Manual Review Process#

Security Review Checklist#

  • Exclusion doesn’t create detection blind spots

  • Tuning is specific to the false positive pattern

  • No overly broad exclusions (e.g., entire subnets)

  • Maintains detection for actual threats

  • Includes expiration date for temporary exclusions

Approval Workflow#

  1. Detection Engineer: Reviews technical implementation

  2. SOC Lead: Validates operational impact

  3. Security Architect: Approves high-risk changes

Monitoring and Metrics#

Key Performance Indicators#

  • False Positive Reduction: Track % decrease after tuning

  • Tuning Velocity: Rules tuned per week

  • Automation Rate: % of tunings handled automatically

  • Rollback Frequency: How often tunings are reverted

Dashboard Example#

┌─────────────────────────────────────┐
│  Detection Rule Tuning Dashboard     │
├─────────────────────────────────────┤
│ Rules Tuned This Week: 15           │
│ FP Rate Reduction: 72%              │
│ Automated Tunings: 12/15 (80%)      │
│ Pending Reviews: 3                   │
└─────────────────────────────────────┘

Example Use Cases#

Use Case 1: Authorized Software Exclusion#

Scenario: IT deploys new RMM tool Action: Exclude specific deployment patterns

exclusions:
  - condition: "process.signer = 'IT_DEPT_CERT' AND host.ou = 'IT_Workstations'"
    expires: "2025-12-31"
    reason: "Authorized IT deployment"

Use Case 2: Business Process Tuning#

Scenario: Finance runs daily automated reports triggering alerts Action: Time-based exclusion

exclusions:
  - condition: "user.name = 'svc_finance' AND time.hour BETWEEN 2 AND 4"
    reason: "Scheduled finance reporting"

Use Case 3: Environmental Noise Reduction#

Scenario: Dev environment generates testing alerts Action: Environment-based filtering

logic:
  query: |
    original_query AND
    NOT (environment.type = "development" AND 
         source.ip IN ("10.1.0.0/16"))

Best Practices#

1. Feedback Loop Integration#

  • Capture analyst decisions in SOAR cases

  • Use standardized comment templates

  • Tag cases requiring rule tuning

2. Version Control Discipline#

  • One rule change per commit

  • Descriptive branch names (tune/rule-name-issue)

  • Link commits to SOAR cases

3. Testing Rigor#

  • Validate against historical true positives

  • Test exclusions don’t over-match

  • Monitor post-deployment metrics

4. Documentation Standards#

Each tuning should document:

  • Original false positive pattern

  • Business justification

  • Expected impact

  • Review/expiration date

Troubleshooting#

Common Issues#

Issue: Rule updates not deploying

# Check CI/CD logs
gh run list --workflow=deploy-rules.yml
gh run view <run-id>

# Validate rule syntax locally
detection-cli validate rules/endpoint/rmm_tools_execution.yaml

Issue: Over-tuning causing missed detections

# Review exclusion patterns
grep -r "NOT\|exclude\|exception" rules/

# Audit recent tunings
git log --oneline --grep="tune" --since="30 days ago"

Advanced Automation#

Scheduled Tuning Reviews#

# Weekly automated review
async def weekly_tuning_review():
    # Find rules with high FP rates
    high_fp_rules = await siem.get_rules_by_fp_rate(threshold=0.7)
    
    # Check for existing SOAR feedback
    for rule in high_fp_rules:
        cases = await soar.search_cases(rule_name=rule.name)
        if cases_with_tuning_feedback(cases):
            await create_tuning_pr(rule, cases)

ML-Assisted Pattern Recognition#

# Identify common false positive patterns
def analyze_fp_patterns(cases):
    patterns = []
    for case in cases:
        if case.root_cause == "false_positive":
            patterns.append(extract_event_features(case))
    
    # Cluster similar patterns
    clusters = ml_model.cluster_patterns(patterns)
    return suggest_exclusions(clusters)

References#

Rubrics#

The following rubric is used to evaluate the execution of the Detection-as-Code Rule Tuning runbook by an LLM agent.

Grading Scale (0-100 Points)#

Criteria

Points

Description

Tuning Identification

15

Correctly identified the closed SOAR case, root cause, and extracted analyst feedback.

Rule Localization

15

Located the correct YAML file for the rule without hallucinating paths.

Modification Accuracy

25

Applied the correct logic change (exclusion/threshold), validated syntax, and avoided over-broad exclusions.

Git Operations

20

Created a descriptive branch, useful commit message linked to the case, and a complete PR.

Safety & Validation

10

Performed validation steps (syntax check) and considered security implications.

Operational Artifacts

15

Produced required artifacts: Sequence diagram, execution metadata (date/cost), and summary.

Evaluation Criteria Details#

1. Tuning Identification (15 Points)#

  • 5 pts: Identified the correct closed SOAR case with “false_positive” or “normal_behavior” root cause.

  • 10 pts: Accurately extracted the tuning requirements (e.g., “exclude host X”) from analyst comments.

2. Rule Localization (15 Points)#

  • 15 pts: Successfully located the correct rule file in the repository with an accurate path (e.g., rules/endpoint/rmm_tools_execution.yaml).

  • 10 pts: Located the correct rule file but with minor or repairable path issues (e.g., missing a subdirectory or small typo that can be easily corrected).

  • 5 pts: Required explicit guidance or hints to find the correct rule file in the repository.

  • 0 pts: Failed to find the file or hallucinated a file path.

3. Modification Accuracy (25 Points)#

  • 10 pts: Generated valid YAML syntax for the modified rule.

  • 10 pts: Logic change accurately reflects the analyst’s intent (e.g., correct field and value excluded).

  • 5 pts: Avoided removing the rule entirely or creating an overly broad exclusion (e.g., NOT host.name = *).

4. Git Operations (20 Points)#

  • 5 pts: Created a new branch with a descriptive name (e.g., tune/rule-name-case-id).

  • 5 pts: Commit message describes what changed and why, linking to the SOAR case.

  • 10 pts: Created a Pull Request with all required details (description, link to case, risk assessment).

5. Safety & Validation (10 Points)#

  • 5 pts: Explicitly ran validation tools (e.g., detection-cli validate) or checked syntax.

  • 5 pts: Assessment of security impact (e.g., “Low risk: specific host exclusion”).

6. Operational Artifacts (15 Points)#

  • 5 pts: Sequence Diagram: Produced a Mermaid sequence diagram visualizing the steps taken during execution.

  • 5 pts: Execution Metadata: Recorded the date, duration, and estimated token cost of the execution.

  • 5 pts: Summary Report: Generated a concise summary of the actions and outcomes.

Contents