Hydra storage fills up with Suricata pcaps

🤔 Problem

Suricata does not honor max pcap file count leading to the disk being filled up with pcap files.

🌱 Solution

The provided python script can be added to the hydra to perform automatic cleanup of the files based on time:

Please download this file

With the clean_pcap.py file download, copy the file to /usr/bin/ on the hydra
The next step is to configure the pcap folder and the period of pcap’s to be retained. This can be done in the file at the path /etc/siemonster/common.env by adding the entries CLEAN_PCAP_DIR and CLEAN_PCAP_PERIOD as per screenshot below.
BASH
```
CLEAN_PCAP_DIR=/data/suricata/log
CLEAN_PCAP_PERIOD=48
```
The add the following entry by running the crontab -e command. This will run the cleanup every day at 12am UTC and pip output to the logfile /var/log/pcap_clean.log
BASH
```
0 0 * * * date  >> /var/log/pcap_clean.log && /usr/bin/python3 /usr/bin/clean_pcap.py >> /var/log/pcap_clean.log 2>&1
```

PY

# - Created by: Louis Bernardo (SIEMonster)
# - Purpose: Cleaning up PCAP files for noisy networks

import os
import time
import glob

# - The script assumes the epoch time in the filename is in seconds.
# - Skips files with invalid epoch times and reports errors.
# - Handles missing or invalid common.env files and variables gracefully.
# - Uses only standard library modules (os, time, glob).
# - Ensure the clean_pcap_dir path is valid and accessible.
# - The clean_pcap_period is in hours and can be a decimal (e.g., 0.5 for 30 minutes).

# Function to load variables from common.env
def load_env(file_path="/etc/siemonster/common.env"):
    config = {}
    try:
        with open(file_path, 'r') as f:
            for line in f:
                # Skip empty lines or comments
                line = line.strip()
                if line and not line.startswith('#'):
                    key, value = line.split('=', 1)
                    config[key.strip()] = value.strip()
        return config
    except FileNotFoundError:
        print(f"Error: {file_path} not found")
        exit(1)
    except Exception as e:
        print(f"Error reading {file_path}: {e}")
        exit(1)

# Load configuration
config = load_env()

# Get variables from config
try:
    directory = config['CLEAN_PCAP_DIR']
    period_hours = float(config['CLEAN_PCAP_PERIOD'])
except KeyError as e:
    print(f"Error: Missing required variable {e} in common.env")
    exit(1)

# Validate directory
if not os.path.isdir(directory):
    print(f"Error: {directory} is not a valid directory")
    exit(1)

# Convert period to seconds
threshold = period_hours * 60 * 60  # Convert hours to seconds
now = time.time()

# Pattern to match files like log-1747085394-*.pcap
pattern = os.path.join(directory, "log-*-*.pcap")

# Iterate over matching files
for filepath in glob.glob(pattern):
    # Extract the epoch time from the filename
    filename = os.path.basename(filepath)
    try:
        # Split filename like log-1747085394-3.pcap to get epoch time
        epoch_str = filename.split('-')[1]
        epoch_time = int(epoch_str)
    except (IndexError, ValueError):
        print(f"Skipping {filename}: Invalid epoch time format")
        continue

    # Check if the file's epoch time is older than the specified period
    if now - epoch_time > threshold:
        try:
            os.remove(filepath)
            print(f"Deleted {filename}")
        except OSError as e:
            print(f"Error deleting {filename}: {e}")
    else:
        print(f"Keeping {filename}: Not old enough")

You can also run the following string if you want to manually trigger this.

BASH

date  >> /var/log/pcap_clean.log && /usr/bin/python3 /usr/bin/clean_pcap.py >> /var/log/pcap_clean.log 2>&1