Disk full on host

Symptoms

• Node-exporter disk usage > 85% (`DiskSpaceUsageHigh`).
• Containers fail to write to their volume — usually PG or RabbitMQ first.

Check

# Overall
df -h /

# Top offenders
sudo du -h --max-depth=1 /var/lib/docker 2>/dev/null | sort -hr | head -10
docker system df

# Which logs are biggest?
sudo du -h --max-depth=1 /var/lib/docker/containers 2>/dev/null | sort -hr | head -5

# Postgres data volume size
sudo du -sh /var/lib/docker/volumes/*identity*
sudo du -sh /var/lib/docker/volumes/*gateway*

Remediation

1. First, prune images + stopped containers:
   prune.sh

   docker builder prune -f
   docker image prune -af --filter "until=72h"
   docker container prune -f

2. Truncate any container log > 500 MB:
   logs.sh

   for f in $(sudo find /var/lib/docker/containers -name '*-json.log' -size +500M); do
     echo "truncating $f"
     sudo truncate -s 0 "$f"
   done

3. If less than 10 GB free remains, grow the EBS volume (Console → EC2 → Volumes → Modify).

Post-incident

• Ensure Docker daemon `log-opts` are set (`max-size=50m`, `max-file=5`) in `/etc/docker/daemon.json`.
• Investigate PG growth — stale `audit_logs`, oversized uploads, un-rotated partitions.