How I Solved Unexpected High Disk Usage on My VPS • CeamKrier

The Unusual Discovery

Recently, I noticed my VPS was consuming more disk space than expected. As a developer managing a lightweight application with a few essential services like Portainer and Traefik, this was puzzling. The disk usage was alarmingly high, with /dev/sda1 showing 32GB used out of 75GB:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        75G   32G   41G  44% /

For a small-scale operation, 32GB usage seemed out of place. Something was clearly wrong, and I needed to investigate.

Initial Investigation: A Disk Usage Summary

I started by examining the disk usage summary to identify which directories were consuming the most space. Using the following command, I hoped to uncover any large directories or files:

sudo du -h --max-depth=1 /

This command provides a high-level view of disk usage, breaking it down by directory at the root level. However, the results didn’t reveal anything unusual. The large usage wasn’t immediately apparent in any specific directory.

The Docker Revelation

Given the unremarkable initial findings, I turned my attention to Docker, which I use extensively on this VPS. Running Docker services often involves multiple images, containers, and volumes that can quietly accumulate over time. To check Docker’s disk usage, I used:

docker system df

This command outputs a detailed summary of Docker-related storage usage. It didn’t take long to identify the culprit: a substantial number of unused Docker images, containers, and volumes were taking up valuable space.

The Quick Fix: Pruning Docker

To free up space, I ran a comprehensive cleanup with:

docker system prune -a --volumes

This command removes all unused Docker objects, including stopped containers, dangling images, and unused volumes. The -a flag ensures that even unused images are deleted, while --volumes includes any lingering volumes in the cleanup. The impact was immediate and significant. After running the prune command, the disk usage dropped dramatically:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        75G  7.6G   65G  11% /

This confirmed that Docker-related files were indeed the main cause of the high disk usage.

The Need for Automation

Realizing that unused Docker objects could quickly accumulate again, I decided to automate the pruning process. Regularly cleaning up would prevent future disk space issues and maintain a healthy system.

Setting Up a Cron Job for Automated Docker Pruning

To automate the Docker prune process, I set up a cron job to run the cleanup script weekly. Here’s how you can do the same:

Step 1: Create a Prune Script

First, I created a script that includes the docker system prune command. This script will be executed by the cron job.

Create the script file:

sudo nano /usr/local/bin/docker-prune.sh

Add the prune command to the script:
```
#!/bin/bash
# Docker system prune to clean up unused objects

docker system prune -a --volumes -f
```
The -f flag forces the prune command to run without needing manual confirmation.
Save and exit: Press Ctrl + O to save and Ctrl + X to exit nano.

Make the script executable:

sudo chmod +x /usr/local/bin/docker-prune.sh

Step 2: Schedule the Script Using `cron`

Next, I scheduled the script to run weekly using cron, a time-based job scheduler in Unix-like operating systems.

Open the root crontab:
```
sudo crontab -e
```
Select an editor: If prompted to choose an editor, I recommend nano for its simplicity. Type 1 and press Enter.
Add the cron job:

Add the following line to schedule the prune job every Sunday at 2:00 AM:
```
0 2 * * 0 /usr/local/bin/docker-prune.sh >> /var/log/docker-prune.log 2>&1
```
This line tells cron to execute the script and append the output to a log file (/var/log/docker-prune.log).
Save and exit: Press Ctrl + O to save and Ctrl + X to exit.
Verify the cron job:

To ensure the cron job is set up correctly:
```
sudo crontab -l
```
The new cron job should be listed.

Step 3: Test the Script Manually

Before relying on the cron job, it’s good practice to test the script manually:

sudo /usr/local/bin/docker-prune.sh

This ensures the script runs without issues and performs the cleanup as expected.

Monitoring and Maintenance

Log Rotation: Over time, the log file (/var/log/docker-prune.log) might grow large. To prevent this, you can set up log rotation using logrotate.
Regular Monitoring: Periodically check the log file and monitor the disk usage to ensure the cron job is functioning properly.

Conclusion

By identifying and addressing the issue of unused Docker objects consuming disk space, I was able to reclaim a significant amount of space on my VPS. Automating the cleanup process with a cron job ensures that the system remains clean and efficient without manual intervention. If you’re running Docker on a VPS, consider setting up a similar automation to keep your system in check.