Read a sample: 1. When a Server Runs Out of Space
A disk-space incident rarely begins with a tidy message explaining exactly what to remove. More often, an application cannot save a file, a scheduled backup fails, or users report that a service has become slow or unreliable. An administrator connects to the server and discovers that even a routine command cannot create its temporary output.
The immediate temptation is to find the largest file and delete it. That approach can turn a capacity problem into a data-loss problem. A large file might be an active database file, a backup needed for recovery, or a log that an application still has open. It might also reside on a different filesystem from the one causing the failure, so removing it would not solve the immediate problem.
The first task is not cleanup. It is to establish what is failing, identify the affected filesystem, and preserve enough information to choose a safe response. This chapter gives you an investigation sequence you can use under pressure and a disposable environment for practicing the early steps.
A filesystem shortage can appear in several ways:
- An application reports “No space left on device” when writing a file.
- A package installation or update stops because it cannot unpack or configure files.
- A database reports write errors, refuses new transactions, or changes its behavior to protect its data.
- A service cannot create a log, socket, lock file, or temporary file.
- A backup job fails while creating an archive or staging its output.
- Users cannot upload files or save changes.
- A monitoring system reports high filesystem utilization or an increasing number of write failures.
These symptoms do not all prove that stored file contents have consumed every available byte. A filesystem can also run out of inodes, the structures used to represent files and directories. Permissions, quotas, read-only mounts, hardware trouble, and application-specific limits can produce write failures as well. Treat the error message as evidence, not as a complete diagnosis.
The location of the failed write matters. A server may have separate filesystems mounted at /, /var, /home, or an application data directory. If /var is full, removing files from /home will not create space on /var. Conversely, a root filesystem with little free space does not necessarily explain a failure on a separate data mount.
A useful initial question is therefore: What path was the application trying to write, and which filesystem contains that path?
When an alert arrives, take a moment to understand the operational impact. Identify affected services, note when the problem started, and tell the appropriate people that an investigation is underway. On a shared system, coordinate before making changes: another administrator may already be responding, or an automated job may be generating the files you are inspecting.
If a runaway process is rapidly consuming the remaining capacity, stopping or limiting that growth may be more urgent than performing a complete analysis. Choose an action appropriate to the service, and understand its consequences. Pausing a nonessential batch job may be straightforward; stopping a database or interrupting an active backup may require an incident decision and an approved procedure.
Avoid actions that make the situation harder to understand:
- Do not delete unfamiliar files simply because they are large.
- Do not recursively remove the contents of a directory based only on its name.
- Do not restart a critical service solely in the hope that space will reappear.
- Do not assume a backup exists or is usable without checking.
- Do not run cleanup commands copied from another system without understanding their scope.
Keep a short incident record as you work. Record the affected host, the time, relevant error messages, the commands or checks that informed your decisions, and any changes made. Captured evidence helps with handoff and later prevention, but avoid writing bulky diagnostic output to the filesystem that is already full.
The df command reports space usage for mounted filesystems. A practical first check is:
bash
df -h
The -h option displays sizes in human-readable units. Look at the usage and available-space columns, but also pay attention to the mount point. A nearly full filesystem matters most when it contains the path where writes are failing.
If you know that path, ask df about it directly. For example:
bash
df -h /var/log
The result identifies the filesystem containing /var/log, even if /var/log is not itself a separate mount point. Substitute the actual existing path associated with the failure. This focused check is often more useful than guessing from directory names.
Next, check inode availability:
bash
df -i
A filesystem with free data blocks can still fail to create files if its available inodes are exhausted. A directory containing a very large number of small files is one possible cause. Chapter 2 examines filesystem capacity, reserved space, mount points, and inode exhaustion in more detail. For now, the important habit is to check both capacity and inodes before deciding what kind of shortage you have.
Also consider whether the reported error is limited to a particular user or application. If the filesystem has capacity and inodes available, investigate permissions, quotas, mount state, and the application’s own limits rather than assuming that deleting files is the answer.
Once you know which filesystem is under pressure, inspect usage on that filesystem. The du command estimates space used by files and directories. For a first look at the…