Read a sample: 1. Start with the Evidence
A service will not start. A user says their password worked yesterday but not today. The computer displays an unfamiliar warning during boot.
Each symptom gives you a reason to investigate, but none tells you exactly what went wrong. A failed login might involve a mistyped password, an account restriction, a service problem, or something else. A service that stopped might have encountered a configuration error, a missing file, or a dependency that was not available. The symptom is the starting point, not the answer.
Linux logs help narrow the possibilities. They record messages produced by the operating system, services, and applications. Some messages describe routine activity; others report conditions that may need attention. Reading them is a little like examining a receipt or a delivery record: the record can tell you what was reported and when, but you may need more evidence to understand why it happened.
This chapter establishes a safe way to begin. You will prepare a practice environment, learn what the command prompts in this book mean, and follow a small investigation from a reported symptom to a cautious conclusion. You do not need to memorize every command. The goal is to develop a repeatable habit: observe first, then investigate.
Imagine someone tells you, “The web service is broken.” That statement is useful because it identifies something to examine, but it leaves important questions unanswered:
- What did the person try to do?
- What happened instead of the expected result?
- When did it happen?
- Was there an error message?
- Does the problem occur every time?
A log can help answer some of these questions. For example, a service might record that it could not read a configuration file. That is stronger evidence than simply knowing the service failed. Even so, the message may not explain why the file could not be read. You might still need to check whether the file exists, whether its permissions allow access, and whether the service is looking in the right place.
Logs also have limits. A program may fail before it can write a useful message. An entry might be stored somewhere you have not searched, or it might no longer be available. The clock on a system may not match the clock used by the person reporting the problem. A message labeled as an error might be relevant to your investigation—or it might describe an unrelated event.
Treat every entry as a statement from a particular source at a particular time. Ask what it actually says before deciding what it means.
Throughout this book, you will return to the same basic sequence:
1. Describe the symptom. Record what was expected, what happened, and when it happened.
2. Identify a likely source. Decide which service, login process, application, or part of the system might have recorded the event.
3. Collect a small amount of evidence. Start with the relevant status or a narrow set of log messages rather than reading everything at once.
4. Separate facts from explanations. Note what the output reports and what you are merely considering.
5. Choose the next check. Look for evidence that could support or contradict your leading explanation.
6. Record the result. Summarize what you established, what remains uncertain, and what you checked.
The steps are not a rigid script. You may discover that the reported time was wrong or that a different service handled the request. When that happens, revise the question and search again.
For example, suppose a user reports that a scheduled backup did not run around 9:00 a.m. Your first question might be whether the backup service started at that time. If the available records show that it started and finished successfully, you have not yet proven that the backup contains the expected files. You have, however, learned something that can guide the next check.
This distinction between what happened and why it happened is one of the most valuable habits in troubleshooting.
You will get more from this book if you can run commands as you read. A disposable Linux virtual machine is a good choice: it gives you a system to explore without requiring you to experiment on a computer that other people depend on. A spare Linux installation or an authorized lab system can also work.
If you use a virtual machine, create it with a Linux distribution of your choice and make sure you can sign in. A distribution is a packaged version of Linux that includes its own selection of tools and configuration. The exact appearance of your desktop and terminal may differ from the examples here.
Many exercises in this book use systemd, a system and service manager used by many Linux distributions, and its journal, a store of log entries. Not every Linux environment uses systemd. Containers, minimal installations, and some distributions may also provide a different set of tools or logs. If a command is unavailable, that is information about your environment—not proof that you have done something wrong.
Before experimenting, consider these boundaries:
- Use a system you own or have explicit permission to inspect. - Prefer a lab that does not contain personal, customer, or production data. - Keep a way to restore the lab, such as a virtual machine snapshot or a fresh installation. - Do not deliberately disrupt a shared or production service to reproduce an example. - Read a command and understand…