strace needs root or the same user that owns the process, and it slows the traced program down a lot, so filter early and detach when you are done.
Attach to a Running Process
Ctrl+C detaches from a process you attached to; it keeps running.
| Command | What it does |
|---|---|
| Attach to PID 1234 and print its syscalls live | |
| Follow its threads and child processes too | |
| Attach to the newest process matching a name | |
| Attach to a systemd service by unit name | |
| Write to a file instead of the screen | |
| Attach, but show only network calls | |
| Count calls instead of printing them; Ctrl+C prints the table | |
| Add clock timestamps and per-call duration | |
| Attach to several processes at once |
Find the PID first with ps, , or .
Find a Missing File or Config
The everyday use: a program starts, says nothing useful, and quits. Trace what it tried to open and look for .
| Command | What it does |
|---|---|
| Show only file opens | |
| Every call that takes a filename (open, stat, access, unlink) | |
| List the files it looked for and did not find | |
| Same idea without the pipe (strace 5.2 and newer) | |
| Save the opens to read later | |
| Files a running process actually reached |
What it looks like:
openat(AT_FDCWD, "/etc/myapp/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/etc/myapp/config.yml", O_RDONLY) = 3The first path is the one it wanted and did not find, the second is the one it settled for. on a path you expected to exist is the answer most of the time; means the file is there and the permissions are not. The same trick finds missing shared libraries, which show up as failed calls on files.
Run a Command Under strace
| Command | What it does |
|---|---|
| Run the command and print every syscall it makes | |
| Follow the children a shell script spawns | |
| The program's own flags come after it, as usual | |
| Only the syscalls you name | |
| Run to the end, then print a summary table | |
| Run it as another user (needs root) | |
| Set an environment variable for the traced run | |
| Print long strings in full instead of cutting at 32 characters |
Ctrl+C here kills the program, unlike Ctrl+C on an attached process.
Filter Syscalls
takes syscall names, or a class that stands for a whole group. Filtering is also the cheapest way to cut the slowdown.
| Filter | What it shows |
|---|---|
| The named syscalls only, comma separated | |
| Anything that takes a filename ( without the % also works) | |
| socket, connect, bind, listen, send, recv | |
| fork, clone, execve, exit, wait | |
| File descriptor work: read, write, close, poll, epoll | |
| mmap, brk, mprotect | |
| Signal delivery and handling | |
| Everything except these; quote it or the shell eats the | |
| Syscalls matching a regular expression | |
| Only calls that returned an error (strace 5.2 and newer) | |
| Drop the lines | |
| Signals only, no syscalls |
Follow Forks and Threads
Without , strace watches one process. A program that does its real work in a child or a worker thread looks idle.
| Command | What it does |
|---|---|
| Follow children from fork, vfork, and clone, threads included | |
| Follow the threads of something already running | |
| One file per process: , | |
| The fork and exec tree alone, no I/O noise | |
| After , find which child hit the missing file |
With , every line gains a prefix so you can tell the processes apart. See grep for sifting the files leaves behind.
Save Output to a File
strace writes to stderr, not stdout, so saves the program's output and none of the trace.
| Command | What it does |
|---|---|
| Trace to the file, program output stays on screen | |
| The shell version of the same thing | |
| Pipe the trace, not the program's output | |
| Append to the file instead of overwriting it | |
| Split per process, one file each | |
| Trace in the background; detaches |
Timestamps and Timing
| Flag | What it adds |
|---|---|
| Wall clock time on each line () | |
| The same with microseconds () | |
| Unix epoch seconds with microseconds, easy to subtract | |
| How long the call took, at the end of the line () | |
| Time since the previous syscall, for spotting the gap | |
| With , summarise wall clock time instead of CPU time |
is the pair for "which call is the slow one". Read as time spent inside the kernel, and as the pause before the call, which is usually where the waiting really happened.
Summary Mode
| Command | What it does |
|---|---|
| Counts, time, and errors per syscall after it exits | |
| The same for a running process; Ctrl+C prints the table | |
| Sort and total by wall clock time, not CPU time | |
| Sort by count (, , also work) | |
| Print the full trace and the summary | |
| Totals across every child process |
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
62.19 0.004312 14 298 12 openat
18.02 0.001249 3 402 read
11.44 0.000793 9 88 3 connectThe errors column is the one to read first: twelve failed calls points at a missing file, not a slow disk.
Full Strings
strace cuts string arguments at 32 characters and marks the cut with , which is why the path or payload you wanted is half there.
| Command | What it does |
|---|---|
| Print up to 4096 characters per string | |
| Also expand structures and environments instead of abbreviating | |
| Print non-printable bytes as hex | |
| Hex and ASCII dump of everything read from fd 3 | |
| Dump everything written to stdout and stderr | |
| Print the path behind every file descriptor number | |
| Also print the address and protocol behind socket fds |
Reading the Output
[pid 2412] openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 2412] read(3, "127.0.0.1 localhost\n", 4096) = 20
[pid 2412] connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 2412] +++ exited with 0 +++| Part | What it means |
|---|---|
| Which process the line came from; only shown with | |
| The system call. explains its arguments | |
| Relative paths resolve from the process's current directory | |
| Flags ORed together, which is why there are pipes | |
| The return value. For an open, this is the new file descriptor | |
| The call failed and the name after is the errno | |
| inside | The buffer size asked for; the is what actually arrived |
| The call is still running; a later closes it | |
| A signal arrived, and the next line says what happened | |
| The process finished, with its exit code | |
| Something else killed it, often the OOM killer |
The fd number is the thread to pull: find the or that returned it, and you know what every later , , and on that number is really touching. prints the path next to the fd so you do not have to scroll back.
Network Calls
| Command | What it does |
|---|---|
| Every socket, connect, send, and receive | |
| Outgoing connections with the resolved address | |
| Only the network calls that failed | |
| Check whether it read the DNS config at all | |
| Catch the DNS lookups themselves | |
| Which address and port it tried to take |
means nothing is listening, means packets are going nowhere, and means something else already has the port. Use lsof to find the owner, and tcpdump when you need the packets rather than the calls.
Common Errors
| Errno | Meaning | What it usually is |
|---|---|---|
| No such file or directory | Missing config, wrong path, or a missing | |
| Permission denied | File mode, a directory on the path, SELinux or AppArmor | |
| Operation not permitted | Needs root or a capability the process does not have | |
| Resource temporarily unavailable | Normal on non-blocking sockets; a bug only if it never stops | |
| Operation now in progress | Normal for a non-blocking | |
| Interrupted system call | A signal arrived and the program should retry | |
| Connection refused | Nothing listening on that port | |
| Connection timed out | A firewall is dropping packets, or the host is gone | |
| Address already in use | Another process holds the port | |
| Too many open files | The process hit its descriptor limit () | |
| No space left on device | Disk or inodes full (, ) | |
| No such process | The PID went away, often mid-attach |
A failed call is not automatically the bug. Programs probe paths on purpose, so a handful of hits followed by a success is normal. The interesting one is the failure with nothing after it.
strace vs ltrace
| Question | strace | ltrace |
|---|---|---|
| Traces | System calls, at the kernel boundary | Library calls: , , |
| Answers | Which file, socket, or permission failed | Which function ran and with what arguments |
| Static binaries | Works | Does not work, it needs dynamic linking |
| Overhead | High | Higher |
| Usually installed | Often | Rarely |
Both drive the same machinery, so both stop the process on every event. For tracing you can leave running on a live server, use or instead.
Gotchas
- blocks most attaches. On Ubuntu and most desktop distros is , so you can only trace your own children. gets around it; lowers it until reboot, and a file in makes that permanent.
- Containers drop the capability. Add to , or put in the pod's .
- The slowdown is real. Ten times slower is common on a syscall-heavy process, and timeouts elsewhere can fire while you watch. Narrow with , write to , detach quickly.
- The trace goes to stderr. Any pipe or redirect needs , or use and skip the problem.
- strace is not installed on minimal images: , , .
- Nothing on screen usually means the process is parked in , , or waiting for work. Send it a request, or check journalctl to see what it thinks it is doing.