Even after having learned many programming languages and contributed to various projects, it was only when I started using strace that I felt like truly, efficiently understand what any program does, and can reliably write programs that do things fast.
I believe that "syscall oriented programming" (making your program emit exactly and only the right syscalls) results in clean, understandable software.
Now I use strace every day, as it is often the fastest way to figure out problems with any tool, written in any language, open-source or proprietary.
- Something hangs? strace shows how it's hanging.
- Computer is slow? strace will likely show who's the culprit spamming syscalls.
- "Unexpected error occured"? Bypass programmers doing poor error handling, and witness the underlying "no such file or directory" directly.
Last week I even used strace to debug why my leisure-time computer game wouldn't load a mod, and as usual, strace did the job.
It's so disappointing that dtrace is neutered by System Integrity Protection on MacOS. When I want to do this I have to stop and transport my workload to a server or VM, which may or may not reproduce the issue.
strace is the first thing I skimmed down the thread for. You can learn a lot about how things work (or aren't working) by getting really familiar with an strace. Some of my coworkers give me grief for how easily I jump to "let me see an strace" but it works.
sysdig is useless without installing nontrivial performance-impacting instrumentation, cannot handle non-IP networking, does not fully report all syscalls, has a license with patent crap in it, has gated features behind a paywall, and cannot inject syscall faults. It's not even in the same class of tool as strace at this point.
> sysdig is useless without installing nontrivial performance-impacting instrumentation
Most reasonable people reading this sentence would come away with the conclusion that strace is fast, whereas sysdig has some inherent overheads. In reality it is strace that has performance and other problems which make it completely unsuitable for production use (strace will slow syscall heavy code down by a factor of over 100; sysdig won't). Sysdig, on the other than can definitely be used in production and I always found the performance overhead minor. Can you point to something showing otherwise? BTW, newer versions of sysdig do not require a kernel module, thanks to eBPF (but I have not used this).
> , cannot handle non-IP networking,
What is an example of a networking related query you can do with strace but not with sysdig?
> does not fully report all syscalls
Can you expand? Are you referring to the fact that sysdig will drop traces if the userland client cannot keep up (which is a feature and not a bug, and something that all production grade tracing tools do)?
> , has a license with patent crap in it,
As far as I'm aware sysdig's core is Apache licensed and the user scripts are MIT and GPL licensed. Apache has a patent grant, which seems better than not having one. What is your specific beef?
> has gated features behind a paywall,
What features that strace offers are behind a paywall in sysdig? What's wrong with a company that provides a tool that massively advanced the (linux, pre-eBPF) state of the art as open source for free to all also provide some paid offerings on top?
> and cannot inject syscall faults.
This is indeed a useful recent-ish feature I did not know about so thank you! But there are other ways to do it, and something that's orthogonal to the core tracing functionality.
> It's not even in the same class of tool as strace at this point.
Indeed -- the only reason to use strace at this point is because you already know it and it is likely available. This may change if strace switches away from ptrace, but for now it is a joke. If you want something that just does strace, but much better (minimal overhead, powerful and intuitive query language with CLI autocompletion) use sysdig. If you want to use the most general and powerful tool that can tell you lots of other stuff besides syscall usage (but has a much worse UX) look at eBPF and perf. If you want to be a serious performance engineer or similar you will have to learn it, but I suspect for most people sysdig has the best ROI. Perf and dtrace are both (far) more versatile but, IMO, (far) less pleasant to use.
Even after having learned many programming languages and contributed to various projects, it was only when I started using strace that I felt like truly, efficiently understand what any program does, and can reliably write programs that do things fast.
I believe that "syscall oriented programming" (making your program emit exactly and only the right syscalls) results in clean, understandable software.
Now I use strace every day, as it is often the fastest way to figure out problems with any tool, written in any language, open-source or proprietary.
- Something hangs? strace shows how it's hanging.
- Computer is slow? strace will likely show who's the culprit spamming syscalls.
- "Unexpected error occured"? Bypass programmers doing poor error handling, and witness the underlying "no such file or directory" directly.
Last week I even used strace to debug why my leisure-time computer game wouldn't load a mod, and as usual, strace did the job.
strace helps to really understand computers.
If you want to learn more about strace, check out Brendan Gregg's pages like http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-..., my presentation for an intermediate-level example (https://news.ycombinator.com/item?id=16708392) or my project to build a programmable strace alternative (https://github.com/nh2/hatrace).