Long before I ever started on dtrace, I used truss, on Solaris.
Brilliant tool. (My hobby/interest has always been tools
to help debug programs or systems). truss has some great
features, but it was very beholden to Solaris.
I wrote my own truss - I called it ptrace; this was in the days
before Linux was a successful and broad operating system. It did
what I would call "hacky" things to enhance what truss did. And
it worked on the various Unix flavors. By the time Linux became
successful and prominent, strace appeared. Strace for many years
was much reduced in functionality and reliability - on Linux - compared
to truss or ptrace (my tool).
I had run out of ideas for ptrace, and I note my last change
was back in 2007.
In 2010 or so I picked up the baton on dtrace.
strace has evolved and so has the Linux kernel; last I looked at the
strace source, it was disappointing - reflecting some deficiencies in
the Linux kernel (in terms of process control and debugging).
More recently I have had a chance to look at strace, and my ptrace, to
reassess the state of the art in user space tracing. (Dtrace, as
good as it is, is somewhat crude for doing what strace/ptrace and truss
do - dtrace doesnt make it easy to create a standalone application
that doesnt need full privs and has a good quality command line
parser; dtrace comes with a truss emulation, but its not refined
and its not good at decoding the arguments to syscalls...but I digress).
Looking at strace, it lacks a facility I was interested in, and to
which ptrace has: stack dumps of the syscalls. I found a package
on google called strace+ but it wont build, and I gave up trying
to figure out what was wrong, such was the brokenness of the build.
So, I re-evaluated my ptrace. Last I had touched it, in 2007, was
a while ago and it just didnt work - reliably. It didnt acknowledge
64 bit processors/processes or a mixed 64/32 bit world. And it didnt
After a couple of days, I got it up and running again; I started adding
the best bits of strace and other enhancements, and now, its pretty good.
Its reliable (thanks to bug fixes but also the kernel ptrace(2)
enhancements make it much more resilient). On older Linux kernels,
kill -9 of strace or ptrace could hang or kill the processes being traced.
On modern Linux kernels, this horrific situation is resolved.
I have an arsenal of LD_PRELOAD bits - which are useful for
debugging or monitoring specific scenarios, but strace/ptrace/dtrace are
great for pure unobtrusive debugging.
I used the strace source to help fix/understand some of the issues in
my own code. And am now considering adding more functionality - much
more than truss/strace has. FYI, heres the help/usage for ptrace as
it currently stands (some features are broken - I need to fix them -
especially the i386 specific code).
I may or may not release ptrace as source - I dont necessarily have
an interest in maintaining it - as its potentially fast paced
for the situation at hand I am debugging. (ptrace gives
me the luxury of writing C code rather than D code, in user space
to do very specific things - similar to LD_PRELOAD, but in a way
that can rarely accidentally kill the target; and ptrace is more
portable than Dtrace to systems where you dont have root access to
ptrace: Trace process execution. (C) 1990-2014 PD Fox, Foxtrot Systems Ltd
Usage: trace [-delay nn] [-d nn] [-gethostid nnn] [-trap] [-gethostname name]
[-llib ...] [-o output] [-fchnt] [-p pid] [-s size]
[-r [!]fd,...] [-w [!]fd,...] [-size nn] [-stack:nn] [-regs] [command]
-a Print argv on execv calls.
-flush Flush output as we go along.
List of functions to trace
-gethostid Intercept gethostid() system call and fake return value.
-gethostname Intercept gethostname() system call and fake return value.
-hex Dump ASCII strings in 1-byte hex
-hex2 Dump ASCII strings in 2-byte hex
-hex4 Dump ASCII strings in 4-byte hex
-name Sort by name.
-nest Allow for nested functions
-time Sort by syscall time.
-tee file Write output to specified file and stdout.
-trap Map SIGTRAP signal
-trace Trace with -l switch
-pc Show PC of system call
-c Display system call counts
-delay nn Sleep for nn msec before each syscall
-d nn Detach after nn calls to gethostid()
-e Dump out exec() functions
-f Follow child processes
-h Display strings in hex/asc (read()/write()).
-llib Preload shared library.
Intercept page faults.
-multiline When printing certain arguments, use
multilines to make pretty printing.
-n Print network addresses numerically.
-nosyscalls Don't print syscalls (monitor page faults only)
-o file Write output to specified file.
-p pid Trace specified process.
-ptr Show pointers for arguments.
-q Quiet mode -- dont print output.
-r [!]fd,... Dump read buffers for specified file descriptors.
-regs Dump registers.
Trace list of signals.
-size size Specify size of strings to print out.
-stack:nn Dump call stack (depth of nn).
-t Print timestamps. (msec accuracy)
-tt Print timestamps. (usec accuracy)
Specify syscalls to [ignore]/trace.
-verbose Add extra detail for some args.
-w [!]fd,... Dump write buffers for specified file descriptors.
-warp YYYYMMDD-HH:MM:SS Warp clock system calls.
-nouse_process_vm Avoid Linux 3.4 dependency
Set PTRACE_OPTS to pass in command line arguments.