PID Provider: Did you call?

dtrace: description 'pid5748::do_nothing2:' matched 9 probes
CPU     ID                    FUNCTION:NAME
  1 250496                do_nothing2:entry
  1 250497                    do_nothing2:0
  1 250498                    do_nothing2:1
  1 250499                    do_nothing2:4
  1 250500                    do_nothing2:9
  1 250501                    do_nothing2:e
  1 250494                   do_nothing2:13
  1 250502                   do_nothing2:14
  1 250495               do_nothing2:return

Spent a couple of weeks to get this going, and its looking postive now.
Let me recount the issues.

First, what is the PID provider? What is USDT? What is a normal probe?

Hopefully, we all understand a normal probe - its effectively
a breakpoint placed in the kernel, eg via the FBT provider.
(Syscall tracing doesnt rely on breakpoints, but thats doesnt really matter).
When the breakpoint is hit, dtrace maps that to a user space caller
who is waiting for the event.

USDT is similar to the normal in-kernel providers, but they occur in
user space. They occur because someone put the probe in their code
(e.g. in the interpreter loop in Perl or Python, or in malloc() in libc).
Because dtrace for linux isnt widespread, few if any, apps have user
space probes.

The PID provider (also known as "fasttrap", for historical reasons, and
the name of the source code file for it) is very similar to USDT.
But instead of manually littering source code with probe points,
a user can drop a probe into a running process, e.g.

$ dtrace -n pid1234::malloc:entry

Along the way to getting the PID provider working, I found some interesting
things.

First, although there is a lot of code in libdtrace to handle supporting
the PID provider, it is actually a lot simpler than I thought. The act
of placing a probe requires finding the address in the target process.
Once located, a breakpoint instruction is placed at the probe address, or
addresses. (PID provider lets you instrument the entry, return, or any/all
instructions of the target function; in fact, its very similar to the
INSTR provider).

dtrace itself doesnt need to ptrace(PTRACE_ATTACH) to the process except
to gain write-access to the target process. On Solaris, using the
/proc subsystem, ptrace() is not used. (Solaris allows two or more
processes to debug a process at the same time, or, at least read/write
memory and control the process; ptrace() does not). Although libdtrace
in my release uses ptrace(), this is a limitation, which I plan to remove.
The reason is that you cannot use dtrace to probe a process running under
a debugger; Solaris lets you do this. Its a silly limitation of DTrace/Linux.

Another thing I found out which is very interesting is that you
*CANNOT* do the following: put a probe on, for example, malloc, for
every process in the system, including those which have yet to be created.
If one examines:

$ dtrace -n fbt::function:entry

for probing the kernel, you get a hit no matter which process or interrupt
causes the function to be called. But there is no syntax to support something
like:

$ dtrace -n pid*::malloc:entry

When using the pid provider, we specify an actual PID, not all PIDs.
Architecturally, dtrace cannot support this. When you put a PID probe
in place, dtrace creates a new probe out of thin air - its automatically
registered in the kernel. These "auto" probes are removed when the
target process or dtrace terminates.

I will continue this entry in the next blog piece...

Post created by CRiSP v11.0.10a-b6436