Saturday, 29 October 2011

Dtrace .. does it work. Yes. No. Yes. No. What?!

Got a strange one here. Nigel Smith reported issues running dtrace
on real hardware (FC15). I could reproduce the issue on
Ubuntu 11.10 (Linux 3.0.0). A simply

$ dtrace -n vminfo:::'{printf("%s", execname);}'

would panic or hang the kernel - not straightaway, but within say 5mins,
especially if doing a:

$ while true
> do
> date
> done

in another window.

I've been staring at the dtrace code and trying various things to
see what causes it. Its an annoying panic because I lose control of the
kernel and no way of figuring out what happened immediately leading
up the issue. The stack trace on the panic doesnt help
(I am seeing the same panic in the e1000 driver cleanup code,
but no references to dtrace causing this).

I suspect dtrace is taking an interrupt, maybe not restoring a
register and sometimes, that register happens to be important.

Especially strange, as, running on Ubuntu 11.04 (2.6.38 kernel),
it works fine. I can really torture the system and it stays up.

I need to dive more into the entry64.S code to examine what changes
happened around the way an interrupt is handled. If I am lucky I may
be able to localise this to a register issue (%GS is a high probability).

Linux is really missing a kernel debugger. Theres kgdb and remote
debugging available, but this is really painful, when you suddenly
need to have to compile a new kernel, waste more than 1GB of disk
because of the symbol table, and then try and get it all "working".

What is needed is a better way to take control on a panic, and
poke around, similar to kadb for older Sun machines.

I might have to start writing a crude debugger to help with
these annoying "you died but I am not going to tell you why" issues.

Post created by CRiSP v10.0.17a-b6103


  1. To save you time writing a crude kernel debugger, you might consider using the Joyent method when they ported Linux KVM to SmartOS.

    * SmartOS host system
    * Virtualized instance of Ubuntu 11.10
    * SmartOS dtrace to trace the virtualized instance

  2. Sorry, I didn't meant to be cryptic. I was referring to the process that Joyent used to port the Linux Kernel Virtual Machine from Linux to SmartOS.

    Joyent is the company that created their own operating system called SmartOS, which is based on an OpenSolaris fork called IllumOS.

    One of the Sun Dtrace creators works for Joyent. As part of the KVM porting process, he used a SmartOS host system, and then virtualized an instance of Linux within the host. This meant that he could use Dtrace (which is supposedly fully functional within SmartOS) to debug and trace kernel issues with the running Linux instance.

    You were referring in your post to trying to debug "you died but I am not going to tell you why" issues; I was suggesting that in lieu of writing your own kernel debugger, perhaps you could use the same method that Joyent uses to debug Linux kernel/KVM issues, especially in light of your familiarity with Dtrace probes already.

  3. Otoburb, has anything been published, that demonstrates how you can use the SmartOS DTrace to trace into Linux running under KVM?

  4. Hi Nigel,

    Unfortunately, details seem a bit sparse on the exact setup. The links I've been able to find are:

  5. The article is an interesting and enlightening read.

    I cant see any relevance to using SmartOS to dive into Linux - it doesnt help for a native Linux port of Dtrace, although it can be useful for other linux kernel work (eg profiling and experimentation).

    I have resolved the latest kernel hangs - this is solely a bug in linux's printk() implementation, which is attempting to be too smart. It is not allowing recursive printks - recursive printks are bizarre, but, when you need them, you need them bad.

    It highlights a naughtyism in my code, so I can now work to remove these (and avoid wasteful pollution of the /var/log/messages file).