The Final Phase of dtrace

I have been writing about the issues of inter cpu cross function
calls (xcall) - a key part of dtrace. This feature isnt used very
often, but its an important part of SMP to ensure consistency in
accessing buffers.

After a lot of effort, writing, rewriting and rewriting yet again, the
code is (nearly) finished. It looks good - it handles arbitrary
cpus calling into other cpus and allows for a xcall to be interrupted
by another call to xcall (effectively a mesh of NCPU * NCPU callers).

However, I have found a flaw. If I modify the dtrace_sync() function
to sync 100-200 times instead of just once, then occasionally there are delays
and kernel printk()s from the code - where spinlocks are taking too long.

Turns out, we could deadlock if we try to invoke an IPI on another
CPU which has interrupts disabled. Not totally sure how Solaris handles
this - I get a little lost in the maze of mutex_enter() and splx() code.

There is a solution to take us to the next level - NMI - the NMI interrupt
is not maskable (unless an NMI is in progress). NMIs are typically used
by Linux for a watchdog facility - make sure CPUs arent locking up, as
well as "danger signals" (like ECC/parity memory errors).

I will experiment to see if I can run via an NMI rather than a normal
interrupt and that should help reduce the problems of lock-busting
significantly.

At the moment dtrace is pretty good - my ultra-torture tests really
are horrible, and most people wont do that in real life.

So, as always, tread carefully until *you* feel happy this is not
going to panic your production system.

Post created by CRiSP v10.0.12a-b6033