Trials and Tribulations of Dtrace

I reported a while back about the issue with cross-cpu function
calls (via the smp_call_function() series of functions).

On Solaris, the way dtrace_xcall() works is to invoke an inter-cpu
interrupt in order to ensure the CPUs are in sync with certain data structures.

Alas, on Linux, the smp_call_function's are not callable from an interrupt
routine. Unluckily, this is necessary and can cause the systems to
deadlock / hang, whilst dtrace breaks the API call.

I tried an alternate implementation of dtrace_xcall() using timers,
but found problems in getting this to work.

I tried a 3rd variant - not invoking cross-cpu calls, but, instead
executing the cross-cpu call 'on-behalf-of' the current cpu. This
looks more promising, but requires putting a mutex (actually, a spin-lock)
around the dtrace_probe function, to avoid one cpu executing a probe, whilst
another cpu is doing the housework to clean up.

This does look better, but I am currently having rare scenarios
where some syscalls can break: a repeated series of process forks can
occasionally give rise to a problem mmap-ing one of the shared libraries.
This is very difficult to debug - as many millions of calls can work
correctly, before one of them fails.

By "thinking about the problem" it is likely that some form of stack
or register corruption is happening somewhere. What might be worse is
CPU cache consistency issues happening - I am fearful of cache consistency
issues being very difficult to debug, but lets see what happens.

I have put extra debug code into dtrace to try and spot the issues.

There is another approach which is to intercept the NMI interrupt which
is where IPI interrupts come from, but I need to think this one through
before attempting this.

Post created by CRiSP v10.0.7a-b5984