So, Nigel got me to check dtrace with real hardware. And my heart stopped.
It ran, and then hung the machine. I narrowed down to an "fbt::form*:" (46
probes) and it kept hanging.
Nothing I could do could make it work. (Relaxing locks, removing
all the actual probe capture code). I assumed the worst - a CPU
L1/L2 caching issue against the SMP cores.
So, I tried on my VirtualBox VMs, and it was the same! Now, this
worked perfectly yesterday, and I hadnt changed anything. Amazingly,
"fbt:::" worked well, but a narrow FBT probe did not.
After narrowing it down further, it transpires that some of the calls
to printk() (which now map to the internal dtrace_printf, rather than
the real kernel printk) were causing some form of recursion fault.
After modifying the dtrace_printf() and printk() code to avoid this,
it worked as I had expected before. Nicely. On real hardware.
Now, something else is strange. I keep talking about rapid-fbt-teardowns
being slow. I had done a lot of work to optimise this, and was proud
of my ~10s of execution, down from more than 30-40s. On real hardware,
this was coming in at sub-second. Now, on my VirtualBox, it was
subsecond too! I dont know if the act of rebooting my laptop and
starting afresh had fixed the issue, or some other nastiness (like
the recursion above) had caused a performance issue to disappear.
I checked the timer (tick-1ms) again - and thats still bad,
at about 350 x 1ms ticks per real second.
(I wonder if my VirtualBox slowdowns could be triggered by a paused VM).