DTrace and Xen: part #2

I wrote a few days ago about DTrace and Amazon EC2 not playing
well together. As I have dug into the problem, it has become clear
what is wrong.

Xen uses paravirtualisation - meaning that priviledged instructions need
to go through the correct API, else the VM may crash abruptly. Its
quite distracting that Xen does this - not even the ability to failback
to more normal semantics at a performance cost.

Over the many iterations of the Linux kernel, these priviledged areas
have been mapped, via #define macros so that if CONFIG_PARAVIRT is
set, then a function call is made, which invokes the hypervisor; if
CONFIG_PARAVIRT is not set, then the direct instructions are executed.

This is mostly straightforward; eg the IDT instructions (SIDT and LIDT)
dont do the right things in a Xen guest; Xen keeps a copy of the IDT
outside of the address space the kernel can see. So, any changes need
to be channelled through the correct API. In addition, direct memory
tampering with IDT entries is not allowed - you must tell the Xen
hypervisor that you just changed an entry.

After making these changes to DTrace, it now loads and unloads,
and, with some corrections to the page-table mapping code, syscall
tracing works.

I have two or three areas to work on to complete this work.

Firstly, the interrupt handlers (for INT1 and INT3) are broken - I
havent played by the Xen API rules, so I need to fix that. (I also need
to make sure that the correct paravirt detection is done; a kernel
which is setup as a Xen guest can be run on physical hardware or inside
Xen; the API functions hide this detail).

Second, multi-cpu operation needs to be corrected. The code in
xcall.c which invokes the NMI based IPI cross-cpu calls is too low level
and doesnt play well with Xen. If I restrict my guest to a single CPU,
things work well; on a multi-cpu, the system locks up because the
cross-cpu calls are not delivered or processed properly.

Lastly, having made these changes, I need to handle old kernels
which lack the Xen API calls, so we dont break compatibility.

I now have a Xen guest on my main machine - a nuisance, because
VirtualBox wont run on a Xen kernel. So I either need to migrate my
VMs to Xen, or migrate to VMWare or KVM.

Post created by CRiSP v11.0.12a-b6455