|I would like to post the availability of Sun's Dtrace for Linux.|
This is a complete port to Linux, and is provided as a source code module which you can compile and load into a running kernel.
It includes the ability to trace using the various providers, of which the key ones at present are the FBT, SYSCALL and INSTR providers. You can trace any syscall or function. The INSTR provider is an instruction level provider, e.g. you can trace any call, jump, LOCK, REP, CLI/STI in the kernel - as many or as little as you like.
There are some parts of the user space dtrace which are yet to work (attaching to a running process, but this is being worked on).
Dtrace runs on 32b and 64b kernels and has been tested with a variety of kernels from 2.6.9 to 2.6.29. No kernel source code changes are required - it is a pure open source kernel module and will work with or without kprobes - it has no dependencies on the kernel, except for compilation purposes.
The software is free, under the CDDL since parts are courtesy of Sun.
Do not try this on a production host without validating until you are happy it cannot crash your test systems.