XFree86 currently accesses your hardware directly, essentially bypassing the kernel. Historically, this is because Linux, BSD, and other unix systems never had their own video drivers.
However, as Linux modernized, it now has video drivers, otherwise known as the "Linux Framebuffer." The idea is really cool. You can insmod / rmmod a video driver, just like any other Linux driver, and the video device is /dev/fb0 (and so on) just like any other device. Unfortunately, the Linux Framebuffer is severely underused, and so many cards are unsupported and the drivers are not very optimized. In contrast, XFree86's drivers are a lot more mature.
DirectFB is an API that layers over the Linux Framebuffer. It's sorta like DirectX, in that there is a HAL/HEL system (if your card doesn't support a certain feature, for instance such as alphablending, DirectFB will do it in software for you). However, DirectFB is not intended for just games, like DirectX often is reserved for.
Oh, and before anyone says that DirectFB is an X replacement, let me remind you:
* XFree86 = graphics layer + 100 other important things
* DirectFB = graphics layer
Personally, I'd just like to see DirectFB used as a replacement for the XFree86 graphics layer. In fact, you can already do this with XDirectFB (it's a port of XFree86 to DirectFB).