I would like to suggest two plugins that could easily help you catch any x-rayers or people who use any other mods.
First off, we have Orebfuscator. This is an anti-xray plugin that when an xray mod is turned on, it shows every block as an ore.
BukkitDev: http://dev.bukkit.org/server-mods/orebfuscator/
Gif examples:
Video Example:
I think this would make it more fair for players who play legit.
Next up is Spectate. This plugin lets you see through the eyes of the victim without them knowing. You see nearly their every move, all besides chat I believe. I think it could be used to find players who have xray mods that aren't blocked by Orebfuscator. You could also watch what they're building without teleporting to them if you're really bored :3
BukkitDev: http://dev.bukkit.org/server-mods/spectate/
Feedback is appreciated.
