For about the last month, the longer I leave an X session running the more swap I have in use. (I use the system monitor applet.) Eventually performance starts to suffer, I restart X, and everything is fine for a couple of days then it starts to creep up.
I'm assuming a recent update has a memory leak, but have no idea how to track down the cause. I checked top and sorted by memory use, and XFree86 was on top. Next was Mozilla. I shut down Mozilla and swap didn't go down. Nothing else showed more than 1% memory use.
It's a fairly crappy old box with not enough memory, but it didn't used to do this.