I'm involved in a project to implement company-wide application monitoring: database, home-grown apps, distributed grids, a JMS service bus, and web servers. Also in the mix is SNMP and hardware monitoring.

The question is, what's the best architecture for this? We need both overview monitoring with alerts, and full data capture for drill-down in the event of a problem.

I'm leaning towards a combination of agents on each machine and/or in each application with a central server polling/receiving data directly. Drill-down would be done via application logs.

Another suggestion is to throw all monitoring information on a bus like the service bus. I'm concerned about number of moving pieces and using an in-band process to monitor essentially out-of-band information (since the bus is part of the architecture being monitored).

I've seen other monitoring systems that use multicast to disseminate agent information as well.

Any comments or thoughts?