Policies are different all around. I've had shell access to production web servers before.


Debugging on production boxes: unfortunately sometimes production is the only place bugs happen. Only if you can exactly duplicate the production environment elsewhere (hardware, software, and most importantly production inputs) is it reasonable to state that in every instance you should be able to debug outside of production.


Logging habits are all over the place. Let me instead describe reasonable alternatives to do logging of Java (or other) applications that do not require access.

1. Allow the applications to log locally to a specified directory, typically located on a NAS or some other central file system. Use a product such as Splunk (http://splunk.com) to periodically gather the log files and index them in a centrally accessible location. This approach has he advantage of being quite simple, but you are dependent on the stability of the NAS. Not everyone has access to enterprise SAN for logging purposes.


2. Create a logging infrastructure using something like rsyslog. Put a local rsyslog daemon on each machine. Applications can then log locally using UDP (this is a simple matter of configuration with log4j). Have the local rsyslog daemons normalize and forward the log messages to a central location. This centralized location can be either log files readable by developers, or something like Splunk. This is my preferred approach, as once you have the log messages in a syslog stream you can tee them off and send them anywhere: Splunk, event correlation systems, databases, whatever. As an extra bonus use RFC 5424 structured data for added craft.

Javaland doesn't signify with respect to shell accounts. Again, these are all operations policies, and if you have well-formed, published operational and development standards for these things then there should be no difference between how things are done in "Javaland", "Pythonland", "PL/SQL-land", or even Cleveland for that matter. If your organization makes a practice of employing developers for production support, then it's probably best to allow them to have access to production machines (with restrictions such as a need-only token based authorization system). There are useful debugging approaches that can only be accomplished by running commands as the process owner (jstack, etc).