IWETHEY v. 0.3.0 | TODO
1,095 registered users | 1 active user | 0 LpH | Statistics
Login | Create New User
IWETHEY Banner

Welcome to IWETHEY!

New Tomcat Questions.
This is a question that I think I know the answer to, but being that you could put what I know about Tomcat and Java in a thimble and have room for your fist, I thought I'd ask. We have a tomcat server sitting in a dmz zone (2 firewalls to the Net, 2 between our LAN and it). A Java developer here finally launched an "application" consisting of about 42 jsp pages and various and sundry other java objects. This is the production server. We have development servers and qa servers on the LAN, but this is the production box. He wrote his own logging system and places the log files in a directory he made off of the tomcat install root ($CATALINA_HOME). He now tells me he needs a shell account on that host. For obvious reasons, as a rule, we do not provide anyone but sys admins a shell account on any host not on the LAN. They run lambdaprobe on this box in the role of "probeuser". There are not administrator roles defined for lambdaprobe on this box. That gives them access to a whole host of things that make me nervous, including viewing and downloading logs in $CATALINA_HOME/logs. I offered an sftp jail account to the directory he created, but that isn't good enough it seems. He wants shell access and I don't want to go through the pain of setting him up a chroot jail on a production box so that he can "debug" a production application on a production server. Call me stupid, but I thought that was what the dev and qa servers were for: debugging applications.

I have some questions for those of you who are tomcat/java goobs:
1) In Javaland, is it typical to debug apps on production boxes?
2) In Javaland, is it typical to create your own logging directories outside of webapps and/or $CATALINA_HOME/logs?
3) In Javaland, is it SOP to hand out shell accounts to developers on boxes that are in production and not on your LAN?

Any help appreciated. This just doesn't pass the smell test for me. But, as I said, maybe this is normal for Java teams. I just don't know, but I do know that this developer has no difficulty lying to me. He's done it before and it was absolutely clear that he was lying to everyone but the CIO. (He said a file format was X, produced the file and it was not in format X. But, both he and the CIO insisted it was. It wasn't even a close call. The files were dramatically different.)

New logging outside the path
meh, no real call for that. As far as a shell account for a dev user in prod? Who promotes the code from dev/qa, him or someone else?
It should be a sysadmin but the developer does need access to the logs without hassle so either syslog to a box in dev, or enable apache so he can view the logs via a browser.
Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free American and do not reflect the opinions of any person or company that I have had professional relations with in the past 55 years. meep
New Re: Tomcat Questions.
None of your questions have anything to do with "In Javaland." You already know the answers to these questions.

As a Java developer, I typically want as little access to production as possible.[*] Developers who attempt to debug in production are ninnies.

[*]: What I want and what I need to do (in production) to accomplish my job are often two very different things. I often have far more access than I desire, because I choose to work in small shops where I need to wear multiple hats. That said, I'm trying to move my current place of employ more towards a DevOps model where we can use our CI server (currently Jenkins) to perform push-button deployments to any environment for us, and everything is painless. None of this should require my access to production anything.
-Mike

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, 1759 Historical Review of Pennsylvania
New Answer: it depends
Policies are different all around. I've had shell access to production web servers before.


Debugging on production boxes: unfortunately sometimes production is the only place bugs happen. Only if you can exactly duplicate the production environment elsewhere (hardware, software, and most importantly production inputs) is it reasonable to state that in every instance you should be able to debug outside of production.


Logging habits are all over the place. Let me instead describe reasonable alternatives to do logging of Java (or other) applications that do not require access.
  1. Allow the applications to log locally to a specified directory, typically located on a NAS or some other central file system. Use a product such as Splunk (http://splunk.com) to periodically gather the log files and index them in a centrally accessible location. This approach has he advantage of being quite simple, but you are dependent on the stability of the NAS. Not everyone has access to enterprise SAN for logging purposes.

  2. Create a logging infrastructure using something like rsyslog. Put a local rsyslog daemon on each machine. Applications can then log locally using UDP (this is a simple matter of configuration with log4j). Have the local rsyslog daemons normalize and forward the log messages to a central location. This centralized location can be either log files readable by developers, or something like Splunk. This is my preferred approach, as once you have the log messages in a syslog stream you can tee them off and send them anywhere: Splunk, event correlation systems, databases, whatever. As an extra bonus use RFC 5424 structured data for added craft.


Javaland doesn't signify with respect to shell accounts. Again, these are all operations policies, and if you have well-formed, published operational and development standards for these things then there should be no difference between how things are done in "Javaland", "Pythonland", "PL/SQL-land", or even Cleveland for that matter. If your organization makes a practice of employing developers for production support, then it's probably best to allow them to have access to production machines (with restrictions such as a need-only token based authorization system). There are useful debugging approaches that can only be accomplished by running commands as the process owner (jstack, etc).
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
Expand Edited by malraux Jan. 27, 2012, 12:09:53 PM EST
New ^Thanks all.
Unified logging is a great idea and we should do that. With enough shovels, I'd *love* to make the possible.

Just a few notes. The QA/Test boxes are identical. The only difference between the QA/Test boxen and the production box is location. QA/Test boxes are on the LAN, production outside the f/w. So, from my POV, debugging *should* be possible on one of the QA/Test boxes. The dev boxes start out in sych with everything else, but the developers pretty much own those boxes. Until the devs break it. Then we clean them return them to being in synch with production.

Box-> Yep, same guy gives official word that app is ready to push to prod, and my group does it.

I wasn't trying to be a smart ass with my qualifier "In Javaland." I wrote it that way because it's been a decade or so since I developed anything other database stuff. I'm really out of touch with any sort of development that doesn't involve sql, but when I was a developer all those years ago, the sysadmins I worked with would have laughed me out of their offices (or just shot me) if I asked for a shell account on a production box. ;0)

Thanks again everybody!
New Re: ^Thanks all.
The boxes might be identical, but without the same data (database, web requests, STP traffic, load levels, and so on) it can be very difficult to find and fix a problem outside of production. Load in particular can be difficult to replicate.

Example: I've seen "identical" environments produce different results when the web request traffic was off by a few milliseconds, where different is defined as "works fine in staging, completely melts down in production". It was discovered that the replay tool was batching its requests in 5ms intervals. Once the interval was improved to 1ms the staging environment began to exhibit the same behavior that was observed in production.

That said there are not many environments that actually involve such tight tolerances.
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New Thanks again.
I'm embarrassed to say this, but the "production" box we're talking about in my case is a single function box with, at present, 1 user utilizing this one batch of jsp pages, 1 or 2 days a week. IOW, it is a *very* lightly loaded box. :0)

We might run into load problems, but they'll be a while off. The plan is to add one user every 1 to 2 weeks up to a grand total of around 90 at the end of the year. And they won't be concurrent users. ;0)
New Yeah, not really necessary for PRD at that point
As long as the data can be replicated.
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New AdminiScott has the best answer.
First of all, custom logging below $CATALINA_HOME is wierd and were I the sysop I would flat-out disallow this. It's too close to the web server being allowed to write to its own files, to my way of thinking.

OTOH, shell access to the webservers for devs is incredibly useful, even if all they can do is look. I have that kind of access on our web servers and it is most helpful.

Logging is the big reason. Logs of events occur in Prod that, not unreasonably, occur nowhere else. We need access to those for diagnosing real issues that occur on Prod. And for "debugging" things that can only occur in Prod often when interacting with third-party links that can only be done from Prod. We also use the access to verify code drops because Operations have a kind of set-and-forget mentality when doing releases.

Wade.
Just Add Story http://justaddstory.wordpress.com/
New Yabut...
If your applications are logging elsewhere, so can Tomcat (set the Juli properties or use log4j) and the other system processes.
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New This is true.
Our Ops have never suggested remote logging. I think they're scared of it. :-) But they do know how to give us shell access without letting us do anything.

Wade.
Just Add Story http://justaddstory.wordpress.com/
     Tomcat Questions. - (mmoffitt) - (10)
         logging outside the path - (boxley)
         Re: Tomcat Questions. - (mvitale)
         Answer: it depends - (malraux)
         ^Thanks all. - (mmoffitt) - (3)
             Re: ^Thanks all. - (malraux) - (2)
                 Thanks again. - (mmoffitt) - (1)
                     Yeah, not really necessary for PRD at that point - (malraux)
         AdminiScott has the best answer. - (static) - (2)
             Yabut... - (malraux) - (1)
                 This is true. - (static)

The only reason the bongos are the worst is because AIs have never tried chicken nuggets before.
57 ms