Post #51,632
9/13/02 10:42:19 AM
|
Build/Release/Promote Process
I'm interested in hearing what everyone's experiences (past and new) on development processes in regards to build, release and promotion through staging, QA, and finally production.
I work for a company where have many sites that are very large, and share code bases and databases. On top of that, we're attempting to do parallel development and find ourselves in a few messes, most of the time.
We're taking a look at redoing our process, but would love to get a feel for other's best practice and learned lessons.
Thanks,
-Jason
----
My pid is Inigo Montoya. You "killed -9" my parent process. Prepare to vi.
|
Post #51,644
9/13/02 11:38:56 AM
|
Re: Build/Release/Promote Process
Important things:
1) If you are going to have a process, *use it*. Everything should go into the system, including vendor libraries, database stuff, tools, you name it. That way you can specify a release with all of the attendant pieces that are needed. What we do during build is produce a huge build archive, then run a command on each machine (web server, process server, db server, whatever) that configures that server individually by pulling what it needs out of the build archive.
2) Automate. There should be NO manual processes. If that means writing some Perl, so be it. If it means buying one tool over another because it is easier to script, so be it. It should be possible to run a single command and produce a working system out of the code repository.
3) Individual builds. Set it up so that anyone can build the entire system on their own development box.
4) Periodic builds. The system should be rebuilt automatically at the very least every night. There are also systems that will do a rebuild whenever a new change is checked in. Broken builds should be a priority to fix.
5) TESTING. Testing should be built for every single module, and it should be run as a part of the build process. Broken tests should be a priority to fix. RFEs should have a testing suite designed at time of creation, by a meeting between the QA people, the developer, and the originator of the RFE. Any defect should have the same treatment; defect fixes must not be released without a unit test that covers the defect.
Here, we have dev, test, QA, and production. The system is broken into modules, each of which is considered an atomic release unit. Stuff can be checked into the repository, but until a module is actually released, when dev gets rebuilt the new stuff isn't used. Release marks a module for inclusion into the build. Local builds can be built against new stuff, however.
When something works in dev and doesn't break the build, it is usually promoted to test. Some clients can see the test system for early testing; generally, however, something must be in QA before it is fully visible.
Every 6 weeks we have a production release cycle. Anything scheduled to be in that release must be in QA by two weeks prior; only defects may be fixed during that lockdown period. No new code.
During the last two weeks, QA is promoted and built in a shadow production environment every night for testing. The goal is for QA and production to match exactly.
Note that we have many sites that share code and databases, as you mentioned that you do.
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #51,654
9/13/02 12:30:46 PM
|
Procedures? Vee don't neet no steenkin' procedures!
The closest to your setup I've worked in was dev->stage->production with dev and production database. 'dev' was on your local machine, pointing to the dev database. 'stage' also pointed to the dev database. dev db changes were copied by hand to the production db.
Where I am now, we have neither source control nor a dev server. Oh, we have a dev db server, but it's standing in for 7 production db servers. The only way to see what happens to queries on large datasets is to run them against the production servers, watch the load, and kill the query if it seems to be taking to long. And all our code has to be in differently-named trees parallel to the main ones. Which means when you move it to production you have to remember to re-write any internal links to point to the right paths.
The things both places have in common is that the guy in charge was/is definitely suffering the curse of the gifted. Brilliant coders who think that everyone should test their own changes before going live with them. "If you test it right, there won't be any problems."* No way to rollback changes except by hand. No way to tell who wrote what except by looking at who last touched the file. Which doesn't show who wrote a particular function, of course, nor when.
The other thing they have in common is that 90% of the code has only ever been seen by the person who wrote it. Every time I bring up code review (at both places) or version control (at the current one), people roll their eyes at me.
* Actual quote.
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #51,667
9/13/02 3:22:42 PM
|
I'm in an extremely large and structured environment....
(yea, corporate life!)
And I wrote a post that almost exactly matched Scott's. So I erased it and wrote this instead:
A couple of (other) thoughts:
The ultimate goal of the process of Build/Release/Promote is Repeatability.
If special steps are required to get a release into QA, they should be documented and followed each time a release goes into QA. Anything that is one-time-only should be viewed with suspicion.
Source Code control (RCS, CVS, PVCS, Clearcase) - absolutely required, in my opinion.
Our code is too large to have everyone compile it. (It would take too much space and time). In these cases, we've modularized our code out and created stubbed out services for others code that we call (but which we won't build and aren't responsible for.)
Testing should be built for every single module. I wish we ran our unit level tests as part of the build process. :-( We do have unit level test automated and we run them every single single time they're a change.
I'm currently working on similar testing scripts for the higher levels of testing. (Perl is wonderful.)
I strongly recommend that users get to see test results as early on in the testing cycle as possible. (We unoffically get them involved at our level 2 testing when process calls for them to be involved in level 5 testing.) (Speaking as a person who's misinterrepted user requirements far too often.)
|
Post #51,668
9/13/02 3:28:42 PM
|
Re: I'm in an extremely large and structured environment....
Ditto on the repeatability... hence the 'automate everything' stance I took. There should be no special steps for any environment. We control everything through local environment initialization files, the values of which can be inserted with macros into just about anything in the build process. As an example, our J2EE server's config directory is created on the fly at build time out of one of the modules in the source code repository. Datasources are built, local machine names, port numbers, admin password, all of that stuff is created by the build configuration script.
We have almost two million lines of code, and still can build the entire system on someone's PC.
Re: test results: the results of every nightly build in each of the environments gets posted to the intranet automatically here.
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #51,656
9/13/02 12:40:03 PM
|
build a usable process, get buy in and enforce it
The following stuff needs to be done and signed off on each step
Needs requirement
High level design document (detailed enough so coder is filling in blanks)
coding with sccs of some kind
coder doesnt release to test until personally satisfied
test (regressive if possible)
qa
production
With parallel efforts and resources the above should help identify the choke points.
thanx,
bill
will work for cash and other incentives [link|http://home.tampabay.rr.com/boxley/Resume.html|skill set]
[link|http://home.tampabay.rr.com/boxley/index.html|boxley's home page]
qui mori didicit servire dedidicit
|
Post #51,660
9/13/02 1:42:12 PM
|
How do you do that in a small shop?
It's hard to define a five-step process when there are only five people in the shop. I'm not just nay-saying, I actually want to know how to do a multi-stage formal process with <10 people.
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #51,669
9/13/02 3:36:35 PM
|
one person might do all 5
it is scalable up and down. Userdefs, write a high level design that shows the methodology to achive the goal. Fill in the blanks, test it with a script or a tool deploy it to production. The idea is to document what you are doing so if someone follows behind you they can see the thought process that got you to where you are.
thanx,
bill
will work for cash and other incentives [link|http://home.tampabay.rr.com/boxley/Resume.html|skill set]
[link|http://home.tampabay.rr.com/boxley/index.html|boxley's home page]
qui mori didicit servire dedidicit
|
Post #51,675
9/13/02 4:00:58 PM
|
Probably
I guess having to have everything done yesterday is going to cause problems no matter what your procedures.
Have I mentioned we are in the process of migrating off of an HP3000-based system? And that the 3000's only allow a 6-digit ordernum? And that we're currently at 934467? And that we're creating orders at the rate of ~500-600/day? How long does that give us to finish rebuilding the system? (A: About 4 months.)
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #51,687
9/13/02 6:18:37 PM
|
well you could ]:->
assuming that the 1st orders were taken a long time ago
setnext ordernum=100 takes about 3 seconds and gives you some time to breath.
thanx,
bill
will work for cash and other incentives [link|http://home.tampabay.rr.com/boxley/Resume.html|skill set]
[link|http://home.tampabay.rr.com/boxley/index.html|boxley's home page]
qui mori didicit servire dedidicit
|
Post #51,885
9/16/02 10:21:56 AM
|
Like we haven't thought of that
Every order we've ever done is still live in both the operations and accounting databases. Have to be able to look up all account history for any customer at any time. So no, we can't recycle early numbers.
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #51,899
9/16/02 11:49:00 AM
|
you mean you couldnt archive 1-5000
into an identical sysytem with a query the grabs order 101 from the archive and makes it 101A on the user screen and in any sets doing totals would differentiate from the "new" 101. I am sure you have all worked on this. What kind of DB is it?
thanx,
bill
will work for cash and other incentives [link|http://home.tampabay.rr.com/boxley/resume/Resume.html|skill set]
qui mori didicit servire dedidicit
|
Post #51,911
9/16/02 1:36:50 PM
|
Too many things would have to change
Every table that deals with order info uses ordernum as the foreign key. Every page that queries any of these tables would have to incorporate logic that says, "if it's less than 5000 then call it 5000a, unless you really mean 5000." And that would only buy us another couple of weeks anyway. Testing alone for the changes would take longer than that.
And that doesn't account for the fact that until we are completely off of the old system everything we do has to migrate to and from that system. Which is maintained by a contractor on an as-needed basis.
Any attempts to re-use low numbers amounts to trying to crash your plane in a valley instead of on a peak because you'll have a little while longer before you hit the ground. This plane is running on fumes, and we have to switch to the new plane before it hits the ground.
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #51,884
9/16/02 10:19:49 AM
|
Re: Build/Release/Promote Process
Thanks for everyone's feedback.
I see lots of things that we're doing wrong.
Namely, our Configuration Manager is our QA Manager. She has somehow wrangled our process so that it's the easiest for her.
*Everything* is manual here. Everytime I change something, I have to keep a list of everything I've touched. I put that into our promotion list. That's given to our QA manager. She then runs a program that compares two directories and returns a list of files that have changed (by the datetime stamp on the file *cringe*).
On top of that, our production system is 100+ servers behind F5 Big IP load balancers (both web and database). Well, we don't have development setup with the same scenario. QA does, though. They have two servers behind a Big IP.
Anyways, lots of good stuff you guys brought up. Lots of good ammo, and points of discussion to bring up in our meetings.
It's very much appreciated!
-Jason
----
My pid is Inigo Montoya. You "killed -9" my parent process. Prepare to vi.
|
Post #52,008
9/17/02 8:37:27 AM
8/21/07 12:46:12 PM
|
What Scott said
Plus:
If you are using Visual Source Safe (and I have reason to suspect you are from your .NET post) ditch it now and move all your stuff to CVS. Even Microsoft doesn't put their code into VSS. Its a toy.
Use as much open source stuff as you can.
You have to have stages like open development, then bug fix only, then showstopper bugfix only. At open development and bugfix it takes two people to check in some code (peer reviews required). At showstopper it takes full team consensus to make a change.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #52,027
9/17/02 11:15:14 AM
|
Re: What Scott said
If you are using Visual Source Safe (and I have reason to suspect you are from your .NET post) ditch it now and move all your stuff to CVS. Even Microsoft doesn't put their code into VSS. Its a toy. CVS, RCS, almost anything would be better than VSS. :=) I've heard decent things about ClearCase's configuration management, but shoot, even clunky Make scripts would be better than nothing. The lawyers would mostly rather be what they are than get out of the way even if the cost was Hammerfall. - Jerry Pournelle
|
Post #52,029
9/17/02 11:17:21 AM
|
I've heard good things about MKS
Morton Kern Systems or whatever it is.
Avoid PVCS too. It's... not good.
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #52,155
9/18/02 3:39:02 AM
|
ClearCase
My experience with ClearCase screams "PASS".
Come to think of it, all of Rational's products leave me feeling this way.
The key problem with ClearCase is that its too clever for its own good. Its implemented as a foreign file system. So it tries to pretend its part of your file system but the illusion isn't quite good enough. You have to use their file manipulation tools. So you can't say rm file, its ccrm file (or something like that). Its easy to forget whether the directory you are working on is a cc thing or a regular thing and pick the wrong tool.
It also entail rather tight OS integration and its upgrade cycle has held back critical OS patches from the dev group.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #52,031
9/17/02 11:26:14 AM
|
Re: What Scott said
Yeah, I've been pushing to get us out of VSS. As you said, I've had nothing but problems with it (versions mysteriously disappear, file corruption, and on and on).
I've been promoting CVS, but our "toy programmers" are spoiled on the IDE integration (argh, silly).
Personally, I'd think that version integrity alone would be enough to warrant switching.
Oh well. Let them get bit a few times, then we'll see how quickly they look to another solution.
----
My pid is Inigo Montoya. You "killed -9" my parent process. Prepare to vi.
|
Post #52,034
9/17/02 11:30:29 AM
|
What Scott said this time
[link|http://www.kryptonians.net/cvs/wincvs_and_ide.html|CVS Plug In for Visual Studio]
Not sure if it works for .NET, but it's a start on finding out.
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #52,035
9/17/02 11:36:19 AM
|
Try TKCVS
Right [link|http://www.twobarleycorns.net/tkcvs.html|here]. There are bunches of other gui frontends for CVS. You also might want to look at [link|http://www.research.att.com/sw/tools/graphviz/|Subversion]: The goal of the Subversion project is to build a version control system that is a compelling replacement for CVS in the open source community. The software is released under an Apache/BSD-style open source license. Much of the new frontend development seems to be moving towards this instead of CVS. ===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #52,037
9/17/02 11:38:40 AM
|
It's not the GUI he needs...
It's the integration of CVS with the dev tools; namely, Visual Studio .NET (or whatever they're calling it these days).
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #52,044
9/17/02 12:14:53 PM
|
Doh!
Of course. I forgot how hard it is to launch two whole applications at once.
Yeah, I know it can be convenient having the source control in the same listing as the dev tools. I'm just so used to keeping dev tools on one desktop, testing on another, and admin on a third that I don't usually think in terms of IDE.
===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #52,057
9/17/02 1:58:34 PM
|
Re: Doh!
Yeah, and most of our senior staff here can handle having more then one application open.
Unfortunately, we've gotten ourselves into a situation where we have alot of legacy developers that have a very, very limited skill set and ability. Over the years, they've attempted to mentor them, but unsuccessfully.
Now, we're dealing with these issues when attempting to migrate away from VBScript to C#, from ASP to .NET, etc... we're growing up, and some of our more junior staff can't.
Very interesting predicament to be in.
Personally, I say trim the fat. But, again, I'm not the CTO (or even a manager) around here. So, meanwhile, my back hurts from having to carry them.
Thanks for the link, Scott! That looks to be a great application to bring up. :)
-Jason
----
My pid is Inigo Montoya. You "killed -9" my parent process. Prepare to vi.
|
Post #52,158
9/18/02 3:48:57 AM
|
I'm in the same fix
This company is 3 years old, has some multi-megabuck accounts, and has a web app written in ColdFusion on windows with vss and all the other lame tools that is (not so) slowly collapsing under its own weight.
My comrade and I are here to:
1) Redesign the database schema (why? I mean, VARCHAR2's make great foreign keys - right?)
2) Rebuild a very complicated module using WebObjects
3) Build a standard widget set in WebObjects
4) Train the developers in basic Java, OO thinking, and modern open source tools
5) Figure out how to move a fair chunk of development to Poland where we also have to do 4
We'll see how the training goes...
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #52,156
9/18/02 3:41:20 AM
|
Subversion
Looks like its going to be really cool when (if?) it grows up. But I've been watching that project for 2 years and I still don't have the confidence in it to trust my code to it. Maybe a couple more years.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #52,038
9/17/02 11:39:04 AM
|
Oh, and one other thing
Oh well. Let them get bit a few times, then we'll see how quickly they look to another solution. Three words: Melissa, ILoveYou, CodeRed. It takes significantly more than "a few times" for Microsoft addicts to look elsewhere. ===
Microsoft offers them the one thing most business people will pay any price for - the ability to say "we had no choice - everyone's doing it that way." -- [link|http://z.iwethey.org/forums/render/content/show?contentid=38978|Andrew Grygus]
|
Post #52,033
9/17/02 11:30:03 AM
|
Visual Source Safe...
...the NotePad of version control software. :-)
|
