Post #150,035
4/4/04 11:39:16 PM
|
Merge/Purge On Linux
MP on Unix VS MP on the IBM mainframe is an apples to
oranges process with apples to apples results.
MF programming tries to limit the amount of memory used.
Most processes try to work through data sequentially
since tape is cheaper than disk and VERY large files
often are tape only. Very large in this case is greater than
44 GB because that is the native mainframe single file
limit. You can dance around and create much larger files,
but it can be painful. Something about max number of extents
VS size of disk packs.
We have software on the MF that makes disk pretend to be
tape. So it it much faster than tape but still forces
certain sequential programming practices.
In our case, we have a process that we run twice a month.
If the MF is empty, it takes about 30 hours, plus time to
move the files to and from, another 12 hours or so.
But the MF is almost never empty. So it usually takes 3 to
5 DAYS.
We bought another MP package that run under NT, Unix, and
Linux. The reason I was so anxious to get the Opterons
working is that this process is VERY cpu and disk intensive.
It also will consume all the memory I can give it for sort
buffer.
To achieve approximately the same results on Linux it takes
8.5 hours to run. With no transfer overhead since the files
started on the Unix side.
I say approximately because it finds some matches the old
software missed, and misses some the old software found.
This is an art, with knobs to adjust depending on the type
of matching you want.
So NOW I can buy 2 dual Opteron IBM 335s to install for real
production now that I've proved it. Wheeeeeeee!
If I want, I can pay a bunch more and parallize some steps
of the process and knock it down to about 4 hours. But it
seems like overkill and not worth paying for it.
I originally tested under NT. It worked but was painful
to control / monitor. Good thing they offered it on Linux
as well.
Hey Lincoln: Feel like telling me I'm full of shit again?
|
Post #150,048
4/5/04 2:57:27 AM
|
" Gleefully participating in the heat death of the Universe!
|
Post #150,052
4/5/04 8:17:10 AM
4/5/04 10:17:33 AM
|
Not Done Yet are you (Now with EIDE and SCSI)
EIDE eServer 325| Base Model Features and Specifications |
Part No | Unit Price | Quantity | Price | | eServer 325 | | Form factor | Rack | | Processor type | Opteron | | Processor speed | 2200 MHz | | Processor cache | 1024 KB | | Hard drive type | EIDE | | Memory | 1024 MB | | Maximum memory | 12288 MB | | Optical drive | 24X CD-ROM |
| 883562X | $3,389.00 | 1 | $3,389.00 | | | | Accessories and Options Selected | Part No | Unit Price | Quantity | Price | | 80GB 7200rpm EIDE ATA/100 HDD | 09N4226 | $189.00 | 1 | $189.00 | | eServer 325 Opteron processor model 248 | 13N0701 | $1,499.00 | 1 | $1,499.00 | | No Internal RAID | 32P9669 | $0.00 | 1 | $0.00 | | 1GB PC2700 CL2.5 ECC DDR SDRAM RDIMM | 73P2267 | $619.00 | 4 | $2,476.00 | | Warranty service upgrade; 3 year onsite repair 24x7x4 hour | 69P9507 | $599.00 | 1 | $599.00 | | |
| | Total: | $8152.00 |
That is without any taxes or shipping. These prices are a bit out of date, as IBM hasn't updated the AMD stuff for the 325 is a bit. SO it'll prolly be cheaper... and you can prolly get the 250GB drives as well.
SCSI eServer 325| Base Model Features and Specifications | Part No | Unit Price | Quantity | Price | | eServer 325 | | Form factor | Rack | | Processor type | Opteron | | Processor speed | 2200 MHz | | Processor cache | 1024 KB | | Hot swap HDD | Yes | | Memory | 1024 MB | | Maximum memory | 12288 MB | | Optical drive | 24X CD-ROM |
| 883561X | $3,399.00 | 1 | $3,399.00 | | | | Accessories and Options Selected | Part No | Unit Price | Quantity | Price | | Integrated Mirroring - 2 HDD's required | 01R1356 | $0.00 | 1 | $0.00 | | eServer 325 Opteron processor model 248 | 13N0701 | $1,499.00 | 1 | $1,499.00 | | IBM 73.4 GB 10K rpm Ultra320 SCSI Hot-Swap HDD | 32P0727 | $439.00 | 2 | $878.00 | | Warranty service upgrade; 3 year onsite repair 24x7x4 hour | 69P9507 | $599.00 | 1 | $599.00 | | 1GB PC2700 CL2.5 ECC DDR SDRAM RDIMM | 73P2267 | $619.00 | 4 | $2,476.00 | | | | | Total: | $8851.00 |
Personally iffn it is only $700, I'd go with the u320 Mirrored Drives meself.
--
[link|mailto:greg@gregfolkert.net|greg],
[link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwethey
"I told my doctor that everyone hates me. He said I was being ridiculous, everyone hasn't met me yet." -Rodney Dangerfield
Edited by folkert
April 5, 2004, 09:27:58 AM EDT
Edited by folkert
April 5, 2004, 09:46:25 AM EDT
Edited by folkert
April 5, 2004, 10:15:18 AM EDT
Edited by folkert
April 5, 2004, 10:17:33 AM EDT
|
Post #150,073
4/5/04 11:15:33 AM
|
The disks are too small
I figure I'll get a 3Ware with an external case.
I need about a TB of usable space for MP tmp files.
And the 3Ware / SATA is FAST!!!
Note: I just discovered that our current IBM 345 with the
optional RAID controller and 6 disks doing a RAID 10 is
HORRIBLE! I don't know if it is the setup or an intrinsic
limitation (yet), but my Bonnie++ tests show it to be about
5 time slower than the nearest machine, and 8 times slower than
my 3Ware SATA setup.
We have 3 of these that were setup at the same time. I've tested
2 of them and they performance the same.
I didn't set it up. The guy who did is gone.
You are going to hate me for this one.
These are the systems I was testing SAP-FB on!!!!!!!
This means I consider the test invalid and I was forced
into NT SQL/Server for no good reason. ARRRGGGGG!!!!
6 months of pain and suffering because I was told to leave
the systems group alone when they set stuff up so I didn't
QC / performance test then. The weren't to be my systems
at all. I had 3 days to install and test SAP-DB then they got
handed to the final users.
Never again.
|
Post #150,081
4/5/04 12:21:52 PM
|
Try the SAP-DB stuff on
The Opterons now. Good a time as any.
--
[link|mailto:greg@gregfolkert.net|greg],
[link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwethey
"I told my doctor that everyone hates me. He said I was being ridiculous, everyone hasn't met me yet." -Rodney Dangerfield
|
Post #150,082
4/5/04 12:28:52 PM
|
RAID 10 == Nice for Data Protection...
But very sucky for performance.
I'd have done something else, but then I always do thing differently.
No biggy either. I knew something was up.
--
[link|mailto:greg@gregfolkert.net|greg],
[link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwethey
"I told my doctor that everyone hates me. He said I was being ridiculous, everyone hasn't met me yet." -Rodney Dangerfield
|
Post #150,092
4/5/04 1:04:27 PM
|
Explain
This should be the BEST of both worlds.
Many stripes across multiple disks while simulatenous mirroring
so reads can be satisfied by alternating disk. No RAID5 calcluation
or parity read/write penalty.
There should be NOTHING better than RAID 0, which means no
protection.
|
Post #150,100
4/5/04 2:05:50 PM
|
RAID 1 can beat RAID 0
At least with some implementations if you plan to read much more often than write, it will.
RAID 0 has faster writes because you stripe the writes between 2 disks. RAID 1 on writing behaviour is pretty close to just having a single disk because it writes to both.
But RAID 1 can choose to stripe reads as well, and can ignore whatever is on the other disk. This lets it dynamically distribute the reading loads between the two disks, giving it better read latency than RAID 0 which has no choice about what disk to access a particular piece of data from.
Whether RAID 1 outperforms RAID 0 in practice depends heavily on your workload and RAID implementation. I know that it can because my boss at a previous job tested both on Linux 2.2 with software RAID and found that RAID 1 was faster, couldn't believe the result, and figured out how it is possible by reading the source-code.
Cheers,
Ben
To deny the indirect purchaser, who in this case is the ultimate purchaser, the right to seek relief from unlawful conduct, would essentially remove the word consumer from the Consumer Protection Act
- [link|http://www.techworld.com/opsys/news/index.cfm?NewsID=1246&Page=1&pagePos=20|Nebraska Supreme Court]
|
Post #150,102
4/5/04 2:32:57 PM
|
Not always so.
With RAID 10, you need to have FULL redundancy. 2 Controllers, 4 Disk (2 on each) for it to be fast. Basically duplicate data paths to make it faster.
Why would you want 1 controller with 4 disk? I am assuming you have an LX IBM ServeRAID card right? Meaning one channel. You have the data going to 2 disks first and then the other 2 second. On a single channel this makes it SLOW SLOW on Writes. No matter if you were on Raid 0 or not. Reads, different story, the controller will just ask the drives with the heads closest to get the data.
BTW, I am not talking about vpaths either. Seperate Bus for seperate instance of data. RAID-5 is different, reads and writes are nominal, but seperate arrays on seperate controllers (maybe just channels) in RAID-50, would beat RAID-10 in both reads and writes.
Of course this all changes when you only have one drive per channel, the 3ware stuff IOW.
Think about this, u320 is actually u80 with auto-matic hardware encoding (u160 uses the same type of blackmagic, but fewer tokens) the clocking is still the same. That is why u320 drives work on u160 and u80 channels. When it comes down to it, the scsi channel gets saturated quite easily in the different configs, unless they are designed specifically for the purpose at hand.
Whereas the Seperate Channel for each drive methodology is only limited by the controllers fabric and main IO capability.
Now, if this was SSA, things would be different, from a stand point of stacking rather than saturation, the SSA controller has two ways to access any device at a time... but still serial is still serial.
--
[link|mailto:greg@gregfolkert.net|greg],
[link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwethey
"I told my doctor that everyone hates me. He said I was being ridiculous, everyone hasn't met me yet." -Rodney Dangerfield
|
Post #150,067
4/5/04 10:39:46 AM
|
Funny you should say something about Disk-as-Tape
[link|http://www.internetnews.com/storage/article.php/3335661|EMC introduces Clariion ATA] and [link|http://www.emc.com/news/press_releases/viewUS.jsp?id=2160|The real PR from EMC] and [link|http://www.emc.com/products/systems/clariion_disk/|the product homepage]
Clearly We could have seen this a while ago. As the prices of ATA Storage GO DOWN DOWN DOWN and the size goes UP UP UP.
Then you can mirror it to an offsite storage too while you are at it.
--
[link|mailto:greg@gregfolkert.net|greg],
[link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwethey
"I told my doctor that everyone hates me. He said I was being ridiculous, everyone hasn't met me yet." -Rodney Dangerfield
|
Post #150,074
4/5/04 11:20:41 AM
|
Could of seen it? DID!
[link|http://z.iwethey.org/forums/render/content/show?contentid=122426|http://z.iwethey.org...?contentid=122426]
|
Post #150,095
4/5/04 1:28:50 PM
|
Speaking of optimization ...
[link|http://www.computerworld.com/departments/opinions/sharktank/0,4885,91868,00.html| Now THAT'S good programming]
===
Implicitly condoning stupidity since 2001.
|
Post #150,161
4/5/04 5:59:23 PM
|
Re: Merge/Purge On Linux
Hey Lincoln: Feel like telling me I'm full of shit again?
I will at every opportunity you give me. lincoln
"Windows XP has so many holes in its security that any reasonable user will conclude it was designed by the same German officer who created the prison compound in "Hogan's Heroes." - Andy Ihnatko, Chicago Sun-Times
[link|mailto:bconnors@ev1.net|contact me]
|
Post #150,186
4/5/04 9:27:41 PM
|
I'm trying!
|
Post #150,286
4/6/04 2:46:10 PM
|
And notice that I didn't here?
Because you started off admitting that you were doing an "apples to apples" versus "apples to oranges" comparison.
I'd still like to see a true "apples to apples" comparison between the boxes: ONLY the same job running ( so the cpu(s) aren't giving cycles to other apps ), break the files up to equivalent sizes ( to overcome the limitations on a single file size on the mainframe - guess IBM shoulda known several years ago that people want to have files over 50 gig in size ), the code as identical as possible, hardware optimized in as similar a configuration, and then drop the starter flag and let 'em go.
Remember: [link|http://z.iwethey.org/forums/render/content/show?contentid=150144|if it weren't for this significant event], we probably wouldn't be here right now.
lincoln
"Windows XP has so many holes in its security that any reasonable user will conclude it was designed by the same German officer who created the prison compound in "Hogan's Heroes." - Andy Ihnatko, Chicago Sun-Times
[link|mailto:bconnors@ev1.net|contact me]
|
Post #150,380
4/7/04 12:01:53 AM
|
Pure apples to apples?
I did it a while back, at least as best I could.
I had the same software running on the MF and Linux.
I seem to recall I was about 5 times faster.
But now we are going to different software. So it
will always be apples to oranges. No point in me
installing the Linux version of the software we
will not buy.
Also, I just realized that about 80% of the time of
the MP process is creating reports. Reports that
we may not have been getting from the previous
vendor. I can probably cut another hour or 2
for reports we weren't getting.
Also, the IO wait time can be helped. I am going
against RAID 10 disk for temp files. I should just
have a pure RAID 0 stripe.
Gotta reconfigure the disk for that one.
Those type of optimizations will NEVER be done on
the mainframe.
RAID 0? Not on the big EMC array!
|
Post #150,644
4/9/04 11:11:07 AM
|
Cut off an hour
9.5 hour process went to 8.5.
Used RAID 0 disks for temp space.
My bonnie++ write rate went from 80MB a second to 300MB per second. But my read read went down a bit, proving alternative reads on mirrored disks work.
I won't keep it like this, too scary.
It is pure temp, but I still don't want to have to recover from a failure.
|
