More info please

Post #123,502 by broomberg 10/30/03 6:24:47 PM Reply	More info please This type of work is very common in my environment. What data needs to be shared amongst the processes if any? Do you truly need to split the data, or can you jump to calculated start points? Are you truly CPU intensive, or do you bottlenck on IO? How often does it run? How much memory does a single process take? Does the memory requirement scale with the data, or does it hit a limit of how much it needs and then remain static? How is the core data shared? Do you have a bunch of compute servers sharing an NFS back end or so you sling the data around via interprocess messages or a large shared segment? How many processes can you kick of at once, ie: Is there an inherent limitation to the amount of data granularity or can you keep sub-dividing? If NFS back-end, what OS / hardware? If you move a LOT of data have you closely monitored the system cycles to see where NFS starts to be the bottleneck. Have you considered bypassing NFS by moving the data using rsh pipes, ie: "(rsh SERVER dd if=source_data ibs=4096k obs=8k) \| dd ibs=8K obs=4096k \| local_program". Note: The BS= are dependant on your network and TCP/IP stack. Test for the best mix by doing: time (rsh SERVER dd if=source ibs=4096K obs=(1-64K) \| dd of=/dev/null obs=4096k ibs=(1-64k) If CPU intensive, are you using the latest and greatest intel or or AMD, or are you putting up with crappy old / slow / Sun gear? A few compute server swapouts can make a HUGE difference when bottlenecking on the CPUs. If Sun, and G-bit, are you using Jumbo frames? Doubtful unless you get 3rd party Gbit cards. You can triple your throughput and drop your kernel cycles dramatically if you do. The list goes on and on. Need mroe info!!!
Post #123,631 by broomberg 11/1/03 9:06:49 AM Reply	Geez. 1 day late and nothin!
Post #123,691 by slugbug 11/2/03 11:25:47 AM Reply	1 day late and.... There is no problem with this app. I just wanted to better leverage available processors and memory since this particular server did not really have too much activity. The data does not need to be shared amongst the processes. It seems to meet expected requirements to run concurrent multiple instances and it is not taxing the machine at all. This is a weekly data scrub that executes prior to updating a data warehouse. The process uses an IFS together with an AIX runtime. -Slugbug
Post #123,705 by broomberg 11/2/03 2:08:35 PM Reply	IFS? Would this be the term for a processors on the mainframe that are supposed to be strictly used for non-MF processing which in turn means they can run Linux or AIX but not MVS/OS-390? Licensing issue on my box. Or is this more of a generic term, since I saw you mention 14 free processors, and MFs that large are very rare. And with that many processors free, rarer. In later posts you mention more of a distributed setup, but that is for later. So, hopefully most of my post will apply to that and be of some assistance. Have you (or your Perl programmer) read "Data Munging in Perl"? [link\|http://www.amazon.com/exec/obidos/tg/detail/-/1930110006/qid=1067799999/sr=8-1/ref=sr_8_1/104-5113305-5796718?v=glance&n=507846\|http://www.amazon.co...v=glance&n=507846] If not, go get it. It validated most of what I thought I knew, giving an ego boost, and taught me a bunch that I didn't, paying for itselt with the 1st 3 lines of code I used from it.
Post #123,728 by slugbug 11/2/03 6:05:46 PM Reply	Re: IFS? IFS is the integrated file system on IBM's midrange platforms. It can integrate with Unix/Linux, Netware, and Windows among other things. One can also create user defined file systems, if needed. These servers also have a POSIX environment and an AIX runtime. In addition, one can set up logical partitions (LPARs) to run multiple OSes concurrently (e.g. Linux, Windows, and soon Unix). This particular app currently has available 14 processors (non-distributed app). However, a portion of those processors will be shifting to another LPAR shortly to handle some other processing. Using the multi-instance approach with smaller files (which is how the data arrives anyway) seems to work well with the current number of processors (hardly taxing the machine at all). I'll have to measure the before/after processor shift performance and tweak as needed. I have not yet read Data Munging in Perl, but will grab a copy. Thanks, Slugbug

Welcome to IWETHEY!