multithreaded Perl?

Post #123,218 by slugbug 10/29/03 9:51:00 AM Reply	multithreaded Perl? I've got a Perl program that I'm using to parse some rather large files (e.g. 25GB). It works fine except it is single threaded. I'm having a tough time finding docs on multithreading and Perl. Any documentation pointers from the Perl gurus herein? TIA, Slugbug
Post #123,225 by deSitter 10/29/03 10:12:22 AM Reply	What the?? What is everyone doing with 25GB files?? What could possibly be 25GB? Can't you make 10 2.5GB files and run 10 instances? -drl
Post #123,227 by slugbug 10/29/03 10:34:11 AM Reply	multi-instance... Actually, yes, I have broken the data down into smaller sizes and am running multiple instances now. However, I'd like to better take advantage of available processors (14 of them) and memory (42GB). This is a repeatable process to parse some data prior to updating a data warehouse, fyi...
Post #123,232 by ben_tilly 10/29/03 10:43:47 AM Reply	If this is a Unixy system, definitely go with fork() The bulk of the interpreter will remain shared between children, saving memory. I don't know the threading implementation well enough to know if that holds there. Also the fact that these are separate processes is a big hint to the OS that it doesn't even have to think about concurrency issues. Some versions of *nix actively try to make it more likely that threads will stay on one processor to minimize locking/unlocking costs. Processes are more likely to be migrated freely to balance the load. Plus fork() has been around for longer, and so has better guarantees of stability. Cheers, Ben "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,247 by slugbug 10/29/03 11:33:16 AM Reply	Yes... ....in this particular runtime, there is a correlation between threads and processors. Not overly familiar with fork(), but will read...thanks again for the help. -Slugbug If you think you can, or you think you can't, you're right. -Henry Ford
Post #123,238 by deSitter 10/29/03 11:00:52 AM Reply	Re: multi-instance... !!! What are you working on, the Forbin project?? 14 processors! Holy Cran! Must be an E10000 with a lot of cooling fans! -drl
Post #123,240 by admin 10/29/03 11:07:31 AM Reply	Er... We've got a 6000 with 24 processors here, IIRC... :-) Regards, -scott anderson "Welcome to Rivendell, Mr. Anderson..."
Post #123,242 by deSitter 10/29/03 11:14:20 AM Reply	DAMMIT! I WANT TOYS! -drl
Post #123,243 by slugbug 10/29/03 11:20:40 AM Reply	toys, toys, toys... ...we have 60+ systems and most of these are 10-36 processor range with tons of disk and memory. Fun to partition some of these and run Linux (except for upgrade issues); these boxes can run multiple OSes concurrently and can shift memory, processor, and disk based on scheduled settings. By early next year, these same boxes will be able to shift memory, processor, and disk dynamically based on workload at any given time. Reminds me.....its about time to get some more personal toys for my lab :-) -Slugbug
Post #123,245 by deSitter 10/29/03 11:25:50 AM Reply	Re: toys, toys, toys... Need any help? Have truck and bike will travel :) -drl
Post #123,250 by admin 10/29/03 11:34:11 AM Reply	Re: toys, toys, toys... (new thread) Created as new thread #123249 titled [link\|/forums/render/content/show?contentid=123249\|Re: toys, toys, toys...] Regards, -scott anderson "Welcome to Rivendell, Mr. Anderson..."
Post #123,294 by jb4 10/29/03 1:59:53 PM Reply	Re: What the?? What is everyone doing with 25GB files?? What could possibly be 25GB? Lemme guess: XML? jb4 "There are two ways for you to have lower Prescription-drug costs. One is you could hire Rush Limbaugh's housekeeper ... or you can elect me President." John Kerry
Post #123,296 by deSitter 10/29/03 2:06:11 PM Reply	That's 4GB of angle brackets alone! -drl
Post #123,302 by admin 10/29/03 2:34:36 PM Reply	Don't laugh... We just had an application for data transfer in which the space for the commas and quotes for a CSV format was a significant factor... Regards, -scott anderson "Welcome to Rivendell, Mr. Anderson..."
Post #123,331 by jb4 10/29/03 5:11:53 PM Reply	Yes, the developers of LISP would be proud... jb4 "There are two ways for you to have lower Prescription-drug costs. One is you could hire Rush Limbaugh's housekeeper ... or you can elect me President." John Kerry
Post #123,303 by slugbug 10/29/03 2:38:09 PM Reply	nope.... ....not XML :-)
Post #123,229 by ben_tilly 10/29/03 10:36:24 AM Reply	Possible, but I wouldn't Perl's threading documentation is available with [link\|http://www.perldoc.com/perl5.8.0/pod/perlthrtut.html\|perldoc perlthrtut]. [link\|http://www.perldoc.com/perl5.8.0/lib/threads.html\|man threads] also is helpful, as is [link\|http://www.perldoc.com/perl5.8.0/lib/threads/shared.html\|perldoc threads::shared]. (Perl's core documentation is usable by default from man or perldoc.) The long and short of it is that to do threading you need to have Perl compiled with threading support. Perl's support for multi-threading is fairly recent, and lots of modules (including some core modules) are not threadsafe. After a lot of trouble trying to make the interpreter threadsafe (retrofitting threading on a large, non-threadsafe program is generally a bad idea...), Perl now does it by having one interpreter per thread. This means that spawning a thread means creating a new interpreter, and sharing data between threads is fairly heavy as well. My suggestion instead is to use [link\|http://search.cpan.org/~dlux/Parallel-ForkManager-0.7.5/ForkManager.pm\|Parallel::ForkManager] to divide the work between processes. If the files vary widely in size, then arrange to process them largest to smallest (decreases the amount of time you spend with just one running). (OK, admission time. If you do this on Windows with ActiveState's Perl distribution, the "fork"ing will be done for you using the above threading technique.) Cheers, Ben "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,244 by slugbug 10/29/03 11:22:48 AM Reply	Ben, thank you.... I'm still leaning toward spawning multiple instances, but lemme read from the links you posted and then decide. I'm definitely not a Perl goddess (yet)... -Slugbug
Post #123,300 by ben_tilly 10/29/03 2:27:10 PM Reply	No problem In other languages or environments, multi-threading would be the right solution. But Perl's historical philosophy is that the OS offers fork(), which gets the job done with no work for Perl. I would still call that the preferred way to work on Perl when the OS supports it. Besides which, forking architecturally scales better. I can just see you asking WTF I am talking about, so let me explain. Multi-threading is a much lighter-weight approach than forking. However multi-threading results in a far larger likelyhood of different threads needing simultaneous access to the same resources (eg memory). As you scale up, this quickly becomes a source of bottlenecks. That is why schedulers go out of their way to keep threads on the same CPU (or CPUs), which means that even with SMP, multi-threading won't distribute work as evenly as forking. It is an issue that becomes extremely relevant as soon as you try to distribute work with a clustering technology like [link\|http://www.mosix.org/\|MOSIX]. (They don't even try to handle multi-threaded applications.) And as [link\|http://lse.sourceforge.net/numa/\|NUMA] becomes more and more prevelant in high-end machines (which it will because NUMA scales far, far better than SMP), the benefits of a forking architecture quickly become obvious. A metaphor that I like is that a scalable solution is like an 18-wheeler, while the non-scalable one is often like a Porche. For getting around town, the non-scalable one is often easier and more fun. But if you need to get a few tons from one coast to the other... So yes, in an environment that supports both, multi-threading is faster. Forking can scale better. Choose whatever is appropriate. Cheers, Ben PS Disclaimer: This was posted by a weenie with a lot of theory and with current computation problems which scale just fine using ye basic load balancer... "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,305 by deSitter 10/29/03 2:42:45 PM Reply	Re: No problem This was an issue years ago - but with 1G of memory and 3GHz processors commonplace on workstations, heavy forking is a moot issue. I can see "last swap" competing with "uptime" for bragging rights :) You can also look at Windows NT and its bastard children - multi-threaded from day 1, butt-slow at real world problems. -drl
Post #123,334 by ben_tilly 10/29/03 5:13:13 PM Reply	The more you can do, the more they ask you to do The big toys that slugbug and Scott play with have a reason for existing. People buy them, take them to their limits, and wish that they had more. As for multi-threading being slow, talk to the good folks at Sun about that. Just because Linux 2.0-2.4 sucked at multi-threading, and just because NT has its problems, doesn't mean that multi-threading can't work well. You just have to do it well. (Linux 2.6 also should do it well. Along with scalability improvements, Sun is going to find itself outperformed farther up the scale...) Cheers, Ben "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,340 by admin 10/29/03 5:19:55 PM Reply	Absolutely. We're constantly maxing out that big machine. Incidentally, we don't exactly play with it... that's the production database machine. ;-) With respect to multi-threading, again absolutely. The threading in Linux is currently atrocious. The scheduling is bad (although there are patches) and the signal handling is completely broken. Tests I've done had Linux threading slower than Win2K by 20%, albeit without patching. Regards, -scott anderson "Welcome to Rivendell, Mr. Anderson..."
Post #123,306 by slugbug 10/29/03 2:43:11 PM Reply	this is an... ...eighteen wheeler scenario. Sounds like forking is the answer. Does this mean I will soon become a forking goddess? <smirk> Thanks for the help. Have folks jumping up and down on this one. -Slugbug
Post #123,335 by jb4 10/29/03 5:13:48 PM Reply	So, Perl assumes POSIX compliance in the OS, eh? Rather presumptious of it, doncha think? jb4 "There are two ways for you to have lower Prescription-drug costs. One is you could hire Rush Limbaugh's housekeeper ... or you can elect me President." John Kerry
Post #123,347 by ben_tilly 10/29/03 5:53:22 PM Reply	No See [link\|http://www.perldoc.com/perl5.8.0/pod/perlfork.html\|perlfork] for how Perl emulates forking on operating systems that don't support it. For a broader introduction to portability and Perl, read [link\|http://www.perldoc.com/perl5.8.0/pod/perlport.html\|perlport]. As usual, Perl's philosophy is to give you enough rope to hang yourself. Perl came from POSIX systems, it offers interfaces to POSIX-specific capabilities. But nobody is saying that you have to use them. And where they don't exist, Perl does its level best to do something reasonable instead. The result is that while it is trivial to write unportable Perl code, nearly all Perl code that you encounter is portable by default to all supported platforms. (Were it not so, then CPAN would never have succeeded.) Supported platforms may be found by scanning the PLATFORMS section of perlport above. Cheers, Ben "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,463 by jb4 10/30/03 2:06:32 PM Reply	Interesting, but looks like a bit of a kluge The fact that the emulator spawns threads instead of processes means that the desired protection between the fork-ed processes from each other does not exist. I am no Perl guru (hell, I barely know anything about it), but from a design standpoint, this means that any Perl script must be designed to the target platform...and I thought that was counter to what Perl was supposed to be about. (Yeah, Perl does not claim to be Java...but still....) jb4 "There are two ways for you to have lower Prescription-drug costs. One is you could hire Rush Limbaugh's housekeeper ... or you can elect me President." John Kerry
Post #123,472 by ben_tilly 10/30/03 2:48:22 PM Reply	It is a kludge It is meant as a workaround for features that the OS doesn't have. As for protection, the fact that each thread has its own interpreter, and interpreters do not by default share memory, means that most of the protection that you want is there. (The page that I pointed you had describes many of the things that don't get protected.) For the record, that multi-threading model was not intended to be used as a fork() emulation. Rather it was meant to make it possible for Perl to integrate well into IIS, which is multi-threaded. But given that they had the capability, the fork() emulation made sense. On Perl's design. Perl was not intended to be "about" anything. (OK, making easy things easy and difficult things possible...) It was designed as a general-purpose reporting tool for use on Unix, and kind of evolved. In the end Perl turned out to be highly portable, but that wasn't a design goal. Cheers, Ben "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,473 by deSitter 10/30/03 2:50:47 PM Reply	Re: It is a kludge There are (sigh, were) similar issue with getting things that were born under UNIX to live in the VMS world. -drl
Post #123,486 by ben_tilly 10/30/03 4:22:25 PM Reply	One of those things was Perl ;-) "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,489 by deSitter 10/30/03 4:36:04 PM Reply	Don't I know! I once downloaded and compiled it on a VAX which had Perl v.low on it :) -drl
Post #123,462 by Simon_Jester 10/30/03 2:06:03 PM Reply	Query: I've played a (small) amount with the multithreading That is why schedulers go out of their way to keep threads on the same CPU (or CPUs), which means that even with SMP, multi-threading won't distribute work as evenly as forking. on the bigger Sun boxes, and I don't remember having much difficulty utilizing the other processors on those boxes when I doing the multithreading. Memory issues were a problem. We utilized Sun's multi-threaded malloc, which helped. But, for our application, most of the memory accesses were tied to a particular thread - so our memory bottlenecks weren't too bad.
Post #123,469 by ben_tilly 10/30/03 2:39:39 PM Reply	That scheduler optimization is pretty recent IIRC "good ideas and bad code build communities, the other three combinations do not" - [link\|http://archives.real-time.com/pipermail/cocoon-devel/2000-October/003023.html\|Stefano Mazzocchi]
Post #123,502 by broomberg 10/30/03 6:24:47 PM Reply	More info please This type of work is very common in my environment. What data needs to be shared amongst the processes if any? Do you truly need to split the data, or can you jump to calculated start points? Are you truly CPU intensive, or do you bottlenck on IO? How often does it run? How much memory does a single process take? Does the memory requirement scale with the data, or does it hit a limit of how much it needs and then remain static? How is the core data shared? Do you have a bunch of compute servers sharing an NFS back end or so you sling the data around via interprocess messages or a large shared segment? How many processes can you kick of at once, ie: Is there an inherent limitation to the amount of data granularity or can you keep sub-dividing? If NFS back-end, what OS / hardware? If you move a LOT of data have you closely monitored the system cycles to see where NFS starts to be the bottleneck. Have you considered bypassing NFS by moving the data using rsh pipes, ie: "(rsh SERVER dd if=source_data ibs=4096k obs=8k) \| dd ibs=8K obs=4096k \| local_program". Note: The BS= are dependant on your network and TCP/IP stack. Test for the best mix by doing: time (rsh SERVER dd if=source ibs=4096K obs=(1-64K) \| dd of=/dev/null obs=4096k ibs=(1-64k) If CPU intensive, are you using the latest and greatest intel or or AMD, or are you putting up with crappy old / slow / Sun gear? A few compute server swapouts can make a HUGE difference when bottlenecking on the CPUs. If Sun, and G-bit, are you using Jumbo frames? Doubtful unless you get 3rd party Gbit cards. You can triple your throughput and drop your kernel cycles dramatically if you do. The list goes on and on. Need mroe info!!!
Post #123,631 by broomberg 11/1/03 9:06:49 AM Reply	Geez. 1 day late and nothin!
Post #123,691 by slugbug 11/2/03 11:25:47 AM Reply	1 day late and.... There is no problem with this app. I just wanted to better leverage available processors and memory since this particular server did not really have too much activity. The data does not need to be shared amongst the processes. It seems to meet expected requirements to run concurrent multiple instances and it is not taxing the machine at all. This is a weekly data scrub that executes prior to updating a data warehouse. The process uses an IFS together with an AIX runtime. -Slugbug
Post #123,705 by broomberg 11/2/03 2:08:35 PM Reply	IFS? Would this be the term for a processors on the mainframe that are supposed to be strictly used for non-MF processing which in turn means they can run Linux or AIX but not MVS/OS-390? Licensing issue on my box. Or is this more of a generic term, since I saw you mention 14 free processors, and MFs that large are very rare. And with that many processors free, rarer. In later posts you mention more of a distributed setup, but that is for later. So, hopefully most of my post will apply to that and be of some assistance. Have you (or your Perl programmer) read "Data Munging in Perl"? [link\|http://www.amazon.com/exec/obidos/tg/detail/-/1930110006/qid=1067799999/sr=8-1/ref=sr_8_1/104-5113305-5796718?v=glance&n=507846\|http://www.amazon.co...v=glance&n=507846] If not, go get it. It validated most of what I thought I knew, giving an ego boost, and taught me a bunch that I didn't, paying for itselt with the 1st 3 lines of code I used from it.
Post #123,728 by slugbug 11/2/03 6:05:46 PM Reply	Re: IFS? IFS is the integrated file system on IBM's midrange platforms. It can integrate with Unix/Linux, Netware, and Windows among other things. One can also create user defined file systems, if needed. These servers also have a POSIX environment and an AIX runtime. In addition, one can set up logical partitions (LPARs) to run multiple OSes concurrently (e.g. Linux, Windows, and soon Unix). This particular app currently has available 14 processors (non-distributed app). However, a portion of those processors will be shifting to another LPAR shortly to handle some other processing. Using the multi-instance approach with smaller files (which is how the data arrives anyway) seems to work well with the current number of processors (hardly taxing the machine at all). I'll have to measure the before/after processor shift performance and tweak as needed. I have not yet read Data Munging in Perl, but will grab a copy. Thanks, Slugbug

Welcome to IWETHEY!