Map/Reduce?
Difficult to tell if that's applicable given your writeup.
Regards,
-scott Welcome to Rivendell, Mr. Anderson. |
|
Pretty much, but in a simpler form
Since I have 4 cpus to fill at the moment, I rewrote using Perl threads, and maintain an active thread list to determine if I should release new tasks.
So on this system I hit ~380% CPU (max 400%) with occasional IO bottleneck. My RPS went from 100K to 250K, I'm not nearly there yet, on performance, but it seems I have a direction. My tasks can be partitioned nicely, the input file can be easily split and then pieces sent to individual threads/and/or processes. So I guess the next step is to implement the simplest/cheapest cluster available for me to program the next step on. Any direction on that? |
|
Something like this maybe
http://bonsai.ims.u-...ster/software.htm
I've not used an Perl clustering software, but there's probably something performant out there for you. Ben may have (probably has) a better idea. Regards,
-scott Welcome to Rivendell, Mr. Anderson. |
|
Life just got easy
My main lookup table went from around 300 million records to a measly 10 million, or maybe even less. And the core specification dropped from a possible 500 comparisons per input record to a max of 10.
I'll get back to you on the final real requirements, but pretty soon I'll be able to code and run it on my phone. |