IWETHEY v. 0.3.0 | TODO
1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User
IWETHEY Banner

Welcome to IWETHEY!

New Map/Reduce?
Difficult to tell if that's applicable given your writeup.
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New Pretty much, but in a simpler form
Since I have 4 cpus to fill at the moment, I rewrote using Perl threads, and maintain an active thread list to determine if I should release new tasks.

So on this system I hit ~380% CPU (max 400%) with occasional IO bottleneck. My RPS went from 100K to 250K, I'm not nearly there yet, on performance, but it seems I have a direction.

My tasks can be partitioned nicely, the input file can be easily split and then pieces sent to individual threads/and/or processes.

So I guess the next step is to implement the simplest/cheapest cluster available for me to program the next step on.

Any direction on that?
New Something like this maybe
http://bonsai.ims.u-...ster/software.htm

I've not used an Perl clustering software, but there's probably something performant out there for you. Ben may have (probably has) a better idea.
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New Life just got easy
My main lookup table went from around 300 million records to a measly 10 million, or maybe even less. And the core specification dropped from a possible 500 comparisons per input record to a max of 10.

I'll get back to you on the final real requirements, but pretty soon I'll be able to code and run it on my phone.
     Goal: Really Fast File Lookup - (crazy) - (11)
         Brute force - (drook) - (1)
             No wetware in the budget. -NT - (crazy)
         Use memcached - (folkert) - (1)
             Not yet - (crazy)
         Map/Reduce? - (malraux) - (3)
             Pretty much, but in a simpler form - (crazy) - (2)
                 Something like this maybe - (malraux) - (1)
                     Life just got easy - (crazy)
         Can you combine methods? - (drook) - (2)
             Build time is once every 3 months - (crazy) - (1)
                 If space is not an issue but speed is - (jake123)

I don't usually win conversations this decisively.
74 ms