IWETHEY v. 0.3.0 | TODO
1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User

Welcome to IWETHEY!

New Map/Reduce?
Difficult to tell if that's applicable given your writeup.
Welcome to Rivendell, Mr. Anderson.
New Pretty much, but in a simpler form
Since I have 4 cpus to fill at the moment, I rewrote using Perl threads, and maintain an active thread list to determine if I should release new tasks.

So on this system I hit ~380% CPU (max 400%) with occasional IO bottleneck. My RPS went from 100K to 250K, I'm not nearly there yet, on performance, but it seems I have a direction.

My tasks can be partitioned nicely, the input file can be easily split and then pieces sent to individual threads/and/or processes.

So I guess the next step is to implement the simplest/cheapest cluster available for me to program the next step on.

Any direction on that?
New Something like this maybe

I've not used an Perl clustering software, but there's probably something performant out there for you. Ben may have (probably has) a better idea.
Welcome to Rivendell, Mr. Anderson.
New Life just got easy
My main lookup table went from around 300 million records to a measly 10 million, or maybe even less. And the core specification dropped from a possible 500 comparisons per input record to a max of 10.

I'll get back to you on the final real requirements, but pretty soon I'll be able to code and run it on my phone.
     Goal: Really Fast File Lookup - (crazy) - (11)
         Brute force - (drook) - (1)
             No wetware in the budget. -NT - (crazy)
         Use memcached - (folkert) - (1)
             Not yet - (crazy)
         Map/Reduce? - (malraux) - (3)
             Pretty much, but in a simpler form - (crazy) - (2)
                 Something like this maybe - (malraux) - (1)
                     Life just got easy - (crazy)
         Can you combine methods? - (drook) - (2)
             Build time is once every 3 months - (crazy) - (1)
                 If space is not an issue but speed is - (jake123)

Coo coo ca-choo!
58 ms