and boom - suddenly I'm connecting at up to 40k. Why? I'm dialing a major ISP so I'm sure the infrastructure is the best.

Pfffffft!

Major ISP's often buy lines from other carriers in some areas, and in any case it's always subject to line quality from your local switch to your home. I wasn't able to get DSL until about a year ago, and I changed nothing at my computer. (I haven't subscribed yet, though I'm going to want to try it. Haven't sorted out local reports of DSL vs. cable.)