I think I understand what you're trying to get at (there are unproductive uses of financial assets), but your analogy doesn't work, IMO.
There are too many people in the world for everyone to grow their own food - enough food for year-round survival. Cities would have to be depopulated. Pol Pot would be happy, but few others. Who would produce the fertilizer, farm equipment, diesel fuel to run the tractors, etc., etc.?
There has to be a way for capital to move in the modern economy. Even if derivatives and the like don't have an obvious direct impact on production of goods and services, they do help spread risk. And risk is the monster hidden in the closet in most economic decisions (always in the background).
Of course, abuses in the system should be removed.
Cheers,
Scott.