How much oil does agriculture require? You have to run tractors and farm equipment. Fertilizer problem can be solved by crop rotation and cultivation of nitrogen fixers. Farms I think can become totally self sustaining by dedicating 1/n farms to production of bio-diesel specifically for ag use.
This implies farming gets less efficient but will still beat 40 acres and a mule per family levels.
Transportation is the big worry. I think we will no longer be able to transport crops long distances like we do now. This goes to the point about suburban sprawl being the clinging vine that chokes the cities.