Sure, when everyone has their own cow some of them aren't going to keep the barn clean. Just like some people don't keep their kitchen clean. Have you ever watched "Billy the Exterminator"? Some people are fucking slobs. I wouldn't eat anything from their houses.
But modern, non-factory farming is less prone to E. coli and other pathogens. Most of the FDA food handling regulations are designed to protect against the problems of factory farming. If you aren't doing that kind of farming, you're protecting against the wrong thing.
There's plenty of evidence that unpasteurized milk is beneficial. So we're destroying any possibility of that benefit, in order to protect against a problem that doesn't exist on the small farms that keep getting raided.