Caveat: I've never seen or listened to his show.
He's a big boy and has been around long enough, and been on the radio long enough, and talked to enough people to know that there are certain expressions that upset people.
"You people" can be quite offensive to some people in certain contexts. Would he have a conversation with Pat Robertson and say "You people..."? I don't think so. There's often a racial undercurrent to it, and people pick up on that. People don't say "Boy" to black men any more, either (and there's a reason why it was never said the same way to white men even though in many contexts it's a harmless word).
This stuff really isn't that difficult to figure out. If you use language with one person that's obviously a member of one racial/gender/ethnic group you wouldn't use with another who isn't, then you should think carefully about the implications of that language. Would Imus want his daughter referred to the same way he referred to the Rutgers team? When he figures out why not, he shouldn't get in these troubles as much.
Sharpton has his audience and his reputation. Imus shouldn't have expected an easy time on the show, and by digging himself in deeper it shows that he still Doesn't Get It yet.
Cheers,
Scott.