The World Bank has a complex and arcane structure, and I'm not sure of the exact rules. But I don't think he could be voted out over the objections of the US, because certain things require a supermajority to pass and the US has just enough votes to block any supermajority they disagree with.

However, the Board of Directors has a lot of power to approve and reject plans and give direction to the division managers, if they choose to use it. If he really becomes that unpopular and the US refuses to remove him, he could be bypassed by the board.

Jay