If they were built to celebrate a culture, they may offend our current sensibilities but I don't think they should be casually destroyed.

If they were built as a warning to a weaker segment of the population that just because we lost the war doesn't mean you aren't still a nigger, let's tear that thing down.