The rule is that if you think you can delete it, you should make sure that it is properly backed up to somewhere else (if need be, just by renaming it) before removing the original.
Many unhappy people have followed your original rule only to find out some time later that they couldn't actually delete what they thought they could...
Cheers,
Ben