You need to be able to input as text, munge the non-conforming addresses and convert to a number for sorting. Not to mention keeping track of street name modifiers such directions: North Main, South Main, 5th Street, 5th Ave. Or even streets with different names for different parts of the street. Or one street name for what is essentially two different roads.
Efficient delivery routes don't always follow street number order, so that's a different attribute. If you need any kind of history, street numbers and even street names do occasionally change.
Alot of details for something so simple as an American style address. And because they are so simple everyone expects them to be handled flawlessly.