with the year and like 1/1/2004 and then ignored the day and month. When doing a query you can do a Year([DateColumn]) where DateColumn is the name of the field you put the date into and then sort by that.
As others suggested, you could have created a number field of four digits and then sorted by that. Would have been a lot easier for you.
Text fields take longer to sort by, numbers are faster to sort by. If you get a lot of records the database is going to bog down in speed when sorting by that text field. If it is a small database, you may not have to worry about it.