If he was just a bit older, I'd suggest that he can take care of himself. But he isn't.

However does he have any friends whose parents you know? You could work out a deal with the other parents where they take your son on Friday, and you'll give them a break with their kid another day of the week. Think of it as a weekly playdate.

Cheers,
Ben