When I read the exchange I just see a boss who is willing to cut you whatever slack you need, but also is feeling swamped. I wouldn't read anything negative into that.
However I would take that as an indication that you need to have effective backup. Give some careful thought to who you can bring up to speed on various parts of your job so that if the company was unexpectedly left without you, they would not be paralyzed. This will both leave you more able to take guilt-free vacations, and also be in a position to move onto any interesting projects that happen to come up.
Cheers,
Ben