I only run sshd right now.

As I understand it, they scan for the ftp and web ports occasionally. You could probably just block those scan IPs at the firewall.