With the ssh public key, people can prove that you're you. That's all. They can use it to give you (and only you) a remote login on their system. To pretend to be you, someone needs your private key. Which is why it is private.

The fact that they are smart enough to not ask for passwords indicates that they have some brains already.

Cheers,
Ben