Since it causes you problem, stay out of it.

You owe these services to no one, you do it to be helpful, but not if it causes you to be attacked. I suggest just opting out until a new pastor arrives, at that point you can get back into the swim or not as you see fit then.