How do I not deal with this pest? Anyone have an idea?

I don't think that you are being unreasonable at all. It is perfectly understandable that you should not try to get mixed in their lives. I think you should just tell him that you aren't his friend and never will be and that what goes on between him and your ex is none of your business so stop calling. Sometimes being a little mean and blunt is the only way to stop these things