Should I let my ex husband have visitation with the kids?

Last Friday my ex husband relinquished all rights to our 2 children so that he didn't have to pay CS. He did this after a long custody battle and a custody evaluation said that he was the crazy one even though he made tons of allegations against me. Then this Friday he shows up at the house demanding he gets his regular weekend visitation. I of course said no. He then proceeded to write a very nasty email about how I was stealing the kids from him and that I was destroying their lives by not allowing visitation with them. He then sent it to everyone we know. Does anyone think that I should just allow him to spend every other weekend with the kids when he took his name off their birth certificates and is no longer responsible for them in any way? In his email he wrote that "just because he relinquished his rights doesn't mean that he is not their dad"