Can I stop all visits due to child abuse by child’s father?

There is a custody order in place giving my daughter’s father weekend overnight visits with her. She’s 2 years old. Based on some of her behaviors recently and some information I’ve recently gotten about my ex, I fear he may be sexually abusing her. I obviously don’t have direct proof, but what I do have is very concerning. I don’t want to send my daughter back to her father’s for this weekend visitation and I told him I wasn’t going to. He said if she is not at his house at 5:00 p.m. on Friday he is going to sue me for contempt of court and send the police to get my daughter and give her to him. Can I stop the visits for now while child abuse is investigated? Can he sue me for contempt and get the police involved if I have good reason to fear for my daughter’s safety with him?