Can I Make This Visitation Only In Public?

My daughter is almost 2 years old and has lived with me her whole life in New York, her father moved to California and is visiting currently. We have not gone to court for custody but she lives with me and I am the one who takes care of her. Am I allowed to deny him taking her for a day if I am concerned about her safety? I have a text on my phone that he sent me while I was pregnant with her stating that his doctor thought he was a danger to others. I am not trying to deny him the right to see her at all as I am trying to make arrangements in a public location. He is just insisting that I have no right to keep him from taking her. I do plan on going to court about this but he is visiting until the 2nd and wants to see her while he’s here so we won’t be able to go to court on time.