Is This Considered Refusal Of Child Access?

Me and the father of my child cannot seem to reach an agreement about our parenting plan. He still lives with his parents We used to alternate days but that stopped when they put our child at risk and kept it a secret, this led to our child being admitted to hospital and put on an oxygen tank. And that’s when they admitted what happened to her.

After her recovery I communicated with him that he’s still allowed full access with the exception that she’s not allowed to go to that house again as I do not trust them and I feel if something were to happen to her again, they would keep it a secret once again. I even suggested we can find a public place, preferably kid friendly places, such as parks, game arcades, etc where we can meet and he spends time with his daughter while I give them space and shop around or whatever. And when he’s done, we can go our separate ways.

He has now threatened to take me to court for refusing access to his child. How do I go about solving this? Can he possibly win the case ?