Ex-father In Law Wants "gifted" Down Payment For House Back Now That We're Divorced?

4 years ago, my now ex-wife and I were married. Her father gave us a "gift" of approximately $48,0000 for a down payment for our house and has never once made mention of being paid back.

Late September of last year - found wife was cheating, I filed for divorce, and 2 weeks ago, the divorce was completed and I have since paid my ex most of her half of the equity in the home as agreed in the divorce.

Between the time of the filing and now, my ex-Fil brought up that he wants to be paid back his money citing that he has no money left and has all this dental work he needs to get done and medication and he has said over the phone to me and through text to my ex-wife that if we hadn't gotten divorced, he would never have asked for it back.

I have said to him I think it is right for us to pay him back, without his help we would've never afforded the house, but it can't happen all at once. I told him over the phone I will pay him back my half over time with 10k now and the rest over 2-3 years. My ex has the other half with me having paid her back so he would have to pursue her for that.

My question is, if I pay him back without a formal written contract, does that open me up to the possibility of him, in the future pursing legal action and me being responsible for paying him back the full 48k?