Absent Father Is Threatening To Fight For Primary Custody. Does He Have Any Chance?

My 9 year old son’s father and I split when our son was less than 1 year old. His father immediately moved across the country, and has been largely absent since. We have joint legal custody, and I have primary physical custody with final decision-making authority. He has no set visitation- it is all up to my discretion. He sees her usually once a year, during a visit coordinated by myself and his mother, who lives nearby to him. My son stays with his mother when he visits the area. My son’s father video calls him every few weeks, sporadically. He does pay child support.

Recently, my son’s father has told me that he thinks it’s in our son’s best interest to move across the country and be with him, because the school quality is better there than where we currently live. He hinted at wanting to lower child support- which is why I think he suggested he move there, so he wouldn’t pay child support anymore. He also expressed concern over my ability to care for our son due to my “health”- I do have chronic health conditions but have been able to care for my son his entire life regardless.

He does have the funds to take me to court over this. I’m wondering if a court would take his request seriously? Should I be worrying about this or is it just his delusion that this is realistic?