Can I Leave My Child With Family For A Vacation Next Month?

We do not yet have custody order. I am planning on filing on Tuesday but we had an issue with my ex’s family and they can no longer look after our child. I am traveling out of the country next month for an important event and will cut my trip short. My ex can not watch our child because he works too much, I’ve asked multiple times and will not sign for a passport for our child so I can take hom. Is it possible to have a family member babysit for 4 or 5 days? Would it affect custody? Would it be possible to take him with me if no one in his side can watch him?