Struggling with Family Ties After My Parents’ Divorce

My parents are divorced, and the judge said I could choose who to live with. I picked my mom because I’m the only man at home and feel responsible for her. Out of nowhere, my dad started demanding I live with him, even threatening me with the police-though he knows they can’t force it. Before my Umrah trip, he was fine with me going, but a few days before the flight, he said he’d stop me because he has legal authority over me. Desperate to go, I told him, 'In sha’ Allah, I’ll move in after Umrah.' During my time in Makkah and Madinah, I made up my mind to stay with my mom, as I can’t leave her without a man in the house. When I got back and told him, he was furious and punched my chest-thankfully his dad was there to calm things down. Weeks later, for Eid al-Fitr, he told my uncle I couldn’t come to my grandfather’s house. But after Eid prayer, my uncle called me to join, and I went. I was enjoying myself until my dad showed up, and I had to leave. I didn’t speak to him for months until the judge told me to visit before Eid al-Adha. Since then, things are still tense, and I just go once a week as the judge said. I feel torn-am I wrong here? What should I do with a father who’s so stubborn?