Hey, about judging others' fate...

Hey everyone, I've got something on my mind that might seem straightforward but I'm seeking some clarity. I know a few folks who sometimes say things like 'you know where you're going' or 'you're heading to Hell' when someone messes up. I even made this kind of 'joke' myself before, and I now realize it's off and I'm working on correcting that. When I try to speak up about it, they accuse me of 'playing the Islam card' or similar. From what I understand, it's clearly wrong and a serious sin to assume where someone will end up in the afterlife, since that judgment belongs solely to Allah (SWT). What are your thoughts on this? How should I respond or handle these situations? Thanks, and although it might feel like a silly question, I'd really value sincere advice. JazakAllah Khair.