Why Do We Separate Ourselves into Sects?

Assalamu alaikum, everyone. I've been thinking... there are so many labels among us-Sunni, Shia, Sufi, Salafi, Deobandi, Barelvi, and more. But in the Qur'an, Allah calls us simply 'Ya Muslimeen' (O Muslims). Nowhere does it tell us to divide into groups. Here's my take: I just want to stick to the Qur'an and authentic Hadith, and avoid adding things that aren't part of our deen. But when someone asks what sect I follow, and I say 'I'm just a Muslim,' they laugh. If I say 'Sunni' to distinguish myself from Shia, they still ask, 'But which type of Sunni?' I know scholars have different understandings, and those differences lead to sects, but then some end up condemning each other. As an ordinary person, do I really need a sect? Why can't we all just be Muslims?