Often, people who are misfits as kids eventually find other misfits to be friends with: suddenly, because they've found a group that accepts them and makes them feel comfortable; they're not misfits anymore. In the early nineteenth century, the word misfit meant "a garment that fits badly," but by the end of the 1800s it had come to primarily mean "outcast" or "oddball." The fit in both cases is wrong, thus the prefix mis-, or "wrong."