While it is true that you can be friends with someone and not necessarily want to have sex with them, those other people probably find it hard to imagine that there is zero sexual chemistry of any kind and of any amount between you two.
That is largely because that is how we are raised, not because that is how we are.
Guys in particular (I say guys because, being one, this is the gender I am most familiar with) seem to be raised to constantly have their "femdar" going. We're supposed to notice women more than men (cause if not that'd be totally gay) and we are supposed to be easily able to classify them into groups that are attractive (that is, attractive in the popular sense, not necessarily ones that the guy himself finds attractive; that usually is in its own category).
It might be supposed that this is due entirely to peer pressure, but that isn't exactly true. The media that children consume often reinforces this point. I remember watching Teenage Mutant Ninja Turtles and it seemed like all the turtles had a thing for April. Gargoyles? Goliath and Demona were clearly a pair, with the other male characters obviously not as important (cause if they were important they'd get their own gal). ReBoot? Bob and Dot, Enzo and AndrAIa. Cecil got Rosa, and you TOTALLY know that Edge had a thing for Rydia. Crono and Marle, Donkey Kong and Candy Kong, Cloud and Tifa, Tidus and every female character (but mostly Yuna), Randi and Purimu, Optimus Prime and Elita One, Link and Zelda, and so forth, and so on. All the cool guys get girls, and if a female character shows up, you know she's going to end up with the hero cause that is just what happens, regardless of if they have anything in common, regardless of if she's even said a word yet, so obviously we're supposed to get girls too, by mere fact that they're female. Personality? What, you mean like Smurfette? Um, sure I guess. What are you supposed to do with them? Dunno, the credits always roll before they get to anything interesting. But that is what puberty is for, to fill in the gaps.
Learning to turn off one's femdar is a necessary step in being able to have friends of the opposite gender. Unfortunately, it is not something that all guys learn to do. This creates a barrier between the genders that is detrimental to both. But it is a learned behavior, and so it can be unlearned as well.