It's interesting how people who don't read a particular genre of literature tend to lump it with other genres, willy-nilly. I've been known to do it myself, in regards to some non-fiction types.
What's tricky about SF and Fantasy is that some authors deliberately try to mix the two, and the genres have a certain fanbase in common. (Of course, there are people who only read SF and not fantasy, or vice versa)