I thought Final Fantasy II was a pretty good game, though they did go a little bit over-the-top in killing off characters in it.
I think that FFV got a bad rap, because by that point the story of the crystals had become old news, so to speak and they really weren't coming up with too many new ideas to incorporate them in. Though while that is a downside, the job system is spectacular, the fights are awesome and the storyline, besides the crystals, is pretty good.
You do notice that after FFV, they branched off of the crystals for the most part, except for giving them a slight nod in FFVI by making the Espers crystalline form resemble that of the Crystals. They didn't pick them back up fully though until FF9, and they did a really good job on that game. I enjoyed it completely. Then they left them alone again after that until FFXII and did a nice job of incorporating them into that while also tying FFTA more firmly into the story.
But as to why people generally dislike FFV: In most cases, it's the fact that the storyline is not too unique, and some people just didn't like the job system. Some even complain that it's too difficult. Personally, I have enjoyed every final fantasy game I've ever played. While I haven't enjoyed EVERY part of every game, all of them are damn good games comparatively. Keep in mind that I am not saying they're the best rpg series, just that they're really good, nonetheless. You can argue this, but seriously, 500 million people can't be wrong.