A big part of the problem on both ends is that neither the critics nor the audience is really judging the movie as a movie. Instead, they are expressing their support or disdain for the political viewpoints of Ayn Rand (something many of the critics, I suspect, know only in parody).
Another big part of the problem, and I say this knowing that it is quite a statement indeed, is that the movie is much, much less well written than the book. Yes, Ayn Rand is hardly Jane Austen or even Charles Dickens, but she is a reasonable writer of genre fiction. In contrast, the writing in the movie is just plain awful.
Here is Roger Ebert's review from the Chicago Sun-Times. It is clear that he doesn't like Ayn Rand's political views, but he still gives a pretty good summary of what is not right with this movie. It was made on the cheap, with weak writing, sometimes mediocre acting and a very low budget. Hence all the landscape shots and soap opera interiors.
My own view is that this was an opportunity missed and missed badly. I do not agree with much of Ayn's philosophy - I support altruism the way I define it, I don't care for her atheism, and I prefer my rationalism to her romanticism. But one could have made a really great movie that mocked those in politics and in the broader cultural world who pander to envy and sloth and who denigrate achievement while elevating mediocrity. Ayn Rand mocks such people in her books with icy enthusiasm; in the movie the corresponding characters come off as simply drones playing their parts in the great beltway pork machine, and thus more banal than evil. Also, Ayn's attacks were aimed directly at the people and ideas of her time. One of her later books contains a chapter that pairs similar quotations from JFK and Hitler, for example. There is none of that in the movie. The bad guys are generic and boring. The only one you really hate at the end of the movie is Hank Rearden's ball-busting wife and you hate her not for her philosophy, as Ayn would have wanted, but just for treating her husband so poorly.
So, if you have read the book, it is worth going to see what they do with it. If not, take a pass.