Just index the vote (that is, if you have a field called "vote" that records which way the person voted, index that column). Then doing
SELECT COUNT(*) FROM poll WHERE vote = 'never'
will get the count from the index and should be so fast you will never care about improving it.
In any case, I would never bother with a separate count unless you expect more than, say, 100,000 votes per poll.