The score is calculated in a complicated way.
First, all scores are kept and when a vote is cast for a story, the system looks at all the votes a story has received and then drops the top 5% and the bottom 5% in order to eliminate extreme votes.
Then the average is calculated from the rest of the votes and then the average is passed through a formula that compares the vote to the other story's votes and creates a more balanced score.
The formula was created as a response to idiosyncrasies in voting behaviour that is impossible to avoid.
Few elements ensure that any voting system that simply calculates the votes' average and display them as is will have all its scores bunch up on the top end of the scoring range and that score 'compression' will get worse over time and will in effect render the voting system useless.
Some of the reasons for this 'compression' is that readers tend to vote more for stories they liked. That's because they stop reading the stories they don't like and switch to something else before reaching the end and casting a vote.
Another reason is the general score range effect. If most decent stories score over 4.5, then no reader in clear conscience can give a story anything less that 5 stars if it's readable. Which exasperates the problem.
There are few other reasons like people reading their favourite author and voting for the author to encourage him instead of being critical.
All these reasons get combined over time and get worse and worse.
To combat this problem, the system was designed to take these 'compressed' scores and represent them in a more humanly acceptable way by spreading them out evenly.
After the formula is applied, most high scores are generally lowered, however, the system ensures that the stories' sorting order when sorted by score remains as though the formula didn't exist.