Sudden rating leap

[revenant]

I'm wondering what could have caused my recent sudden Standard Rating leap of 100 points or so.

(Cool rating graphs on the user stats pages, by the way.  I like the blue line on the grey background.)

During the leap, I thought to myself "Am I suddenly better?!  I'm not doing anything any different."  I leave problems up on the screen while multitasking.  Sometimes I spend hours (!) staring at a single problem.

Then the graph leveled off and I thought "No, must have been an effect of something else."

drahacikfm's Standard Rating graph shows a recent leap of 200 points and then a similar leveling off.  He's tops at the moment.  The other top-10 standard users don't show such a pattern, though.  However, I seem to remember like a week ago the top-10's were some 2000's, some 2100's, then the tops.  More recently we're all or mostly in the 2100's.

Is it maybe something to do with a recent generator run that adjusts the ratings of certain problems to more realistic values?  Are sudden rises or falls in rating to be expected whenever general parameters in the system are changed?

[drahacikfm]

My rating increased after the June 15 new feature which stopped marking wrong the alternate moves that are clearly winning.  Before June 15, almost every problem I missed was when I played a winning move that was not the "best" move, and I got marked wrong.  Now I get "good move try again" and a chance to get the problem right, instead of losing rating points.

These days I get about one problem wrong for every 10 correct, and that seems to keep my standard rating around 2350, plus or minus 30 points.  Because the average rating of the problems I am served is about 2000.  I get about one point for a correct problem, and lose about 10 points for a failure.

[richard]

Hi Revenant,

I think most of those rating jumps were caused by the June 15th update Drahacik mentioned.  That update had two impacts , it removed a number of problems that ended up having too many alternative lines and it began accepting alternative moves when the second or third best moves were also deemed to be winning. Previously these types of problems had become quite highly rated (as even usually accurate users would understandably get these wrong quite often, having seen a win and snapped it up even though there might have been a win that took slightly more material available). So high rated players had a much lower % success rate than they would have otherwise. After these problems were removed their success rate went up and their rating followed.

This raise in rating is unfortunate from the point of view that it does make it hard to use your rating as a measure of improvement while these types of updates are occurring , for now I usually look more closely at rank and the "better than x %" figure as indications of progress.  The next generator update may cause another slight upward shift as it removes a little more ambiguity from the problem set and fixes two bugs which led to some ambiguous mate problems getting through (valid alternatives were not always getting allowed - although this happened relatively rarely). I'd expect the impact of the coming update to be much less than the June 15th update and I'm hoping that it will be the last update of this type (there will be future improvements, but they are likely to be around better opponent responses in some problems rather than looking at the users available options).

It is also worth noting that "easier" problems do not necessarily cause rating inflation once the problem's ratings have settled down as the glicko system adjusts to this.  In Drahacik's case there is not enough highly rated standard problems to serve up problems at his rating level so he gets problems a bit below his current level.  However as he points out this doesn't allow his rating to keep going up as he gets very few points for the problems he gets right and loses a good deal for problems he gets wrong. I'm planning to use this nice property of glicko to allow premium users to tweak the problem selection algorithm to optionally choose "harder" or "easier" problems, because of the way glicko works this will still provide a fair rating system but allow users a bit more flexibility on the level of problem difficulty they face.

Regards,
Richard.

[revenant]

Thanks guys, that explains it.  An additional factor I thought of since posting is that I was away for 2 weeks (June 25-July 11) and the 100-point leap occurred after I got back and started solving problems again.  Whereas the other users have been climbing too but their graphs show a steady incline because they were solving every day.   I guess all together, our results track whatever the average problem rating is, and we all take a certain amount of "lag time" to catch up with a fluctuation.

I wonder what the graphs would show if you could put some sort of time markers on the X-axis.  In this idea my 2-week vacation would show up as a horizontal gap in the graph.  Might also be interesting to somehow indicate the user's RD at the time they solved any given problem.  Perhaps a light-yellow "river" around the blue rating line would do it.  Low on the "to do" list, obviously, but worth considering.

[richard]

Hi revenant,

Making the x-axis a fixed time unit would be a good option.  The current presentation which squashes up all the empty periods is ok for some purposes but does make it a bit tricky to see at what time-points in your rating history changes occurred.  The current presentation was simply the easiest to write a database query for at the time.  I've added a strict time based presentation alternative to the todo list.

Regards,
Richard.