Inconsistent problem ratings?

[chipschap]

Does anyone else feel that, at least at the moment, some difficult problems are rated way too low while some relatively easy ones are rated high?

I wonder if the problem ratings simply haven't settled out; I notice that many problems change rating by dozens of points subsequent to my attempting them.  So it may be a matter of time and usage until things stabilize.

[richard]

Hi chipschap,

This was discussed in another thread yesterday.  At the moment the new problems haven't had time to settle down.  While the total number of problems didn't change much almost half of the problems are essentially new.  In some situations ratings for the new problems were guessed from the old problems they replaced, however this is still very much a guess as after adding alternatives to a problem it may become easier, in some cases MUCH easier.  I will still take a little longer for things to start settling down - probably just long enough for me to have the next problem set shakeup ready to go :-P

Richard.

[stormcrow]

I wonder if it would be possible to get problems into the right ballpark by running them through a chess program set to a certain play quality.  Perhaps the initial ratings could be set to 1250, 1500, or 1750 (assuming 1500 is the median problem rating...)

[tmr]

I wonder if it would be possible to get problems into the right ballpark by running them through a chess program set to a certain play quality.  Perhaps the initial ratings could be set to 1250, 1500, or 1750 (assuming 1500 is the median problem rating...)

Very interesting idea.  This would also allow for a rough coversion between CT ratings and say USCF ratings.

Another item for the to do list richard?

[richard]

I think the chess program set to a play quality is an interesting idea, but it has one practical drawback that I'm not sure is resolvable.  My understanding of how chess programs handicap themselves to play at different ratings is that it will not necessarily reflect the kind of mistakes that humans will make. I think humans miss tactics for different reasons than handicapped computers do, so this process will not necessary generate initial ratings any more accurately than the current simple regression equation does.

Regards,
Richard.

[hexalingual]

Another idea for assigning initial ratings to problems would be to ask some users to volunteer to be laboratory animals.  Volunteers could be assigned a new unrated problem randomly, on average every 5 or 10 (maybe let them choose the number themselves) tries, and they wouldn't be told it's new until after their attempt.  A high problem number might be a giveaway, but maybe that could be fixed by changing the numbering system.  The idea would be that during the experimental phase the problem rating would change, but the volunteers' ratings would be unaffected.  This would prevent ripple effects on other problems and non-volunteers' ratings.  After a certain number of volunteers had tried a problem, an initial rating could be calculated and it could be released for all users.

If not enough people volunteer across the entire rating spectrum, you could introduce something like "site helper points" (one for each new problem tested) that would be shown on their profile and that might help motivate them to show their community spirit.

This would actually be in everyone's interest, since users would like to have an ongoing input of new problems, but would not want solver's ratings to be affected each time.

[stormcrow]

Or Richard could simply ask some trusted helpers to assign ratings.  The problem is assigned the average.

This is only meaningful if the points lost (or gained) due to a insufficiently rated problem matter.

Or, Richard could add an option that allows (paid?) members to refuse problems that have an RD that is too high.

[richard]

I mentioned this on another thread but it is worth repeating here that there was a provisional rating system put in place last week where new problems have a reduced impact on the user's rating until they have had a chance to be seen a few times.  The problem gets it's full rating adjustment each time but the user gets an increasing % of their rating adjustment the more the problem has been seen. At the moment the user gets 20%,40%,60%,80% for problems that have been done 0,1,2,3 times and 100% thereafter.

This has some of the advantages of the lab animals/problem assignment approach but with very little coding changes required from my end. Essentially it enlists everyone as volunteers and the volunteers get a bit of protection from the rating impact of unstable problems.

Regards,
Richard.