New Problem Selection Algorithm

[richard]

I've just made this announcement in the news subforum, but I'm link to it here in case people miss it.

Please post in this thread if you notice any issues with the problem selection system, especially if you suspect a bug is in play, but also I'd appreciate feedback on users' experience of the new selection system and how it impacts their tactics solving here.

As of this posting these changes should be live on the site.

Regards,
Richard.

[pollock]

  Im goner lose my overrated rating *sobs uncontrolably*. But it'll be good in the long run 

[pawnmancer]

I am standad puzzles solver, my rating is around 1870.
I don't know whether it's not because the problem's rating isn't stabilized yet, but today I was served mostly tough puzzles with rating aroun 2200+ (and I got only 5 of 25 right-usually I was able to score at least 75% right).

Is this because of the new selection algoritm or just because problems rating is not stable yet?

[richard]

I think you were probably just unlucky here and had a nasty run of problems over 2000. Overall your average for these 25 problems was only a bit over 100 ratings points above your average.  25 problems isn't realy enough to form an opinion on the distribution.  I'd expect things to even out for you in the longer term. There was certainly some tough problems from #4509 to #9325 !

Regards,
Richard.

[oblonsky]

Hi


My new selection of problems is definitely a lot tougher - for standard option anyway.   This is good as it really does force me to play at my level and beyond.  I'm no longer getting problems 6-800 points below my rating, which, if I get right give me no reward, but if I get them wrong lose me a lot of rating points.

I'm not one of those people who takes no notice of rating points.  To be truthful the rating system helps guide me on how good my tactics are and if I am improving.

Maybe some stats on the kind of motif errors I make may be helpful in the premium version.  I would want to know where my tactical blindspots are.

Congrats again on another genuine improvement which can only benefit me on my chess journey.

best regards
Bernard

[richard]

Hi oblonsky,

I also think that getting a lot less of the "too easy" problems is a fairly big improvement.

The premium features do have a number of canned problem sets included and has one for each of the motif tags and there is a per problem set success rate graph so you can see how you go on the "forks" set compared to the "skewers" for example.  This isn't exactly what you want however as this will only show up if you explicitly do problems in that problem set.  I've added an item to my todo list to add per motif stats across all problems irrespective of problem set they were done in.

Of course these stats are only as good as the motif tags and at the moment there is a relatively small number of tagged problems  and some of them are mistakes!

Regards,
Richard.

[tmr]

Well the change was very obvious and I was wondering if you had changed the selection algorithm as I was served up hard problem after hard problem.  Because of the increased difficulty I only got to 10 problems (with way more incorrect ones than recently) but it seems as if they're much more centered around my current rating.

A couple things I noticed about some of the tougher problems:

1) The line of many doesn't seem to extend out very far, say only a move or two in what really is a many move line.  I don't think this effectively tests whether someone has grasped the situation rather than just taking the most obvious piece (a tactic which seems to be quite prominent in the problem set now by the way).

2) In comparing some of the blitz vs standard ratings the blitz success rate on some of the very tough problems is higher than the standard.  At first blush this doesn't make much sense.  I wonder if this could be due to the limited problems at the high end and thus highly rated folks have memorized these problems having been served them more frequently.  I think this would tend to skew upward the ratings of higher rated players compared to lower rated players.

[richard]

There are two reasons for problem 1), the first is that often the computer line for a problem really doesn't have much more of interest after the end point due to the computer deciding not to take back where a human play might have, i.e. the tactic really has finished as far as the computer was concerned as it has given up the material to avoid further material loss down the line. Sometimes these computer line is not really early pruning but just the sensible thing to do for a human and computer but to see the tactical point requires a lot of analysis. The second reason is that I may have pruned it too early, I have to do pruning to avoid the tactic rolling along long after the point has been made and whilst I go to some effort to make sure I got most of the important action I do err on the side of too early rather than too long when things are unclear as I think it is much more annoying to have a problem where you only have to play the first 2 moves in a 3 move tactic than have played all 3 moves and then have to play a boring pawn move for the last move.  Usually you can tell which of the two sources of early pruning you are looking at by using the computer analysis button, if the computer analysis line makes it easier to see what was going on then its probably my own early pruning, if the computer line doesn't make it much clearer then it is probably the first reason.  Pruning is actually the trickiest part of the generator, the material balance issues that have been discussed in other threads are relatively easy in comparison (they are just of matter picking the appropriate cutoffs), but pruning is a bit  of a black art and much harder than I had expected when I started writing the generator.  Overall after looking though some of the CTS problems (especially the higher rated ones) I think the CT problems much more often reach their conclusion, but I'm sure there is still more room for improvement.  The next improvements are likely to tackle the alternative winning lines that are still getting through (I should point out that there are still a LOT that are getting detected, just not as much as there should be), after that is dealt with I can look back at the early pruning issues.

I'm also planning to store the analysis of more than just the best line but this will also have to wait as I need to urgently deal with some of the winning alternatives that are still appearing i the current generator.


I have noticed the same thing about some blitz versus standard, I'm not really sure why some problems have higher success rates at blitz than standard.  Maybe there is something common with these such that overthinking them is leading to failure on standard (maybe they have an alternative winning line that users are seeing when they look at the problem  bit longer).



Regards,
Richard.
 

[tama]

Before the algorith came out i was at 1600+ strugalling to move up because all the problems i was given were around 1200-1500. In about 3 days my rating jump to 1850 on standard instantaly. And now all the problems is get are around 1700-2500 witch is 10 times better for me because when i solve  a problem i get +2 or +4 before i was getting +0.2 and max +1 lol.

[pollock]

being hi rated and getting a barrage of problems when people solve at 0% percent or sub 10% is not fun. Especially when the puzzles may be flawed by countless winning moves after you have demonstrated the main tactic in position. NOT FUN. Offcourse i could just be annoyed cause am decimating my rating  - pollock

[richard]

Pollock: Unfortunately the new selection code has been somewhat more annoying for higher rated users.  The multiple winning line problems were all hiding up in the high rated problems so now that the selection code chooses a lot more high rated problems for high rated users they have become a lot more visible. I'm hoping that some of these will be forced even higher up the rating scale but unfortunately I can't rely on that as with the multiple winning lines some users will see the "best" line and others will not so some of them will not move much.  The 0% success problems SHOULD push into higher rating territory now that they are being seen a bit more often. None of this is much consolation to losing rating points on a winning line though :-/

I am currently doing a verification run with a new generator which explicitly marks the alternative lines in problems and will allow the UI to not mark a problem as failed just because the user chose an alternative (the UI will notify the user they found a good move but that they should keep looking for another good move).

I'm still probably 1-2 weeks away from completely this run and I still need to make the UI changes required to support the alternatives, I'm hoping to make them before the verification run is complete so that I can release the new problem set as soon as the generator is finished.

Regards,
Richard.

[pollock]

Thanks for answer Richard  . Sometimes can get over not finding the computers best move. But when sit there having calculated a forced win or evaluated a king and pawn endgame to a win and not playing that move cause just "know" its not the computers move isn't good.  - pollock

[tama]

Pollock its not fun solving easy problems that are under 1700 standard. Its just boring and a waist of time for high rated players. I love fully calculating problem even if i dont pick the best move, witch is not as hard as u think it is.

Ill give u a trick, When ever u see a good move LOOK FOR A BETTER ONE! all problems over 1800 standard are generaly like that, and i fined them the funnist to solve.

But i do have 1 or 2 problems that i just cant accept and are way to inhuman. Like problem http://chesstempo.com/chess-problems/2581 the move is Nb3 atacking a rook but the computer does not move it because in the long run after u win his queen side pawns he will be forced to give it up for your passed pawn anyways.

No human in this position would leave the rook hanging, besides Kasporav, and even he will simply resighn.

[dlester]

A lot of the feedback here on problem selection is coming from more advanced players, which obviously makes sense since they are the ones who practice and improve their game.

I am around 1600 standard, which is 300 points higher than my CTS rating.  I haven't really done much of the blitz here.

At the more experienced level, which I am just on the lower cusp of, getting lots of problems over a fairly wide range of difficulty is a good thing (I think).  You get some around your level, you get one over 2000 thrown in every once in a while, and occasionally you get one of the lower ones that is nothing more than capturing a hung piece, but you have to capture that piece and accept there isn't a better move, which of course sometimes there is and you get the problem incorrect.  The difficulty of the 'easy' puzzles has gone up considerably since the rating of the puzzle is now hidden.  You can't just glance at it, know it is an easy recapture, and then do it.  You have to treat every one as if it were a nail-biter.

For my son, who is just starting to play and is six years old, this problem selection method is impossibly cruel.  The skill range at the lower level needs to be much more granular and never range more than 200 points or so from their rating.  When it gets much higher they simply have no prayer of solving the problem.

I don't know your source code so I don't know how complicated it would be, but if it isn't terribly difficult is there any chance the rating scale of presented problems could dramatically narrow the lower your rating is?

[tmr]

At my current rating, 1800ish, the problem selection algorithm is giving me problems +/- 500 to 600 points or so.  And without the problem ratings shown I find the problem set quite challenging.  I find I'm getting about 50% of the problems correct with the new update versus about 75% correct before. 

I think implimenting a selection algorithm that selects from a narrower range of problems the lower rated you are could increase the likelihood of younger players not getting discouraged but I'm not sure the implimentation would be that simple if it is to be done correctly.

For example, to make the rating scale consistent overall it seems as if this implimentation would have to include at least:

1) a variable selection range so that as your rating increases the width of the range of problems gradually widens.  This will help prevent any discontinuities in the ratings.

2) The reward/penalty would have to change to reflect the relativly easier problems set being presented at the lower level.  I haven't thought through this but it seems that the reward/penalty would have to be less the lower rated you are (that is a lower rated user would receive less reward for getting a problem 200 over their rating than a higher rated player gets for a problem 200 points above their rating).  I think this has the problem of getting a user stuck in a rating hole that one could find more difficult to move out of.