Pruning algorithm and one-move problems

[drahacikfm]

Too many one-move problems.  I think the pruning algorithm, which decides when a problem ends, could be easily improved.  Just continue the problem for as long as the usual generator parameters are satisfied by each position.  Those parameters are +1.0 evaluation difference between first and second-best moves, and not more than 3 moves with evaluation +1.75 or better.

A typical one-move problem is 40777.  After 1...Rxd5+, the problem ends.

But let's look at the computer analysis given to the right of the board:

2.Kxd5 a4  (the key move the user should have to play, the only winning move!)
3.Kc4 a3     (only one other move wins here, 3...axb3)
4.Kxb4 a2   (the only winning move!)

Now why couldn't the problem's solution contain all of those moves?  2...a4, 3...a3, and 4...a2 are all +1.0 better than the second-best move, and except for the one alternate 3...axb3, they are all the only winning moves!  Not until move 5 does the problem become ambiguous with more than 3 winning moves for Black.

So in this case, if the pruning algorithm just followed the usual generator rules (+1.0 and +1.75) this problem would have 4 moves and would be a much better problem.

[richard]

Drahacik,

Looking at the gap between best and second best moves is a good suggestion and one I hadn't looked at before (currently pruning is mostly based on static material evaluations with some king safety checks thrown in).  I'd have to do another generator run to play with this as I only capture evaluations for the pruned lines at the moment (to save CPU), I'm also a bit worried that this could lead to underpruning and I really think that underpruning is MUCH more annoying than overpruning, although I'm open to other opinions on that.

Regards,
Richard.

[drahacikfm]

currently pruning is mostly based on static material evaluations with some king safety checks thrown in.

That's probably why most of the endgame problems are pruned too short.  Material and king safety often are not important, but pawns running for queens are.

I'm also a bit worried that this could lead to underpruning and I really think that underpruning is MUCH more annoying than overpruning, although I'm open to other opinions on that.

I would say the ratio of the number of too-short problems to too-long problems in the current set is 50 to 1.  So go for a little under-pruning

If the best move is +1.0 better than the second move, and there are less than 4 winning moves, nobody can complain about having to make another move!

I would think the pruning should be based on the computer evaluations of the top 4 moves, just like the generator is.  Not on a stripped down static evalution.

To avoid a complete new generator run, you could try the new pruning on only the 1-move problems.  Much fewer problems, and much fewer positions in each problem.  Could probably finish the run in a couple days.

[richard]

I would say the ratio of the number of too-short problems to too-long problems in the current set is 50 to 1.  So go for a little under-pruning

The current pruning is deliberately conservative in this regard at the moment, too-short problems are therefore much more common than too long. So I agree that currently too-short problems are a more common annoyance if there is an increase underpruning this is likely to make people much angrier than overpruned problems.  Having to find a pointless pawn move with no relevance to the tactic you've just successfully completed and then being marked wrong when you don't find it is likely to be highly frustrating (even if there IS a point, if it is a side-issue and nothing to do with the tactic you've just played I think this will still be quite frustrating).

Looking at evaluation gaps for pruning could be risky in some situations, for example there may be a gap which is due to the user having to carefully respond to threats rather than finding threats of their own.  While I'd like to start including defensive tactics at some point, I'd rather not have them mixed up in the same problem.

I'm still trying to avoid anything but minor bug fix changes to the current generator as the code is now particularly difficult to maintain and really needs to undergo a major rewrite in order to provide a more change-friendly base for future generator improvements.

Richard.

[revenant]

Not to flog a dead horse (or perhaps only a sleeping one?), but this is one area where rating each position within the problem individually might make your task as administrator easier by mitigating the issue of user anger/confusion.  If drahacikfm (currently the strongest Standard player) is recommending that you extend a certain well-defined class of problems, and your automatic generator concurs that they can be extended without introducing ambiguities or going below the threshold, then rating each position would let you breathe freely as you allow the extension.  You wouldn't have to worry about the potential for large-scale rating drift (like the 150-point migration we've observed in Standard recently) because you wouldn't be changing the characteristics of a whole problem, you would only be adding new, independent positions that then settle to their own appropriate ratings.  The "problem set" becomes a more malleable "position set".

Users who proceed to miss the "pointless pawn move" after the main tactic in any given problem, would feel less cause for complaint because they would only get penalized for that particular position in the sequence (perhaps dropping a couple of points).  And users who *do* find the move will be strengthening good habits, namely looking at each position with fresh eyes and no presumptions.  In chess, the past is past and what matters is what's on the board in front of you right now.

When we were all discussing this idea in the "Rate the position, not the problem" thread last month, apprehensions were raised that it would reward guessing on the first move.  On reflection since then, I think we were missing the forest for the trees (here come the botanical metaphors again!).  Actually, won't the problems where guessing works be compensated for by the ones where it doesn't?

For example, in a number of CT problems, I've tried (or considered) a "Greek Gift" (Bxh7+) sacrifice without working out the concrete variations beforehand.  Bad habit.  Sometimes it works, but usually it doesn't.  Sometimes it would work partway but there's a clearer, better move available that wins a whole piece on the other side of the board.  The CT problem set is so huge and varied (kudos to its creator!) that we get the opportunity to be presented with every conceivable configuration.  Over time, the result has been that I'm now much more cautious about firing off that move.  From a ratings point of view, if each position were rated separately, I would have lost as many points (or more) to wrong guesses as I gained from right ones.

Meanwhile, aren't there situations where CT *should* reward good guesses?  Chess is a combination of accurate calculation and intuition.  Intuition comes from experience with a lot of different kinds of positions.  It's the "whisper" drahacikfm was talking about in another recent thread.  We want to tune in on the whispers that are right and reward them.  Rating points are a good reward.  Next time around, the whisper will speak up more confidently.  It will become a proper voice and contribute to the development of one's style.

As a specific example, consider http://chesstempo.com/chess-problems/30641.  I got it right, but not because I saw the whole tree of variations by any means.  1. Rd7+  *felt* like the right move after a lot of mental thrashing around with mating possibilities that didn't quite gel.  So I played it.  I didn't even see that white could capture the b8 rook until move 3 when the 2nd rank was visually cleared out enough to let the king in to c7.  That is, I didn't see it consciously.  I think some other part of my mind, the intuitive part, *did* "see" it at move 1.  The welcome result solidified my confidence with this type of position which will occur again in the future as a matter of course.  Surely other users can confirm that they have had similar experiences.

Wake up, Mr. horse!  (nudge, nudge)

[drahacikfm]

Richard:

I don't think "pointless pawn moves" will be included when lengthening problems according to the generator rules.  The "pointless pawn move" has to be at least +1.0 better than every other move.  In that case it's not pointless.  It could even be considered to be "forced".  Problems should continue as long as the user's moves are "forced".  (Actually, as long as there are less than 4 winning moves and the best move is "clearly" best).

I don't agree that it's mixing defensive problems and offensive problems into one problem.  That's basic chess calculation.  You see a combination that wins material, and weak players stop there, and play it.  But strong players first think:

"What can he do to me after I win that material?  Ah, he can threaten mate.  Can I stop the mate or not?"  Or "After I win that material, he pushes his pawn to the 7th rank.  Can I stop it from queening or not?"

You have to ask those questions to be a good tactician.  In games between strong players, your opponent almost never just gives you a combination "for free".  He has some sneaky threats at the end that you better know how to meet even before you play your first move.  Or else you lose, because all you did is fall into his trap!

[richard]

drahacik: "Pointless" was probably a bad choice of words, obviously the computer wouldn't have seen it as pointless.  My concern is that someone would end up playing say a move which wins a rook and then get marked wrong because they didn't follow up with a pawn winning tactic, the pawn move wasn't pointless but it was also not the main point of the tactic so marking someone wrong for missing it is likely to lead to tactics-rage :-)  This is an area where revenant's idea would help, however I'm still not ready at this point to make such a large shift in how the tactics work.  I do think the idea has some nice features , but I'd probably implement it as an alternative tactics solving approach to start with rather than a replacement and there are quite a few other things I need to get to first.

Regards,
Richard.

[revenant]

"tactics-rage"...  LOL...  Now I know the name for the emotion I'm experiencing when I curse at the onscreen chessboard loud enough for people in the next apartment to hear.    (Etymological offshoot of "road rage" I guess)

Prolonged interaction with thousands of Chess Tempo problems actually seems to confer the benefit that one begins to feel that emotion less and less.   After a while the tactics-rage is worked out of your system and you begin to have a different, healthier response of "Wow, that's a beautiful solution, it's amazing what computers can show us.  How does this work?!"  And then you investigate.  Each problem is like a unique, intricate puzzle or glockenspiel.

Just one of the many ways CT is making the world a better place.

[richard]

Ahh revenant,

What lovely euphemisms you've crafted! Next time you find a less than satisfactory problem you can simply post, "This glockenspiel is in need of service" :-)

Richard.

[drahacikfm]

Problem 50038 is a perfect example where the pruning algorithm can be improved with the simple idea that if the user's next move is completely forced (in other words there is only one winning move), then include it in the problem.

1.Kxd5 and the problem stops.  Most people will probably think they just took a hanging rook for free, and that's all there is to the problem. But then after 1...d2 they might think they have to resign because they can't stop the pawn!

The only move then is 2.g4+! and after 2...Kxg4, the only move is 3.Rc4+ and 4.Rd4.  Any other second, third, and fourth moves by the user lose!  So those should all be included in the problem.

This idea, that if the user's move is forced include it in the problem, might introduce some spite checks at the end of some problems.  Well, in real life games (at least in blitz) people often throw in spite checks before resigning, so it's good training

Another requirement for the pruning algorithm should be that problems cannot stop when the user is down material.  49343 ends after 1.Rxe2 and the user is down the exchange and two pawns after 1...Kxe2.  Very weird.  The only case where it would be acceptable for the problem to end with the user down material is when there is a passed pawn that obviously can't be stopped from queening.

[richard]

These types of pruning issues are probably the lowest hanging improvement fruit still available in the generator. When I get around to the generator rewrite, this will likely be the first area I tackle, as it will give the most benefit with the least amount of extra effort.

Richard.

[oded ross]

Here's a similar topic.
The subject is also being addressed here.

I'll start by saying I fully agree with every word that drahacikfm have written in this thread.
Hopefully after a future update manual pruning of a problem  would be just as simple as manually disabling it. This is without touching its analysis - cases where the computer gives away material instead of making a more "human" moves that still requires some thought are probably harder to deal with, although some good ideas have been brought up.
I'd try to make a list of problems that would be better off longer or shorter. Prolific solvers such as drahacikfm and revenant could probably add many more.

(Number of problem [d=currently disabled], current number of moves -> desired number of moves)

Overpruned Problems
50038 1->4
32642 2->5
54295 2->4
23621 1->2
49343 1->3 ? (it is a strange problem)
47985 1->2
52004d 1->3
55183 1->5
44861 1->2

Underpruned Problems
1394d 15->3
48499d 3->2
48956 2->1 (What purpose does the second move serve? Who in is right mind wouldn't take the queen?)

[revenant]

Who in is right mind wouldn't take the queen?

I resent that!   

                                   Signed,

                                   Revenant
                                   (a.k.a. the "Hanging Queen Misser
                                    and Subsequent Head-Banger")

P.S.  See (very) partial list of relevant Problems here

[oded ross]

Who in is right mind wouldn't take the queen?

I resent that!   

                                   Signed,

                                   Revenant
                                   (a.k.a. the "Hanging Queen Misser
                                    and Subsequent Head-Banger")

P.S.  See (very) partial list of relevant Problems here

(Slightly off-topic)
I've gone over your list. Although some (if not most  ) of the misses are surprising for a solver at your level, and some of them are rated much higher than expected (seen several 1500 problems that are much easier than 1200 ones), there is a big difference between a problem at which the point is you need to see you can capture a queen (most often because a line has opened), and the one on which I commented where you are practically forced to take the queen.
The only one on your list that has some similarity to my case is 11184, which can be shortened to one move, but maybe there's a small chance that someone would get his math wrong and play 2...Bxf2+ ...

[revenant]

Well I will say it's happening less and less often (the EMT paddles must have had some effect).  45301 bit me a few days ago (tunnel vision...  eh? He left his Q en prise to a PAWN?, way over there on the EDGE?).  I'm pretty sure I've been served-up your 48956 at some point but I don't remember what happened.  I could easily have slipped up even there, though, and perhaps you can now see how.  You find 1. Rf3! and you're all pleased with yourself.  Then 1... Qxf3+ and panic!  These "computer punts on 4th" type moves are still a shock to me--  is my clever "reculer pour mieux sauter" move really sound after all?  I'm in check!  Run away!  2. Kh2, red flag "Failed", ow.