Alternative winning lines - question on approach

[richard]

Hi all,

As users go through the new problem set and comment on the bad problems still left in the set, a couple of things have become apparent to me:
1) There are at least two bugs in my "eliminate alternative winning lines" code, one impacts only mates with alternative winning lines and the other only non-mates. Many alternative winning lines WERE being detected and problems which had them filtered from the problem set, however due to the two bugs, too many are still sneaking through.
2) Many of the "best moves" in these situations are quite nice tactics but are undermined by the alternatives winning moves that are going undetected.

To deal with 1) I'll have to do some fixes and another problem verification run. While doing that I'm considering looking at changing the UI to allow the possibility of alternative winning lines, thus allowing me to let these problem through when I identify the alternative winning lines instead of throwing what would otherwise be a useful problem away.

My question is, how would users prefer to see the UI present these? The easiest for me would be to detect when an alternative line exists, record the first move in the alternative  line and then detect when the user plays a winning alternative in the UI and deal with that different from a normal deviation from the "best line".  So users don't get a chance to play out the alternative lines under this model and would not necessarily get credit for them, but do avoid losing points for choosing the alternative line. 

The alternative to this is to record (and run disambiguation) on all alternative winning lines and treat them on an equal level with the "best move" (where the best move is always the move with the best evaluation. From a users point of view I'd say this is what most people would prefer, but it has a few disadvantages from my point of view. Firstly it greatly increases the CPU hours required per tactic found in the problem generator as alternative lines would have to be processed  (and recursively also, if you allow alternatives to be treated on the same level as the 'best move' then those alternatives may also have alternatives further along their path :-)) [actually the CPU hours to number of problems ratio impact is a bit hard to predict as this would also allow an overall higher yield from the set].  Secondly there are also major database and UI changes required to accommodate the second approach, I'm concerned that the UI will have to get so complex that it becomes too hard to use.  I think one of the reasons CTS has so many users is due to the simplicity of the UI (it makes it easier to "just focus on the tactics") - of course I'm hoping to add some power without complicating too much, but the CT UI is already starting to get a bit cluttered (although better UI design could help here).

If I decided to go with the first solution as an interim solution, would users find that acceptible (i.e no ability to see full disambiguated lines - at most user can see starting evaluation and disambiguated move order , may not be able play them in UI, may just see alternatives in notation, possibly no credit for getting the alternative move [but also no less for finding it instead of best line)?  Or would allowing moves with alternatives without giving full credit and allowing the tactic to continue down the alternative lines be just too annoying and users would prefer to have ALL problems with winning alternative lines to be removed? If I go with the interim approach how should  I tell the user there was a "better line", in situations where the second best line wins but (hopefully) not as quickly, is it ok to say, "that was a good move but there is a better one" or is the idea of better when both moves eventually win just a nonsense?

Feedback from both high , medium and low rated players would be very useful here as what is good for one group of users is not necessarily good for another.

Regards,
Rchard.

[tmr]

I see that this is a tougher problem than I envisioned.  I'd say filter out the ambiguous problems that don't have a clearly better line than the alternatives (say at least a point better).  Folks know that the objective is too find the best move.  In this case if you find the second best move you shouldn't be that upset because you could have done better by at least a pawn.

I don't think I've encountered much of this yet.  In each case I've found another line it has been inferior to the best line.  I did comment today on one problem where the opponent's (white) move in response to my first move was clearly inferior.  The tactic would have been longer with a better move so perhaps that was the reason.

[slacker00]

Richard,  I think your simplest solution is the best.

For problems with multiple "winning" solutions, only giving full  credit for the "best" solution is satisfactory,  with secondary solutions maybe counting as a draw, or as a non-play.  I think the frustrating part with the prior system is finding a clearly winning  move, yet LOSING points.  It just didn't seem  intuitively fair.  But we are all here to learn and improve, so I think we can accept this kind of compromise, philosophically.

Also, the simplest solution is better as far as keeping the UI as simple as possible.  I think a site like this needs  to cater to the lowest common denomenator, the beginning player.  Partly due to beginner-level players being more  likely find this type of site useful, but  also partly due to advanced players being able to and wanting to do further analysis on their own, rather than needing ChessTempo to guide them in a significant way.  Keeping this site fast & easy to use is the best bet,  IMHO.

[richard]

tmr:  This is not about choosing the "best line" which the current generator does a reasonable job of , the "best line" is always either a shorter mate or has a large evaluation gap to the second best move (sometimes the engine still gets things wrong in that if I'd looked at the position for 10 times as long the evaluation might have changed enough to prefer a different move).  The problem is the situation where the second best move although being behind the first move in evaluation is still a move which would put the player in a game winning position.  I've had heuristics to try and remove positions that had this possibility but a) the heuristics have had flaws and b) it throws away many tactics whose "best move" is quite beautiful.  Some users don't get too worried by second best moves as long as they can see why the "best move" was given preference. However others strongly dislike these positions , wondering why they spent some time finding a move that would be good enough to win the game only to be told they are "wrong". Over time I've developed a lot more sympathy for the second opinion. I think that going through the tactics is probably much more enjoyable if when you see a winning move you can just take it without being scared you'll be marked wrong because of a "better" winning move lurking in the background.

slacker00:  Thanks, the simple approach is certainly the direction I'm leaning towards. I've had a bit more of a think about it and the idea I'm liking the most at the moment is to simply tell the user "good move, but please keep looking." when they find an alternative winning move and let the tactic continue on as if nothing had happened, i.e. alternative moves are treated in a similar way to illegal move attempts (in the new UI), they are simply ignored (but with a message to the user letting them know why).  I think the wording is important as I want to get away from the "best move" idea as the counter argument is that what is "better than winning?", I imagine some users would still argue they should have got full points for the alternatives, but I think the "don't punish" approach is a reasonable compromise.

I still have a few things to get to before I can look at the generator again so there is plenty of time for further thoughts and comments by users.

Regards,
Richard.

[slacker00]

I like the idea, richard.  I can't wait until I get to try one of these problems.  I wonder  what my reaction will be when a message pops up,  "Good move, but there is a better  move.  Keep looking.".  lol.  As if the chess board is talking to me.  Plus it's almost like getting to do 2 problems in 1.  I found the "good" solution, now I get to find the "best" solution.  It's almost a bonus, depending on how you think about it.  I'm 100% behind you  on  this idea.

[drahacikfm]

simply tell the user "good move, but please keep looking." when they find an alternative winning move and let the tactic continue on as if nothing had happened, i.e. alternative moves are treated in a similar way to illegal move attempts (in the new UI), they are simply ignored (but with a message to the user letting them know why). 

Yes, I like this solution.  The user does not lose any rating points for choosing a winning move that is not the computer's number one choice.  And the user gets to continue looking for the computer's best move, and still has a chance to get rating points.

If you don't want to use the word "best" in the message, maybe something like "That's a good move, but not the computer's first choice.  Please try another move."

There are still decisions about which problems to keep.  Do you keep a mate-in-5 problem that also has a mate-in-6, and tell the user to keep looking when he plays the mate-in-6?  Or do you plan to use this only for mate problems with material-winning alternatives?  Or problems where more than one move gives a winning material advantage?

[richard]

Slacker00: Glad you like the idea, it will still be a while before you get to see it in action as a full problem set verification run can take a few weeks.

Drahacik: Which problems to keep is certainly an interesting discussion. I'd be inclined to keep the mate in N+1 as alternatives instead of rejecting them as I do now.  I'd also consider making the mates that fall outside the ambiguity window as winning alternatives so if you do find a mate in 9 when there is a mate in 3 you get told about it rather than marked wrong although I'm a little more ambivalent about this, the user is more likely to be making a mistake than playing what they've seen as the first move in a 9 move combination, especially if they missed the 3 move mate.  It is important that accepting alternatives doesn't turn into an exercise in not punishing for imprecision...

Generally I'd like to use it for all "winning advantage" situations. I think the mate in N versus mate in N+big_number_of_moves is probably the only tricky one, I can't think of other situations where I'd not want to take the alternative. 

Regards,
Richard.

[tmr]

If I understand your idea correctly you're looking at first move alternatives, correct?  Or do you envision winning alternatives as the line progresses?  This would seem to get complicated.  It would be nice though since I've seen several problems where there are several ways to complete the mate and all too often I'm picking the slightly longer one.

[richard]

If I understand your idea correctly you're looking at first move alternatives, correct?  Or do you envision winning alternatives as the line progresses?  This would seem to get complicated.  It would be nice though since I've seen several problems where there are several ways to complete the mate and all too often I'm picking the slightly longer one.

This would be for all points in the "best line", the generator already operates like this as I have to progress through all moves in the "best move" sequence to make sure ambiguities don't come up on later moves that were not apparent on the first move. This is why it is rare to find moves that have equal evaluations as "best move", there are certainly still alternative lines which also win but they are almost always a couple of pawns under the evaluation of the "best move" line.  This is not always the case as depending on how long you look at a position, the difference between best and second best can fluctuate, I try to look at each move long enough for this not to be a problem but  without an infinite amount of CPU time I can't be perfect here (in practice evaluations don't seem to change too much after about a minute).

So as the generator is doing most of this already (just not completely successfully for some situations :-/ ) it is not much harder for me on the generator side.  The most time will probably be spent changing the rest of the code to deal with the alternatives but as the underlying board can handle alternative PGN lines it shouldn't be super difficult.

From the user's point of view I think the extra complexity should be minimal, for each move there is 3 options, correct, incorrect and now "try again."
 
Regards,
Richard.