Chess as a Testbed for Language Model State Tracking

Transformer language models have made tremendous strides in natural understanding tasks. However, the complexity of makes it challenging to ascertain how accurately these are tracking world state underlying text. Motivated by this issue, we consider task modeling for game chess. Unlike language, chess notations describe a simple, constrained, and deterministic domain. Moreover, observe that appropriate choice notation allows directly probing state, without requiring any additional probing-related machinery. We find that: (a) With enough training data, transformer can learn track pieces predict legal moves with high accuracy when trained solely on move sequences. (b) For small sets providing access board information during yield significant improvements. (c) The success is dependent entire history i.e. “full attention”. Approximating full attention results performance drop. propose testbed as benchmark future work development analysis models.