Extended Data Fig. 4: Comparison of the performance of different methods for Multi-MNIST-Sudoku.
From: Automating crystal-structure phase mapping by combining deep learning with constraint reasoning
We show the “solving time” for unsupervised DRNets and its ablation variants and “test time + training time” for supervised baselines. The test time for CapsuleNet/ResNet + local search includes the local search time. Note that we used two different local search algorithms for 4x4 cases and 9x9 cases. “local_search1” performs an enumeration for the top-2 likely digits in all 16 cells to try to satisfy Sudoku rules. For 9x9 cases, it is impossible to enumerate the top-2 likely digits for 81 cells (281). Therefore, “local_search2” conducts a depth-first search for digits in each cell from most likely to less likely until it finds a valid Sudoku combination, which is faster than “local_search1”. For 4x4 cases, we also applied exhaustive search for all methods, where we enumerate all possible 4x4 Sudokus and return the one with the highest likelihood given our predictions. Note that such strategy is not feasible for 9x9 Sudokus, given there are around 6.67 × 1021 9x9 Sudokus. The ablation study of removing the reasoning modules (DRNets w/o Reasoning) shows that not only does the Sudoku accuracy degrades, the digit accuracy also degrades, especially for 9x9 Sudokus. The ablation study of replacing the cGAN with a (weaker) standard learnable decoder, without prior knowledge about single digits (DRNets w/o cGAN) shows that both the Sudoku and digit accuracy degrades dramatically.
