Recent studies on the 't Hooft anomaly matching condition for 4D SU() gauge theory have suggested that the phase structure at should be nontrivial. Namely, some symmetry will be spontaneously broken, or gapless modes will appear. In the large- limit, it is known that CP symmetry at is broken in the confined phase, while it restores in the deconfined phase, which is indeed one of the consequence of the anomaly matching. However, at small , one may find a qualitatively different phase structure, which will be another possible scenario consistent with the anomaly matching. Here we investigate this issue for by direct lattice calculations. The crucial point of our method is that the restoration of CP symmetry can be probed by the sudden change of the topological charge distribution at , which can be seen by simulating the theory at imaginary without the sign problem. Our results suggest that the CP symmetry at is restored at higher temperature than the deconfining temperature unlike the situation in the large- limit.