Topological phase transitions and quantum oscillations in systems with broken time-reversal symmetry

We study the effects of the Zeeman field on the Kane-Mele model with a next-nearest-neighbor Rashba spin-orbit coupling (SOC) and find that two different types of quantum anomalous Hall (QAH) insulators appear and that there is a gap oscillation region with relatively small Rashba SOC in nanoribbon systems. Calculations of the total and spin Chern numbers show that the oscillation appears in the two QAH phases and near the phase transition between a metal and the two QAH insulators. Interestingly, the closing and opening of the energy gap just corresponds to a metal-insulator transition. Therefore, the gap oscillation is also a kind of quantum critical phenomenon, which is caused by the competition between the Zeeman field and the Rashba SOC. Besides, through investigating the Zeeman-field dependence of the bulk gap in real materials, we predict that silicene, as a quantum spin Hall insulator, can become a QAH insulator with a sufficiently large Zeeman field.