Rashba coupling and magnetic order in correlated helical liquids

We study strongly correlated helical liquids with and without Rashba coupling using quantum Monte Carlo simulations of the Kane-Mele model with a Hubbard interaction at the edge. Independent of the Rashba coupling, we find that interactions enhance spin correlations and suppress the spectral weight at the Fermi level. For sufficiently strong interactions, a gap can be observed in the single-particle spectral function. However, based on a finite-size scaling analysis and theoretical arguments, we argue that this gap is closed by order parameter fluctuations in the Luttinger liquid phase even at zero temperature, and filled in by thermally induced kinks in the order parameter in the Mott phase at finite temperatures. While the bosonization suggests an umklapp-driven Mott transition only in the presence of Rashba coupling and hence an important impact of the latter, our numerical results are almost unaffected by Rashba coupling even at low temperatures.