Inspired by Kitaev's argument that physical error correction is possible in a system of interacting anyons, we demonstrate that such "self-correction" is fairly common in spin systems with classical Hamiltonians that admit the Peierls argument and where errors are modelled by quantum perturbations.