You should consider using pogies (ex. "Bar Mitts" brand). Neoprene pockets on the handlebars that enclose your brakes/shifters and hands. You can wear lighter gloves than you otherwise would, retain dexterity to operate the controls, and you stay warm.
It's also still easy to pull your hand...