This study addresses a significant open question in soft magnetorheological elastomers (MREs): how microstructural rearrangements during magnetic actuation influence their viscoelastic behavior. Through experiments on mechanically confined MREs, we show that magnetic actuation increases relaxation times by more than one order of magnitude compared to purely mechanical cases. Our findings reveal that this modulation of viscous response can be precisely controlled by adjusting the strength and frequency of the magnetic stimulus and is inherently tied to the microstructural rearrangement of magnetic particles. Inspired by these results, we developed magnetic actuation protocols that endow soft materials with force-memory capabilities. Specifically, we achieved magnetically-driven yielding that introduces material hardening under cyclic loading due to sustained magnetically-induced viscous relaxation. This mechanical memory effect can be released by removing the magnetic stimulus for one hour, fully restoring the material's original properties. These mechanisms are clarified through diverse experimental approaches and theoretically supported by a novel continuum model. The findings presented here address fundamental questions in soft MREs and establish a transformative approach for creating a new class of soft sensor-actuator systems.