Short Term Political Memory and the Inevitability of Polarization