“I’m like, ‘Should I just quit? Because it’s just I feel like it’s breaking me down,'” Judge said of her future on the Bravo series.