If you decide to remove an existing curse, you can do so in Evermore (Bangkorai), Rawl’kha (Reaper’s March), or Riften (the Rift). Once you’ve settled on a location, visit the town’s Mages Guild and enlist the services of Prelate Sabinus.
In exchange for a small fee, Sabinus will remove all traces of the cursed affliction—including the related skill line. You’re then free to remain as you are, seek out a different curse, or reacquire your previous curse.