If the bearing is rusted to the driveshaft... then you may have to cut it... and replace all of it.
Even if you break the bearing lose, and get it to slide out... the inner bearing race, is the driveshaft. So, if it's rough... you can't reuse the dirveshaft anyway.