Not sure if you are a car guy, but swapping impellers is just like swapping gears in your rear differnential.
If you go with a higher gear, you take off faster (out of the hole) but then you end up with going 60 mph at 3500 rpms, rather than 2000 rpms. So top speed is lower, but you have more UPMH! when taking off.
I have a 96XP and the stock impeller is MORE than efficient for hopping out of the hole, if the wear ring is in good shape. Mine actually hops OUT of the hole when I gun it. It will hop 6-8 inches out of the water 2 or 3 times before it grabs and takes off, and maxes out around 55 mph. In about 3 or 4 seconds. That is fine for me.
If anything, a higher speed impeller might be better for the XP. It has plenty of spin and grab wtih the stock pitch. Not sure what that pitch is...
A higher speed top end impeller that may lose some bottom end grab would likely be fine for the XP, if you want faster top end speed, since the stock tends to grab too much, for my taste.
But I use my XP primarily for jumping waves, so that grab in the low end is what I need for fast take-offs and to make sharp turns and to get back into the next wave. For my Challenger, I opted for a better grab in the lower rpms and a lower top-end speed for pulling skiers and a faster hole-shot.