Well, it can be caused by either (or both). Even with high powered, clean amps, some speakers can distort. I won't get into the technical reasons why, but realize there are 8" 4ohm 100 watt speakers that sell for $5 and 8" 4 ohm 100 watt speakers that make up sound systems that sell for over $500.
Before you go even more in debt, consider some realities of physics:
Big powerful sound systems with subwoofers use a lot of power. Not a problem for a home system on 115 volts or a car with a 100amp alternator. But for boats, especially SeaDoo boats with small 12 volt batteries, limited current voltage regulators, and under voltage/over voltage sensitive mpems, appreciate the following.
The formula for instantaneous power is P=I*V where P is power in watts, I is current in amps and V is voltage in volts. The reason I bring up this techno mumbo jumbo is to illustrate as simply as possible why 12 volt high powered sound systems in boats are impractical. To develop full power output with a 400 watt amplifier, it would take 33.3 amps (using the above formula 400 divided by 12), and that is at 100% efficiency which most amps are not. So you could be drawing 33 to 60 amps from your battery which will mean you will need at least another battery, a battery switch to isolate your starting battery so you can get home, and a box full of spare voltage regulators and possibly mpems. My 200HP Mercury has the optional 40amp double voltage regulator setup and it still blows regulators if I run the battery down with a 120 watt stereo, tube inflator, small cooler and running lights which all told don't come to 200 watts.