Those little cageless needle bearings were all over the inside of this crankcase, I'm sure they had something to do with it. The previous owner must have made no attempt to catch them when he put in new pistons, and then somehow thought it would be fine that they all fell in and ran it anyways. Surprisingly enough, the crank "feels" fine and smooth when you rotate it by hand, but its obviously not fine upon inspection.
Looks like a used good crankcase (including rotary shaft) can be had on eBay for around $130, so that's not too bad and saves me the hassle of replacing that brass gear.
I bought the ski for $600, and it was supposed to be an easy quick fix. I was so sure it just had carb issues and I could rebuild the carb and have a running ski for less than $1000. I'm still going to fix it, but its definitely not the quick fix I thought it was, and definitely not cheap anymore lol