I was unaware of the set up and without thinking pulled my CV joints to replace them the whole time with the car not on jack stands. I rolled it in and out of my garage also a few times and now there is a good amount of play. Do I even try to put it all back together and tighten the castle nut? Or is the bearings in the hub messed up for sure?

If you just rolled it around the garage, you should be fine.

Put the axle back in and tighten up the nut and see if you still have play.
Make sure you check both the 9-3 o'clock as well as the 12-6 o'clock plane for play.

Great, thanks for your advice I’ll put it back together and cross my fingers.

Everything is a matter of degree.

Hubs and CV's are fine.

Rolling the rear wheel bearings with the full weight of the vehicle on them HAS damaged the wheel bearings. Split race, double row ball bearings as are used on the rear axle are not meant to bear load on them without a fully torqued hub in place. Just installing the hub or the inner CV without having torqued them will still cause damage if rolled without pre-load on the bearing.

Lack of pre-load causes the races to separate and the balls then roll on the weakest portion of the bearing race in a place they were not designed to operate.

The bearings will not fail immediately but I assure you as a guy that used to do engineering of hub and bearing assemblies as my day job -- the life of that bearing has been substantially shortened.

Plan to replace those bearings in the future.