Kritz, just a suggestion if the security protocol isn't totally to blame for your issues...
I had a similar problem with my original iPad. It would just have issues with the home network. guess what. the router couldn't handle so many devices. one new Billion router later and I have more devices connected all working fine than I could ever conceivably find a use for.
Try it on some other network, or on your own network, try turning everything connected to the router off, power cycle the router and turn on the 3DS first. If the 3DS gets on the network then your router can't handle so many signals. A router with QoS might help
Not saying buying a new router will fix your issues, just that it fixed mine
At the moment there's only about four devices connected to it, three of them wifi, one of them the 3DS. It's handled loads upwards of six devices before, between my Xbox, PS3, bluray player (for some reason), PC, two laptops, two ipads as well as an assortment of phones. Except for the PC and Xbox, all of those were going off wireless. But, after a system update, the thing's going fine. I connected to the eshop and the browser and watched a trailer and it all went fine. This was after changing the setting back to WPA2 AES. Between it not working, and it working, the only thing I changed was the system update. The router itself gets rebooted regularly due to the bi-monthly dropout.
I had it update over my phone, as suggested, which (as said) I did try previously but didn't delete the old connection and simply added another one, thinking it'd work out that one connection didn't work so it might try the other. I'm not really sure how you tell it what connection to use when you have multiple set up. I also tried, of course, deleting the connection and recreating it, both using the automatic settings and also using manual settings, which included the nintendo troubleshooting stuff. So, I don't fucking know. Maybe launch 3DSs sucked ass. Maybe there was an incompatibility with my dlink router's WPA2 AES setting and the 3DS's WPA AES setting that has since been fixed by a system update. I did actually try tuning on/off my router's QOS setting with no result either way. The router itself is somewhere between 4-5 years old. The 3DS is less than two. I have no idea.
Also eshop games are expensive I ain't buying any of that.