Well the computer stopped booting into recovery mode altogether now and just goes to Internet Recovery, so there's that.[...]
Sounds like your HDD is broken.
I've tried to mount the mechanical drive and couldn't. Do I need to plug both?
Yes, you would have to have all components present to get the Fusion Drive up. From everything you've said at this point I doubt it will work. If you only had one of the parts plugged in, the hardware should still be visible in diskutil ('diskutil list').
Isn't there some rescue software that ignore partitions errors and such and just tries to read whatever data is salvageable?
Photorec, from the testdisk/ddrescue guy. Despite its name it will work on a variety of files. The problem is that everything is unsorted. You'll just have file1-600000.whatever in a fairly flat directory structure.
I would first run ddrescue on the HDD to a blank disk/image of sufficient size, then try rescuing the files there/restoring the FD to functionality, etc.
There is a chance that ddrescue would get you a copy of your drive that works well enough that the Fusion Drive can be fixed (fsck/diskutil repair) enough for you to get your files off.
FWIW, I do most of my disk rescuing on PCs I boot into Ubuntu live CDs (well, USB sticks). Disconnect the internal drive if it's important (I've rescued a drive onto a co-worker's "C drive" (whatever that is) instead of the intended disk by accident).
I just don't now how to mount a drive that is composed of 2 physical units with OS X CoreStorage, because obviously USB enclosures can't emulate that and target disk mode doesn't work with it either.
The following is my understanding and it may be wrong: The Fusion Drive/any CoreStorage volume group is set up by recording the drives' UDID's to some place in the header of the physical volumes, like the partition map. It shouldn't matter how the drives are attached to the Mac so long as that information is passed through the connection. A multi-drive enclosure may or may not allow that, depending on what it does. Once all the drives are connected, telling diskutil to mount the FD by name might bring it up.
I don't have any method of testing my supposition; my only FD equipped Mac is in use.