I'm interested in this thread as I'm also thinking of slowly relieving Ableton Live from its FX duties.
-- Mazz
Well, maybe the simpler ones...there's still plenty of things that software can do that hardware can't.
As for what's needed to really integrate hardware with a software "engine," that would be a mixer that has ample sends and returns. Or you can use some spare I/O channels to establish send/returns from inside Ableton to outboard devices. I don't think it's a very good idea to try and get audio out of and then back into the modular just by adding a matrix mixer and plugging effects processors into that. For one thing, there's a big level mismatch, so you'd need extra modules to drop/raise the levels so that you weren't prone to overloading the effects or not having enough level on the return to work within the synth. That's going to cost, both in terms of $$$ and hp.
But there IS a way to make this work with a minimum of additional I/O modules, and that's by putting together a parallel mixing setup for your effects processors.
Let's say for a minute that you could do this exact thing by adding only TWO channels of I/O level shifting, but you've got a PILE of processors. So, drop in a stereo I/O...something like a Ladik A-595, for example. Now you've got what's needed to move a stereo signal out and back. Then, add a distribution amp...it's sort of the rack gear equivalent to a buffered multiple...and route your effects outputs from this as it splits your stereo Eurorack send. Then on the other side, use a small rack mixer (I prefer Rane's SM 26B for this, as it works as either a signal splitter OR a 6-input stereo submixer) to submix your processing back to the stereo Eurorack return. The fact that the Ladik module is both cheap AND small can make this work, and all you'll need "outside" would be those two devices, and maybe a 1/4" patchbay if you want to make changes to your routings easily. Alternately, you could use Ladik's P-520 to have four channels of output, and still use a stereo line return while using just the outboard submixer for FX summing; if you had four mono-in/stereo-out processors (something like certain stompboxes, or classic rack processors such as Yamaha's SPX90), one P-520 would handle that and, by using the outboard submixer, you can create a fairly elaborate AUX setup for submixing different voices. And so on...it's the sort of setup that's really only limited by imagination and $$$.