Thanks Lugia, that's really useful feedback.
More than anything else, my constraint is space, which I have just about for the 104HP. I guess I was too optimistic with what I could get away with a modular of that size. Sadly, that means getting multiple devices like you suggest is also out of the question. I might have space for one more synth, definitely not three.
That said, I am not unhappy with my current setup... I love my Rytm and the posibilities for modulation and performance that it has and I'm also very happy with the range of sounds I get out of my Digitone. Also, I'm happy with the music I have released using these two alone. I just have a feeling that going forward, if I want to make things sound different, I have a gap in analog synth sounds, and thought a skiff of modules could sort that.
I guess I will just hold off it until one day I have a dedicated studio space where I can fit a PROPER system as you say.
Again, thanks for the help!