Too many voices in a small space without enough modulation and utilities. Think of how many different things are on the MS20 just to make that one voice (for the most part) feel alive and musical - envelopes, an LFO, two filters, sample and hold, "ring mod", and the external signal processor, plus a pitch wheel thingy. That's the sort of approach you want to take to modular if you want to get more out of it than you would just with complete synths and pedals. Voices and effects are cool, and they can sometimes even be used as really interesting modulation, but for the most part you'd need at least 2-3 times this space to feed these sound sources and effects. Maths can do a lot of things, but not all at once, and certainly not enough for all these. Pam's is also very powerful, but you don't want to rely on it for basic modulation, and it isn't very hands-on. None of these modules are necessarily bad, but you should probably use this much space to make one or two excellent, super-flexible voices rather than 4 with what is probably slightly under the bare minimum of support.
Watch videos about complete modular systems, like the Make Noise Shared System/Tape and Microsound systems, or even do a bit of research on the suggested systems from Buchla (careful not to get sucked into that, though, because, while Buchla stuff is awesome, it makes Eurorack look like smart financial planning). Semi-modular synths as well. Get a sense of what people are using and calling "complete instruments" and why designers and users feel that way. It will help you design your system's vital plumbing in an informed way and make these modules feel and sound more like yours.