Also I highly suggest you to check VCV Rack.
-- gila_crisis
This, definitely! You can try Marbles and Clouds in VCV rack. There's also verbs and delays, filters, sequencing, S&H's... It'll give you instant feedback about how to build your modular. I think your idea is good, but don't forget about utilities and basics to control your patches if you want to be precise about the sound. Especially some dedicated attenuverters to mix and attenuate cv sources, lfo's (batumi, ochd...), some mults, vca's etc. Maybe also a clock source and/or divider. Last but not least: a good (stereo multimode) filter can go a long way.