That all makes sense and does make a good argument for a professionally designed and self contained system to start with.
The Make Noise Tape & Microsound Music Machine is a great foundation. You'll find resources on patching the system, along with many resources for patching Maths alone
Adding Pam's and Midi and you've got everything you need for clocked modulations and synchronization with your DAW
You can always expand the system when you're ready