That's true, but if we compare modular synths to an instrument, it's like you have to build your guitar every time before you start composing. Surely for each song you had in mind, you would choose one type of wood or another, more or less strings, etc... This is the theme, you have a sound in mind, and you are looking for how to achieve and you end up acquiring new gear.
There are sounds that can be achieved with your current equipment, but especially at the beginning, there are sounds that you can only get with a specific setup.