For keeping the groovebox feel, I would also look at a Launchpad Pro. The Nerdseq is designed to do lots of cool stuff in tandem with the Launchpad for performing and general usage.
I still can't help but feel like traditional modulation would be helpful here. Maybe an Ochd or a Batumi? A Vostok Fuji? Obviously space is limited (as is always the case in a case). No hate against the Hypster, and I'm glad to see it still in here, but playing your modulation is a big part of what will make this different from, say, a Elektron box or a Polyend Tracker.
I agree with the person above that the Bitbox is cool for this, but there is always the option of pairing this with another sampler/drum machine for those duties. Something simple like a Circuit Rhythm or Model:Samples, or even a Blackbox to get most of the Bitbox function minus the CV stuff (but plus sequencing lanes separate from the Nerdseq). You can even use the MIDI out of the Nerdseq to sync the sampler and modulate MIDI CCs on the sampler for that "modular modulation" feel. Frankly, the Nerdseq and Morphagene offer plenty here for sampling while you get comfy with all this, so I wouldn't worry about this now. It's something to consider, though.