My take on the 4 hp and down world is that, when they have a function you need and space is limited, they're a blessing. Take Erica's mixer modules, for example; that would be about as small as I would think could work as a simple mixer. But when you start getting into shrunk-down versions of modules that have major functionality in their full-sized versions, I'm a lot more skeptical. For example, 2hp is great for things like their dual VCA, as you don't adjust a VCA all that much, and there's nothing complicated there: in, out, modulation. But I wouldn't consider their version of the Turing Machine to be a contender against a fully-blown Turing Machine, as there's a lot missing that really makes that module (module group, in truth) what it is. Small also works as "hole-fillers"...need an extra basic VCO to mix against a bigger, more featured one? Sure, drop in an Erica PICO if you've got 3 hp to spare. But I'd never use one as a primary VCO unless I was building something in the size of a single Moog 60 hp skiff or smaller.
And then there's manufacturers that specialize in small, aside of 2hp of course; one that comes to mind is Ladik. Ladik has things that fit in 4 hp that really don't exist anywhere else. In such cases, those ARE your primaries, and there's nothing wrong with using them to get that extra bit of functionality. One just ignores the fact that it's in a small footprint. One other one that comes to mind, also, is Konstant Labs' 1 hp power checker; if your cab has no power rail indicators and you do happen to have an extra 1 hp free, then I'd call that a "must-get". Slip it in on a row end, and you get valuable rail voltage info...again, tiny as hell, but tiny works in this case.
Otherwise, yes...if we're talking about something that needs a lot of interaction, like a VCF, then small probably isn't the way to go. But again, different builds and/or different users call for different things, and there's not really a hard and fast rule about size. If you've got Trump hands, then PICO might be the way to go!