Traxams right - I think too much modulation as well!
Pams is a great shout and you should defintely get it. It's actually got built in sample and hold via its cross-ops functions, alongside a variety of other tricks like min/max, logic, euclidean functions, random etc. Great for creating accent variations! I think thats a good place to start, along with a single ochd+expander. 3x Mia is also a great a shout but again I think you should get one and see how you go.
You'll find you wont stick to this plan anyway and your needs will change :)
Recomendations of modulators would be Tides, Batumi+expander, quadrax or a nekiya obsidian (havent got one of those but its feature set looks hyper useful).
Function generators you cant really beat Maths. Schlappi boundary is great as well but only one channel (although built in envelope following, and bound which makes bouncing ball effects easy to patch). I have a boundary and currently have a delta-v in my case which is great, but i would get maths again in my next case.
Utilties aside from pam's and 3x mia which would have been my suggestions anyway, arkan, veils v2 and what the hell i'd suggest manic (gives your drums an extra sparkle, although you might not find its wavefolder useful). I'd look up monotrails on videos to get ideas on utilities which suit you- i basically learned everything from this guy. He covers modules like utilities you might not find useful in short term, but will maximise your rack in the long run. A lot of eurorack principles may not apply to a drum rack in first instance, but its good broader knowledge.
Another utility to research is matrix mixing. Can't beat the doepfer a138m. I personally have fewer modulation sources, however I derive variations through min/max or mixing them together them so they relate. Invetering an lfo for instance, and sending the inverted along side the postive to different sources makes your sounds dance along together. Its more clean and subtley brings your sounds together than having a billion different unrelated resources. The matrix mixer is good for this. Samara II is a good shout to look into as a mixer. Has built in sample and hold and also min/max features and its only 10hp.
I run my drums through a cosmotronic messor which is an amazing compressor. It uses the same chip as some boutique compressors, but comes in at half the price. I'd recommend that or something similar to glue everything together.
Of course a lot of these suggestions wont work for you, which is fine! Thats the beauty of modular- highly customisable. I love this hobby and happily fall down a eurorack rabbit hole everyday.
Start slow, work out your needs, and plot your course as you go. The thing you want now, might not necessarily be the thing you need later :)
EDIT:
Oo sorry forgot to mention your sequencer plan. I think that sounds sensible. Personally I'd stick with the Oxi One still due to the four sequencers. You could have one set up for each rack and use the Oxi Split to seperate out the different channels, with an additional one going to the pipe. For me it would mean everything is in one place and would be one less thing for my brain to handle. Although admittedly its more buttons to press and pages to keep track of, so it has its downsides!