For a small effects rack, especially if you want to continue to use Ableton, you probably want to focus on effects that you can't get on your computer. FX Aid XL is a good in-rack choice for its size and variety, but I don't think it does anything that you can't do better outside a rack. You can try VCVRack for free and see how you like it. That might also be a good way to learn what modulation and utilities you need, though some effects or processors you might want to use in a physical rack won't be available, like Beads.
Maths is big for a rack this size. I prefer Falistri, but it's only slightly smaller. These both have many uses but you may not need their full functionality. Do you know what you want to use them for? Cosmotronic Delta-V is half the size of Maths and does much of what it does. If you just want basic envelopes, you can get smaller and simpler modules (and of course you can get envelopes and LFOs out of the computer also).
In summary, I think your modulation needs depend heavily on the particular physical effects modules you wish to use, and some manual reading and demo watching might shed light on that.