It's a good idea. You can also loop internally with modules like the 1010 Music BitBox. It can look in sync with your CV clock or MIDI (if your sequencer supports MIDI). It offers a 16 cell grid of sample playback, so it can handle loops AND one shots like drums.

The only issue would be wanting to change your tempo later. But a quick trip to and from Ableton Live will solve that. You can open up a 16 cell kit in Ableton Live, warp your samples to another tempo, then save and replace your samples at the new tempo.

Oh and if you're handy with editing XML, you can quickly assemble new kits from any sample collection you have and save them to BitBox's SD card.