Normal, actually. When you turn the "Inject" control up, you're adding a DC offset to whatever signal is there. And if there's no signal present, the wavefolder is trying to make something out of the DC offset...which is what you're hearing. The key to avoiding this is to keep the "Inject" down for most uses, unless you're trying to shift the waveform above or below the zero-crossing line. In those cases, that would be when you'll use the DC offset to "push" the waveform to where it needs to go.
Also, some fairly aggressive wavefolders can create waveform deformation to such a degree that it would SEEM as if there's noise present in the signal, but the issue really stems from some sort of very tight and very high-amplitude "scrunch" that's going on in part of the waveform. Ultimately, the "fix" is to simply note where the "don't" position is on the "Inject", then don't put the knob there...unless you WANT the noise, in which case...