LADSPA vocoder in JACK Rack has the same problem. Being already pretty experienced in this regard (hehehe) I can say that this is because of an effect which I forgot how is called when the plugin stops receiving the signal but continues to try to process data and starts going into calculations of really small numbers and it triggers a software part of your CPU and it uses a lot of resources, these calculations. The developer should find a way to fix this somehow.