I fixed my issue with stuttering by setting maximum pre rendered frames to 1 in Nvidia inspector. Not sure if this will help anyone else
That's interesting to know. I guess that having it to its default value of 3 will consume more VRAM if things are constantly loaded/unloaded from memory, since the card has to keep storage for 3 frames ahead.
It's possible the default value of 3 has been chosen to suit more modern engines like Unreal or Crytek, when a lot of data is cached on the VRAM, so it makes sense to render ahead, but with what we are forced to do to save VAS, a lot of memory management is made on the CPU instead, since we cannot obviously act on the graphic engine itself, so it wouldn't be very efficient to pre-render "future" frames, with things that might appear/disappear at any time, without the video card having any control over it.