First, you can be 100% sure the problem is NOT GSX, since all its memory allocation happens outside FSX, since the whole program is running under the Couatl interpreter which, being a separate .EXE file, is NOT taking from the FSX memory pool (up to 4GB), but has ITS OWN 4GB, entirely available for itself, which on a 16GB system like yours, are comfortably available.
So, your problem is surely caused by another addon, that put FSX on the borderline of memory exhaustion so, a small bit more memory requested by creating a GSX object (that's the only memory that GSX allocates in FSX, the whole program code is running in another dedicated memory space), ended up in OOM. But you were already dangerously close to memory exhaustion.
I think I've read some posts on non-sim related forums about some video memory settings for users with 3 screens, in order to better control VRAM allocation, but I really don't remember the name of the tweak used, it was something in the nVidia settings and/or using a tweaker.