GSX doesn't do anything if you don't call it, and doesn't take any memory away from the sim, since it's running entirely external to it, being a Python program that runs inside the Couatl interpreter, which also loads it in the background, not stopping/slowing the sim in any way.
Until actual GSX objects are being created, which is happening only if you call GSX from its menu, it won't take away any resources from the sim.
The uploading time for my FSX takes longer due to the Airport CACHE.
This happens only when you install or remove a new scenery. Are you doing that often ? Or, are you maybe using a 3rd party utility that keeps changing the Scenery Library at each start ? Using that would cause the GSX airport cache to refresh at each start but, it's just a message on screen, you can still use the sim, just GSX won't be available again until the cache refresh it's done.
In any case, you can try disabling GSX from its menu, but if you have FSDT airports installed, it's best not doing that, since they won't work without it, especially the ones using SODE jetways (KIAH, KMEM, KCLT and KSDF)
