The FSX airport standard doesn't support anything like a "baggage handling facility" location and, even if it did, we doubt it would ever work with 3rd party sceneries, when some developers even forgot to add things like vehicle path lanes...
So, the only thing GSX can do, in order to not unrealistically have such vehicles pop-out in front of you, when they are supposed (exactly) to come from "somewhere" in the terminal, they start on the nearest parking of the "Vehicle" type, which is obviously the most logical choice.
If you use a 3rd party scenery the original developer forgot to place *any* parking of the "Vehicle" kind, or if the nearest parking is too far, GSX will make them start from a random parking which is reasonably close, but not too close that you would see the vehicle pop-out in front of you, precisely for the reason you mentioned, they should come from away, but not too far away.
So, it seems that GSX is doing exactly what you are expecting...