Creating Scenery cache takes 20 minutes!

[gdavej]

I have GSX installed in P3D v2.5. Everything works fine until I change the scenery, which, of course, forces GSX to recreate the scenery cache. This recreation takes a very long time. Last night I turned on the ORBX Oceania scenery, and when I started a flight, GSX took over 22 minutes to recreate the cache. I checked the log, and there were no errors being shown, although there were a large number of messages about bgl files being very large, and thus could probably be ignored. During the cache recreation, the banner at the top of the screen reports the progress, and effectively blocks out the optional messages from the sim about fps, heading altitude, speed, and position. Since I like to use this information, I have to wait until the banner disappears after the cache has completed, ie 20 minutes or more. And of course, if the flight I want to start is cold an dark parked at the terminal, then I have to wait over 20 minutes before I can start to use the features of GSX. This means that I don't change the scenery very often.

I'm currently using the beta version of Coautl, but this cache creation time has been a problem since I purchased the software when I was using the production version of Coautl. Of course the time to recreate the cache has grown with every scenery add-on that I've installed.

I know that this isn't a typical time for cache creation, but also I've seen that it is a recurrent problem in the forum. If these excessive times can't be drastically reduced, then I don't see any option other than removing the add-on, since there is no way to temporarily disable it, and yet keep the other FSDreamTeam addon active. Obviously I'm not happy at removing software that I paid a lot of money for. Hopefully you can provide an alternative solution.

[virtuali]

GSX took over 22 minutes to recreate the cache. I checked the log, and there were no errors being shown, although there were a large number of messages about bgl files being very large, and thus could probably be ignoredπ

22 minutes to create the cache is unheard of. The most I ever saw on my system is like 40-50 seconds, some users reported 3-4 minutes.

You said you checked the log. Is the LOG saying it took 22 minutes, or you just timed it/guessed it ? Because, we found an issue (which will be surely fixed in the Beta), that displays the progress MESSAGE longer than it *really* took to regenerate the cache ESPECIALLY with a very high number of sceneries installed.

The real regeneration time is the one indicated in the log file, but if the message stays longer than that, it can be confusing because you think you can't use GSX (because the messages says so), when in fact it's ready.

So, first check what the time was in the Couatl.LOG file. That's the real time it took on your system, and if it's really 22 minutes, there's something really wrong happening (unless you have *hundreds of thousands* of scenery files, but then you have a far bigger issue than GSX, since such high amount WILL affect your VAS JUST because of their number, since a "directory" of available file names is always kept in memory), but if the time in the log is much shorter than it took for the message to go away, then it's just the issue with the message, which we already fixed in our internal version, so it will appear in the next Beta update.

[gdavej]

You said you checked the log. Is the LOG saying it took 22 minutes, or you just timed it/guessed it ? Because, we found an issue (which will be surely fixed in the Beta), that displays the progress MESSAGE longer than it *really* took to regenerate the cache ESPECIALLY with a very high number of sceneries installed.

Just checked the log and that gave a timing of:

Airport cache regeneration completed with success in 337 seconds

the time I quoted was from a stopwatch. So I guess the message was displaying way longer than it should. But even so, 5-6 minutes is a bit long. I do have a lot of sceneries, the scenery.cfg file has 388 entries, when Orbx FTX Global is active (455 when Orbx Oceania is active), but I don't think that's excessive.

Have you ever considered having the option of having the scenery cache generated outside FSX/P3D (ie something like FSTramp)? Just a thought, but it would make it less intrusive for users like me.

[virtuali]

Airport cache regeneration completed with success in 337 seconds

Ok, it's 5-6 minutes then, which is a bit long, but surely much better than 22 minutes. So, it's really just the problem of the message, which it's basically fixed now.

But even so, 5-6 minutes is a bit long. I do have a lot of sceneries, the scenery.cfg file has 388 entries, when Orbx FTX Global is active (455 when Orbx Oceania is active), but I don't think that's excessive.

No, it's not so much, I have 282 areas, and tried to force a complete regeneration, removing *both* the GSX and the FSX cache, and the GSX regeneration took 2 seconds, less than the FSX one. Are you sceneries installed on an SSD or regular drive ?

The number of files is also affecting the cache regeneration, not just the number of areas.

Have you ever considered having the option of having the scenery cache generated outside FSX/P3D (ie something like FSTramp)? Just a thought, but it would make it less intrusive for users like me.

The scenery cache is regenerated only if you change/add/remove sceneries. If there was an external program to regenerate it, you would still have to wait 5-6 minutes on another program before being able to fly or, at least, being able to use GSX. And, strictly speaking, the cache IS regenerated by an external program, because Couatl.exe it's an external program...

However, if you find the *message* to be intrusive, because you cannot see other notifications while the cache is being regenerated, you can simply turn down the "Message Verbosity" slider in the "GSX - Settings"->Simulation page.

This will remove the message. You will not be able to start GSX until the cache is ready (which of course would be the same if you had to start a separate program and regenerate on request), but no notifications from other programs will be lost, since there's no message. And, this will also fix the issue of the message staying for longer than it should, at least until we'll release the fix.

[gdavej]

[No, it's not so much, I have 282 areas, and tried to force a complete regeneration, removing *both* the GSX and the FSX cache, and the GSX regeneration took 2 seconds, less than the FSX one. Are you sceneries installed on an SSD or regular drive ?

The number of files is also affecting the cache regeneration, not just the number of areas.

I'm just using a regular hard drive for sceneries. It's probably all the Orbx stuff that's consuming the time. I have FTX Global Base, Global Vector and Global OpenLC Europe as well as a number of their airports. I'm sure there are a mass of bgl files here. So I guess I'll just have to be patient when I change sceneries.

Thanks for the advice about the top banner.

[virtuali]

I'm just using a regular hard drive for sceneries. It's probably all the Orbx stuff that's consuming the time. I have FTX Global Base, Global Vector and Global OpenLC Europe as well as a number of their airports. I'm sure there are a mass of bgl files here. So I guess I'll just have to be patient when I change sceneries.

That gave me an idea, like adding some kind of user-configurable setting, like a "blacklist" of scenery areas that shouldn't be checked, so GSX won't even try to look into there. Such terrain scenery might hardly contain AFCAD files, so they might just be skipped...

[toby23]

You could also just let the user activate all sceneries they own once, let the sim rebuild the scenery cache and then give the user the option to run the scenery database update as and when they add sceneries, would that work?

If you change FTX Regions often, as I imagine most people do, it is annoying to have to wait 5 minutes before every flight, I have never seen it faster than that.

[virtuali]

You could also just let the user activate all sceneries they own once, let the sim rebuild the scenery cache and then give the user the option to run the scenery database update as and when they add sceneries, would that work?

No, it MUST be automatic, otherwise, each time someone will report a problem with an airport, we would have to start any troubleshooting with a "have you refreshed the cache ?" question.

[toby23]

From a developer providing support point of view, i can understand but we all use a lot of add-ons that require manual updating of the scenery database, for example, VoxATC, Pro-ATC/X, FSC9 etc. These all require manual updates whenever there are changes to the scenery BUT they do not require updating UNLESS you install new scenery. If you just change FTX Region, there is no need to update the whole database again.

I just loaded a flight after deactivating unused scenery in my scenery.cfg, not adding or removing scenery files, and timed GSX, it took 7 minutes and 8 seconds to refresh the database. What is it doing? I only have 2 airports active... Even running MakeRwys is faster than this.

[virtuali]

we all use a lot of add-ons that require manual updating of the scenery database, for example, VoxATC, Pro-ATC/X, FSC9 etc. These all require manual updates whenever there are changes to the scenery BUT they do not require updating UNLESS you install new scenery. If you just change FTX Region, there is no need to update the whole database again.

That's exactly how GSX works too. It doesn't recreate the whole cache, only the files that are changed. The issue is, when the amount of files is very large, it takes time just to check if they *are* changed.

Again, as I've said in my previous message, the best solution might be doing an exclusion list, to skip folders that are known not to contain any AFCAD files, and terrain/mesh sceneries are usually like that.

I just loaded a flight after deactivating unused scenery in my scenery.cfg, not adding or removing scenery files, and timed GSX, it took 7 minutes and 8 seconds to refresh the database. What is it doing? I only have 2 airports active... Even running MakeRwys is faster than this.

Again, please read this thread. Do not confuse the issue of the duration of the MESSAGE (that has a bug, that we already fixed, and will be included in the next update), with the REAL duration. The REAL duration is the one indicated in the Couatl.LOG, and you can use GSX after this time.

And, as already explained in this thread, you can fix the problem of the progress message staying too long (thus blocking other notifications), if you put the verbosity slider down, so the message will not appear, at least until we'll release the fix.

This, unless the *real* time it took to regenerate the cache was 7-8 minutes in the Couatl.LOG too, which again is unheard of. The previous user, said that while the message stayed on screen for 22 minutes, the time in the log was way shorter, like 5-6 minutes.

[toby23]

Thanks Umberto, where do I find the Couatl.log file and is it then still wise to start the flight before the updater has reached 100% on screen notification?

[virtuali]

Thanks Umberto, where do I find the Couatl.log file and is it then still wise to start the flight before the updater has reached 100% on screen notification?

The Couatl.LOG must be enabled in the Troubleshooting section of the GSX Settings page, and it's located under %APPDATA%\Virtuali.

And yes, it's safe to start a flight, it's not as if the cache regeneration is blocking FSX or slowing it down ( it's done as a background thread so it can use a spare core on a multi-core CPU ), the worse it can happen is that, if you call GSX, it won't display its menu until the cache is ready so, even if you turn down the verbosity, as a temporary measure to not have the cache regeneration progress on screen, you would know if the cache is ready or not, if GSX works. We also added an additional notification in the next Couatl upgrade, that even if you have the verbosity down, you'll see a cache completion message anyway.

[pete_auau]

Don't know why users are complaining about  5  minutes   surely  you cant   get everything  ready  eg  set aircraft, set fmc    etc   under  5  minutes, go make  a cup of  tea if your bored

[virtuali]

Don't know why users are complaining about  5  minutes   surely  you cant   get everything  ready  eg  set aircraft, set fmc    etc   under  5  minutes, go make  a cup of  tea if your bored

The issue is, we are trying to understand why it takes so long for some users. The most I ever see on my system, it's 20-22 seconds, but it's usually 4-5 seconds.

But we are preparing an update to Couatl, that will indicate in the log how much time it took for each scenery folder to be processed, that might give us some hints if a problem are specific sceneries, or just their number, or the number of files in a folder, etc.

[virtuali]

Maybe we are getting somewhere....I managed to test cache regeneration times with both Orbx Global Base + Global Vector installed, and at a certain point, it seemed the cache regeneration was "stuck" for a long time.

With the help of the Microsoft performance analyzer tool, I found that, while the cache regeneration looked like being stuck, the Msmpeng.exe process was accessing to ALL the files in the Orbx folders. Since there are folders with tens of thousands of files, this took a while.

The Msmpeng.exe process is the Windows Defender real-time protection, which was scanning those files to be sure they weren't threats. Of course, as long as the antivirus is checking the files, it won't allow any other program to access them, that's why Couatl was stuck: it was WAITING for the Windows Defender to end scanning the files, before being able to read them. As soon as the antivirus finished the scanning, the whole OrbX APT folder (28.000+ files) was scanned by Couatl in JUST 6 SECONDS.

I added the whole FSX folder to the antivirus exclusions, and this seems to have fixed the problem so, if you had a very long cache regeneration time, try the same: regardless which antivirus you own, try to add the whole FSX folder to the exclusions list, which will likely have a benefit on loading times, regardless of GSX.

If you use a 3rd party antivirus AND Windows Defender, be sure you exclude the FSX folder in both products.