I currently have DX12, so I would have to downgrade to DX10 to save some memory?
You don't have DX12 in FSX. Even if your OS and video card and drivers supports DX12, in order to use it, the game/app must support it. And FSX, being a program made in 2006, by default runs in DirectX 9 and, optionally, is possible to set it to work in DirectX 10, which might have a bit of compatibility issues, but mostly with sceneries originally made for FS9.
So yes, you should UPDATE to DX10 ( in the FSX graphic settings ), to save some memory, because normally it uses DX9.
As far as add-ons go, I don't really have any addons other than Imaginesim KATL. I don't have any payware aircraft and the only other programs I run with FSX-SE are related to VATSIM and OBS since I livestream my flights on YouTube.
You mean you are flying only with default airplanes ? Well, that's weird but, on some sceneries it still possible to go low on memory with a default airplane.
Based on my system specs, which version of Prepar3d would you all recommend?
I think your system could run MSFS 2020 and, considering it the Standard version can be downloaded for FREE if you have a Game Pass subscription, my suggestion would be to start a 1 month Game Pass subscription, which costs about 10$ and TRY MSFS first, see if you like it. Note that, you'll need to purchase GSX Pro.
If your system can't handle MSFS, or you don't want to switch to it for any other reasons, a suggestion about P3D is not the easiest thing to provide right now because:
P3D V6 just came out, but lots of developers have been slow to release updates to support it (we supported it the day it came out), so you might need to go through some hacks to install products that hasn't been updated yet.
P3D V5 has likely the best support and lots of add-ons and supports DX12, so it also has the best performances. However, with an 8GB video card, since DX12 is quite picky with VRAM, you might risk crashes as well, not because of RAM, but because of not enough VRAM, and you can't select DX11 in P3D5, which is usually more reliable.
P3D V4 is the version I liked most, because it's very reliable and it works in DX11, and there we have some of the more advanced features in GSX, like Render To Texture. However, it's slower because of this, and it might not be a good idea to purchase something made in 2017 today.
And, since updates from major versions require a new purchase, if you buy an older version like V4 or V5 and later on want to update to V6 (maybe because more addons would have been updated to support it), you'll have to buy a new license.
That's why, I think the easiest upgrade path for somebody using FSX until today, would be MSFS 2020, which is the most supported by 3rd party developers right now.