I understand that the scenery library is the place where all these data files are kept in bgl form, but how do I know which ones to get rid of, in which folders, etc?
Only you can possibly know what you have installed in your FSX. Unfortunately, there's no way to automatically telling which .BGLs contains mesh data. The FSX Airport Scanner utility search for airports, but there's no similar utility to search for mesh files.
So, the only way is to disable each item in the Scenery Library, one by one, and proceed by exclusion, until you find the offending one.
And of course, you mesh settings should be set as explained in the KDFW manual, in the "Design Notes" paragraph.