Boarding time for passengers is controlled by the Passenger Density, the higher the density, the shorter is the boarding, with of course a bigger impact on fps.
The loaders carts will take a bit more to drive around the airplane, because its cargo door is on the left side, but that's shouldn't add more than 10-15 seconds so, I'm sure this is not the problem.