Exactly, it would be unrealistic to allow boarding of passengers while servicing the plane. In fact, if you paid attention, it IS allowed, instead, to board passengers when loading luggage, which of course is permitted.
In some cases, taking extra precautions, it might be possible to refuel with passengers on board, but it's a fairly rare procedure. Some airports requires the presenze of a fire brigade, for example. So, we assumed the most used case, and don't allow boarding passengers while refueling.