Having read-up on the subject, I believe this might be due to the solar charger being a PWM type - not very efficient. I've just ordered what claims to be a MPPT solar charge regulator (much more efficient).
MPPT does not make the charger more efficient. It loads the solar panel in a way that draws the maximum power that the solar panel can provide for a given level of solar radiation. The only way to verify that the charger is doing that is to measure the solar power being received by the panel and comparing that with the power being delivered to the charger by the panel. Then determine from the panel's characteristic curves if the maximum power is being delivered for the measured level of solar radiation.
The efficiency of the DC to DC power conversion by the charger is another issue. The efficiency is likely to be highest when the input power is highest. When purchasing a charger, you can only really determine how efficient it will be if that is stated by the manufacturer. It efficiency is stated at all, it will likely be only the best efficiency that can be realized, not the efficiency for every operating condition.
You also must consider limits that might be put on the charger by the battery. If charging the battery too quickly will shorten the battery life, operating at the maximum power point of the solar panel might not be the best strategy.