I would think that the UK Passport Office, would have given a reason why it was rejected.
My wife has just got a new passport in her married name and they wrote to her explaining that the charge had gone up and they required extra money.
They should have given you a reason. If it is how you have filled it in, I would pay the extra charge and ask the Post Office to check your application.