You say the envelope was under 5mm.  Did the envelope freely pass through the small envelope slot in the letter gauge?  If AP deems a small envelope as not freely fitting through, then they will consider it a large letter.

Having said that, the maths still don't add up in this instance.  Certainly, a mystery.