An option exists for you to 'Exclude P.O. boxes' when you create a listing... so it might be useful to use that in the future. I have a PO box as a primary address, and whenever I bid on an item with 'Exclude P.O. Box', I have to confirm that I'm able to get it sent to a street address. This would seem to be the simplest option to avoid confusion.

As for what to do now, I agree that the best thing to do is absorb the extra cost to avoid a defect.