Perhaps modify your listings to exclude PO Boxes and remove Australia Post from your postage options if you don't use it. The few listings I looked at had AP as an option in the description and postage options

I have a PO Box and there is nothing to indicate I couldn't buy from you - my delivery address is my PO Box

Domestic 

• Australia Post regular letter (default)
• Australia Post express
• Sendle
• Local pickup
• Any other service of your choice