I said I would refund as soon as he sent stamps back to me after they had arrived

And just how were they meant to do that if they never received them? On the rare occassions I send things as unregistered letters I refund after 10 working days.

I don't know if you are aware but buyers who use bank deposit so have no recourse through ebay (if they are buying on the Au site) can make a report through the QLD police site for auction fraud and without any proof of delivery you could theoretically end up in trouble although they are unlikely to do more than contact you for a relatively small, one off sum of money.

The feedback was nothing but factual in the first instance and your comment that you could not be blamed for AP is probably what incensed them and caused them to call you a scammer. You are responsibl for delivery and if you don't want this type of thing to happen you have to refund without question or registr.