How do you know it was not damaged in transit? How do you know there was not a minor fault you did not notice? If it is the hand made hat are you absolutely infalible and couldn't possibly have got a easurement wrong?

As the buyer would have to return it at their own cost in order to get a refund what would be the point of them saying it was not as described if there was indeed nothing wrong with it?

Politely ask them what the problem is and ask them to supply photos. You can tell them they may return it and once you confirm the faut you will refund them in full, if there is no fault then you will refund them item price only.