You have a defect just because the buyer was inconsiderate and opened a dispute instead of just asking where it was.

Even if you did refund immediately the defect is still there.