There are a number of ways to send items in envelopes.  C5, B5, B4 or C4 cost different amounts for different weights.   You SHOULD have a set of scales, even kitchen scales to check the weights before posting.  Ask the PO for the gauge they use to check thickness of the envelope before sending.  Maximum thickness should be no more than 20cm.  

To flatten out clothing you can place in cliplock bag and press out the air before sealing.  This seems to work for many and is probably less weighty/messy than the other alternative which is to put clothing between card and sticky tape.  However there are others who will share with you tricks/tips on how to send things such as lightweight clothing.  

Cost for prepaid envelopes purchased over the counter at the PO will vary according to size (C5, C4, B4).  Prices have just gone up for these so you will need to check the Aust Post website or get the Post Charges booklet from the PO.

What is likely to happen if your envelopes are picked up at a Distribution Centre for being too thick is (a) if you have your return address on the back you will receive an invoice from Australia Post, for the underpaid amount of postage plus $1.50 admin fee.

So if you put 3 stamps on the envelope and it is over 20cm thick you will be charged parcel rate (less the cost of the 3 stamps), plus the admin fee.  (b) if you haven't put your return address on the back your buyer is likely to be charged the extra.  (So would not make for a happy buying experience).  I would keep my fingers crossed that they send you the bill rather than your buyer getting hit with it, as they could also refuse to accept the parcel.

The weight factor can be anything up to 500g with Prepaid Envelopes (as above) but still need to be less than 20cm thick.  There is always the Click and Send option which I am sure someone who is more familiar than I can advise you on.

The PO could have enlightened you right at the start about their requirements for sending items in envelopes, but then you should have made the appropriate enquiries and avoided what might be an expensive mistake.

A bit slow on the typing.  Questions seem to be already mostly answered.