I never thought of 'buttoned down shirt' as being an item that has a buttoned down collar. I assumed that the term meant a garment that buttoned down the front. I think of a blouse as something that doesn't necessarily have buttons down the front. So I have always used the item specific 'buttoned down shirt' for all my shirts/blouses that button down the front. It probably doesn't matter, as who searches for 'buttoned down shirts' anyway?