Yes but the OP said they are asking for "Personal Details"

When you add account numbers with address, phone number DOB you have proof of ID.  This can gain someone access to your account by phone banking.

I know this because I know someone who this has happened to.

You are right though, just the account number will do nothing however using it as part of proof  can be used as identity theft.