I haven't ground a gear *not* using the clutch either... it I miss a shift, it doesn't sound like you think. It sound just like when you miss a gear. There is metal to metal contact when you miss a gear as well.
Especially with these modern manual transmissions.