Ha ha, yeah, I was thinking just the same thing today. Even though I'm aware of my time, the sound causes some sort of weird panic, and I freeze, giving away about three seconds. I think it might be the type of sound rather than the fact that there is a sound. It's a strange chipping noise that works like chinese water torture, boring into your brain.
I think instead that there should be some sort of soothing voice saying things like "Relax , you still might win" or "It's only a game of chess, who's going to remember this in even five minutes" Or maybe "I love you, even in time trouble"