The Novell server under the desk that nobody was allowed to reboot

In 1997 I took a job looking after the systems at a mid-sized firm in Hobart. On my first day the office manager walked me round, pointed under a desk in accounts at a beige tower with a handwritten sign taped to it, and said: "That one runs payroll. Nobody touches it." The sign said DO NOT TURN OFF in capitals and was signed by a man who had left the company two years earlier.

It was a NetWare 3.12 server. It had been up, as far as anyone could tell, for well over a year. Nobody knew the supervisor password. The documentation was the sign.

I asked if we could schedule a reboot to check it would come back. I was told, in the tone you use with a child, that it was too important to reboot. I wrote down in my notebook, "payroll box, no password, no backup tested, no reboot in 600+ days". Then I got busy with the forty other things that were on fire, as you do.

Eight months later, on a Friday afternoon, a new office manager decided to tidy the cables under the desks in accounts. Lovely job she did. She unplugged everything, bundled it neatly with cable ties, and plugged it all back in. Including the payroll server's power cord.

It did not come back.

The disk had spun for years without stopping, and when it cooled down, the heads stuck to the platters. Stiction, we called it. Old drives did that. I spent that Friday night on the floor of the accounts office with a torch. In the end I did the thing every sysadmin of that era will recognise and none will admit to in writing: I took the drive out, held it firmly, and gave it a sharp twist in the plane of the platters. It spun up. I did not turn it off again until every file was copied to the new server, which took until Sunday.

Payroll ran on Tuesday. Nobody outside IT ever knew how close it was.

What I took from it

• If you are afraid to reboot it, that is exactly why you must reboot it. Planned, at a quiet time, with a tested backup and a plan for when it does not come back. Better to find out on your terms than on a Friday afternoon on someone else's.
• "Tidy the cables" is a change. So is "move the desk" and "just swap the power board". If it touches something important, somebody responsible should know before it happens. We did not call it change control then. We do now.
• Write the passwords down, properly. In 1997 that meant a sealed envelope in the company safe, signed across the seal. Today it means a password manager with an emergency access arrangement. Either way, the knowledge must not walk out the door with one man.
• Documentation that is a sign is not documentation. What it does, who owns it, how to restore it, written where the next person will find it.
• Old hardware fails when it cools down. Still true of anything with a spinning disk or tired capacitors. If you have a box that has run for years, expect it to fail on the first restart, and plan for that.

The new office manager and I became good friends, as it happens. She never touched a cable again without asking, and I never again left a "no reboot" server alone for eight months because I was busy.

Cheers, G.