Dear friends, a story from 1984: a power cut, 40,000 punched cards and the word CHECKPOINT

Dear friends,

My grandson tells me that nobody reads long posts any more. I have told him that I am 62 and I have time, and so, kindly, do you. This is a story from my first year in the bank, and it has a lesson at the end which I believe is still useful, although the machines have changed beyond recognition.

I joined the data processing department of our bank's head office in Madras (it became Chennai only later) in 1984, as a trainee programmer. I had a degree in mathematics and had never touched a computer. For the first four months I did not touch one either. I wrote COBOL on coding sheets, in pencil, in capital letters, and the sheets went to the punching section, where six ladies converted them into punched cards at a speed I have never seen equalled on any keyboard since. You collected your deck, held together with a rubber band, and submitted it to the operators through a small window, like a ration shop. The next morning you collected your listing. If you had made one mistake, you had lost one day.

Power cuts in Madras in those years were not an event, they were the weather. The machine room had a generator, but the switch-over was not instant, and the air conditioning was the first thing to go and the last thing to come back.

In my seventh month I was given my first real job: a program that read the day's transactions from cards (about 40,000 of them for our city branches) and posted them to the account master file on tape. The night I am speaking of, the power went at about 2.40 am, two hours into the run. The generator came on, the machine was restarted, and the operator, a very senior and very sleepy gentleman, did what seemed obvious. He put the whole deck back in and ran it again from the first card.

You can guess. Around 23,000 transactions were posted twice. Some customers in Mylapore were very rich for one morning, and some were very poor.

My section head, Mr. Venkatraman, did not shout. He was not a shouting man. He sat with me for two days while we reversed every duplicate by hand from the listings, and on the third day he gave me a sheet of paper with one word written on it: CHECKPOINT.

What he taught me, and what I have repeated to every young person who would listen for the next thirty years, was this:

• A long batch job must know how far it has got. Every N records, write down (on a separate file, which in those days meant a separate tape) the position you have reached and the totals so far. We used every 1,000 transactions.
• Restart must be a decision the program makes, not the operator. On start, the program looks for the last checkpoint and continues from the record after it. The operator should not need to know anything.
• Totals must reconcile at the end. Count of records in, count posted, count rejected, sum of amounts in, sum posted. If they do not agree, the job does not say "completed". It says something unpleasant, in capital letters.
• Posting must be safe to repeat. Each transaction carried its own reference number, and the master update refused to apply a reference it had already applied. This was the most important one, and I did not truly understand it until many years later.

Later, on the mainframe with CICS and DB2, all of this had proper names and proper tools. Today my grandson tells me the word is "idempotent", and that his Kafka consumers commit offsets, and I nod. It is the same idea. The machine will stop halfway. The power will go, or the network, or the cloud (which I am told is also somebody's machine room). The only question is whether your program knows what it has already done.

There is a Tamil saying: "Aatril pottaalum alandhu podu". Even if you are throwing it into the river, measure it first. Mr. Venkatraman never said it to me, but every reconciliation total I ever wrote was that proverb in COBOL.

With regards,
S. Ramanathan