by Paul Gibson
Home >> $38 Billion Mistake
“Hey boss, I accidentally deleted a file this morning. Nobody’s perfect, right? Rome wasn’t built in a day. Don’t worry we have a backup right?â€?
“Could you lend me a hand? I have a slight problem; I misplaced $38 billion… I could have sworn I put it in the first drawer, but now I can’t find it!â€?
Would you just blame it on somebody else and nonchalantly act like you are the hero of the situation: “Hey you guys, I just noticed that the files for the Alaska Permanent Fund are missing! Do you know where we can get the backup data, ASAP?�
Well this is really a true story which occurred in Juneau, Alaska. A computer techie was going through a routine reformatting of a disk drive and “accidentally� erased all the information of a very important account at the Department of Revenue, containing $38 billion in payouts. The account was funded with oil money, belonging to the Alaska Permanent Fund. The payout was imminent, but all the information for beneficiaries of the money (cash dividend distributions) was gone.
Amy Skow, Director of the Fund said they didn’t panic – they were just trying to look for 9-months worth of applicant information for the annual payout: close to 800,000 electronic images. All of the applications had been scanned into the system.
The situation seemed simple at the beginning. Just a few files missing – that was until they discovered the technician had also re-formatted the backup disk. Uh oh! Big mistake!
So now with no backup other than 300 boxes of paper documentation, the Revenue Service realised they were in a catastrophic situation – up the creek without a paddle, if you will. It took a team of 82 people working extra hours and overtime to meet the deadline, but the cash dividend payments went out on schedule.
The recovery lasted for six weeks and cost the revenue service an additional $220,000, including $128,400 in overtime and $71,800 in computer consultant fees.
The Revenue Service claims that no one was ever blamed for this incident. But it would appear that dividend checks will be docked an extra 37 cents to cover the additional cost of information recovery. All’s well that ends well, I guess.
Released under the following licence: Creative Commons Attribution-NonCommercial-NoDeriv
You are free to copy, distribute and display the contents of this article but you must give credit to and mention the original author. You are not allowed to use these contents for commercial purposes, and you may not modify them to make any derivative works.
For full licence description, go to: http://creativecommons.org/licenses/by-nc-nd/2.1/es/deed.en
Posted on http://www.weeklyletter.com at 2007-04-03 10:00:00 +0200
Copyright (C) ITT (http://www.itt.es) and Planet Lingua (http://www.lingua.es)
We have more weekly letters by Paul
Poll for this weekly letter
I have tried to cover up a mistake at work before.
Licencing
You are free to copy, distribute and display the contents of this article but you must give credit to and mention the original author. You are not allowed to use these contents for commercial purposes, and you may not modify them to make any derivative works.
(click the above link for more information)
Add a comment
Hello Paul,
Scary story, specially for those of us who work in the IT department :(
I think that this nasty things do happen.
Maybe, in order to prevent this $38 billion mistake the Company should have spent $1 million on some type of contingency plan. I think we should think ahead and spend some Euros in order to prevent these catastrophies and not leave it all to a human fault.
Regards,
Cristina
Thanks for your comments Cristina. I guess the real problem here was that their “backup” was not reliable. Many companies nowadays resort to “silos” to store their information, using automatic backup systems that automatically transmit information to another physical location for safekeeping.
There is really a rather large market here in Spain for these silos, I wonder if this is a good business idea..
Thanks again!
Paul
I agree…. a good way to avoid this kind of events it a good backup system…. But don’t forget: Murphy (the one of the Law…) is always there, waiting the best time to occure: if you have the backup, you won’t need it; if you have the backup and you need it, it wont work… etc.
We use two complimentary systems to backup our files: one of them, automaticaly programmed, is saved in other building, in other city,.. The other, in our building, a local copy…. I hope it covers every event!
Thanks for your comment Elena!
Murphy’s Law: If something can go wrong, it will.
Sounds like you have a backup for your backup: in which case you need a backup to back up your backups. When is too much caution superfluous?
This scenario reminds me of the foods at the supermarket that offer “extra Vitamin C”. Everyone thinks they are healthier for you, but they don’t realize that once their body reaches the appropriate level or requirement of vitamin C, it simply gets rid of the rest. Too much backup can be a costly mistake.
Great comments Elena!
Paul
Companies spend more and more money to avoid the impact of human mistakes but human incompetence is above all these measures :-)