An unprecedented incident involving Google Cloud resulted in the deletion of the entire customer account, including all backups, of the $135 billion Australian pension fund UniSuper. This deletion caused services for over 615,000 members to be disrupted for nearly two weeks.
UniSuper’s decision to use Google Cloud for its digital services was significant enough to warrant its own press release, with the fund outsourcing crucial infrastructure to the internet giant, a move similar to many companies’ reliance on Amazon Web Services.
While UniSuper stated that pension payments were not disrupted by the outage, customers were unable to log into their accounts on mobile or desktop. The fund’s outage update timeline suggests that transactions were also unable to be processed during the blackout, with a data rollback to the end of April required once services were restored. Fortunately, UniSuper had kept a backup of its cloud data through another service, which aided in the full resumption of its operations.
The exact cause of the incident remains unclear. The deletion of a multi-billion dollar client’s account and every backup, bypassing all safeguards, and knocking out financial services to hundreds of thousands of pensioners for two weeks is a scenario that no one wanted to occur.
A joint statement from Google Cloud and UniSuper described the deletion as resulting “from an unprecedented sequence of events whereby an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.” However, the statement’s vague language has left many questions unanswered.
Software developer Daniel Compton has pointed out that the statement does little to explain the incident on a technical level. He argues that one likely explanation was an error on UniSuper’s part when using a common cloud infrastructure tool. Despite this, Google Cloud seemed to accept some responsibility, implying that it was a bug on their end.
This incident serves as a reminder that no technology service is infallible, highlighting the importance of maintaining redundant backups of critical data.
Source: Ars Technica
