A report in the UK’s tech magazine and website
the Register has suggested that web host 123-Reg has been accused of deleting some of its customers’ websites. The report suggests that a "Rogue Script" was responsible for the company’s issues.
123-Reg, which has headquarters in Hayes, Middlesex, United Kingdom, was established in 2000. Managing over 3 million domains, the company is recognized as one of the UK’s leading registrars. The company prides itself on having “drawn together a range of easy to use tools and services to help our customers to get online, create their first website, and make a success of their business online”. The company caters for around 800,000 active websites.
On 16 April 2016 123-Reg reported "VPS connectivity issues" on its support page. These issues had cause a service failure for some of customers. However, more seriously, a posting on the blog of
InnMaster – a “cloud-based financial management and stocktaking service designed to be used by restaurateurs, licensees” and a 123-Reg customer – suggested the following:
“At 7am on Saturday morning someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers! This is a major embarrassment for a large web hosting group and judging by what we have seen on Twitter it has left some of their customers who had no backup or emergency plan with no website and potentially no business any more! – We can assure you, we are not one of those!”
The blog posting continues by suggesting the company makes daily backups and has a disaster recovery procedure that it was following:
” Our system is now up and running from our backup server – this is a short term measure and although we do have a backup server for this backup we don’t want to be running on “the spare tyre” for any long period of time! An automatic backup of our database ran at midnight on Friday, so if you entered any data between midnight on Friday and 7am on Saturday this may need to be re-entered.”
The incident appears to be very similar to a recent news item where a web host purportedly deleted all his servers’ data using a single string of code. That turned out to be a hoax and some believed InnMaster’s news might be a marketing stunt off the back of this hoax. However, online magazine
Softpedia confirmed the news to be true:
“UPDATE: Ian Hamilton and James Tanner have told Softpedia that 123-Reg has actually deleted all VPS servers and had no way of restoring their data from backups.”
Softpedia’s news item points to an email sent to 123-Reg’s customers:
“So what happened to some services? As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore. What have we done? We have been working with an extended team of experts and have left no stone unturned. Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.”
At present 123-Reg there is no indication as to the extent to which 123-Reg’s issues have been resolved but the company must be applauded for its open communication policy in what must be a difficult time for its team members.
Do you know of any other companies having technical issues? Let us know the details. Add your comments below.
