January 26 2019

Doubting backups. If something can go wrong, it will go wrong.

A case study that reminded us once again how important it is to have a disaster recovery plan.

When you write on a corporate blog you always tend to have that pun in a corporate / institutional style, telling everything and nothing with cold and really not very compelling words. Unlike the big American corporations and the small companies that mimic the big corporations, we want to share in an empathic way the professional experiences of real life that will allow you to live real situations in the third person in order to learn from the experiences of others. Because in a world made up of companies, customers and suppliers and mere VAT numbers, there are stories of people who deserve to be told.

Today I want to talk to you about backup and data security telling you what I have and have to say about an all too underestimated problem.

If something can go wrong, it will go wrong.

This pseudo-scientific axiom better known as Murphy's law was the cornerstone of the organizational processes regarding the management of security and backups.

With this awareness in the last 6 years we have evaluated and implemented robust and proven backup solutions to ensure integrity of customer data.

Operating with the old Dreamsnet.it brand since 2005 and with previous systemic experiences since 2000, up to now we had always used an incremental snapshot backup solution that allowed a selective restore of both individual files and entire images. At the database level, for example, we used (and still use) tools that allow you to take snapshots quickly and respecting the logic of hot backups and integrity, that is to backup the DB without having to turn it off and making it always available the service even if it happens at 4 am (maybe the users are asleep, but the search engines index).

In short, not exactly the last of the class, if we think that today we have famous Italian hosting that still use the ancient and ancestral mysqldump. Really unbelievable.

In thirteen long years, in short, and after several hundred restored backups, no customer has ever lost a single file of his projects.

In the summer of 2018, something new, unexpected and upsetting happened that would forever question ourselves and anyone else after that ugly misadventure.


It was a sunny day, warm but not muggy, short-sleeved T-shirt and shorts with the notebook over the shoulder, I had just finished lunch at the river park that borders the house in Arad in Romania. I remember I was walking back in office at the Arad Business Center, along the beautiful bike path of the city very popular at that time by all those who like me were on their lunch break.

The view towards the river in full tranquility inspired peace and tranquility, the sight of people relaxed on the benches conveyed calm and positivity, I was literally immersed in that moment of bliss when the mobile phone starts ringing who was hooking up a call forwarding directly from the main office.


Return to planet earth, the one made of other people's problems to solve (after all, work generally has this purpose, right?) and I respond with a warm response

Hello, I'm Marco from Managed Server, how can I help you?

A male voice replies, it was a male, a boy roughly my age, between 30 and 35 I would have estimated by ear. We immediately overcome the formalities of the Italian language and put ourselves at ease by calling each other immediately (by the way, on the net and on social networks we call each other TU says Netiquette did you know?)

Start with a bit of agitation to talk to me about Backup, restore restore, failed restore, lost data and data recovery. Too many abstract and confusing concepts, too many fragmented inputs, and a lot of confusion in my head. I do not understand.

Who is the one who is phoning me? Is he our customer?

What happened?

Talk about backups, but do you need to restore a backup? Lost a backup? Don't have a backup?

Could it be one of those kids who can't tell the difference between a washing machine and a VCR?

I immediately stop this flood of random terms and phrases and ask you to calmly explain everything that has happened from the beginning, and ask me what your needs were.

Let's start again much more calmly and finally establish a discussion made up of sentences that make sense and above all logically connected.

In simple terms

This guy, hosted by a well-known French Hosting company, said that the night before he had made a mistake with the production of some FTP files and taken by frustration in solving the problem he had decided to make a "clean sweep", or delete everything And restore a backup to the day before.

Therefore, he is about to connect with his FTP client, delete all the folders of his site and once finished, from the customer area, execute the backup restore procedure from the comfortable Web interface.

Four clicks and you're done!

After a few minutes of progress of the task, a message announced the completion of the restore operation.

Oh, joy, glee and jubilation! What better news?

What happened is easy to understand and as you have already guessed, the backup was damaged or rather EMPTY. The restore wizard had restored exactly ZERO FILES leaving the destination directory completely empty, instead of the files from the previous day.

He doesn't get discouraged, doesn't give up, and tries again. Maybe maybe something went wrong.

Same procedure, same archive, same restore message. Then you check with the FTP client and… EMPTY. Again no file. Nothing, Nada, zero, nisba.

Return to the interface and select the backup of the day before, same restore procedure. Same successful completion message, same result. EMPTY.

Select the backup of the day before the day before. Restore. EMPTY

And so on with the backup of the day before the day before the day before yet. 3 days before, 4 days before, 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30, XNUMX days before.

Same result: EMPTY.

The archives to be restored had run out, and the conclusion was only one: none of the 30 backups up to 30 days before had been possible to restore. This person had no way to restore backups of him, site of him feeding him.

The problem had become something really big as it was apparently without solution.

At that moment I felt a mix of emotions between them. Sadness mixed with joy.

You know when you experience some bad event directly or indirectly in which out of empathy you feel sadness about what happened but deep down also a pinch of joy in knowing that that bad thing didn't happen to you? Here, just that feeling, that mix of emotions at the antipodes, which collide with each other.

I was happy that he was not a customer of ours, I would not have been able to face such a situation in which the backup service exists but none of the archives can be restored. What could I have said to a PERSON who lives on the site and feeds their family who was on the phone with me at the time?

I immediately thought that this moral ethical problem, that sense of responsibility would not have affected the huge hosting company of which this person was a customer in the least.

In the end it was clear how the story would go: this person would have spoken to the telephone support, some guy put there a few coins a month, would have forwarded the problem to the Italian technical department who would have forwarded it to the French one and one of his technicians . The technician would probably have noticed that something was wrong and would have sent the message back to the Italian technical support who would probably have stated the lack of data and possibly asked for a refund or open a legal dispute.

Legal dispute. In Italy. Already here it would be funny, if it were not the tragic situation. Instead, it makes you cry when you think back to the stipulation of a contract and an SLA (written in such tiny fonts that they should be illegal), in which the supplier company indemnifies itself from any damage of this kind and any request for compensation.

It would have been easier to climb Everest with bare hands. (Okay, the one in the picture is not Everest, but it gives the idea).

The CEO of the company would never even know about the affair of this father of a family desperate for having lost 5 years of work. One of the many among millions of customers, a drop of the ocean. What do you want a problem like this to be for a company that makes hundreds and hundreds of millions of revenue a year with millions and millions of customers? Absolutely insignificant. Unfortunately.

What could we do?

Nothing. Faced with that complex situation and absolutely beyond mine and our margin of maneuver (the supplier was another, we didn't know what backup system he used, what the storage system could be, how to have access to the media, and why had this bad mishap happened), what could we do?

The most correct thing to do was obviously to ask for a comparison with their technical support to understand if there was any possibility of recovery, and in the meantime to look if CASOMAI had a local backup on your PC maybe a few days or months before.

The first suggestion did not give any positive results, in fact the support limited itself to saying that there was no file within the backup that had been done correctly anyway. This affirmation was also confirmed by their top-level French technical support after about two weeks who simply liquidated with a "there is nothing“, An absolutely grotesque and unpleasant situation.

A little better instead for what concerns the advice to look for a backup locally, since actually with a lot of dedication and various rummaging through the files of his PC he managed to find and restore a backup of a few months before, which although it was not was the optimal situation of how to repair such damage, however, it allowed this person to get back on track and avoid bankruptcy inevitable if this were not the case.

All's well that ends well!

The lesson we have learned

Blackboard Lesson

In all this ugly history, our role has been absolutely spectator. Absolutely irrelevant, in short, for the outcome of the story fortunately resolved if not for the best, certainly not for the worst.

All this, however, gave us the opportunity to learn some important considerations hitherto underestimated. Essentially we asked ourselves the following questions:

1. Why can't what happened to them happen to us?

Why should a backup system fail only happen to other hosting providers and not to us?

It would be hypocritical right? It's like saying that we could easily avoid not wearing seat belts when we drive because accidents happen only to others, don't they? Instead, the most correct and appropriate answer to the case is that until then, that memorable summer day, the only reason it didn't happen to us was chance, pure and simple luck. In short, the fact that a very advanced snapshot backup system has never generated corrupted backups. Certain remote eventuality, improbable, but not impossible as we saw on that very day.

2. How much does data loss affect people's lives?

Losing a website or data can spell the end of a business. This means creating an economic damage on people's lives, probably putting them in a position not to be able to afford goods of primary necessity. This cannot and must not happen, at least it must not happen through our fault, neither as a cause nor as a contributing cause.

3. How much does data loss affect our company?

The loss of a customer's data can most likely mean a legal dispute. It is irrelevant to be wrong or right or to go and see the contracts for the various indemnities and SLAs signed, there are clear legal obligations such as those imposed by the new European GDPR law that would see us accused of various omissions and therefore condemned to fines and compensation. . Better to invest in data security by allocating 10% of the turnover on monitoring systems, RAID systems, redundant storage, multiple backups, rather than risking courts, litigation, compensation and fines.

What did we do then?

Having understood the three points above loud and clear, we “simply” added a secondary one to the already functioning backup system, in turn redundant on a C14 data storage system with military grade certification in France.

In short, if before on our RAID1 systems we had only one backup that flowed into a storage area in RAID5, today we have three backups with two different technologies that converge on three different RAID5 storage systems and in turn one of these is mirrored on a service from C14 military grade secure anti-nuclear storage redundant in France.

In short, in this modus operandi, an unfortunate customer who needs to restore a backup can rely on a remote backup which, if it doesn't work (a very remote hypothesis), could rely on the second backup system, which in turn, if it doesn't work ( hypothesis itself very remote) could rely on mirroring with Rsync on another storage.

If there is a major attack (let's assume a nuclear explosion) on the datacenter, we would still have a weekly mirroring on the C14 anti-atomic storage in France.

In short, the possibility of losing customer data has really been reduced to an almost impossible limit. More than these precautions do not currently exist to our knowledge in Italy but also internationally, seen and ascertained that even very large companies with turnovers of hundreds of millions of euros continue to use a single and unique backup as a definitive solution to protect themselves itself and its customers.

In terms of internal policies (as we have always done) we continued to offer the Backup service included in the offer. It does not exist that it can be an additional value-added service with a separate price. THEWe offer backup included, it will be our care to protect customers in the best possible way, regardless of their optimism that nothing will ever happen to their data, or that it won't help.

In short, atomic bomb proof!

nuclear explosion

Do you have doubts? Don't know where to start? Contact us!

We have all the answers to your questions to help you make the right choice.

Chat with us

Chat directly with our presales support.


Contact us by phone during office hours 9:30 - 19:30

Contact us online

Open a request directly in the contact area.


Managed Server Srl is a leading Italian player in providing advanced GNU/Linux system solutions oriented towards high performance. With a low-cost and predictable subscription model, we ensure that our customers have access to advanced technologies in hosting, dedicated servers and cloud services. In addition to this, we offer systems consultancy on Linux systems and specialized maintenance in DBMS, IT Security, Cloud and much more. We stand out for our expertise in hosting leading Open Source CMS such as WordPress, WooCommerce, Drupal, Prestashop, Joomla, OpenCart and Magento, supported by a high-level support and consultancy service suitable for Public Administration, SMEs and any size.

Red Hat, Inc. owns the rights to Red Hat®, RHEL®, RedHat Linux®, and CentOS®; AlmaLinux™ is a trademark of AlmaLinux OS Foundation; Rocky Linux® is a registered trademark of the Rocky Linux Foundation; SUSE® is a registered trademark of SUSE LLC; Canonical Ltd. owns the rights to Ubuntu®; Software in the Public Interest, Inc. holds the rights to Debian®; Linus Torvalds holds the rights to Linux®; FreeBSD® is a registered trademark of The FreeBSD Foundation; NetBSD® is a registered trademark of The NetBSD Foundation; OpenBSD® is a registered trademark of Theo de Raadt. Oracle Corporation owns the rights to Oracle®, MySQL®, and MyRocks®; Percona® is a registered trademark of Percona LLC; MariaDB® is a registered trademark of MariaDB Corporation Ab; REDIS® is a registered trademark of Redis Labs Ltd. F5 Networks, Inc. owns the rights to NGINX® and NGINX Plus®; Varnish® is a registered trademark of Varnish Software AB. Adobe Inc. holds the rights to Magento®; PrestaShop® is a registered trademark of PrestaShop SA; OpenCart® is a registered trademark of OpenCart Limited. Automattic Inc. owns the rights to WordPress®, WooCommerce®, and JetPack®; Open Source Matters, Inc. owns the rights to Joomla®; Dries Buytaert holds the rights to Drupal®. Amazon Web Services, Inc. holds the rights to AWS®; Google LLC holds the rights to Google Cloud™ and Chrome™; Microsoft Corporation holds the rights to Microsoft®, Azure®, and Internet Explorer®; Mozilla Foundation owns the rights to Firefox®. Apache® is a registered trademark of The Apache Software Foundation; PHP® is a registered trademark of the PHP Group. CloudFlare® is a registered trademark of Cloudflare, Inc.; NETSCOUT® is a registered trademark of NETSCOUT Systems Inc.; ElasticSearch®, LogStash®, and Kibana® are registered trademarks of Elastic NV Hetzner Online GmbH owns the rights to Hetzner®; OVHcloud is a registered trademark of OVH Groupe SAS; cPanel®, LLC owns the rights to cPanel®; Plesk® is a registered trademark of Plesk International GmbH; Facebook, Inc. owns the rights to Facebook®. This site is not affiliated, sponsored or otherwise associated with any of the entities mentioned above and does not represent any of these entities in any way. All rights to the brands and product names mentioned are the property of their respective copyright holders. Any other trademarks mentioned belong to their registrants. MANAGED SERVER® is a trademark registered at European level by MANAGED SERVER SRL, Via Enzo Ferrari, 9, 62012 Civitanova Marche (MC), Italy.

Back to top