This past Thursday we had a major hardware failure that affected not only our main storage server, but our backup storage as well. We’ve spent the weekend trying to recover the data, but unfortunately have not been able to do so as of yet. We have now sent the disks to a data recovery service, that will diagnose and try to repair the hard drives.
We have managed to restore all data, meaning users, keywords, descriptions, folders etc. that you have created in Razuna, but the actual asset files have still not been recovered.
In the meantime, in order to get Razuna running, we have decided on an emergency solution for those who need to use the service. We’ve provided a tool that will allow you to re-upload files to Razuna. These uploaded files will inherit the data of the file that was already in the system. Razuna will automatically place the uploaded file in the same folder along with all meta-data – keywords and permissions, etc., that you previously defined.
New files will be uploaded and treated as such, so you can continue to use Razuna normally for new media.
Please log-in to your Razuna account where you will be presented with the emergency upload tool. Below are some more details in a form of a FAQ.
This past Thursday we had a major hardware failure that affected not only our main storage server, but our backup storage as well.
We use RAID 6 servers, so a failing harddisk is normally no problem. In our case, four disks failed at the same time, which means we can no longer run a repair to get the files back.
Don’t you have a backup?
Yes, we have a mirrored RAID 6 backup server, which had exactly the same failure at the same time. We use a mirror, so that if the main server fails, we can immediately point to the other and customers will then normally not see any downtime.
Unfortunately, we were hit with a failure of the main storage and the backup server at the same time.
What happens to our data?
What’s been affected is the files. The data such as users, folders, groups, meta-data, labels, keywords etc. has been successfully restored. The defective disks have been couriered to a leading data recovery company. They are now analysing them and will then try to restore the files.
Will I lose my files?
The honest answer is that we don’t know. This is unmapped territory, as a four-disk failure in a RAID 6 system is very rare.
If you have files that were uploaded to Razuna on your harddisk or other servers, we suggest you upload them via the emergency upload tool. Razuna will then map it against its records, and if you are re-uploading a file, all the metadata will be restored and the file will automatically be moved to the right folders and collections with the same asset-ID and permission levels.
When will you know, what can be recovered?
An analysis and diagnose typically can take from one day to many days. As we need to have 39TB analysed, it will probably take a few days. Once we have the result of the analysis, we will inform you about the next steps.
Will Razuna work normally going forward?
Yes. The Razuna service itself was never affected as it was not a software error. The Razuna service is now fully restored, including uploading files, will be handled as normal going forward.
What have you done to prevent this from happening again?
Well, theoretically it wasn’t supposed to happen to begin with. We are absolutely shocked. In order to prevent something like this from happening again, we have implemented a double backup. I don’t think there are a lot of companies with double backups, but that’s what we’ve done.
For performance reasons, we are still using a mirror with RAID 6, so a similar breakdown in one server, would not be noticeable. If two servers break down, we have a cloud backup via Amazon S3. So a similar, but still theoretically close to impossible, failure would mean that we would have down-time but that files would be recoverable.