backup

Backup Storage Solutions and Costs

Old Technology

More than a year ago, I wrote about some experiences I’d had with backup storage and thoughts about “off-site” back-ups. It prompted a couple of responses on the blog, and a few more on social media. Not the most exciting of subjects, but an important one, as I still believe that in this age of digital collections, backup will be a key issue. Just one that most people don’t spend much time thinking about.

To recap.

I use a lot of storage:

  • I shoot video and although I upload edits of what I shoot to Vimeo or YouTube, I like keep the original materials (in one instance this paid off, when I sold a shot to a company who’d found part of it on Vimeo).
  • I shoot photographs. Lots of photographs. Anything I take on my phone gets backed up to my Flickr and Google Photos accounts. (They also go temporarily to a Dropbox account.) But that doesn’t help for shooting with my cameras, where I prefer RAW. These large files end up on a local NAS drive once I’ve moved them off the PC where I edit them. I should note that I do delete files. If I shoot a batch of ten broadly identical photos, then assuming I like any of them, I’ll only keep one version. So for every 100 photos I shoot, I perhaps keep 10, and upload 5 to Flickr.
  • I have a digital music archive. I keep a local copy of it in iTunes – loathsome as that software is. And it’s mirrored on Google Play Music where they upped the limit to 50,000 tracks (I have more than 25,000 on there currently). I mostly play back music via Google Play Music either on my phone or via Chromecast devices. Beyond music, I have lots of other audio, both field recordings and radio recordings.
  • I have other legally purchased commercial video files – including iTunes downloads and DRM-free mp4 files.
  • I have a myriad of other documents, mostly much smaller in size.
  • I have several PCs and tablets, including the hard drives from a number of older computers, some of which files I’d like to have retrievable.
  • I have around 100 MiniDV tapes that I’d like to “rip” and store (A project for another day).
  • I have a large box of film photos that I want to scan at some point (Another project for yet another day).
  • Probably other stuff too…

While some files sit on various computers, I try to mostly copy files onto one of two Synology NAS drives. Cumulatively they have about 6TB of space, of which I’m using more than half. One NAS in particular will need its 2TB drives replaced by 3 or 4TB drives fairly soon (Yes, I Know 6TB drives are available, but I try to be cost efficient).

I also have a plethora of old hard drives and expansion drives that I have been slowly consolidating onto larger, newer portable hard drives (Three old small hard drives might now sit on one larger capacity drive).

Services I use that offer storage:

  • Google Drive. Through various promotions and purchases, including a Chromebook, an HTC phone, doing work for Local Guides, and running security check-ups, I have a cumulatuive 2.2TB of space. However, this space is not permanent, with 1TB disappearing at the start of next year when my Chromebook will be two years’ old (Plus, they don’t hand out 1TB for new Chromebooks now, in case I was thinking of upgrading).
  • OneDrive. Beacuse I have a paid subscription to Office 360, I get 1TB with this. In fact, I could have five accounts, each with their own 1TB. But that would be painful with multiple logins required. For those with lower usage needs, this offers a great deal however.
  • Dropbox. Just the basic free account with 2GB of space. Although I have occasionally gone on the paid plan for a podcast I work on.
  • BT Cloud. My ISP gives me 500GB because of the plan I’m on. I’ve never used it, and have heard it’s a bit slow.
  • Evernote Premium. Used for note taking and scanning. Not for files, but scanning magazine articles I want to keep and making them searchable, for example, does take space. But it’s not for files per se.

What I need:

Let’s say I need 6TB of space. It’ll probably more than that in due course, but 6TB is a good starting point. Let’s try to price that up at the various big vendors.

  • Google Drive is £7.99 a month per TB. So for 5TB of additional storage (1TB drops off my account in January don’t forget), we’re looking at £479 pa.
  • OneDrive is £1.99 per 50GB. So for 5TB of additional storage, it’d be £2388 pa although I feel sure that there must be most cost efficient ways of buying storage from Microsoft. I just can’t see them – aside from opening multiple Office 360 accounts which seems mad.
  • Dropbox seems to push me towads a Business plan at £110 pa for “As much as needed” (which may not include VAT). Still, that seems to be pretty decent value.
  • Amazon Cloud Drive has finally updated its UK offer and either gives Amazon Prime users their free “Prime Photos” service which is unlimited space for photos, and 5GB for other files including video, or their Unlimited Storage offer which is £55 a year, with a free three month trial.

There are of course lots of other providers which are and aren’t aimed at the consumer end of the market. And different not all cloud services are aimed at the same use cases. I’m not really looking for a shared working environment so much as reasonably priced archival space.

I’m looking for something that’s reliable, that allows me a clear understanding of cost, and doesn’t place many limits on what I store and how large the individual files are. While I’ve not shot many yet, a 4K video file lasting more than a few minutes is likely to take up an enormous amount of space. I don’t think I have any single file that’s larger than 4GB, but some services top out what they’ll accept at 5GB per file.

I also want to be certain that a provider I use is going to be there next year, in five years and in ten years.

Amazon has actually built an enormously profitable business offering cloud servers and storage. However, S3 storage pricing is a little opaque – essentially it’s based on how much access the files have. While Amazon Glacier is intriguing because it’s a very low cost backup solution at $0.007 per GB / month. It’s based around archival needs, and the data might take a few hours to become available once you’ve requested it.

It seemed pretty clear to me that Amazon Cloud Drive is the winner here. £55 for unlimited storage is a good deal from my perspective. But there are a few provisos worth getting into. This is for non-commercial use – they particularly seem to be concerned about people running photography businesses who no doubt generate lots of files all the time. I’m not running a business, but am an enthusiastic amateur. I assume I won’t be in trouble!

They’re also unsurprisingly worried about people sharing illegal files, or storing such files in the first place. I’d guess that this is not going to be home for your massive collection of torrents. That said, I’m not clear how they can determine a film bought legally and an illegal one. If I rip my DVD collection, can I store it in the cloud?

If you have large files, then you won’t want to use their web interface to upload them (But you didn’t want to do that anyway). And files with more than 255 characters or special characters, unsurprisingly, cause problems.

If you stop paying Amazon, they may delete your files. Kind of obvious, but worth knowing anyway.

To be clear, I’m using this as purely a backup service. There will be no files on Amazon that I don’t also have locally. Indeed the NAS drives I own are in RAID arrays themselves (although other files that will eventually reach Amazon may only have a single local backup).

Finally there’s the small issue of getting terabytes of data into the cloud in the first place. My BT Infinity 2 service is fibre-to-cabinet, and claims that it has no upload or download limits. But if I’ve got something like 6TB of data I want to initially back-up, that’s going to take quite some time.

At time of writing, I’ve only been uploading a 2TB NAS drive for less than 24 hours, and it’s not clear how long it’s going to take me in total. But it could easily be a full month before that first NAS is in the cloud. It does seem that Amazon “drip feeds” the files across. This is probably a good thing, because I don’t want all my bandwidth used up all the time as files fly back and forth.

Indeed I’m actually using a Synology app called Cloud Sync to do the backups, and that can limit the number of simultaneous files being uploaded. Furthermore, I’ve scheduled the syncing to stop at various points during the week when I’m more likely to want unfettered internet use myself. So I’ve got three hours in the morning and another six in the evening when it pauses syncing. At weekends, the syncing process will also be halted during the day.

To be honest, I was happily streaming Netflix while syncing over the weekend, so it’s probably not really necessary. But it lightens the load on the network during peak hours.

The good thing about using a NAS drive app is that only the NAS needs to remain on while the upload happens. Amazon does have a PC/Mac app for backing up your local PC, but there seems to be a question mark over whether it keeps your computer “in sync” with the cloud, or simply does a one-time back-up, requiring you to manually sync additional files at a later date. Either way, I’ve not bothered yet, since I’ll do one at device at a time. The good thing is that there are plenty of third party applications that will work with Amazon and will probably do what you need.

Amazon also has a mobile app which will dutifully send through all your photos and videos, but as mentioned above, I’ve got these covered already.

I will report back on the success of Amazon Cloud Drive and any issues I have with it in due course. In the meantime… let the upload continue!

Note: All prices and offerings are correct at time of writing. This is clearly a very fluid market.

Storage and Backup: Some Personal Experiences

[Note that this is likely to be duller than usual! It’s mostly written up so that anyone who searches for similar problems might find it useful.]

This all began last Wednesday when I noticed an email from Synology inviting me to update the firmware that runs that two NAS drives I own – a DS210j and DS214se. They’d just released DSM 5.2, their proprietary software that allows you to do all sorts of clever things with their products.

I’ve had nothing but good experiences with Synology. They’re reasonably priced, and they offer a safer way to store data than simply using an external hard drive.

So I ran the updates. My new DS214se updated no problem, but the older DS210j had a problem. It got to about 20% of the process and then stalled. And it wouldn’t restart. The status light was flashing orange which is not a good sign.

Now I was unable to get into the volume at all. I was concerned.

I had previously put in place a monthly “off-site” back-up. The NAS had 2 x 2TB drives in a RAID 1 array. That gave me a 2TB volume safely mirrored on the two drives alleviating most hardware failure. Additionally, I would monthly plug a 2TB external hard drive into my computer and ensure that all the contents were backed up. This 2TB volume would sit safely in my desk drawer at work – “off site”. Unfortunately, I left my old job, and hadn’t carried on with my back-ups, so my most recent back-up was over a year old. Furthermore, I’d since bought a second NAS, and that had no back-up at all. But that was working fine, so a problem for another day…

At one point in the procedure, I’d attempted to reinstall DSM using the software provided. But then I’d been presented with the five stages the software would take – including re-partitioning the drives! I pulled the plug – literally – when I saw that. I didn’t want my drives to wiped!

My workaround after a bit of Googling was as follows. I bought another 2TB hard drive and a SATA dock for a hard drive from my NAS. Because I’d used RAID 1, using one of the drives should be fine. I then downloaded a bootable version of Ubuntu onto a memory stick, and followed instructions to boot into the OS on my PC and get access to the drive. Ubuntu us seemingly better for reading the RAID file format.

My files were safe! Ubuntu wasn’t reading my external HD though. I put this down to buying a special kind of HD that I’d my eye on anyway. It’s WD WiFi model that can deliver files to portable devices easily. More usefully to me in photography was the built in SD-Card reader which can hoover up any files on a card while you’re in the field. While SD cards are cheap enough, it’s good to have a backup on the road, and reading the resulting images into Lightroom afterwards via USB will be quicker. However Ubuntu couldn’t read this new purchase, so I had to get another cheap 2TB drive which it could see.

I then used Ubunutu to copy down the files to my external hard drive. This obviously took many hours. But now I had at least the safety of knowing that they were backed up before I went about rebuilding my NAS.

Then it was a question of reinstalling DSM 5.2 onto my Synology. I was expecting the software to reformat the drives before I had to copy them back to the NAS. However, when it came to the re-partitioning step… it skipped it! It took a while, but it reinstalled DSM onto my NAS, and my files were safely sitting there without me having to copy anything back. I hadn’t actually needed to buy either of the two 2TB drives I now had, boot into Ubuntu or buy a HD dock. I felt much safer for having a backup though.

There was another problem now – one of the hard disks was reported as physically degraded. That was because I’d foolishly yanked the HD out of the dock before making sure it was switched off. You know instantly if you’ve done this because the spin speed imparts some serious gravitational forces on the drive and you can feel them in your hand. I really can’t explain how weird this feels. I don’t suggest you try it unless you don’t care about your hard disk.

Had I completely messed up my drive in this process? Was I going to have to buy yet another 2TB drive?

Possibly.

I went through the Synology repair process – it took a painfully long 18 hours – and all seemed fine. The drive is no longer reported as degraded (although I will keep a close eye on it), and once I’d reinstalled Plex and Download Station onto my NAS, I was back up and running.

Searching my records, it appears that I bought this NAS in 2010, and the drives at the same time. I suspect that I need to replace them anyway in the near future. So a couple of 3TB or 4TB drives is on the cards. Storage is so expensive, yet so dull.

Do I get a new NAS at the same time? Getting a Synology NAS that supports Plex is a more expensive proposition these days, despite the advances in processing power. I’ll see depending on what comes out.

In the meantime, I’m now faced with another dilemma. What do I do about my “off-site” back-ups?

I reckon I currently have between 3 and 4TB of data stored across various devices that I’d like safely backed up. But what’s the most cost effective way of doing this? I could revert to portable hard-drives sitting in a locker at work again. But cloud computing costs are coming down, and my fibre connectivity is unlimited. Even then, I realise that it’s likely to initially take a couple of weeks to fully back-up my current file usage. But having files in the cloud would be massively useful. And much safer.

I currently have a Google account with a bit over 1TB of included storage (a Chromebook promotion), of which I’m using a grand total of 7%. They charge $9.99 per TB per month for storage additionally. So that’s an option, although getting on for $500 (£320).

I also have 1TB of storage with Microsoft’s OneDrive because I have an Office 365 Personal subscription that came with a computer. They came out as Labs winners in the most recent PC Pro magazine. I’m using even less of that 1TB, but I’m a bit confused about OneDrive’s options.

Logging onto OneDrive to look at additional storage options suggests that I have to pay either £1.99 a month for another 100GB or £3.99 a month for another 200GB. Yet this piece from their blog in October last year says that they are rolling out unlimited storage! It still seems to be in beta, so I’ve added myself to the list. £60 a year for Office plus unlimited cloud space would be a great deal (or even £80 for 5 PCs). [Update – November 2015: I never got my “unlimited” storage. I got the 1TB when I later bought an MS product that came with 12 months’ of Office 365. Microsoft has now pulled unlimited, and is sticking with 1TB for Office 365 users, and 5GB (down from 15GB) for free users.]

The other option seems to be Amazon where I pay for Prime. But again the situation is a bit confusing. Logging into my account I seemingly have 10GB of total storage space, of which I’ve got 9.7GB free. The remainder is mostly made up of documents I’ve sent to my Kindle in the past.

Amazon does allow free photos to Prime members in the UK. But there are question marks about some file formats. Amazon recognises some RAW file formats but not others. Video is excluded though, and although I mostly keep it out of Lightroom, there’s a little in there. Perhaps Amazon could take care of my Lightroom photo library though for no additional cost.

What’s odd is that in the US you seem to be able to pay $60 a year for unlimited storage of any kind of file. This just doesn’t seem to be an option in the UK.

Otherwise, there’s DropBox, but it’s too expensive on current price plans. There are smaller companies, but it feels safer to go with a big company. If Microsoft does properly roll out a full unlimited OneDrive offering, then I’m in. Otherwise it could be Amazon as a partial solution.

Decisions, decisions.