backup

NAS Upgrading Fun

I’m documenting this, just in case there’s someone out there for whom it’s useful.

I had two NAS drives, both made by Synology. I have a DS210j that I bought back in 2010. It had a pair of 2TB drives in it with a mirror RAID array, and perhaps the most important thing stored on it is my offline music library. The drive was about 75% full. The RAID setup means that if one drive fails, the data should be safe on the other. I can then replace the failed drive and get back to parity.

The other drive is a DS214se bought in 2014. This had a pair of 3TB drives also in a mirrored RAID array. This was over 90% full. It’s home to a lot of my archived video projects as well as much of my substantial Lightroom photo library (essentially, everything I’m not currently working on).

Diskstation Manager (DSM) is Synology’s software that runs on these NAS drives, and it was not happy that I have less than 10%. In any event, it was clear that I needed to expand my storage in the near future. So I’d been keeping my eye out for cheap large format hard drives. I reckoned that 8TB is probably the sweet-spot at the moment in terms of value per TB. But there’s a very odd thing in the hard disk market.

If you go to somewhere like Amazon, and look for a WD Red 8TB drive or Seagate Ironwolf 8TB (both versions of their drives designed to run in NAS drives), the drives will cost you around £218-240. Now those prices can fluctuate, but north of £200 is common.

However, having spent some time in the Reddit Datahoarder group, there are plenty of folk on there who will tell you that the drives that come in some WD MyBook or WD Elements external drives are actually pretty good for using in NAS drives. Indeed at one time, they were actually Red drives. These days, they’re not strictly the same drives as you can buy on their own, but they’ll still pretty good.

More importantly, the price of these things can drop very low indeed. If you use a service like CamelCamelCamel to keep an eye out, you can stay ahead of the game. Last week, the price dropped to as low as £129 for an 8TB WD MyBook. I didn’t get onto it in time, and the price jumped back up to over £150. But then it came back down to £139, where at time of writing it still is. So I jumped in and bought two.

When they arrive, they’re of course designed for use as external drives. They come with smart plastic cases and power adaptors. Fortunately, there are YouTube videos that show you how to open these cases non destructively, a process called “shucking.”

This video worked well for me – the design in the video still being the current WD design. You just need a spudger or knife, and something to keep the tabs open. In the video, he uses guitar picks. I used a few business cards.

Then it’s just a question of sliding the case apart and unscrewing the drive from the plastic container.

Obviously you could use brute force, but that runs the risk of damaging the drive. Furthermore, if you keep the drive cases intact, you can re-purpose your old NAS drives as external hard drives. I put my two 3TB drives back into the cases as 3TB external hard drives are useful to have around. However, at time of writing, I’ve yet to work out how to format them away from Synology’s Hybrid RAID format.

Now you have your two new 8TB hard drives out. So it was a question of replacing the drives in my newer DS214se. Now I did wonder whether it was worth upgrading my NAS at the same time. As I mentioned above, my NAS is now getting on for 5 years’ old. There’s only 256mb of RAM in it, and it’s not powerful enough to do anything fancy like host 4K video and then transcode it. But in fact, that’s not a problem. I do keep some videos on my NAS for Plex (the in-home streaming platform), but I use an Nvidia Shield TV to run Plex server (i.e. run the main software) rather than run it on the NAS drive itself – something that many more powerful NAS drives can do. Instead, I just point the Nvidia Shield TV towards the video library stored on one of my NAS drives.

Similarly, I’m not doing anything like hosting a website on NAS (again – something that’s completely do-able). But I asked the question. The equivalent Synology NAS today to my DS214se is probably the DS218j. This is another £150 and in actual fact, isn’t that much more powerful than my older one. I’ll probably upgrade at some point, but not today.

Instead, I needed to replace the 2 x 3TB drives in the existing DS214se with my 2 x 8TB drives. This is relatively straightforward – but it just takes a while.

The process is basically:

  • switch off your NAS (actually – it quietly shuts itself down properly)
  • open it up (give it a good clean while it’s there – I had lots of dust in mine)
  • take out one of the 3TB drives, and replace it with an 8TB one
  • close it all up and switch on

Once it boots up, it start beeping at you. It’s noticed that one of the drives has “degraded.” It hasn’t really of course – you’ve just taken one away. Nonetheless, your next step is to “repair” it.

In essence, “repairing” it is just a matter of backing up all the data from the remaining 3TB onto the new 8TB drive.

This takes some time. 8+ hours in my case.

So I went to bed.

The next morning, there once the final couple of percent in the process had completed (you can monitor it all within DSM), it was time to switch out the other drive.

Another 8+ hours later, I was out. But the device sends an email when anything major happens. So I got the email and knew that all was fine.

At this point the system also notices that it has 8TB on each drive, so it ups the overall size available, and I was only using 35% or so of my space.

But I wasn’t quite finished. My older DS210j has been annoying me of late. While it’s not full, the discs rattle constantly – it’s noisy. Remember, you need these drives on at all times if you want to access the data, and in general, I tend to leave them on. Indeed, I can remotely access files on them if I need to.

For me, the next step was going to be copying all the data across from one NAS to the other.

The key thing with this process is not to do it via your PC (or Mac). Certainly, I have both NAS drives attached to my PC and I could do the whole thing in File Explorer in Windows 10. But that’s not smart. Instead, there’s a built in app called FileStation.

You just need to make sure that from your destination NAS (the DS214se in my case), you’ve mounted a remote drive to a folder on it. Once you’ve done that, you can then copy (or move) the files from one drive to another.

My older device actually has two Volumes on it – effectively two separate drives, although the second volume is tiny. I chose to do this one step at a time. Copying the first volume across resulted in FileStation telling me it would take 17 hours or so. A little more than making a cup of tea. And this is only an unfilled 2TB drive. But it’s a one time thing. While the estimate was initially 17 hours, the process actually took more than 24 hours. I had to leave it copying files for two consecutive nights.

Fortunately, I didn’t need the second volume – system files for a Plex server I was no longer using. So finally, I was able to retire the older NAS drive. I’m left with one NAS, and I still have about 46% of space left.

A final note is to say that I sync my Lightroom library to Amazon Photos – mainly because it’s an inexpensive bolt-on for Amazon Prime members, giving unlimited storage for photo files (including RAW photo formats). What it doesn’t include free is video.

I used to use Amazon Drive to back up both NAS drives completely, when Amazon Drive allowed unlimited storage. But that’s gone. I have to pay a modest extra fee for all the sidecar data that Lightroom stores. Non destructive editing requires a lot of these files. But it’s not too bad.

My ideal would be to securely backup all my files offsite. This is very do-able, and there’s an app within DSM precisely for that. However the costs begin to mount up. For 4.5-5TB of storage, you’re looking at £/€300-350 a year. Getting the data back might increase those costs. Amazon, for example, has Amazon Glacier. I assume that it’s stored on a tape format. The idea with Glacier is that you don’t need to get the data particularly regularly – just know that it’s there. That’s as well, because recovering the data is actually quite expensive.

In my instance, this is really about securing photos, music and video – with a few documents thrown in. If my flat where to burn down, or someone steal my NAS drive, then I’d know I could recover everything. But doing that in a cost-effective manner is tricky.

One solution is finding a friend with a big internet pipe who’s willing to let you put a second NAS drive in their place. You could then keep the two in sync. You could obviously “host” a NAS drive for your friend in your house at the same time.

A previous “off-site backup” solution I’ve used in the past is to simply buy another external hard drive (perhaps one of those 8TB WD MyBooks!), back the NAS up to that, and then keep that drive securely in your workplace. If you then brought it home once a week/fortnight/month, you could update the backup and return it to work the next day. You’d never be more than a week/fortnight/month behind if the worst happened. Indeed, you could buy two drives and shuttle them back and forth – keeping one at home and one at work at all times. But that may be over-kill.

I realise that this isn’t the most exciting thing to write about. And when I talk to people, I realise that relatively few people really look after their data. If you don’t have all that much, you can perhaps rely on cloud storage services. Google Drive, Dropbox and iCloud all give you a bit of free space, and will charge you for more. OneDrive can be good value if you want Microsoft Office. But many of these max out at 1TB in space. Even Google, who used to be quite generous with Google Drive space, giving it away with Android phones or Chromebooks, seems to be more inclined to charge these days.

Your parents’ or grandparents’ photo albums may seem quaint, but those albums will be much safer than a lot of photos that are only stored safely while users hang on to their social media accounts…

Backup Storage Solutions and Costs

Old Technology

More than a year ago, I wrote about some experiences I’d had with backup storage and thoughts about “off-site” back-ups. It prompted a couple of responses on the blog, and a few more on social media. Not the most exciting of subjects, but an important one, as I still believe that in this age of digital collections, backup will be a key issue. Just one that most people don’t spend much time thinking about.

To recap.

I use a lot of storage:

  • I shoot video and although I upload edits of what I shoot to Vimeo or YouTube, I like keep the original materials (in one instance this paid off, when I sold a shot to a company who’d found part of it on Vimeo).
  • I shoot photographs. Lots of photographs. Anything I take on my phone gets backed up to my Flickr and Google Photos accounts. (They also go temporarily to a Dropbox account.) But that doesn’t help for shooting with my cameras, where I prefer RAW. These large files end up on a local NAS drive once I’ve moved them off the PC where I edit them. I should note that I do delete files. If I shoot a batch of ten broadly identical photos, then assuming I like any of them, I’ll only keep one version. So for every 100 photos I shoot, I perhaps keep 10, and upload 5 to Flickr.
  • I have a digital music archive. I keep a local copy of it in iTunes – loathsome as that software is. And it’s mirrored on Google Play Music where they upped the limit to 50,000 tracks (I have more than 25,000 on there currently). I mostly play back music via Google Play Music either on my phone or via Chromecast devices. Beyond music, I have lots of other audio, both field recordings and radio recordings.
  • I have other legally purchased commercial video files – including iTunes downloads and DRM-free mp4 files.
  • I have a myriad of other documents, mostly much smaller in size.
  • I have several PCs and tablets, including the hard drives from a number of older computers, some of which files I’d like to have retrievable.
  • I have around 100 MiniDV tapes that I’d like to “rip” and store (A project for another day).
  • I have a large box of film photos that I want to scan at some point (Another project for yet another day).
  • Probably other stuff too…

While some files sit on various computers, I try to mostly copy files onto one of two Synology NAS drives. Cumulatively they have about 6TB of space, of which I’m using more than half. One NAS in particular will need its 2TB drives replaced by 3 or 4TB drives fairly soon (Yes, I Know 6TB drives are available, but I try to be cost efficient).

I also have a plethora of old hard drives and expansion drives that I have been slowly consolidating onto larger, newer portable hard drives (Three old small hard drives might now sit on one larger capacity drive).

Services I use that offer storage:

  • Google Drive. Through various promotions and purchases, including a Chromebook, an HTC phone, doing work for Local Guides, and running security check-ups, I have a cumulatuive 2.2TB of space. However, this space is not permanent, with 1TB disappearing at the start of next year when my Chromebook will be two years’ old (Plus, they don’t hand out 1TB for new Chromebooks now, in case I was thinking of upgrading).
  • OneDrive. Beacuse I have a paid subscription to Office 360, I get 1TB with this. In fact, I could have five accounts, each with their own 1TB. But that would be painful with multiple logins required. For those with lower usage needs, this offers a great deal however.
  • Dropbox. Just the basic free account with 2GB of space. Although I have occasionally gone on the paid plan for a podcast I work on.
  • BT Cloud. My ISP gives me 500GB because of the plan I’m on. I’ve never used it, and have heard it’s a bit slow.
  • Evernote Premium. Used for note taking and scanning. Not for files, but scanning magazine articles I want to keep and making them searchable, for example, does take space. But it’s not for files per se.

What I need:

Let’s say I need 6TB of space. It’ll probably more than that in due course, but 6TB is a good starting point. Let’s try to price that up at the various big vendors.

  • Google Drive is £7.99 a month per TB. So for 5TB of additional storage (1TB drops off my account in January don’t forget), we’re looking at £479 pa.
  • OneDrive is £1.99 per 50GB. So for 5TB of additional storage, it’d be £2388 pa although I feel sure that there must be most cost efficient ways of buying storage from Microsoft. I just can’t see them – aside from opening multiple Office 360 accounts which seems mad.
  • Dropbox seems to push me towads a Business plan at £110 pa for “As much as needed” (which may not include VAT). Still, that seems to be pretty decent value.
  • Amazon Cloud Drive has finally updated its UK offer and either gives Amazon Prime users their free “Prime Photos” service which is unlimited space for photos, and 5GB for other files including video, or their Unlimited Storage offer which is £55 a year, with a free three month trial.

There are of course lots of other providers which are and aren’t aimed at the consumer end of the market. And different not all cloud services are aimed at the same use cases. I’m not really looking for a shared working environment so much as reasonably priced archival space.

I’m looking for something that’s reliable, that allows me a clear understanding of cost, and doesn’t place many limits on what I store and how large the individual files are. While I’ve not shot many yet, a 4K video file lasting more than a few minutes is likely to take up an enormous amount of space. I don’t think I have any single file that’s larger than 4GB, but some services top out what they’ll accept at 5GB per file.

I also want to be certain that a provider I use is going to be there next year, in five years and in ten years.

Amazon has actually built an enormously profitable business offering cloud servers and storage. However, S3 storage pricing is a little opaque – essentially it’s based on how much access the files have. While Amazon Glacier is intriguing because it’s a very low cost backup solution at $0.007 per GB / month. It’s based around archival needs, and the data might take a few hours to become available once you’ve requested it.

It seemed pretty clear to me that Amazon Cloud Drive is the winner here. £55 for unlimited storage is a good deal from my perspective. But there are a few provisos worth getting into. This is for non-commercial use – they particularly seem to be concerned about people running photography businesses who no doubt generate lots of files all the time. I’m not running a business, but am an enthusiastic amateur. I assume I won’t be in trouble!

They’re also unsurprisingly worried about people sharing illegal files, or storing such files in the first place. I’d guess that this is not going to be home for your massive collection of torrents. That said, I’m not clear how they can determine a film bought legally and an illegal one. If I rip my DVD collection, can I store it in the cloud?

If you have large files, then you won’t want to use their web interface to upload them (But you didn’t want to do that anyway). And files with more than 255 characters or special characters, unsurprisingly, cause problems.

If you stop paying Amazon, they may delete your files. Kind of obvious, but worth knowing anyway.

To be clear, I’m using this as purely a backup service. There will be no files on Amazon that I don’t also have locally. Indeed the NAS drives I own are in RAID arrays themselves (although other files that will eventually reach Amazon may only have a single local backup).

Finally there’s the small issue of getting terabytes of data into the cloud in the first place. My BT Infinity 2 service is fibre-to-cabinet, and claims that it has no upload or download limits. But if I’ve got something like 6TB of data I want to initially back-up, that’s going to take quite some time.

At time of writing, I’ve only been uploading a 2TB NAS drive for less than 24 hours, and it’s not clear how long it’s going to take me in total. But it could easily be a full month before that first NAS is in the cloud. It does seem that Amazon “drip feeds” the files across. This is probably a good thing, because I don’t want all my bandwidth used up all the time as files fly back and forth.

Indeed I’m actually using a Synology app called Cloud Sync to do the backups, and that can limit the number of simultaneous files being uploaded. Furthermore, I’ve scheduled the syncing to stop at various points during the week when I’m more likely to want unfettered internet use myself. So I’ve got three hours in the morning and another six in the evening when it pauses syncing. At weekends, the syncing process will also be halted during the day.

To be honest, I was happily streaming Netflix while syncing over the weekend, so it’s probably not really necessary. But it lightens the load on the network during peak hours.

The good thing about using a NAS drive app is that only the NAS needs to remain on while the upload happens. Amazon does have a PC/Mac app for backing up your local PC, but there seems to be a question mark over whether it keeps your computer “in sync” with the cloud, or simply does a one-time back-up, requiring you to manually sync additional files at a later date. Either way, I’ve not bothered yet, since I’ll do one at device at a time. The good thing is that there are plenty of third party applications that will work with Amazon and will probably do what you need.

Amazon also has a mobile app which will dutifully send through all your photos and videos, but as mentioned above, I’ve got these covered already.

I will report back on the success of Amazon Cloud Drive and any issues I have with it in due course. In the meantime… let the upload continue!

Note: All prices and offerings are correct at time of writing. This is clearly a very fluid market.

Storage and Backup: Some Personal Experiences

[Note that this is likely to be duller than usual! It’s mostly written up so that anyone who searches for similar problems might find it useful.]

This all began last Wednesday when I noticed an email from Synology inviting me to update the firmware that runs that two NAS drives I own – a DS210j and DS214se. They’d just released DSM 5.2, their proprietary software that allows you to do all sorts of clever things with their products.

I’ve had nothing but good experiences with Synology. They’re reasonably priced, and they offer a safer way to store data than simply using an external hard drive.

So I ran the updates. My new DS214se updated no problem, but the older DS210j had a problem. It got to about 20% of the process and then stalled. And it wouldn’t restart. The status light was flashing orange which is not a good sign.

Now I was unable to get into the volume at all. I was concerned.

I had previously put in place a monthly “off-site” back-up. The NAS had 2 x 2TB drives in a RAID 1 array. That gave me a 2TB volume safely mirrored on the two drives alleviating most hardware failure. Additionally, I would monthly plug a 2TB external hard drive into my computer and ensure that all the contents were backed up. This 2TB volume would sit safely in my desk drawer at work – “off site”. Unfortunately, I left my old job, and hadn’t carried on with my back-ups, so my most recent back-up was over a year old. Furthermore, I’d since bought a second NAS, and that had no back-up at all. But that was working fine, so a problem for another day…

At one point in the procedure, I’d attempted to reinstall DSM using the software provided. But then I’d been presented with the five stages the software would take – including re-partitioning the drives! I pulled the plug – literally – when I saw that. I didn’t want my drives to wiped!

My workaround after a bit of Googling was as follows. I bought another 2TB hard drive and a SATA dock for a hard drive from my NAS. Because I’d used RAID 1, using one of the drives should be fine. I then downloaded a bootable version of Ubuntu onto a memory stick, and followed instructions to boot into the OS on my PC and get access to the drive. Ubuntu us seemingly better for reading the RAID file format.

My files were safe! Ubuntu wasn’t reading my external HD though. I put this down to buying a special kind of HD that I’d my eye on anyway. It’s WD WiFi model that can deliver files to portable devices easily. More usefully to me in photography was the built in SD-Card reader which can hoover up any files on a card while you’re in the field. While SD cards are cheap enough, it’s good to have a backup on the road, and reading the resulting images into Lightroom afterwards via USB will be quicker. However Ubuntu couldn’t read this new purchase, so I had to get another cheap 2TB drive which it could see.

I then used Ubunutu to copy down the files to my external hard drive. This obviously took many hours. But now I had at least the safety of knowing that they were backed up before I went about rebuilding my NAS.

Then it was a question of reinstalling DSM 5.2 onto my Synology. I was expecting the software to reformat the drives before I had to copy them back to the NAS. However, when it came to the re-partitioning step… it skipped it! It took a while, but it reinstalled DSM onto my NAS, and my files were safely sitting there without me having to copy anything back. I hadn’t actually needed to buy either of the two 2TB drives I now had, boot into Ubuntu or buy a HD dock. I felt much safer for having a backup though.

There was another problem now – one of the hard disks was reported as physically degraded. That was because I’d foolishly yanked the HD out of the dock before making sure it was switched off. You know instantly if you’ve done this because the spin speed imparts some serious gravitational forces on the drive and you can feel them in your hand. I really can’t explain how weird this feels. I don’t suggest you try it unless you don’t care about your hard disk.

Had I completely messed up my drive in this process? Was I going to have to buy yet another 2TB drive?

Possibly.

I went through the Synology repair process – it took a painfully long 18 hours – and all seemed fine. The drive is no longer reported as degraded (although I will keep a close eye on it), and once I’d reinstalled Plex and Download Station onto my NAS, I was back up and running.

Searching my records, it appears that I bought this NAS in 2010, and the drives at the same time. I suspect that I need to replace them anyway in the near future. So a couple of 3TB or 4TB drives is on the cards. Storage is so expensive, yet so dull.

Do I get a new NAS at the same time? Getting a Synology NAS that supports Plex is a more expensive proposition these days, despite the advances in processing power. I’ll see depending on what comes out.

In the meantime, I’m now faced with another dilemma. What do I do about my “off-site” back-ups?

I reckon I currently have between 3 and 4TB of data stored across various devices that I’d like safely backed up. But what’s the most cost effective way of doing this? I could revert to portable hard-drives sitting in a locker at work again. But cloud computing costs are coming down, and my fibre connectivity is unlimited. Even then, I realise that it’s likely to initially take a couple of weeks to fully back-up my current file usage. But having files in the cloud would be massively useful. And much safer.

I currently have a Google account with a bit over 1TB of included storage (a Chromebook promotion), of which I’m using a grand total of 7%. They charge $9.99 per TB per month for storage additionally. So that’s an option, although getting on for $500 (£320).

I also have 1TB of storage with Microsoft’s OneDrive because I have an Office 365 Personal subscription that came with a computer. They came out as Labs winners in the most recent PC Pro magazine. I’m using even less of that 1TB, but I’m a bit confused about OneDrive’s options.

Logging onto OneDrive to look at additional storage options suggests that I have to pay either £1.99 a month for another 100GB or £3.99 a month for another 200GB. Yet this piece from their blog in October last year says that they are rolling out unlimited storage! It still seems to be in beta, so I’ve added myself to the list. £60 a year for Office plus unlimited cloud space would be a great deal (or even £80 for 5 PCs). [Update – November 2015: I never got my “unlimited” storage. I got the 1TB when I later bought an MS product that came with 12 months’ of Office 365. Microsoft has now pulled unlimited, and is sticking with 1TB for Office 365 users, and 5GB (down from 15GB) for free users.]

The other option seems to be Amazon where I pay for Prime. But again the situation is a bit confusing. Logging into my account I seemingly have 10GB of total storage space, of which I’ve got 9.7GB free. The remainder is mostly made up of documents I’ve sent to my Kindle in the past.

Amazon does allow free photos to Prime members in the UK. But there are question marks about some file formats. Amazon recognises some RAW file formats but not others. Video is excluded though, and although I mostly keep it out of Lightroom, there’s a little in there. Perhaps Amazon could take care of my Lightroom photo library though for no additional cost.

What’s odd is that in the US you seem to be able to pay $60 a year for unlimited storage of any kind of file. This just doesn’t seem to be an option in the UK.

Otherwise, there’s DropBox, but it’s too expensive on current price plans. There are smaller companies, but it feels safer to go with a big company. If Microsoft does properly roll out a full unlimited OneDrive offering, then I’m in. Otherwise it could be Amazon as a partial solution.

Decisions, decisions.