When Things Go Wrong: aka Thank Goodness for Amazon Drive

I have a NAS drive, and it holds all my worldly digital goods. That is, it has all my photos, videos, documents and other things.

I still haven’t found a cost-effective off-site backup solution beyond the NAS drive, although I do need one. Something that I can back up, upwards of 6TB of data.

Amazon Drive used to allow unlimited backup for a modest fee on top of their Prime offer. I assume that this got abused by people storing petabytes of data, because they reduced this to unlimited photo backups for Prime members. The good news is that any photo format, including RAW, is allowed.

So I pointed by Lightroom folder (I use Lightroom to catalogue all my photos) at Amazon Drive and used it as a backup for my pictures.

Then something went wrong.

It was first obvious when I noticed that I was no longer using 67% of my NAS drive, but only just over 50%. Where had that space come from?

I realised I’d tweaked some settings in my Synology Cloud Sync app -attempting to do a two-way backup -and for some reason, files were now being deleted on my Synology NAS drive.

I killed the process, and took a look at what had happened. Photos from my Lightroom catalogue had been deleted all over the place. Usually some here and some there.

Aaarggh!

Then I had a look at Amazon Drive. They still seemed to have copies of everything including my most recent pictures.

Phew.

But here’s the thing. I now needed to download those files and re-populate my Lightroom catalogue. Lightroom is useful here, as the actual “catalogue” file is held on my local machine. The photos themselves are mainly offloaded to my NAS drive (My usual process is to only keep photos on my local drive while I’m still editing them, or they’re awaiting editing. Then, once done, I will move them over to the NAS drive).

Anyway, I could easily identify the missing photos. In many cases I had low quality images of them via the “catalogue” file’s previews which are stored with the catalogue file.

So it was now a question of downloading the pictures from Amazon, and putting them back on my NAS. This was to prove a slow process.

While Amazon Drive has a web interface, if you want to download any volume of folders, you need to use their app. With this, you can download large numbers of files without having to zip and unzip them as the web interface makes you.

But it’s also slow.

My photos, mostly shot in RAW (or RAW+JPG) since I bought my first DSLR camera back in the early ’00s, take up a reasonable amount of space. And regardless of my ISP’s 75 Mbps or so download speeds, Amazon seems to limit the speed you can get your data at.

I dusted off an old PC, went through the process of updating the Windows 10 install on it, and installed the Amazon Drive app. I also changed the power settings to ensure that the machine never shut down. I did all this to have a dedicated resource for grabbing all my photos.

At around the same time, I picked up a cheap WD 8TB My Book, which had been discounted to £113 in a Black Friday Amazon sale. This would act as a temporary intermediary for now. And then later, I planned for it to become my “off-site” backup drive.

I found a quiet corner of my flat to place the laptop and hard drive, and then set it to work downloading all my photos from Amazon. I ended up doing this folder by folder. That’s because some of my bigger folders seemed to need as many as 3 days to download. As I say, I’m sure that this is an artificial limit placed on them Amazon. This is probably also worth knowing if you use a service like this for backups. I note that a company like Backblaze provides the option of them shipping a hard drive of up to 8TB that you pay for, should you need your data restored. That seems fair.

Anyway, I got there eventually. I now had a hard drive with all my photos on it. At least I hoped I did.

One slight issue, that I would have to overcome later, was that some folders were duplicated because I changed some file structures at some point. It seems Amazon kept copies of both old and new file structures. Having duplicates would not be a problem from my perspective, but would eventually need cleaning up.

Now it was a question of copying my photos from my hard drive back to the NAS drive and into my Lightroom folders.

Windows 10 (and some prior versions) have a built in command line app called Robocopy. This lets you copy files between folders, allowing you to do things like only copy files that aren’t already there. It’s much nicer than using the regular File Explorer. I used an app that puts a Graphical User Interface in front of Robocopy, called RoboMirror.

I felt that it was safest to again only do this folder by folder. I dutifully made sure that my laptop wouldn’t shut down, and set the process going. The pictures were returning.

In Lightroom there were two further issues.

First, I wanted to be absolutely sure that all my photos were back in place. I followed the instructions in this YouTube video to search for missing photos. It uses Lightroom’s built in functionality to build a temporary collection called “Photos Missing From Folder.”

This brought up a few oddities, which may or may not have been related to my second issue.

In some folders, there were only one or two “missing” images, while in others, there seemed to several hundred. Yet each time I looked, the images were actually still there, in folders where Lightroom expected them. They weren’t really “missing.” I’m unclear why Lightroom thought that they were missing, since the images were always in place in the folder where Lightroom thought they were. Strange.

The other issue was to do with metadata. There were sometimes “conflicts” in metadata between the image in the library file and the image in restored from Amazon. These were indicated by either an exclamation mark (much like a missing image identifier) or an arrow.

I’ve come to the conclusion that some of these were possibly issue was how Lightroom was handling RAW + JPG files in some older folders. Lightroom normally handles having two copies of an image in RAW and JPG format very well, only showing one image in the catalogue even though there are two files behind the scenes.

I’m wondering whether that link was somehow broken via a trip to Amazon Drive and back?

Right-hand clicking on the metadata icon and “overwriting” the library data back to the file cleaned things up, although the missing photos warnings continued. It’s actually a slow process to update this metadata. Another folder by folder process for me I suspect.

I think that re-importing the problematic folders might do the trick, although I would then lose those images from any collections they were in. Not a massive problem, but worth knowing.

A job for another day.

One way or another, no picture was actually missing.

This whole process took a lot of time. Recovering those photos from Amazon was slow, but very much worth it. Then importing them back into Lightroom was slow.

I’ve still go the job of removing some duplicates that are sitting outside my Lightroom catalogue, but are hidden within the folder structure. I’ll do that slowly to ensure nothing important gets missed.

Unfortunately, I’m now dubious about Synology Cloud Sync backing up my folders to Amazon, but I may carefully reinstate it if I am satisfied it works. My mistake, incidentally, was trying to sync in both directions. I will definitely not be doing that again.

I could use Amazon’s own app to backup my NAS drive Lightroom folder, but it’s a PC app and so needs the PC to be awake to perform the backup. Plus it’s yet another background process clogging things up.

I will definitely take regular “snapshots” of my entire NAS drive. It it 8TB when full, so my standalone WD hard drive will manage. The first time around, this will be slow to do, but it will be worth it. My current thoughts are to store it at work, bringing it home to update the snapshot every few weeks on a schedule.

Anyway, lessons have been learned!


Posted

in

Tags: