DP | @ekhoophotography

A few people have asked me to elaborate on my storage strategy so here it is. Some background information is needed first. I started taking photography seriously in 1998. Buying my first SLR back then was a big thing and working through the lenses and other options. In late '99, was basically scanning the prints on a flatbed scanner. Quality was amazing, or at least what I thought was amazing until I stumbled upon a very expensive (at that time, I wasn't making much a month) option; the slide scanner. By skipping the printing stage and scanning directly from film, it delivered unmatched results from anything else unless you're comparing to a drum scan. Since then, I've been scanning almost all my slides and negatives but most of what I shot were from my travels. Film costs a fair bit, especially if you shot transparency film such as Fuji Velvia, Kodachrome or Elitechromes, so the everyday life doesn't often get captured on film.

So with digital scans dating all the way back to 2000 (the 2700 dpi scans tend to produce on average 14MB LZW compressed TIFF files), storage is key, along with a proper backup strategy. I went with CD-Rs and then DVD-Rs but they never were convenient (not fun, not quick, and laborious process) meant that you will almost never will do it timely enough so I decided to stick with hard drives. They get cheaper and are inherently quicker to access if they're stuck into a spare machine that can be booted up anytime you needed to.

I upgraded scanners so the file sizes got bigger and films like Velvia and EliteChrome 100 pushed the resolution (grain?) further. Then came my first DSLR, a 300D (my shortest-lived DSLR ownership) and my 20D several months later. So came RAW files, 16-bit PSDs, and the story continues until I hit my 5D Mark II with 21-megapixels, Full HD movies. Backup is getting more important but also important is accessibility.

I needed to backup and offload my main PC but also need to access the "older" stuff occasionally. Leaving your main storage server switched on/off daily or 24x7 wasn't something I want to do on a daily basis so I got a NAS which I quickly outgrew within a year so that too got upgraded to a 4-bay unit.

I have a storage server with 6x1TB disks. It's all RAID 1. Mirrored. So I get a net space of 3TB. I have a "_photos" folder shared out that further breaks down by year, i.e. "2001", "2003", along with some odd directories. I also have a "_videos" folder where I back up my created DVDs (ISO format) and Adobe Premiere Pro project files. The problem is occasionally accessing stuff from say, 2006. My main PC only stores past 16-odd months so I ended up going to my storage server regularly. So what's running 24x7 is my NAS box with 4x500Gb in RAID5. That gives me about 1.4TB of disk space. That way, I get that past 2-years on it; i.e. 2008 & 2009 along with my 600Gb music collection on the built-in iTunes server. My IP cameras also record to NAS and they keep the past-7 days recording. I also have imaged backups of all the machines in my home so that if the primary disk crashes, recovery is just a click away. Keeping everything in sync is the unix utility rsync (the windows equivalent is DeltaCopy).

With the growing file sizes, I'll be moving my storage server to 1.5TB disks by Q4 this year and my NAS will grow to 4x1TB*... one can never have enough disk space!



* That's one thing I like about my NAS. I can live swap the disks and expand it without powering down and losing my data. I can easily move from 500Gbx4 to 1TBx4 and later to 2TBx4.

After I mentioned on Facebook and Twitter about my recent Western Digital hard drive failure, someone asked me what are my other disk failures. So happens, I do keep records of my disk failures. So here goes, all the way from 2003.

2002 : Nov : Quantum Atlas 10k II QM318400TY-LW, 10,000rpm USCSI3. Some bad sectors, recovered from spare sectors.
2003 : June : Western Digital WD1200JB - 120GB PATA, 7200rpm. Disk crash after bad-sectors detected.
2004 : Oct : Western Digital WD2000BB - 200Gb PATA, 7200rpm, Bad-sectors, discontinued using
2005 : May : Seagate Barracuda 7200.7 (ST3160023AS) 160Gb SATA, Bad sectors, Warrantied, replacement retired.
2008 : May : Western Digital WD2500JD - 250GB SATA, 7200rpm. Bad sectors, retired
2008 : Aug : Western Digital WD5000AAKS - 500GB SATA, 7200rpm, Bad sectors. Replaced under warranty
2010: May : Western Digital WD5000AAKS - same drive above. Retired.

What drives do I have running on my desktops, server & NAS right now:

Hitachi Travelstar 5K100, 5400rpm, 100Gb, SATA (2.5") - 1 drive
Seagate Barracude 7200.9, 7200rpm, 500Gb - 3 drives
Seagate Barracuda 7200.11, 7200rpm, 1TB - 5 drives
Seagate Barracuda 7200.12, 7200rpm, 1TB - 2 drives
Seagate Momentus 5400.2, 100Gb, SATA (2.5") - 1 drive
Samsung SpinPoint F3, 7200rpm, 1TB - 1 drive
Western Digital WD10EACS, 5400rpm, 1Tb - 1 drive
Western Digital WD5000AAKS , 7200rpm, 500Gb - 1 drive
Western Digital Raptor WD740, 10000rpm, 74Gb - 1 drive
Western Digital Raptor WD1500, 10000rpm, 150Gb - 1 drive

Guess what brand I'll stick to for now?

If you've been following this blog for a while, you'll have noticed a few posts alluding to the search of some network storage to replace my 2-drive unit. I finally settled on the QNAP TS-439 Pro. Read the previous post about it or go to QNAP's site to read more. Basically, I needed a compact, four-drive unit. Why four drives? It allows either two mirrored sets of disks (note: not necessarily RAID1) or RAID5 (with or without a spare). Going larger (5, 6 or 8 drive units) adds exponentially to the price but I wanted the use of the few 500Gb disks I have lying around. 500Gb is quite small but if I put them in a RAID5 set, I get 1TB (n-1). Later when I move some of my main storage disks to 1.5TB disks, I get another 500Gb to slot in as a spare drive (3x500Gb + 1 spare).

Other highlights for me were the hot-swap disk trays (lockable), multiple USB ports for printer sharing and dual-Gigabit LAN ports. The other stuff are frills. The dual gigabit Ethernet ports give you a fair bit of flexibility. You can configure two separate links for two separate networks, or some form of load-balancing, etc. Mine is configured for link aggregation group (LAG) mode. If you want to read more on that, which can get technical, Google for 802.3ad. In a nutshell, what LAG does is to create an aggregation group of network links that share the same speed and duplex settings. This requires a switch that supports LAG though and few consumer grade switches support it. The TS-439 Pro does have another option of using Adaptive load balancing. This bonds the network interfaces on the 439 and allows it to aggregate without any switch support. There's a slight drop in performance but if you don't have a switch that supports 802.3ad, it is the way to go. Did I mention two links means redundancy?


My primary use is to serve as my iTunes and media server (uPNP/TwonkyMedia), download stuff (BitTorrent), IP camera recorder, and a common file store to use between my PCs and notebooks in the house. My plan is to copy my photos there from camera, then once they're sort of done (processing, etc), they can be moved to my File Server (which has 6x1Tb) in a RAID1+0 config. I've used jperf to do some testing to check the LAN bandwidth/switch speeds and for the TS-439, some regular file transfer timings.

First off, my gigabit LAN between Win Server 2003 R2 (dual-core Athlon, 6x1TB, 3Gb RAM, single link, the onboard NIC is died earlier this year so I'm running an Intel Gigabit CT off the PCIe x1 slot) and my Vista 64 SP2 client. I ran jperf over a few minutes with a five second interval. Min speed reported at 601,274Kbits/sec, max speed reported at 879,761Kbits/sec. Average the 30 readings and I get an average of 795,208Kbits/sec, or 776Mbits/sec or about 97Mbytes/sec. So let's see how the 439 Pro transfers -- remember, I have link aggregation on the TS-439.

1. Transfering 14,521Mbytes of photos. What I have is a mix of small and large files sizes. Large as in on average 26,912kb and small as in about 9kb (the XMP sidecar file). Transfer took about 320 seconds. 45.3MBytes/sec.

2. Transfering only the CR2 files took faster, clocking in about 261 seconds, or 55.5MBytes/sec.

3. Dual client access. Transfer a directory of photos and sidecar XMP files (12,672Mbytes and 11,760Mbytes). Start the transfer on client A then client B. Same target on NAS. Client A finished at 262 seconds. Client B finished at 302 seconds. Note that client B started transferring at roughly the 30 second mark of Client A. Average transfer speed is about 48.3Mbytes/sec for client A and 38.9Mbytes/sec for client B. Not too shabby considering both were hitting the NAS at the same time for about 87Mbytes/sec. Good enough for me.


I've used it as a mapped drive for working on my photos and from Lightroom 3 beta 2 and Bridge CS4, the images feel very snappy. Not as quick of course compared to my local SATA drive but it's not that obvious you're working through a network. The lag on bridge in terms of refresh for example is noticeable only when you have three clients hitting the NAS but the fact is that its not terribly slow. My guess is that if Bridge refreshes my 800 photo directory on my local drive in 5 seconds, it takes about 8 or 9 on the NAS and perhaps 15+ seconds if you have multiple clients hitting F5 at the same time. Overall, I'm pretty satisfied. Maybe it's time to look for dual-Gigabit port NICs for my client machines as well but I think the NAS will get faster when I up the RAM to 2Gb.


Overall, I'm pretty happy. It serves up my music to my iTunes, I can also listen to music/look at photos/watch movies on my Popcorn Hour C-200 via Twonkymedia server, record from all four of my IP cameras and work on a shared folder of photos. Not cheap but probably more cost effective to run than a standalone PC or server (if you factor in licensing, power, heat and noise). You can do other things with it as well though even thought I don't such as running a Web or FTP server, etc.

If you want a more in-depth review, you can read it here.

From Wikipedia : A NAS unit is a computer connected to a network that only provides file-based data storage services to other devices on the network. Although it may technically be possible to run other software on a NAS unit, it is not designed to be a general purpose server.

NAS vs SAN
NAS provides both storage and a file system. This is often contrasted with SAN (Storage Area Network), which provides only block-based storage and leaves file system concerns on the "client" side. SAN protocols are SCSI, Fibre Channel, iSCSI, ATA over Ethernet (AoE), or HyperSCSI. Despite their differences, SAN and NAS are not mutually exclusive, and may be combined as a SAN-NAS hybrid, offering both file-level protocols (NAS) and block-level protocols (SAN) from the same system. Considering I'm writing for home-users, a NAS makes more sense.

---

With the amount of data many photographers (and users) are generating, it's become obvious that storage is a problem. For casual users, it probably may not be a problem in the beginning but it soon will be. I know of friends who after a few years, a kid or two and the amount of photos, videos, music, documents, scanned PDFs, etc start to add up to gigabytes. Burning to DVD (gosh, who burns CDs today?!) can alleviate some of the problem but with 4.5Gb (dual layer disks aren't common as they're not so cost effective), it'll take a few DVDs for the average persons' backup. Worse still, those DVDs may not be readable after a few years. A friend of mine burns two discs for each copy -- and 3 years later found out that some are not readable. Good thing he had two copies as he's managed to combine them to even out things so no loss occurred. So, he consolidated the storage. Where? Good old spinning platters. With the falling prices of hard drives, it's a no brainer that disks are still the most reliable, cheap and effective storage for backup for most users today (most users here I mean home and small business perhaps). Reliable? Yes, if you have two disks and you replace them religiously every 36-months. That's what I do. The benefit is that you'll end up having larger disks too. My backup server has 1TB disks now. It started out with 160Gb and 250Gb disks (two pairs) for a capacity of 450Gb. My data very important to me so I change disks every two years. I then went to 320Gb and 500Gb (still two pairs) and now, I've got four 1Tb disks plus a pair 500Gb disks (these store my music, video and misc files). They're going to 1.5TB disks soon (best price per Mb).

The problem is usually, you won't want your backup server running non-stop. Mine's on a UPS along with my network, and more but most casual users also don't need access to their backups every single instant. They also normally don't have another spare machine lying around. In many organizations today, they use the term near-line storage for things that can be retrieved faster that tape but slower than current online storage. Of course big organizations use fast disks arrays for online and slower and usually SATA disks for near-line storage. In my case, I'll call my storage server near-line because it may not have been started up. Where's my online storage? Well, it sits on a NAS, a QNAP TS-439 Pro. This is a bit too much for most home-users (they can go with a TS-210 or TS-219) instead. I've a RAID 5 set of 500Gb disks in there, giving me just under one Terabyte of available storage. It makes a good always on file-server, repository and immediate backup (Downloader Pro immediately makes a copy of my photos there). When time comes, it gets rysncd to my near-line storage server. The reason I got the 439 Pro was because it was on promotion. It's been replaced by the Pro II (better CPU) and was the same price as the 419P. To me, the benefits were the faster (compared to 419P) CPU, and larger RAM (1Gb 439 Pro compared to 512Mb on the 419P) which translates to faster transfer speeds. The 439 Pro also has the memory on a SODIMM which means I can bump it up easily to 2Gb. QNAP's disty over here still has a few units of 439 Pro if anyone is interested.

So what's for the average home user? Well, if you have more than one computer in the house, would like to share some storage, do some backup, stream iTunes around the house, etc, then any one of these NAS units over here, here and here would serve you well. You'll also notice that I've avoided any form of single drive device. That's because most home users "think" that once it's backed up, it's going to be safe. Two drives are always better than one. If you're on a tighter budget, get a pair of USB drives. Seagate or WD are good options. Just buy two and make sure your data is identical on each one. And, yes, please store them in separate places -- at least a different room. My storage server and NAS are three walls and fifty-feet apart. Note: RAID does not prevent idiotic errors like deleting your only copy of the most important file or photo you ever have -- the array isn't going to care and will promptly erase all for you. In my case, well, I can still go back to either my storage server (100% read-only) or desktop.

My choice of 4-bay, well it's more for RAID 5 as I'm using older disks for it (i.e. the ones retired from my storage server) so it's good to have RAID5+1. RAID 5 capacity is (n-1) resulting in the available space of two disks, i.e. in a RAID 5 set of 3x500Gb, I get 2x500Gb usable. The +1? That's the hot-spare in case something dies which is for the paranoid like myself. If you go with plain RAID5 on my setup, you'll have 3x500 resulting in a usable 1.5TB.

I'll post again why I chose the QNAP NAS plus a mini-review... I'll stop here... more to come while I leave you with a pretty picture of my network appliances

The problem about ever increasing mega-pixel camera counts and media (including HD video) is that the disk space is getting smaller and smaller each day. The added complication is when you have more than one user trying to work on the same set of images at the same time. This becomes an issue with local storage hence the need for networked storage so that you have less issues with trying to sync two or more working sets of raw, tagged, marked-up or processed images. So, the result is that my dual-drive network storage is running out of space and expandability.

Before I go further, "online" storage in my terminology means always-on, but it's on the local network, not in the cloud or on the internet. Keeping a large server or PC is not cost effective (not to mention power, heating and noise issues) for home users or even home-office users where the number of users or devices is fewer than 5.

So, what do I need in my online storage? Considering it's not going to be primary backup (my backup machine is a dedicated PC with 2.5TB of storage), I think a RAID setup is fine. You can read here where it says smart SOHO people don't do raid. Anyways, I'm also looking for a 4-drive unit. It also needs to serve iTunes as well as provide network and iSCSI fileshares. In addition, I'll need it to able to handle the small number (and growing slowly) of IP cameras I have littered around the house. This limits me to several NAS models but right now, I'm looking specifically at a QNAP unit.

More soon...

I think it's safe to say that most people would like to travel. Most people would also love to travel abroad. If you love traveling and photography means that you'll definitely bring along a camera. If you really enjoy traveling and really are into photography, you will probably end up taking a lot of photos. You'll also wake up earlier and stay out longer.


Back in the old days where photos (not images) were captured on chemicals, the average storage capacity was thirty-six photos. If you lucked-out, you could sometimes get 37 before the rewind kicks in. Rewind? WTF? I remember lugging forty rolls of film to Spain. It was estimated at that time that I'll shoot on average 2 rolls a day. This is to negate out time spent traveling between destinations (we almost always self-drive), chilling out (hey, you need to take a break and enjoy your time too), recovering from jet-lag and so on. I digress. Fast forward ten years and I can now have 630, yup, over six-hundred photos (21-megapixel RAW images) on my 16Gb CF card. Only that I tend to shoot a lot more and more little things get shot, food, street signs, mundane things, etc. Then came HD video. So unless you're super rich, buying twenty 16Gb CF cards is not the solution.

My travel buddy has always been Breeze Systems Downloader Pro (DLPro). I also use it for wedding and other large shoots coupled with several card readers. Today's multi-core computers can handle quite a bit of simultaneous downloads. So far, I average 16MB/sec per CF (8Gb SanDisk Extreme III) with five SanDisk All-in-one USB card readers downloading simultaneously into a striped disk array. Not very fast, but very convienent. I can go shower, have lunch, nap while the download is running. I've tried those image tanks or similar devices. Never worked out for me. Some were horribly slow, others just had poor UI and with the costs of today's disks drives and netbooks, it makes sense to go with that route. For trips where I'm with a rental car (i.e. Australia or the US), I'll normally take my full-size (14") notebook but trains, backpacks and mountains don't really agree with heavy notebooks.

For travel, I'm using DLPro to download from a card reader into two hard drives. Here's my typical desktop configuration (Quad-core, 8Gb RAM, etc) so I won't be configuring it to show thumbnails on my Mini9 for example.

Downloader Pro basic job-code, download preferences, IPTC keywords, etc.

Downloader Pro Back-up Plugin allows downloads to be replicated onto a second or even third storage location. My Wedding shoots get backed up immediately to a NAS this way. So here, mine goes to the NAS drive immediately. For my Mini9, I'll configure it to go to the second hard drive.

You can launch multiple instances of DLPro, hence the beauty of this solution. My Atom powered Mini9 is going to choke on multiple card readers so I stick with one reader for now. I can't wait for the dual-cores though. For now, weight is the main factor so have to make do with the tools at hand. More to come.

With the explosion of Digital cameras, digital video recorders, iTunes, MP3 players, Youtube and more, storage of digital media is going to be very important, if not downright critical. How many people do you know who really backup what digital media files they have? It doesn't need to be photos or videos. I know this entry is filed under Photography but digital media can mean documents and spreadsheets as much as photos and video. It can also be your digital audio collection.

Primarily, the backup we're talking about is just a second copy of your information somewhere else. Where? Well, anywhere other than your PC is good. I don't agree much with same PC backup (and generally, many users can't tell the difference between physical drives and partitions, etc). So the question is where and how to store the backup.

WHERE?
The most simple location is probably very ubiquitous, the USB drive (aka thumb drive, usb stick, etc, etc). With drive capacities at 32Gb and beyond, and at affordable prices, it makes is ideal for users with limited amount of information. My parents for one can do with a 4gb USB drive. If you have more stuff to backup, get a portable or external hard drive. Here are some that I know of that users have had good experiences with: Western Digital MyBook Essentials or Seagate FreeAgent Go if you want something portable. For larger disk capacity, Western Digital Essential, Home or Studio or Seagate's FreeAgent Xtreme make good buys. For heavier disk storage and on a networked environment (like me), you can go with a Network Storage Appliance (NAS). I recommend QNAP's TS-209 or Synology's DS-207. Those on a tighter budget can opt for DLink's DS-323 or single drive versions of the QNAP and Synology models I mention previously. However, those require a bit more know-how to setup and manage. Me, I run a custom built NAS which is actually a dual-core box (speed doesn't matter) with 8-SATA disk ports (these matter, RAID is not required) and 4GB RAM (this also matters). Read the full entry to see how it is setup.

HOW
This is as important as where because if you don't store your backup properly, you'd find yourself in trouble when you need the backup. Generally, most people can get by with weekly or even monthly backups. We aren't large corporations needing daily or hourly backups so a week or so is good for 99% of home users. Some probably can get by with once a month. If using a portable hard drive, make sure you keep that in a safe place. I'm not going into backup software but because I don't use them myself -- primarily, my goal is to mirror (copy all) the data across to another disk. I use Acronis TruImage to backup the system drive but applications can be re-installed, data can't so my focus here is on data. Firstly, ensure your data is stored logically and in an organized drive. This makes backing up easy. My favorite tool is robocopy. It works very reliably, and the ability to monitor folders for changes is invaluable. I use robocopy to copy from my network PCs directly to the backup server regularly. If you want a prettier utility, there's Allways Sync.


Continue reading "Safe-storage" »

In research independently commissioned, 63% of digital camera owners risk losing a lifetime of memories as they opt for saving images on a computer or printing images using an inkjet printer at the expense of obtaining true quality prints. For the 39% of those surveyed who rely on storing images on a PC there comes a strong warning from computer experts who pinpoint the potential risk of hardware failure and data loss. Computer consultant Adam Olliffe says: "The hard drive is a mechanical device prone to failure. Technology over recent years has concentrated on squeezing more and more data onto the same disk surface area."
Adam Olliffe added: "When it does fail all data is then lost with no chance of retrieval. Businesses will always consider data backup as a crucial part of their IT infrastructure, however most home users will not address this issue until it is too late." Alarmingly, 81% of those who regularly saved images on a PC did not have any form of back-up. A further concern is for the 24% of those surveyed who print images using an inkjet printer.
So, backup, backup, backup!