Posts tagged storage

Network Attached Storage

Don't Panik

From Wikipedia : A NAS unit is a computer connected to a network that only provides file-based data storage services to other devices on the network. Although it may technically be possible to run other software on a NAS unit, it is not designed to be a general purpose server.

NAS vs SAN
NAS provides both storage and a file system. This is often contrasted with SAN (Storage Area Network), which provides only block-based storage and leaves file system concerns on the “client” side. SAN protocols are SCSI, Fibre Channel, iSCSI, ATA over Ethernet (AoE), or HyperSCSI. Despite their differences, SAN and NAS are not mutually exclusive, and may be combined as a SAN-NAS hybrid, offering both file-level protocols (NAS) and block-level protocols (SAN) from the same system. Considering I’m writing for home-users, a NAS makes more sense.

With the amount of data many photographers (and users) are generating, it’s become obvious that storage is a problem. For casual users, it probably may not be a problem in the beginning but it soon will be. I know of friends who after a few years, a kid or two and the amount of photos, videos, music, documents, scanned PDFs, etc start to add up to gigabytes. Burning to DVD (gosh, who burns CDs today?!) can alleviate some of the problem but with 4.5Gb (dual layer disks aren’t common as they’re not so cost effective), it’ll take a few DVDs for the average persons’ backup. Worse still, those DVDs may not be readable after a few years. A friend of mine burns two discs for each copy — and 3 years later found out that some are not readable. Good thing he had two copies as he’s managed to combine them to even out things so no loss occurred. So, he consolidated the storage. Where? Good old spinning platters. With the falling prices of hard drives, it’s a no brainer that disks are still the most reliable, cheap and effective storage for backup for most users today (most users here I mean home and small business perhaps). Reliable? Yes, if you have two disks and you replace them religiously every 36-months. That’s what I do. The benefit is that you’ll end up having larger disks too. My backup server has 1TB disks now. It started out with 160Gb and 250Gb disks (two pairs) for a capacity of 450Gb. My data very important to me so I change disks every two years. I then went to 320Gb and 500Gb (still two pairs) and now, I’ve got four 1Tb disks plus a pair 500Gb disks (these store my music, video and misc files). They’re going to 1.5TB disks soon (best price per Mb).

The problem is usually, you won’t want your backup server running non-stop. Mine’s on a UPS along with my network, and more but most casual users also don’t need access to their backups every single instant. They also normally don’t have another spare machine lying around. In many organizations today, they use the term near-line storage for things that can be retrieved faster that tape but slower than current online storage. Of course big organizations use fast disks arrays for online and slower and usually SATA disks for near-line storage. In my case, I’ll call my storage server near-line because it may not have been started up. Where’s my online storage? Well, it sits on a NAS, a QNAP TS-439 Pro. This is a bit too much for most home-users (they can go with a TS-210 or TS-219) instead. I’ve a RAID 5 set of 500Gb disks in there, giving me just under one Terabyte of available storage. It makes a good always on file-server, repository and immediate backup (Downloader Pro immediately makes a copy of my photos there). When time comes, it gets rysncd to my near-line storage server. The reason I got the 439 Pro was because it was on promotion. It’s been replaced by the Pro II (better CPU) and was the same price as the 419P. To me, the benefits were the faster (compared to 419P) CPU, and larger RAM (1Gb 439 Pro compared to 512Mb on the 419P) which translates to faster transfer speeds. The 439 Pro also has the memory on a SODIMM which means I can bump it up easily to 2Gb. QNAP’s disty over here still has a few units of 439 Pro if anyone is interested.

So what’s for the average home user? Well, if you have more than one computer in the house, would like to share some storage, do some backup, stream iTunes around the house, etc, then any one of these NAS units over here, here and here would serve you well. You’ll also notice that I’ve avoided any form of single drive device. That’s because most home users “think” that once it’s backed up, it’s going to be safe. Two drives are always better than one. If you’re on a tighter budget, get a pair of USB drives. Seagate or WD are good options. Just buy two and make sure your data is identical on each one. And, yes, please store them in separate places — at least a different room. My storage server and NAS are three walls and fifty-feet apart. Note: RAID does not prevent idiotic errors like deleting your only copy of the most important file or photo you ever have — the array isn’t going to care and will promptly erase all for you. In my case, well, I can still go back to either my storage server (100% read-only) or desktop.

My choice of 4-bay, well it’s more for RAID 5 as I’m using older disks for it (i.e. the ones retired from my storage server) so it’s good to have RAID5+1. RAID 5 capacity is (n-1) resulting in the available space of two disks, i.e. in a RAID 5 set of 3x500Gb, I get 2x500Gb usable. The +1? That’s the hot-spare in case something dies which is for the paranoid like myself. If you go with plain RAID5 on my setup, you’ll have 3×500 resulting in a usable 1.5TB.

I’ll post again why I chose the QNAP NAS plus a mini-review… I’ll stop here… more to come while I leave you with a pretty picture of my network appliances 🙂

CF, SD, Batteries…

Photography

A sign of the times. It used to be little kiosks selling CR123, CR5 or CR2 lithium batteries (remember those) and AA, AAA batteries alongside boxes and boxes of Kodak and Fuji film. Today, if you find your fancy-pants digital camera running low on juice or memory cards, you go look for a little stall like this one.

This little kiosk in the Yuyuan area in Shanghai was selling all sorts of media cards (CF, SD, SDHC, etc) along with 3rd party made-in-China batteries in various Canon, Nikon, Olympus, Panasonic flavors. They do say that the batteries are “charged” so you can just pop them in and use them. This particular kiosk still had some film products but they were heavily showing off their instant inkjet printing service. While Kodak may be a icon of the film era, its transition to the digital era has hit some turbulence. So while they had a couple of rolls of Kodak film, majority of their products are for digital cameras. I wonder when I might see a Sandisk branded kiosk… soon perhaps?

Run out of batteries and media? Head for the kiosks!

p.s. what “real” photographer would only have one battery and one memory card with him/her at one time?

Online storage shopping…

Don't Panik

The problem about ever increasing mega-pixel camera counts and media (including HD video) is that the disk space is getting smaller and smaller each day. The added complication is when you have more than one user trying to work on the same set of images at the same time. This becomes an issue with local storage hence the need for networked storage so that you have less issues with trying to sync two or more working sets of raw, tagged, marked-up or processed images. So, the result is that my dual-drive network storage is running out of space and expandability.

Before I go further, “online” storage in my terminology means always-on, but it’s on the local network, not in the cloud or on the internet. Keeping a large server or PC is not cost effective (not to mention power, heating and noise issues) for home users or even home-office users where the number of users or devices is fewer than 5.

So, what do I need in my online storage? Considering it’s not going to be primary backup (my backup machine is a dedicated PC with 2.5TB of storage), I think a RAID setup is fine. You can read here where it says smart SOHO people don’t do raid. Anyways, I’m also looking for a 4-drive unit. It also needs to serve iTunes as well as provide network and iSCSI fileshares. In addition, I’ll need it to able to handle the small number (and growing slowly) of IP cameras I have littered around the house. This limits me to several NAS models but right now, I’m looking specifically at a QNAP unit.

More soon…

Disk storage and travel

Photography

Looks like wet weather put a damper on my predicted shoot rate. The other factor being the red-eye flight contributed to a wash-out day in terms of photography. How much was I off? Well, I predicted for 10-days I’d shoot approximately 280 Gigabytes. I returned with 93 Gigabytes. Looks like off by quite a bit but then, we had spent over 12 hours and 1500km on the road and rail. How much can you shoot from a moving vehicle? Also, I realized that my GF1 cannot be counted to be a third camera when estimating disk/shoot usage. You can’t use two cameras at the same time so the 60 gigs I estimated for the GF1 dropped to only 21 gigs — and I had 30 gigs of SDHC available. Anyways, always good to be prepared so while I could have saved some cash by not getting my 320Gb Seagates, they always will come in handy next trip… or anyone interested buying them off me? They are in black and silver colors.

Downloader Pro running at Hongcun Village with my two FreeAgents…

Cards, and more cards…

Photography

It wasn’t very long ago that RM1,000 (about US$300) bought you a 128Mb Sandisk Compact Flash card. Go a bit more and that same amount got you a 1Gb Sandisk Ultra II that wrote at about 5mb/second. Today, you can get a 32Gb Extreme IV (the Extreme Pro’s are a bit more than that) that has a 60Mb/second speed. That’s the good. The thing also is that when I had a 128Mb CF card, I had a 1.6 megapixel digital camera, shooting JPEG only. I think it also only had ISO 100, 200 and 400. Those images were about 500 Kb in size. Now, I shoot 21 megapixel 14-bit RAW images, ISO 50 to 25,600. Oh, plus HD movies that chew 320Mb per minute. So, the point is basically this – you can never have enough storage cards. I’m packing over a hundred-twenty-odd gigs (CF and SD combined)… and sure, I could use a couple more 16-gig cards 🙂

Safe-storage

Photography

With the explosion of Digital cameras, digital video recorders, iTunes, MP3 players, Youtube and more, storage of digital media is going to be very important, if not downright critical. How many people do you know who really backup what digital media files they have? It doesn’t need to be photos or videos. I know this entry is filed under Photography but digital media can mean documents and spreadsheets as much as photos and video. It can also be your digital audio collection.

Primarily, the backup we’re talking about is just a second copy of your information somewhere else. Where? Well, anywhere other than your PC is good. I don’t agree much with same PC backup (and generally, many users can’t tell the difference between physical drives and partitions, etc). So the question is where and how to store the backup.

WHERE?
The most simple location is probably very ubiquitous, the USB drive (aka thumb drive, usb stick, etc, etc). With drive capacities at 32Gb and beyond, and at affordable prices, it makes is ideal for users with limited amount of information. My parents for one can do with a 4gb USB drive. If you have more stuff to backup, get a portable or external hard drive. Here are some that I know of that users have had good experiences with: Western Digital MyBook Essentials or Seagate FreeAgent Go if you want something portable. For larger disk capacity, Western Digital Essential, Home or Studio or Seagate’s FreeAgent Xtreme make good buys. For heavier disk storage and on a networked environment (like me), you can go with a Network Storage Appliance (NAS). I recommend QNAP’s TS-209 or Synology’s DS-207. Those on a tighter budget can opt for DLink’s DS-323 or single drive versions of the QNAP and Synology models I mention previously. However, those require a bit more know-how to setup and manage. Me, I run a custom built NAS which is actually a dual-core box (speed doesn’t matter) with 8-SATA disk ports (these matter, RAID is not required) and 4GB RAM (this also matters). Read the full entry to see how it is setup.

HOW
This is as important as where because if you don’t store your backup properly, you’d find yourself in trouble when you need the backup. Generally, most people can get by with weekly or even monthly backups. We aren’t large corporations needing daily or hourly backups so a week or so is good for 99% of home users. Some probably can get by with once a month. If using a portable hard drive, make sure you keep that in a safe place. I’m not going into backup software but because I don’t use them myself — primarily, my goal is to mirror (copy all) the data across to another disk. I use Acronis TruImage to backup the system drive but applications can be re-installed, data can’t so my focus here is on data. Firstly, ensure your data is stored logically and in an organized drive. This makes backing up easy. My favorite tool is robocopy. It works very reliably, and the ability to monitor folders for changes is invaluable. I use robocopy to copy from my network PCs directly to the backup server regularly. If you want a prettier utility, there’s Allways Sync.
Read More