Picade Member Extranet

I can't help myself ... I'm still trying to pack too much "information" into what I write, it's still too geeky by far, so I'll try and adhere to the spirit of my recent pledge by writing a really short summary and then my longer "mad scientist"/business version with all the gory details.

The Really Really Short Version

I'm changing our agency image filenames for really good reasons. What our photographers do will continue unchanged. Without re-compressing them, we'll rebuild our images with cleaned-up metadata and reload them to PhotoShelter and to Amazon S3 for backup.

The "Long Winded", gory details version

I'm trying to cleanup our images and image metadata to continue to make Picade an industry leader in using metadata, and to eliminate problems with Digital Asset Management (I'm going to have to skip the D__ acronym - the blogging software doesn't like it and it won't display this posting otherwise!!!) systems, and to eliminate problems with our photographers getting properly paid and protecting their work. Now while we have a relatively few thousand images online is the time to do this, rather than later when we have hundreds of thousands or even millions of images online.

Digital computers are relatively rigid, and essentially do exactly what they are told to do (even if that is not exactly what their owners intend) every time they do it with the "same input", and therefore sometimes "do not like" input they are given to deal with because of rigid two-valued "Boolean logic" embedded in their programming that does not accept input variations created by humans.

Humans on the other hand frequently do things differently every time they do them, and are very much more flexible than computers and "recognize" close matches to an expected situation (so called "fuzzy logic" ), and therefore sometimes as a result can make mistakes with regard to working within rigid specifications or guidelines for input to computers.

Picade photographers and Picade customers are humans and make human type mistakes, and as a result we have to mitigate against the results of these types of mistakes when working with Picade computer programs that get their input from these human beings. Further, Picade customers and vendors utilize their own separate computer systems and programs that get their input from human beings and from Picade which have unknown to Picade requirements, and we have to mitigate against problems with those systems and our images as best we can.

One of the principal business plan requirements for Picade to exist and function successfully in the current marketplace was to eliminate as much as possible of the agency "business overhead" of human employees by having the photographers of Picade do as much of the agency work as possible, having "automatic" computer processes handle as much of the rest as possible, thereby leaving essentially only a basic management, marketing, and negotiation process to be "hired out" to paid human beings.

Principal among the agency "work" items to be handled by individual photographer's was the preparation of images for the website and delivery to end customers who would license the images, something that in other smaller agencies is responsible for as much as 75-80% of the total agency "overhead".

This requires the Picade photographers to properly prepare their images for final delivery to customers, embedding in the images sufficient copyright licensing and creator identification metadata to prevent "orphan works" issues, as well as automating image metadata input into Picade's and customers DAM computer systems to enable both the Picade website search routines to function, and to enable proper management of the images within a marketing and licensing context.

Among other things, this process is extremely equitable in sharing costs and work among Picade photographers: everybody's images benefit exactly the same, and no one bears the cost burden of processing or preparing images for another photographer who uploads more images than they do, because some photographers by the nature of their work generate very few images, while others generate very large numbers of images.

There have been some significant overall agency workflow errors resulting with this approach, some of which have been mitigated against by redundancy built into the original specifications for Picade photographers, and some of were not foreseen to be a problem or which subsequent events have caused to become a problem which have required significant manual processing to overcome.

Currently the image filename we have Picade photographers create for images uploaded to Picade's PhotoShelter MU account is composed of three parts and is supposed to fit a rigid, 21 total characters long filename format plus the ".jpg" filetype suffix:

1. The 4 character Picade photographer Id
2. A single character agency exclusivity flag: ['E'|'N']
3. Exactly 16 characters for the photographer's personal ImageId, padded on the left as needed with zeros, using only the specific character set [0..9,A..Z,'-']
4. The four character ".jpg" filetype suffix

The issues with the images

Now at 7000 images uploaded to PhotoShelter MU for Picade there have already been:

1. A single case of an image incorrectly tagged with an incorrect 5 character photographer Id in the filename - if this information is not correct we can never properly pay the photographer for their share of licensed uses!
2. 45 cases where the image filename was the photographer's own raw filename with no preparation at all (including not having the photographer's Id and exclusivity flag character embedded - see item #1 above about payment)
3. Nearly a thousand images where the image filename contained characters other than the approved character set of [0..9,A..Z,'-'] or that was incorrectly formatted, potentially causing problems with end users internal DAM systems, and definitely with Picade's.
4. Nearly 1500 images where there were incorrectly formatted or missing photographer contact metadata embedded.
5. About 400 images that were uploaded more than once, in once case, 5 times

A Picade photographer may upload an image to Picade's MU account on PhotoShelter more than once and we have to properly deal with this both in our own record keeping and on PhotoShelter to track metadata changes.

On PhotoShelter, when a photographer uploads images to their personal account, they are given the option to "overwrite" a previously uploaded image of the same filename.

When a photographer copies an image multiple times from their personal account on PhotoShelter to Picade's MU account on PhotoShelter, they do not get the same "overwrite" option, with the result that Picade will have multiple copies of the same image in its MU account that will appear multiple times in search query results for customers, and that requires substantial human effort to first identify, then fix the multiple uploaded image issues.

The Picade photographers may also make errors in naming the file (viz: the filename is too long or too short, may contain "disallowed" characters, or even be improperly formatted), or changes in the filename for a given image (viz: making an 'Exclusive' image 'Non-Exclusive' or vice-versa, with a matching change in the 5th character of their filename, but with the actual image remaining the same) becuase of changes in agency exclusivity.

Actually determining that a given image is uploaded under different filenames, or different images are uploaded under the same filename based solely on the visual aspects of the image is very difficult and consumes a lot of computer resources and programming to do automatically, and as we get more and more images we will in fact likely have to implement such a solution, but at present we are still using the standard issue "Mark I, Mod I, Eyeball, Human" to do so.

What we will be doing about the issues

As a result of these issues and other considerations external to Picade's and Picade's photographers workflow, it is now necessary to change and simplify the filenames of the images we present to the general public for licensing consideration, thereby requiring renaming of the images and thereby simplifying some issues of the existing problematic photographer image naming: we can control the exact character set and length of a delivered filename so as not to cause problems for an end customer or for ourselves.

We will not change Picade photographer requirements for naming image files for upload to Picade: these requirements contain necessary information redundancy to help eliminate human errors and need to be preserved.

On the other hand, for online imageid display and delivery to customers and for internal marketing management we are going to be renaming all Picade images in the following manner:

1. The 4 character Picade photographer Id
2. A single character agency exclusivity flag: ['E'|'N']
3. A 7 character unique numeric ImageId encoded in base 32 that uses a specially selected character set to eliminate multiple human and computer input related errors. The character set we will use is "0123456789ABCDEFGHJKLMNPQRSTVWXY", and a six character "number" in base 32 using this character set can encode 36,507,222,015 possibilities before we run out of numbers, and a seven character "number" can encode 1,168,231,104,511 unique possibilities: it will be a very long time before we exceed those limits!!!
4. The four character ".jpg" (or possibly in the future: ".tif") filetype suffix

That will mean that we will maintain internally the Picade photographer's own originally created and uploaded image filename in our Digital Asset Management system as image metadata, but will present a different, more easily read one to customers that will eliminate certain types of common errors. We will also make available to Picade photographers a cross reference between the ImageId that they uploaded, and the new ImageIds that Picade will be utilizing, and all reports of licensing to photographer's from Picade will contain both references.

For example, a recently uploaded image:

1028E00000RNA_1920_07.jpg

uploaded to Picade's PhotoShelter MU account by Picade photographer Richard Anderson as the 6888th unique image for Picade will become internally:

1028EA000757.jpg

and will eventually get downloaded to end users as:

www.picade.com-1028EA000757.jpg
or
www.picade.com-1028EA000757.tif

This image file renaming alone will require that we rewrite the embedded metadata in the images (without jpeg image re-compression/re-encoding by using the Bert Bos wrtjpgxmp tool I recently wrote about) to account for the embedded imageid title metadata attribute.

At the same time we will take the opportunity to write into each image a uniformly formatted set of photographer contact and licensing metadata to correct missing or poorly formatted metadata, and to eliminate Picade specific metadata or camera and software processing metadata (added by some Raw image processing programs: 1100 lines of unneeded metadata text in the case of an image recently uploaded), and default whitespace padding from the images delivered to customers and stored online. Among other things this will save a few thousand bytes of storage and bandwidth for each image stored online, which in the long run really adds up.

We will upload these rewritten and slimmed down images to our backup archive on Amazon S3, and finally to Picade's PhotoShelter MU account after first deleting the existing photographer named images (or subsequent Picade renamed images), thereby cleaning up online issues with our images on PhotoShelter in terms of duplicates, and guaranteeing proper image metadata for delivery to customers.

This is a lot of extra work to do over what we originally envisioned when Picade was founded, but it will result in far fewer problems with our Digital Asset Management system, our customer's Digital Asset Management systems, and with PhotoShelter's systems.

Once we implement this as part of our standard agency workflow, and get the current relatively small set of images re-processed and re-uploaded, it will not add a lot of overhead work for the future. I would hate to have to think about reprocessing hundreds of thousands of images, or even millions (which of course where we want to get to as an agency <GGGGG>).

-30-

Michael Beasley (who just wants to get back out shooting pictures!)

It should come as no surprise to anyone who has talked to me or worked with me regarding digital images or stock photography that I am a great proponent of embedded metadata in digital images, and that in my work as a consultant to other photographers and stock photography agencies I have implemented strong use of embedded metadata in digital images to help solve some of the problems of "orphan images". It certainly will not be a surprise to the Members of Picade who have had to struggle to come up to speed with our requirements to do so since prior to our inception.

In a major Chicago ad agency conference room recently I was asked a fundamental question by an art buyer I have heard countless times in many forms in the last ten years as digital images have became the norm, and that has recently given rise in the US Congress to the specter of "Orphan Works" legislation:

"One of my [art directors|researchers|secretaries|friends|...] found this image that they want to use but they [didn't|can't] tell me where they found it. How do I find who owns it and how to license it?"

The fact was that the digital image in question (which they thought might have been one of mine and was unfortunately not) had no identifying characteristics: no visible or invisible watermark, a numeric filename that gave no clues about anything, and no embedded metadata.

The image was an "orphan" that had to be discarded for consideration as the anchor image in a major ad campaign because it was impossible to find the owner [cue sound of gurgling water from a flushing toilet].

Photographers and agencies: every image that leaves your hands or those of your agents or distributors must have embedded in it as metadata your copyright notices, your contact information, image identifying information, and basic terms of licensing AND be properly registered with the Copyright Office. Anything less means that you will likely lose license income and may suffer from image theft or the penalties of any "Orphan Works" legislation that may be passed in the future - which might even strip you of your copyright in the images or the ability to derive licensing or penalty income from prior uses that were not properly licensed.

Adding other identifying features such as a "digital slide mount" with your name and contact information visibly displayed outside the image area, a distinct and recognizable watermark, a filename that incorporates your web domain as well as a unique image identification, all are helpful as well.

A sample "digital slide mount" comp image of my own, similar to what I eventually hope to produce automatically for Picade's website for downloadable comp images:

I am in the implementation phase of Picade's first customer oriented website that will customize our presence on PhotoShelter, and some of the additional steps that I referred to the immediate paragraph are not immediately possible because of the design and functionality of PhotoShelter's system, but will definitely be part of the implementation of Picade's first stand alone website presence [also concurrently in development]. All of the previously mentioned embedded metadata are there in our images now, and I had a very significant struggle last year on behalf of all photographers to keep PhotoShelter from routinely stripping all of the embedded metadata from displayed images, which they no longer do as a result.

One of the competing interests in designing and hosting a website is to make the pages (and displayed images) download and display as fast as possible for the viewer and which minimizes the cost of bandwidth for the website owner.

This means the html markup and the images themselves should be as lean as possible, and our industry's primary image tool Photoshop has had a feature since about version 6 called "Save for the web" to make images that were "optimized" for the web by among other things, stripping out all the embedded metadata in the image to make the filesize of the image smaller and to therefore make the image download and display on the web faster.

This has put Adobe squarely at odds with the competing interests of the original image creators and the webs' design and web hosting side of the business, and Adobe have grudgingly and somewhat sheepishly acknowledged this fact by burying within the "save for the web" interface of recent versions of Photoshop the option to include some XMP metadata within the stripped down image. In a posting on his Adobe blog, John Nack references how to do this.

Unfortunately, as I discovered last week while helping one of our Members and a web developer he was working with on an image metadata display issue, the resulting embedded XMP metadata in a CS3 "save for the web" image does not contain all XMP metadata, only effectively the metadata that corresponds to the original Photoshop "fileinfo" binary IPTC metadata, and explicitly not the complete set of the newer IPTC4XMP metadata including the "Creator Contact" information.

Early this week a reference to the John Nack posting on the Adobe blog referenced above showing how to add XMP metadata to a "save for the web" image was made on another private professional group forum, and I posted some information there regarding the issues I had found and a workaround involving an open source command line tool, binaries of which I have posted here on the download files section of the Agora for both Mac PPC and Win32. If I can find a way to get a binary compiled for Mac Intel I will post that as well.

The workflow is basically four steps, and uses Photoshop CSn both to create an XMP metadata packet and to create stripped down "save for the web" jpeg images, and the open source wrjpgxmp command line tool from Bert Bos of the W3C that enables you to embed an XMP metadata packet into the stripped down jpegs without recompressing and re-encoding the jpeg image itself.

1. Get the Bert Bos jpeg xmp toolkit either from his website as a C language source distribution and then compiling it, or as immediately useful compiled binary executables by downloading from the files section here. With respect to the Mac PPC binary, thanks to Aaron Bieber for compiling it and sharing. This is a one-time step.
   
   
2. Use Photoshop CSn to create an XMP metadata template of the information you want to embed in the image(s) - a step that should be very familiar to Picade Members from their workflow of preparing images for Picade. If you are using Photoshop CS, you will need to obtain and install the IPTC4XMP metadata input template that comes already installed in Photoshop CS2 and CS3. If all you are doing is embedding essentially static contact and copyright information metadata, this is a one-time or at worst a single yearly step, but the ease of creation makes it possible to use this for shoots, assignments, or projects with relevant specific additional metadata as well.
   
   
3. Create optimized jpegs for your end website use by soft previewing in Photoshop using either the generic or PC monitor profile to properly visualize your work results, converting to sRGB profile, then using gamma, levels, curves, and color correcting your image for Gamma 2.2 display, and appropriately sharpening and resizing the image to your final output size. "Save for the web" as a jpeg without using the option to embed XMP metadata, at a size/quality setting that is appropriate for you purposes, in a directory where you will do your later work in this workflow. This is essentially what you have always done (or should have done) in the past.
   
   
4. Embed the XMP metadata template you created in step two in your images using the Bert Bos wrjpgxmp command line tool. In the case of the specific metadata packet that I personally use (derived from Photoshop CS, CS3 might make a smaller packet), this increases the size of the stripped down jpeg image by 3793 bytes per image after embedding.
   
   

To expand on the hard bits of this a little, you create a minimal XMP metadata template in Photoshop CSn containing your personal contact and licensing information to use with the workflow by:

1. Create a "new" image (Control-N, Option-N) - this has no embedded Exif or other metadata from previous work, a very important step to minimize the amount of "stuff" in the generated XMP datafile. Just use the defaults in the new image dialog, we'll be throwing the file away later.
   
   
2. Open "fileinfo" (Alt-Control-I, Alt-Option-I) for the new image. Using the IPTC4XMP and other metadata templates, create your contact information, copyright status flag as "marked", copyright notice, copyright information url, rights usage terms, and special instructions and any other information you want to embed in the processed jpegs.
   
   
3. Save the generated "fileinfo" metadata as a template using the arrow button in the upper right hand corner of the fileinfo dialogs to an appropriate name that you will recognize. Cancel the fileinfo dialog, delete the temporary imagefile, and exit Photoshop.
   
   
4. Find the generated "fileinfo" XMP metadata template file on your computer (it will be named as you named it with an ".xmp" filetype extension, on Windows it will be under "C:\Documents and Settings\\Application Data\Adobe\XMP\Metadata Templates") then copy it where you will be working later.
   
   

Screen captures of the process follow (I cheated and loaded a pre-existing metadata template to save retyping, so the initial Description fileinfo dialog has an anomalous "Created" date from last year at the bottom at the bottom), and I am presenting them at "life size" so that they are readable, even if it will mess up the web document display a bit.

Preparing your images prior to "save for the web" should of course be done on a calibrated, profiled, and color managed system. The biggest part of image preparation is properly previewing your visual changes to the image by soft proofing, by first setting it up for either a Windows RGB or a generic Monitor RGB profile, and then turning on soft proofing (Control-Y, Option-Y) while you are working on your images, and in my opinion, by using a "stair step sharpening" image reduction process similar to the Photoshop action you can buy from Fred Miranda, or that is detailed in principle by David Riecks on his Controlled Vocabulary website, to counter the detail blurring that occurs when you reduce a digital image in size.

The final step requires opening a "command line window", "dos prompt window", or "terminal window" to your working directory and running a command similar to the following (assuming the xmp template file is named "template.xmp" and the minimal "save for web" jpeg is "input.jpg", and your desired final filename is "output.jpg") and everything is in the working directory:

wrjpgxmp -cfile template.xmp input.jpg >output.jpg

A screen capture of preparing the "digital slide mount" image used at the start of this article:

Obviously this final step is a one-by-one image step that can be automated using various forms of scripting specific to a particular operating system or scripting language, but that is "an exercise left for the student" <GGGGG>.

What I will say is that I personally name my imagefiles by the image identifier used in my DAM system, with a postfix to the filename name that indicates the class of the imagefile (I store several levels of prepared imagefiles for use by reproduction size for print or the web), and that downloadable "comp" images prepared by this workflow get named with my domain_name+imageid+size_usage_suffix.jpg.

As an example, here is a web preview sized image of one of my Chicago subcollection #D001 images #C030 maintained internally by me:

CHGOD001C030_P.jpg

For Picade, internally this images' master would become:

1002E00000CHGOD001C030.jpg

presented externally as a download protected "preview sized" image or swf:

1002E00A012FGM_P.jpg
1002E00A012FGM_P.swf

and the downloadable "preview sized" comp image (which in the future will be a "digital slide mount" image) as delivered to an end user would be named:

www.picade.com-1002E00A012FGM_P.jpg

When named in this manner, your image filenames are essentially globally unique and contain information about their source (the domain name), your personal image id, and a filesize reference that makes the image unique within your own filesystem. There is other proprietary information embedded that helps us when a customer requests pricing or licensing information from Picade, but as noted, that is proprietary.

HTH.

Michael Beasley

Picade's First Anniversary

The 16th of June officially marks Picade's first anniversary in existence as a legal entity, and it is a time to reflect on where we came from and where we are going.

There are congratulations to be given out to all of us, but one person in particular stands out: Brian Seed.

Picade started life as a series of postings on the SAA mail list/forum by Brian, and he has vigorously advocated and pursued the idea of a photographer owned, true stock photography agency ever since. In the truest sense of the phrase, without Brian Seed we would not exist today, and we all owe him an immeasurable debt of gratitude for his vision and for the job that he has done.

Kudos friend Brian, many, many kudos for a job extraordinarily well done!!!

Congratulations also, to all of our pioneering Founding Members for having the faith and conviction to take the risk and try and create something new and benefical to photographers:

Paul Avis,
Cristian Baitg,
Michael Beasley,
John G. Blair,
Burgess Blevins,
Beth Dixson,
Rob Doda,
Nick Dunmur,
Laura Dwight,
Elizabeth Etienne,
Garry Gay,
Thomas Hallstein,
Michael Hart,
Mike Hill,
Spencer Jones,
Ali Kabas,
Bill Lesch,
Peter Miller,
David Paterson,
Marc PoKempner,
Clayton Price,
Nat Rea,
Antonia Reeve,
Marco Secchi,
Brian Seed,
Terry Thompson,
Michael Vitti,
Andrew Wakeford,
David H. Wells,
and Harald Woeste

Along the way to where we are now we have weathered a number of contentious trials and tribulations that threatened to sink the business stillborn, and have lost the energy and future contributions of people who might still make a positive contribution to us if they choose to join, and we offer them an open hand of friendship and welcome if they choose to do so.

Where we are now

Today we have a real legal business that is a year old, with 30 founder owner/member photographers, with money in the bank, with a web presence via PhotoShelter that offers (04 June 2007) 6769 professional images currently online and searchable for rights-managed licensing with the capability to do full online ecommerce payment.

Our images are backed up in three geographically diverse locations, including in our physical office in Chicago, where we have business phone and fax numbers now appearing in the printed telephone books.

Our Board and management committees are in existence and functioning and working very hard for our mutual benefit.

Finally, we have standing in the wings a professional stock photography agency business manager with extraordinary credentials, Michael Jungert, who will soon begin to help us transition the agency to the sales and marketing phase of our history, and then to take over the day-to-day marketing and sales management responsibilities of our agency.

We have worked hard on creating and embedding significant metadata in our images, and have fought and gotten major structural changes at PhotoShelter to preserve that embedded metadata in delivered images such that we should not have to worry about the movement to create an "orphan works" exception to copyright, and are thereby leading most if not all other agencies in this area.

Our own website has been in existence for 7 months and currently is configured primarily to attract new Members, and to provide a mechanism for communication and collaboration among the Members, and is in significant use by the Board in our own collaboration and in working to obtain new Members.

Where we are going

The growth of our Membership and our image collection is key to our survival, and while Brian is retiring as Board Chairman, he intends to keep working on our mutual behalf by heading our effort to get quality new members and their images. Key to that effort will be our website and starting to make actual licensing sales.

I am currently working on the design and implementation of a new customer-centric focus for our website that will integrate with our images and presence on PhotoShelter to provide a seamless design and experience for professional buyers wishing to license the work that we do, while still affording us our private communication areas and the ability to attract new photographer Members.

This new website design will shift the primary function of our site from obtaining new Members, to attracting buyers and facilitating licensing our images. It will be structured to place minimum barriers between buyers arriving on our site and their ability to find and license our images.

Other primary design considerations are on continuing to be able to continue to attract new photographer Members, and continuing to support our private Extranet capability, which is intended to be significantly enhanced to add individualized online reporting of image licensing for Members, and the ability of the individual Members to maintain their own personal information.

On a technical basis, the design is being aimed at ease of maintenance and expandability, reliability, speed of performance, low cost of operation, and ultimately, growth and migration.

Planning for the future growth of the agency and particularly our image collection and agency web presence is a major consideration. At present we are very small, but if we are successful and continue to grow as we plan to, we will be faced with extremely rapid changes in the underlying requirements for hosting our web presence and delivering our images to our customers, and if we do not plan now for that future growth, it will overwhelm us when it happens.

Of all ecommerce, business oriented websites, a professional stock photography agency website places some of the most extreme technical demands upon the infrastructure of the website, exceeded possibly only by a video streaming and sharing website of the same general scale.

We require very significant online storage and bandwidth for image delivery, and the "compute power" necessary to search for and then deliver customized search result thumbnail pages is extremely high. Making the infrastructure "scalable" to meet the growth in demand, and to provide for "fail over" protection in case some functional part of the website fails adds a level of complication that most business ecommerce websites never have to deal with at the same level.

As an example, look at the number of SKU's (Stock Keeping Units) that a normal reasonably large scale online retailer like Land's End might have to maintain: there will be a base of a few thousand individual items, in a few different sizes and color choices each. Picade currently has (04 June 2007) 6769 images (SKU's) and aims to grow to around 500,000 selected images.

An online retailing customer will typically browse to find a product result set that will typically have as few as 3-7 "items" encompassing text and a thumbnail image for each item, with possibly a display of color choices in another image for the selection of items, all from a static html page (possibly generated dynamically on a regular basis from a database, but then cached).

The typical stock photography website customer will do a keyword database search involving potentially numerous, Boolean logic terms that will generate a dynamic result set of from 24 to 100 thumbnails.

The data transmission for the two different pages will be many times larger for a small stock photography image search result set like 24 images, and dramatically larger for a large scale 100 thumbnail result set as typically used by professional researchers, and database utilization to generate the result set is orders of magnitude greater as well.

Given that a database query for the stock photography website is done to generate each search result page for a customer, and is typically done once per day for the typical online retailer, the differential in database utilization is readily seen to be many orders of magnitude between the two types of sites.

Returning to the idea of growth and the need to plan for it, in our initial period of existence [i.e.: now] working out of our own pockets without any venture capital or loans, we have to use low cost shared web hosting to be able to have a web presence at all.

As we grow, we will very rapidly outgrow the capabilities offered by shared hosting, and will have to move first to a single dedicated server, then ultimately to many servers with distinct separate functions, all probably "co-located" within a larger hosting facility. At some point in that growth timeline it is certain that it will be an economic requirement to replace PhotoShelter as our search engine/online image host/ecommerce workflow provider and do all the same functions for ourselves.

All of that is much easier to do, less traumatic, and less costly in the long run if we properly plan for and structure what we do now, and that is what we are currently doing.

Michael Beasley

On the virtual hat tree in my virtual office there are many virtual hats that I wear, which is a way of saying that I often function in many different professional roles while working for clients (which includes myself: the hat of a client is on that virtual hat tree too).

Today I'm going to be putting on several of those hats to begin to talk about the new world that as working photographers we now find ourselves in, and especially what it means with respect to backup and reliability. It's long and geeky and technical but fundamental, and I'll try and make it as simple, understandable, and as painless as possible so we don't have to repeat it. Unfortunately, I'm not a Hemingway, so bear with me please!

For example (wearing the hat of the Chief Financial Officer of the firm "Me, Myself, and I, LLC"): clients do not like it when you really mess up and lose or fail to produce something that is important to them, and they tend not to pay their invoices and to spread the word around that you are not a reliable business provider. This generally makes your business owner(s) unhappy, because they tend to be losing money, and your business employee(s) unhappy as well, because their paychecks tend to be thinner or non-existent.

As a result (grabbing the hat of the location oriented photographer from the tree) I tend to shoot with multiple camera bodies, bracket exposures, and when shooting film I shoot test Polaroid's, shoot the same setup on multiple bodies and multiple rolls of film, snip test rolls, and split the take and process half the take before committing the other half of the take to the tender mercies of the underpaid and overworked guy back in the darkroom (who might be me wearing yet another hat).

When shooting still life in the studio (grabbing another hat), I tend to leave the set setup in the studio until the film finally come sback from the lab the way I want it, or I shoot brackets and do sheet tests or roll snip tests and split processing of the film take just as if I was on a location shoot.

As working photographers, we all do these things, because we've learned (sometimes the extremely hard way) that they are a functional necessity for getting paid and continuing to be able to work.

(Grabbing the hat, slide rule, and pocket protector of the graying nerd ÜberGeek) Why as photographers should we work in the digital world any differently than we do in the analog film world??

I'm not saying anything new when I say that one of the major paradigm shifts in our universe as working photographers has been the arrival in full force of digital photography and all the new and changing tools and technologies that are inherent in that new way of working and creating. It is not going to go away, much as many of us might like it to, it is a permanent change in the way we have to work, and we need to deal with it appropriately.

At another time, I will talk about digital shooting workflows to implement similar safeguards to the practices mentioned above, but today I'm going to begin to talk about what we do with our images after we create them and we have them "back in the office/studio/...": filing them away so we can reliably get them back at a later time

Back in the halcyon days of the 1990's, Brian Seed published a subscription newsletter on the stock photography industry called Stock Photo Report, and I wrote some initial tutorial articles on digital imaging, with the idea we would try and get ourselves and Brian's subscribers ahead of the coming digital imaging wave. We also tried to start a new subscription newsletter or magazine called Digital Imaging, but we were ahead of our time and it did not succeed in getting off the ground.

In 1992, and later in 1998, Brian wrote in his newsletter about "Archival and Long Term Storage of Photos", and mentioned things like the dangers of formaldehyde and hydrochloric acid vapors from filing enclosures and what they could do to your color film.

In the 1998 article Brian mentioned US$400 CD writers, and US$1/disc CDRom media storing 650mb of digital images saying "Digital imaging has come of age ...", and offering some guidance on how to store and handle writable CDRoms for long term use as a way of protecting your digital images. You all know what media costs are now, and how much larger the media are, and how many more "things" we now create and store digitally.

(Putting on the hat and long white robe of a "Geezer"/prophet/sage/seer/"Consultant" for the coming "sermon on the mount") Digital images are very different from physical images in many ways, and can theoretically last indefinitely with no degradation, but one of the ways that they are not as good, not as archival as physical images is that they are infinitely more "fragile" and much easier to lose or destroy than a physical piece of film.

There are two components about that last statement that are key to our (and your own) stock photography business:

1. The "volatility" of digital files (both images and other appropriate software or "data").
2. Finding digital files on your storage media once they have been created and saved.

and I will address item #2 in yet another posting, but I want to focus now on item #1.

We all should know by now how easy it is to delete a digital file accidentally, or to have digital disk media fail for a variety of reasons and become "unreadable". With the increasing capacity of storage media available in a given small physical volume, it is also easy to physically lose through theft, fire, flood or other hazard significant parts of your digital image "archive" in a single incident.

Like many, I came to photography professionally from a different initial starting place: I was going to be a professional physicist working on the extreme outer edges of high energy particle physics.

Physics is about understanding and mathematically describing how the Universe seems to work, and one of the requirements for working in that field was and is a very good understanding and use of computers: your "experiments" might involve literally trillions of billions (10e12 * 10e9 = 10e21 = 1,000,000,000,000,000,000,000) of "events" that would give you maybe millions of "event data points" to work with to maybe find a single "event" that would prove or disprove a theoretical assumption about how the universe actually seems to work. To sift through the data from such an experiment in a relatively efficient manner, you have to effectively and efficiently use a computer.

My first 'personal computer' at Columbia University, later at Tulane, then IIT: the IBM 650 with 2000 words of 12,500 RPM magnetic drum memory (random access time of 2.496 ms!), photo courtesy IBM.

As a result, I started working with computers very early on, literally decades before most photographers started using them other than as a photographic subject, and very early learned the fundamental tenants of backup:

1. All digital files are very "volatile" and easily destroyed or deleted.
2. All digital filesystems and digital storage media that contain them are both volatile AND degrade over time.
3. Digital storage media containing your digital files can be lost, destroyed, or become unreadable.
4. The hardware used to access digital files and digital storage media may become obsolete over time, or fail, and therefore become unavailable for use.
5. Software (including operating systems) used to create, read, and write digital files on digital media is dependent on the hardware used at the time, and may itself change over time in such a manner that it can no longer reliably read files previously created by it.
6. Only files that are backed up can possibly be recovered if damaged, lost, or destroyed.

The answer to the first three items above is common sense redundancy: multiple "backup" copies of important digital files on multiple media located in appropriate physically separate location(s). How many "backups" you make, how you make them, how you organize and catalog them, and how and where you store them is dependent upon what risks you are trying to avoid and mitigate against and how easily you want to be able to find and retrieve a specific backed up file.

Items 4 and 5 are less commonly thought about by most people, but should enter into your backup planning as well: you may want to have spare current hardware and software, or to move or "migrate" your "static" backups to new hardware and or software as time passes so that the files themselves are still (easily if at all) accessible at a later date or if you have catastrophic hardware failure.

Item 6 is also common sense: no backup "system" or workflow is going to help you recover a file, if the file is not backed up in the first place by the constant, consistent, and appropriate application of your backup system/workflow to your digital work.

Belt, suspenders, and hold onto your pants with both hands and pray: An abridged sermon on backup

Besides the work I am currently doing building Picade's business systems and websites, I've recently been building some new server systems for myself and clients, and as some of you may know or have already guessed from the preceding, I'm very vocal about system reliability and recovery planning, and backup of critical files (like our Picade archive of master digital images files): a favorite saying of mine is "belt, suspenders, and hold onto your pants with both hands" in describing backup systems and system recovery planning.

What follows is based on practical real world experience rather than theory in keeping my trousers up.

For servers and for workstations where functional reliability and instant availability for business use are essential, for storage I use RAID 1 [disk mirroring], preferably hardware based, for my operating system and virtual memory cache filesystems, and hardware RAID 5+"online hot spare" for my data filesystems that hold anything that I cannot afford to lose or that would take longer than a single day to recreate in total from scratch.

As reliable as these two RAID type filesystems are at preventing data loss from the usual types of computer hardware failures, they will not protect you from catastrophic system hardware failure (think lightning strike), fire, theft, or the 747 or asteroid that falls out of the sky and onto your house or studio. They also will not protect you when, in a moment of weariness you overwrite some irreplaceable file: oops!

That means I make backups of the critical stuff: periodically both in an automated fashion onto other systems on my network, and onto some form of portable media and store them somewhere "offsite" so that anything that takes out my office will not destroy my files.

Offsite is relative: if you are worried about an earthquake, a "Dirty Bomb" terrorist strike, or a Chernobyl type event contaminating a major metropolitan area or region, you want to have your offsite backups hundreds if not thousands of miles away, otherwise a few blocks away may suffice. (A sotto voce note: for Picade, we have three primary imagefile archive locations: our office in Chicago, and at PhotoShelter's facilities in New York and San Francisco, and I may add at least two more for good measure in the near future.)

When creating backups, I only use the highest quality media, and I always verify them by reading back the backup and comparing the backed up items to the originals: if they compare 100% identically you've got an acceptable backup, anything less is worthless because you can't later trust the backup to be able to reliably retrieve anything if any single item is not reliably retrieved.

The offsite verified backups are stored in a sheltered, dust and moisture free "shirt sleeves" environment along with the complete minimum necessary software and hardware to use the backups in restoring a system. Periodically, the backups are evaluated again to make sure they are still retrievable, and if necessary, migrated to new media or systems or converted for new software environments.

Thus endeth the sermon on backup.

Notice that I didn't talk about operating systems, manufacturers, or technology save in the most generic manner possible: the abridged sermon on backup is applicable to EVERY computer system that we use, even our iPods or other personal media players!

In the past, I tended to use tape based backup systems: 9-track, QIC-150, Exabyte 8505, DAT, DLT, and LTO Ultrims in both single tape and autoloader "library" systems.

Unfortunately, tape systems which have been the stalwart of backup since the dawn of the computer age are rapidly falling behind the size of the data systems that need to be backed up completely, and the amount of data that must be backed up in a single day. Today it is not terribly uncommon to find individual creatives, who have disk filesystems of several terabytes, who need to backup every day more data than a tape drive can write to and read back from tape in 24hrs.

Tape is slow, and frankly, not very reliable: a few tens or a hundred passes of the media over the read/write heads and you begin to have media flaking and "dropouts" where you loose information permanently. Accessing a backup on tape is very slow because tape is a linear sequential, not random access sequential medium, so access to the last file written to tape from the head of the tape is always dramatically slower than accessing a file on the head of the tape.

The latest high speed, high capacity tape drives and libraries necessary to keep up with the burgeoning disk systems now cost nearly as much or more than the computers that they backup, and the media costs are nearly as expensive as the hard disks as well: a basic HP LTO-3 400GB tape drive costs US$2,998.00, and a Maxell 400GB LTO-3 Ultrim tape to use in the drive costs US$69.95, but a Seagate Baracuda 7200.10 ATA-6 400GB hard drive can be had for only US$89.96 (Frys.com, 17Oct2006, I bought 10 of them for US$899.60 delivered).

The HP drive can read/write uncompressible data (like a tiff, psd, or jpeg imagefile) at 80 megabytes/second, so backing up a complete 400 GB hard drive takes a minimum of ~10,000 seconds (2 hrs 47 minutes) for a full verified backup if the host system can sustain such a data transfer rate.

Since my existing 2TB (2 terabyte) RAID5 image archive systems [based on 7200 rpm SATA150 drives] can deliver a sustained raw read rate of about 64 mb/s [they are designed and configured for low cost and high reliability rather than maximum performance], and the OS and the backup software must do some work, it is more likely that backing up completely and verifying the 2 TB RAID archive to LTO-3 tape will take about 2 * 5 * 1.25 times as long as the above estimate for a single 400GB hard drive.

Put another way, it takes about 34 hrs 43 minutes minimum and 10 tape swaps for a full verified backup of the entire 2TB volume exclusive of the tape swap time, and means a person has to be available about every 3 hours to do the tape swap. Adding a tape library/autoloader to the tape drive to automate tape swapping will easily more than double the cost of the tape system (about US$7500 for an HP LTO-3 library system and drive).

On the other hand of course we have the current crop of large capacity hard drives, viz: a 7200 rpm Seagate SATA2 750gb hard drive (currently available for US$280). With these hard drives I can build a very capable host computer system holding 4TB of usable space for about US$3500, or upgrade my existing image archive RAID5 systems from 2TB to 4TB formatted capacity for about US$2240.

If I needed to backup my entire RAID5 system at once, given the cost of an "adequate" 2TB tape system, and what it would cost to duplicate my entire RAID system at 4TB capacity (including host computer system which I can use for other distributed processing tasks) AND upgrade my existing RAID5 system to 4 TB, for my needs it's a no-brainer: it costs less to use RAID5 hard drives for 4 TB capacity upgrade of my existing system AND for backup than to use the tape drive and carousel hardware for 2TB backup only.

Since most of the capacity of the RAID5 system as used by a stock photographer does not all change at once, it is both feasible and reasonable to do incremental backups of new or changed imagefiles to externally mounted hard drives that can be swapped to offsite storage: I currently use 400 GB Seagate drives for this purpose and can easily add new capacity incrementally at relatively low cost (say about US$150 for a Seagate 400 GB hard drive in an external USB2/Firewire400 enclosure). Total offsite backup cost for my 2 TB RAID5 in this manner is just about US$750, and for both offsite and local backup just US$1500 and is what I currently do. The only caveat is that you should do a complete read of the backup drive every six months or so to "exercise" the drive and keep it properly functional and accessible according to the geeks at the drive manufacturers.

HTH.

Michael Beasley