storage and organization

[John King]

On the contrary, if there were significant benefits, everybody would do it.

It is my suspicion that 99.99% of photographers do not even know that keywording (tagging) exists, and don't know what it is, or what benefits it might convey for them if they do!
Sad, but probably the case, Tony.

Nowhere in this thread has there been any sensible attempt to quantify the cost of keywording. Having tried it for myself, I know for a fact that in my case (and probably in 90+ percent of other cases), on a time = money basis keywording an entire collection is the equivalent of paying $1000 a year to insure your car for $100.

The time cost when even a sizeable set of images has been uploaded is negligible. Doing it while looking at what one has caught becomes second nature - a matter of selecting a keyword from a list and L/clicking on it to apply. Since about 5% of my images are of the children (the cats, AKA Household Gods), I usually just select all of these first up and tick "Cats". That fixes that group up immediately. Then seek out groups and do similarly. It really is quite fast.

Once he actually understood the benefits to him, my brother set out from scratch to keyword all his images. He's older than I am, and usually cannot be bothered doing anything like this ...

@arthurking83 Arthur, I have installed XnViewMP on my Win7 Pro w/s. It will keyword RAW files, but only by creating XMP files, i.e. it does not embed the keywords into the actual RAW file. When one searches, it obligingly finds the XMP file, not the RAW file! I'm probably doing something wrong ...

At least Bridge finds the RAW file associated with the XMP file, rather than finding only the XMP file ...

[Hawthy]

It is my suspicion that 99.99% of photographers do not even know that keywording (tagging) exists, and don't know what it is, or what benefits it might convey for them if they do!

Correct!
Personally, I can think of nothing more tedious than tagging each photo with a keyword. Instead, I download my raw photos into a folder called something like ANDREW>> PHOTOS>> MAUI>>RAW FILES if I am lucky enough to have been to Maui. If not it will say ANDREW>>PHOTOS>>WYNNUM MANLY.
Then after looking at those 1,000 raw images in Bridge and finding maybe 20 keepers, I process them.
I save those into a folder like ANDREW>>PHOTOS>>MAUI>>PROCESSED PHOTOS.
I can always revisit the raw files but I really can't see myself sitting down and giving each a label like ; Lahaina : Luau: Sunset; Sunrise; Surfer; Meal; or any combination of that. The number of keepers that I get from every trip are usually so insignificant that labeling them is unnecessary.

[Tannin]

You raise a salient point, Andrew. I'll get to it in a moment. First, a not-quite-distraction.

On a good day, I might take 1000 pictures. Of those, I'll probably want to keep 100. Of those 100, there might be 5 I'll eventually want to publish (this is a good day - on a bad day it might be one, sometimes none at all) with another 15 or 20 that are genuine keepers. The remaining 80 are context - valuable and meaningful to me but seldom of interest to anybody else. (You probably have pictures of your children that you wouldn't dream of deleting, but wouldn't bother posting here or showing to your friends. Same thing. After all, I love the Australian landscape the way many people love their children.) They also serve a vital purpose insofar as they locate the 5 or 20 best pictures in time and space. For example, they might remind me that the beautiful Spotted Harrier was half an hour and 15km south of the ford over the river, not far from where the saltbush plain gave way to sandy mulga country. The whole collection of 80 or 100 forms a narrative through which I can revisit any of a hundred trips over seven states and getting on for 20 years, and does so in a way no possible tagging system could remotely reproduce.

But (getting to your point now) I am nowhere near mentally capable of reducing 1000 pictures to 100 in a single session. (I wonder how many of us are? Rather few, I suspect.)

The most I can do before my eyes glaze over and I start making bad decisions is throw out some of the worst ones. I always aim to cull 50% in one go, but more often only achieve something like 30-40%. Then I put the day's work aside to be revisited another time, when I can see it with fresh eyes. The first pass is the easiest - there are always obvious duds which need no thought at all. After that it gets harder: the easy choices are all made. But slowly, over several stages, I reduce the thousand to 600, 400, 250, 150, 100. It usually takes about that many sessions, spread over anything from several weeks to several years. Often there are fine distinctions to be made, and it's hard work. My brain overheats. So when I start struggling to make good decisions about the northern New South Wales pictures from last month, I'll flip over to Tasmania the year before last. And so on. Little by little, the collection shrinks and improves in quality. Every now and again, a shot will stand out from the crowd and demand to be published (here, or to my website, or to my private collection of favourites). Maybe I've always had my eye on that one and always intended to do the full PP thing with it when I got around to it; maybe it's one of five or forty-five very similar ones and I've only just decided that #32 is the pick of the crop; or maybe it's a little hidden treasure which has sat there on my hard drive for months or years and I've only just realised how good it is. These last are the best of all: often you had some particular vision in mind when you took it and that didn't quite work the way you wanted it to, but now, months later, you look at it afresh without preconceptions and recognise it for what it is - not what you wanted but something else as good or sometimes better. There is a special joy in these neglected gems: finding one is like finding an unexpected $50 note in your pocket - you might have had $300 in your wallet and thought nothing of it, but that bonus $50 makes your day.

[John King]

Good description, Tony.

Specially love the $50 note analogy.

My method is a bit different, in that any photo that makes it onto my HDD tends to stay there forever. Duds tend to get deleted in camera. The one below nearly met that fate as almost everything about it is "wrong", but something stopped me. It is Rosa sitting on my chest in bed. It captures something of her unique personality - specially her innate ability to be very comfortable ...

[John King]

Tony, that's where we are very different. On a good day, I might take 100 pics, maybe up to 200 if it's an exceptional day. I keep 95%+ of them. I am a contemplative shooter.

[Tannin]

Just so, John. In large part, this is no doubt because of our different subjects. With birds, especially small birds, there are many factors working against any one shot. You are usually working right at the technical limits of your equipment: you are pretty much always too far away, using more lens than you'd really like to, cropping harder than desirable, using higher pixel density than is ideal, pushing the ISO higher and the shutter speed lower than you'd want to, and/or shooting a few stops wider than your desired depth of field. Yes, you have IS/VR but at these focal lengths it is nevertheless imperative to keep the shutter speed up to avoid camera-movement blur and you have to skate as close to that edge as you dare lest you sacrifice too much noise or too little DOF.

And you are often doing all that in a few fleeting moments, nevertheless trying to stay relaxed and move slowly because if you don't the bird takes fright. Meanwhile, you are also guessing exposure compensation on the fly.

All that said, the biggest factor is that creatures the size of birds move rapidly and unpredictably. With something the size of a horse, a human, or a dog, you have a fighting chance of anticipating movement such that you can click the shutter just at the precise instant the horse clears the fence, or the father-in-law kisses the bride. You can do that with birds too, but with a much lower probability of success. And you have a near-zero chance of predicting the vital small movements that (for example) see a thornbill's head turned at the right angle to catch the light in its eye. This last is the key. Look over this forum's bird section: you'll soon observe that one of the biggest differences between an experienced bird photographer's work and that of a beginner is ... well, I hate the jargon term for it, but people call it "the head turn". It's the bird photography equivalent of, in humans, catching the right moment where the subject isn't blinking or looking awkwardly out of frame. And you can't predict it far enough in advance to just wait for the perfect moment.

In short, doing birds, you wind up with a vast number of pictures, many of them similar but (for example) marginally sharper or with slightly different angles, and trawling through deleting them is a slow and tedious task.

In general, I don't delete anything in-camera. There are two reasons for this. (a) You can see the angles (head-turn and such) on the camera screen, but you can't tell the difference between sharp and almost-sharp, no matter how hard you try. (b) A Canon peculiarity: if you shoot with dual cards, raw on one, JPG on the other, and delete in-camera, only the JPG gets deleted. Why? Ask Mr Canon. All Canon cameras are the same in this regard. So I can't delete in-camera on any of my best three bodies - only if I happen to be using one of my elderly spare bodies for some reason (7D and 5D II). So I just don't do it.

Managing raw + JPG collections is a topic in itself. If I remember correctly, Lightroom handles it in a fuss-free and very sensible way. (See? I just said something nice about an Adobe product, and it didn't even hurt.)

My approach is to work with and manage the JPGs, never touching the raws except to copy the odd one to a scratch folder for PPing into a finished publication-standard image. The raw files just stay in (for example) x:/2018/11-November/22-Lightning Ridge/raw, unseen and untouched. (It is far quicker and more convenient to do sorting and deleting with the JPGs only.) Then, at some convenient time when I've made good progress on the sorting, I tell the computer to make the organisation of the raw files mirror that of the JPGs. It does this using, of all things, a batch file. (Remember batch files? Am I making you feel old?) It does a few things but the primary task is a series of variations on this theme:

SET HOME=%CD%
IF NOT EXIST raw GOTO finished
:step9
IF NOT EXIST x9 GOTO step8
IF NOT EXIST raw\x9 md raw\x9
cd x9
FOR %%Z in (*.jpg) DO MOVE "%HOME%\raw\%%~nZ.cr2" "%HOME%\raw\x9"
cd %HOME%

:step8
IF NOT EXIST x8 GOTO step7
IF NOT EXIST raw\x8 md raw\x8
cd x8
FOR %%Z in (*.jpg) DO MOVE "%HOME%\raw\%%~nZ.cr2" "%HOME%\raw\x8"
cd %HOME%

.... and so on ...

It took a bit of fiddling around to get set up, but it's simplicity itself to run: just right-click on the head folder ( x:/2018/11-November/22-Lightning Ridge) and select it. From there on, the whole process is automatic. A few moments later, the raw files are organised exactly the same way as the JPGs.

Note that it doesn't delete the culls. In my book, deleting things is a human-being level decision. But it would be easy to modify. There is also a related script on the archive server which deals gracefully with partial data sets imported from the primary laptop.

File management and backup software generally is very, very good at dealing with increased data sets - for example, dealing with a folder full of documents and making sure that the backup contains all the new and modified files - but is very seldom designed to sensibly deal with workflows where the added information is negative - i.e., where the new update since last backup/archive constitutes removal of certain files rather than additions: and this is of course exactly what happens with a folder full of pictures as you gradually cull it down. This is why in the end I gave up on finding a ready-made solution and wrote my own. That was ten years or so ago and it has been a great success. I can't remember the last time I needed to update or modify it, it just works.

(PS to anyone following this discussion who uses raw + JPG: setting up a similar system is quite simple. The only technical part is writing the script part-quoted above. Sing out if you'd like a copy of it - you don't need to understand it to use it anymore than you need to know how Kym wrote the AP competition scripst in order to vote for a picture.)

[arthurking83]

....

@arthurking83 Arthur, I have installed XnViewMP on my Win7 Pro w/s. It will keyword RAW files, but only by creating XMP files, i.e. it does not embed the keywords into the actual RAW file. When one searches, it obligingly finds the XMP file, not the RAW file! I'm probably doing something wrong ...

At least Bridge finds the RAW file associated with the XMP file, rather than finding only the XMP file ...

Weird issue with XnView you had.

I use XnV myself for error checking my image files.
It's one of the only programs that can view files recursively within hundreds of smaller/child directories, and with any usable performance.
I've added keyword to raw files in XnV(of course it didn't add it to the raw file), but it did find that raw file .. not just the side car file linked to the image!
But I don't really use it for that purpose anyhow .. mainly for the ability to visually check that the raw file is not corrupted.

But again, as my raw files have the tag data embedded, XnV can see that data, as can (or in my case .. could!) Bridge ... and Nikon's software, and Gallery, Explorer .. etc. etc.

[Liney]

Well folks, I must commend you all on a very interesting thread.

What I have learnt today is that a) there are photographers who take a lot of images and quite a lot of them are very good (unlike myself, I get the occasional "oh I'll keep that"), b) we all tend to hoard the images we've taken, and c) there are many different ways to skin a cat (with apologies to cat owners)...

Personally I was using Picasa for many years until it recently started throwing a hissy fit and dating recent downloads back in 2015 (don't ask, I don't know why). So I went on the hunt for another viewer, which I wish I had done after I'd read this thread! My personal modus operandi is built around the fact I have two computers, a small mac laptop that I use on a daily basis, and a larger laptop which now serves as a desktop that I use as the hub of all things photographic.

I'll confess now that I don't pay for any applications, I only use free stuff because I find it hard to justify the cost for the amount I use.

When Picassa started giving me issues I went searching for a replacement and actually settled on Bridge because it had some good reviews and was free to download. So far (three months in) it's worked reasonably well apart from a couple of intermittent issues downloading images from the camera to the laptop desktop. As I tend to stick with images in folders dated when they were taken I'm quite happy, I can roughly guess when they were taken so i can go back and usually find the one I want after a couple of guesses.

However after reading this thread I'll probably go back and check out some of the other applications mentioned. I have recently found that I need to rescale a number of images, which at the moment is time consuming as I need to open a processing application, load the image, rescale it, export it then open the next etc. When I'm focus stacking 20 or so images that takes a lot of time, so if any of these viewers allows me to do that quicker I'll be interested.

In general though I think the idea of tagging images is worth while if (and here is the condition) you have a lot of good images and you need to search for something specific and you have the time and discipline to tag them as you go along. For me at the moment none of those are true, I would be quicker having a single folder called "good stuff" that I copy the images I really like into....

If I had the time I really need to go back and delete all the dross that didn't work out and were instantly forgettable, if nothing else it would save me disk space (seeing as I have two backups of everything that counts for a lot on gash images).

Please continue the debate, seriously it is very interesting to see how everyone thinks and works.

[arthurking83]

@ Liney.
Be careful what you call dross and forgettable too.
I have a fair few images that easily fall into those categories, if not for the fact that there is something in those images that force to hold onto it for a while longer.

eg. I have 5 images of a red robin taken a couple of year back, terrible images in terms of presentability, grainy to high heaven, blurry, and not enough focal length .. etc.
But they are the only images I have of a red robin, and is the only reason I keep them(I'll cull that down to 1 image soon too).
Reason for keeping them is literally for the keyword I've added.
Now that I have that keyword and image, if ever I get a better image of red robin, I don't have to go searching for "red and black small bird with white stripe on head" on the interwebs again!
(reason is that; as far and bird ID goes .... well, I know what a seagull looks like! )

I open Photo Gallery, and on the keyword/tagged data column, I go to 'Birds' click this keyword and all the images of birds display on the page. I scroll through that and look for similar bird images.
Under the main 'Bird' keyword are all the sub keyword entries of the main bird species name(eg. red robin, eagle, seagull, etc)
ATM it takes me about 2mins to scroll through all the images as there aren't all that many ... maybe a thousand or so. I can also click each species that is sub-listed under the main Bird title.
That's the way I've set up my keywording/tagging system in my images.

So, the keywording is not only relevant for future searches, but also helps me in identifying stuff I have zero clue on to begin with.

ps. one key point with keywording is to add as much necessary data as possible, without overloading the image with fluff!
As an example of what I mean.
I have an image of a landscape(my primary interest).
Lets say I have a scene with a church in the background, it's old historic fence/gate in the foreground and a nice landscapey distant backdrop.

Obvious keywords would be church, and the location of the church.
But, if the fence or gate has a high level of appeal, then I'll also add those 'values' as keywords too .. and for any other content of interest.

[Tannin]

Which red robin?

Do you mean the Red-capped Robin?

Or the Scarlet Robin?

Mind you, the Rose Robin is pretty red too.

Not to mention the Flame Robin, which is ... er ... flame red.

[arthurking83]

Actually scarlet robin was what it was.
Went onto a bird ID site found on the interwebs .. but they do my head in.
Seagulls, albatrosses .. scarlet robins! .. they all look the same to me.

[jim]

Gets me the same way Arthur. “Ooh, Scarlet Robin! Aw, just another bloody albatross”.

[ameerat42]

My refrain is usually, "Oh, a sparrah!" for small birds

[landyvlad]

Great thread. I wish all my images were tagged as it would make it much easier to find what I want eg land rover, shotgun, motorcycle etc

[Mark L]

"Was that an albatross?" Robin asked.
"No" said Scarlet

[landyvlad]

(PS to anyone following this discussion who uses raw + JPG: setting up a similar system is quite simple. The only technical part is writing the script part-quoted above. Sing out if you'd like a copy of it - you don't need to understand it to use it anymore than you need to know how Kym wrote the AP competition scripst in order to vote for a picture.)

Mate consider this a "shout out" for the script, and how to install it (and remove it if desired).

While I don't use RAW and JPG together NOW I may well do in future so this would be good to have

[arthurking83]

No-one has yet satisfactorily addressed the massively time-consuming task of tagging many thousands of files. .....

There kind of are two solutions that make it less painful to do, but it's not painless.
Those two programs are Microsoft's Photo Gallery software, and IDImager's Photosupreme.
It's tedious work, but as John(and I) have said, once the process has been started, the benefits are clear.
The only point that John and I disagree on, is what is important to tag, and how it's tagged. I only think in terms of Raw file = important, any other (child) file type = unimportant .. tag the stuff that you NEED to keep in a worst case scenario.

NOTE: tagging thousand of files isn't a case where you tag each individual file singularly, tagging can be done in batches on many files at once. My only proviso is that it's done on the raw file itself.

On the contrary, if there were significant benefits, everybody would do it...

The problem with this argument is that it's 'chicken and egg' syndrome.
That you don't do it NOW, doesn't mean that you see some benefit at a point in the future. The problem as always with the future we know nothing about it!

I think of that argument in a similar manner to (say) electric cars. On the current path we travel on, it's massively obvious that electric cars are the future. Problem is they cost a fortune(to buy). Every day cost to run is much lower for those of us that rely heavily on the benefits of transportation by car. I'd much prefer for it to cost me 10c per day to travel, rather than the current $5 per day it does .. but I'm not yet willing to pay $50K for that 'benefit'

...
Bridge handles IPTC keywords and searches better/faster than any other program I have come across. ...

I really don't think you've tried enough.
You listed some timings there which are in some ways humorous to read.
The old discontinued(but still usable) and free Windows Photo Gallery is instant. Not only instantaneous, but also predictive. As you type a tagged word type it lists all the tagged data with the letter combination already typed.. ie. before you've typed the keyword you're after, it's already given to you.
Not only that, but you don't even need to type a search. all keywords are listed in a pane on one side that you scroll down too. click it, and all images for that data are displayed.
The only tradeoff is that it takes between 30-60sec for the software to load. .. but on reflection that is about half the time most Adobe software takes to load for me anyhow!

It wouldn't work for everyone, but I store everything - absolutely everything - by date, with folders further labelled by location. ....

I do the same, as I think many folks probably do too. It is by far the best form of organisation to begin with.
But it has failings.
Reason why I replied again, is that on another forum a comment was made re: a new train service from Maryborough to Avoca. An area Tannin probably knows very well.
I also know very well.
Those of us that know the area well enough, know that this new train service isn't new, it's being reopened. That is, there's an old line that is being renewed.
I happened to have an old photo of that old line. The buckled twisted remains of the old rail lines, easily viewed and photographed at a crossing between Avoca and Ararat. I know the road, even tho I also geotagged it, I didn't need too. I know the road well, get me into Beaufort from the north side.

In the last 3-4 years, I've 'explored' the area 5 times, and I know I have an image of the rollercoaster train line in it's dilapidated state .. but looking through 5 folders of images taken in the area over that time revealed nothing due to the quick glance method I used to view the images there.
So I searched all the folders I have of the area thinking that I may have lost track of time and the image I have of the rollercoaster was taken earlier(within the last 12 years), so searched all folders of that area I have.
Nothing!
Turned out that it was in one of the folder I originally glanced at, and my mistake was to view the last few images of the folders, as I thought the images of the railway were my last images for the day. I had about 15 more after the rollercoaster images.

Short version:
Moral of the story; no matter how good you think you are at remembering stuff, time is better at 'helping' you forget it all, or confusing small details about a general idea.
Keywording(or tagging) eliminates the effects of time. How valuable this resource is, comes down to the individual themselves.
Tagging images in their multiple thousands is a monumental task. The tools you use make a world of difference. now that I've started the process, which has taken me over 5 years to do less than half the actual job I'd like to do. I don't do it all the time, fits and spurts of it is probably the best way to describe my method.
I've subsequently tagged the dilapidated rollercoaster images of this train line ... stupidly tho I had tagged the other images in the folder with place names, and other stuff .. just for whatever absent minded reason, not the railway images.
I've also made a mental note to get back to this same crossing location to get more boring images of the now updated line, and have before and after shots.
But what I'm kicking myself over is not having gone back earlier .. pre line update to get proper images of the interesting looking old crossing.
Again it drive home the accurate adage about putting off what you can do now, for a later time .. sometimes it just doesn't pay.