storage and organization

[poider]

G'day, just wondering what are the better options for storage and organization of photos?
I have over 40,000 photos from my wife and I and our travels, usually I put the photos in folders with place names etc, but find it excrutiating when I want to look at said photos, any folder with 200 or more photos takes ages to load in and settle, as I am scrolling it continually updates and changes the pictures order etc.
Is there a way that I can put in a search in my main (parent folder) that will search for and tell me how many photos I have taken at 300mm or how many with centre weighted or whichever perimeter I Choose, I know if I put all my photos in one folder I can group them by many different perimeters but having 40,000 photos in one folder would probably crash??
Peter

[ameerat42]

This will do all or most of that.
Faststone Image Viewer.

It also does some basic editing. Lots of people use it on AP.

[Tannin]

Ameerat42 is spot on.

What you need is an image viewer with decent performance. There are dozens of image viewers, many of them very good. Rule One is don't even consider anything from Adobe - Adobe products are a byword for bloated, slug-like performance which would put a sloth to sleep and bring a fast computer to its knees.

Faststone is fine. XNView is excellent. I still love the ancient PMView because of its outstanding ergonomics. Then there is ACDC, IrfanView, and various others I can't remember. Any one those mentioned will be around 1000% faster than any Adobe product this side of Pluto. None of them have the slightest trouble loading large folders.

(To be fair to Adobe, none of their products are actually designed-for-purpose in the way that the programs mentioned above are. Image viewers are not the same thing as image editors. Adobe doesn't make an image viewer (unless you count Bridge, which would be stretching a point), they only make image editors which can, at a pinch, be pressed into service as a second-rate viewer. But if they did make one, there is no reason to suspect that it would be any less bloated, glacial, and awkward in use than the likes of Lightroom, Acrobat, or Photoshop. Go with one of those I mentioned, or an equivalent of your choice.)

[paulheath]

Lightroom

[poider]

Thank you all I will look at faststone, I have photoshop elements 14 for editing and have never really liked light room.
Thank you
Peter

[ricktas]

Thank you all I will look at faststone, I have photoshop elements 14 for editing and have never really liked light room.
Thank you
Peter

Photoshop (even elements) is an editing package. Lightroom is an asset manager with editing capabilities. Lightroom does have wonderful cataloguing, when used correctly.

But as with everything from camera brand to lens selection, it is personal choice.

[arthurking83]

For file viewing I have a preference for Fastone Viewer too.

BUT!! .. (big but here). If I need to view a lot of files in one go, nothing beats XNViewerMP.
What I mean by that is that as you store files in specific folders, most viewer software forces you to view the images from within each folder.
So if you have 20K image files, in 1000 folders with XNViewMP you can leave it to display only the files within each folder, OR you can set it to display all the files within the most senior folder recursively(it's your choice).

It does take a while to build up it's cache when viewing a lot of files in one go(ie. recursively) but it still displays a very high number as you view them whilst it build it's cache.
On my last usage it took 10mins to build the thumbnail cache for about 1000 80+ Mb NEF images.
If you only have jpg files saved, it's super duper much faster. Raw files take a bit of time to decode/read.

What I didn't like about Lightroom is that it forces you to make a catalog .. ie. that you can view all files at once once they're catalogged.
it's OK in what it does, but I much prefer the option to do so or not.
And it's catalog is not just a simple catalog .. it's a monstrous behemoth of a database.
Mine once ended up at over 6Gb with barely half my 200K images forcefully catalogged. I used to regularly delete it(only as I never wanted/needed another catalog of images).

On my primary and secondary archive locations I have about 200K images all up(jpgs, tifs and mostly NEFs).
XNViewMP will take about 20-30mins to rebuild the thumbnail cache(noting that i always clear it out, or delete it). So every time I use XNViewMP I basically start over from scratch.
For me this is important for one specific reason .. and why I use XNViewMP.
I once had a data corruption whilst moving images from one drive to another. 100+ K images to view in their entirety is too big to go through individually via each separate folder, so unknown to me I'd lost about 20 NEF images due to that data corruption.
Literally .. too many folders too many images to sift through each one.
By the time I'd found this out, I'd already updated my archives a few times over, but in that update I saved corrupted images where before the corruption they weren't.
So I was backing up corrupted images.

For me, XNViewer was the only practical way to locate every corrupted image I may have had.
Problem with Lightroom is that it builds it's own database, and relies on it's thumbnail which is stored separately as it's own file.
People may think this is good, but if the original file is corrupted, displaying the perfectly fine thumbnail doesn't show you that the actual raw file is corrupted.
That's where XNViewer comes into it.
Having to build the thumnail cache from scratch it showed a non image thumbnail. That is, for those corrupted raw files, it didn't display a thumnail of the raw file, it displayed a generic icon.
From that, I know there's an issue with the raw file data.
So that's my primary use for XNViewer. I set the thumbnail size to the smallest size, and open my primary and secondary photo archives recursively. I can sift through all 200K images in about 10 mins on that page.
Once I feel secure that they aren't corrupted I resave one of those those archives again to another location as a backup and then update those archives again once a year.

Hope that made sense.

Basically:
1a. For viewing and very basic edits of NEF files ; Nikons' ViewnNX2
1b. For general viewing of many image formats and device types ; FSViewer. it's the smallest fastest viewer that you don't even have to install(you can ruin it as a portable install).
3.. For the more complex viewing tasks ; XNViewerMP.


Question. Do you shoot in raw(NEF) mode or jpg?
I have to make dinner now, but a very important point to note for you is Keywording/Tagging, and IPTC.
I've found over the years, that catalogs the way that programs like Lr and other do them are pretty much useless unless you plan to lock yourself into those programs systems for ever.
Keyword raw files with embedded data, not thirdparty external data!
On a raw file this is not so easy to do.

[John King]

Keywords in Bridge are industry standards compliant. That's a big plus.

Keywording is a PITA!!

However, once done, retrieving any subset of files is very fast. Bridge can search using multiple keywords and logical associations between them.

Far be it from me to defend Adobe in any way, but this works.

Bridge CC is a free download, and can use any editor linked to it.

[arthurking83]

Keywords in Bridge are industry standards compliant. That's a big plus.

....

When they set the standards, then yes. But interoperability with other software is next to zero(unless the other software has an import feature) .. even then, sometimes the imported data becomes garbled in some ways.

For true(er) 'industry standard' stick with ITPC data tagging into the raw file and you're safer for the future.

My history was similar to Tannin, been to many places multiple times, and it then becomes hard to differentiate some tiny insignificant detail difference between dates.
But I do the same.
Uppermost folder structure starts with dates, and then locations.

[Tannin]

It wouldn't work for everyone, but I store everything - absolutely everything - by date, with folders further labelled by location. I practically never have to spend more than a few moments searching, because all I have to do is remember the place, which leads in turn to the journey, which provides me with the approximate date - and I never, ever have to enter a keyword. Once in a long while I trip up over a place I've been to many times and have to look harder. When (to take an example that occurred just yesterday) did I see that vast flock of Flock Bronzewings near Cunnamullah? I've visited the district perhaps six times, and passed through on the way to somewhere else another dozen times, often camping there overnight. Took me about ten minutes to track the pictures down last night, which is very unusual. Are they good enough to publish? Hmmmm .... maybe - which is exactly the same conclusion I failed to come to the last five times I looked at them

Come to think of it, it's not quite as primitive a system as that. If I'm uncertain, provided that there was at least one picture I thought worth posting on that day or close to it, my website is database-driven and readily searchable by things like species, lens, time of day, habitat type, geographical location, and so on, so it's a simple task to find that picture, and thus the date, which leads directly to the other one I'm after. So having said "I don't use or need tagging", I actually do use a form of it (albeit for a very small subset of all pictures) and it is very useful. The point here is that if the task of tagging a gazillion pictures seems way, way too difficult, provided that your basic storage system is robust, you usually only need to tag a very few representative examples. 98% of the benefit for 1% of the effort.

[arthurking83]

No-one has yet satisfactorily addressed the massively time-consuming task of tagging many thousands of files. .....

There kind of are two solutions that make it less painful to do, but it's not painless.
Those two programs are Microsoft's Photo Gallery software, and IDImager's Photosupreme.
It's tedious work, but as John(and I) have said, once the process has been started, the benefits are clear.
The only point that John and I disagree on, is what is important to tag, and how it's tagged. I only think in terms of Raw file = important, any other (child) file type = unimportant .. tag the stuff that you NEED to keep in a worst case scenario.

NOTE: tagging thousand of files isn't a case where you tag each individual file singularly, tagging can be done in batches on many files at once. My only proviso is that it's done on the raw file itself.

On the contrary, if there were significant benefits, everybody would do it...

The problem with this argument is that it's 'chicken and egg' syndrome.
That you don't do it NOW, doesn't mean that you see some benefit at a point in the future. The problem as always with the future we know nothing about it!

I think of that argument in a similar manner to (say) electric cars. On the current path we travel on, it's massively obvious that electric cars are the future. Problem is they cost a fortune(to buy). Every day cost to run is much lower for those of us that rely heavily on the benefits of transportation by car. I'd much prefer for it to cost me 10c per day to travel, rather than the current $5 per day it does .. but I'm not yet willing to pay $50K for that 'benefit'

...
Bridge handles IPTC keywords and searches better/faster than any other program I have come across. ...

I really don't think you've tried enough.
You listed some timings there which are in some ways humorous to read.
The old discontinued(but still usable) and free Windows Photo Gallery is instant. Not only instantaneous, but also predictive. As you type a tagged word type it lists all the tagged data with the letter combination already typed.. ie. before you've typed the keyword you're after, it's already given to you.
Not only that, but you don't even need to type a search. all keywords are listed in a pane on one side that you scroll down too. click it, and all images for that data are displayed.
The only tradeoff is that it takes between 30-60sec for the software to load. .. but on reflection that is about half the time most Adobe software takes to load for me anyhow!

It wouldn't work for everyone, but I store everything - absolutely everything - by date, with folders further labelled by location. ....

I do the same, as I think many folks probably do too. It is by far the best form of organisation to begin with.
But it has failings.
Reason why I replied again, is that on another forum a comment was made re: a new train service from Maryborough to Avoca. An area Tannin probably knows very well.
I also know very well.
Those of us that know the area well enough, know that this new train service isn't new, it's being reopened. That is, there's an old line that is being renewed.
I happened to have an old photo of that old line. The buckled twisted remains of the old rail lines, easily viewed and photographed at a crossing between Avoca and Ararat. I know the road, even tho I also geotagged it, I didn't need too. I know the road well, get me into Beaufort from the north side.

In the last 3-4 years, I've 'explored' the area 5 times, and I know I have an image of the rollercoaster train line in it's dilapidated state .. but looking through 5 folders of images taken in the area over that time revealed nothing due to the quick glance method I used to view the images there.
So I searched all the folders I have of the area thinking that I may have lost track of time and the image I have of the rollercoaster was taken earlier(within the last 12 years), so searched all folders of that area I have.
Nothing!
Turned out that it was in one of the folder I originally glanced at, and my mistake was to view the last few images of the folders, as I thought the images of the railway were my last images for the day. I had about 15 more after the rollercoaster images.

Short version:
Moral of the story; no matter how good you think you are at remembering stuff, time is better at 'helping' you forget it all, or confusing small details about a general idea.
Keywording(or tagging) eliminates the effects of time. How valuable this resource is, comes down to the individual themselves.
Tagging images in their multiple thousands is a monumental task. The tools you use make a world of difference. now that I've started the process, which has taken me over 5 years to do less than half the actual job I'd like to do. I don't do it all the time, fits and spurts of it is probably the best way to describe my method.
I've subsequently tagged the dilapidated rollercoaster images of this train line ... stupidly tho I had tagged the other images in the folder with place names, and other stuff .. just for whatever absent minded reason, not the railway images.
I've also made a mental note to get back to this same crossing location to get more boring images of the now updated line, and have before and after shots.
But what I'm kicking myself over is not having gone back earlier .. pre line update to get proper images of the interesting looking old crossing.
Again it drive home the accurate adage about putting off what you can do now, for a later time .. sometimes it just doesn't pay.

[John King]

Arthur, that's exactly what Bridge does.

Almost no other software supports IPTC keywords. Picmeta does, but it's glacially slow at searching compared with Bridge, and does not support multi-level searching or logical operators.

Credit where and when it's due ...

[arthurking83]

Arthur, that's exactly what Bridge does.

Almost no other software supports IPTC keywords. ..

maybe they changed
In my experience Bridge never added IPTC data into NEF, CRW nor PEF files that I've tried .. but in saying that, I'm talking at least 3years ago now.
Could do jpgs and tiffs .. but who needs jpgs and tiffs for archiving(well OK, tiffs maybe, but camera files .. not tiff .. always only only ever raw)

Maybe they have better interoperability with Olympus raw files?

Only software that I've ever found to allow IPTC data entry into raw files has been:
PhotoSupreme(now) .. but back in the day it used to be IDImager(which I still have a license for) PhotoSupreme wasn't a good upgrade path tho, may have to eventually reconsider it as an option.
I haven't yet tried Pentax's own software, but Canon and Nikon's raw file software allows IPTC data to be embedded into their respective raw files.
The only other software that allows embedding IPTC data into the raw file is M$'s Photo Gallery(free), with the stipulation that you have your manufacturers raw codec installed on Windows.
(no idea if this is possible in a Mac environment).
They are the only few software I've tried(and I reckon I tried them all! ) that will add keyword data into a raw file. (as well as Exiftool based software).

So my primary and secondary keyword/tag info is entered via Nikon's software and Windows Photo Gallery(but Gallery is about to be completely run out of town with the next incarnation of Windows .. so I'm on borrowed time using Gallery)

I have a sneaking suspicion that you may be confusing embedded IPTC data into a raw file(if that's what you were referring too) with Bridge's usage of 'linked data'
So in the instance(of linked data) Bridge knows of the file that you seemingly have entered IPTC data into .. but the reality is it's linked to that file, not embedded. Break the link and you see no IPTC data in that raw file.

What I'm referring too is:
Keyword with either Nikon software or Tag with Window Photo Gallery(makes no difference which).
Once done in one, the other sees it too. (that's really all I use).
But! .. open the folder of the raw files, click on a raw file in Windows Explorer and the data entered via Galley of Nikon's software is also usable in Win Explorer.
Almost all other software will also see the embedded data too. I've yet to come across software(of any use) that doesn't use that embedded data.
This is the major difference!
Add the data via Bridge, and Windows can't see it .. matter of fact no other software that I know of can either(other than Adobe software).
I know that soem software can be made to use the database of Adobe metadata, but it's an arduous process and basically regurgitates all that data into a new database. IDImager had a tool to do this. But it couldn't see Adobe 'linked' metadata.

Anyhow, this is the way it works on Nikon/Canon/Pentax(PEF!!) files
If it's different in ORW land .. could be a good reason to seriously look into Olympus!

Note tho .. I have limited experience with DNG, have converted some files, downloaded others(I don't have a DNG capable camera) .. but some data gets garbled in the conversion from CRW and NEF to DNG, but IPTC data is seen on some other DNG capable software.
Actually, my real hope is that one day camera makers will just suck it up and adopt DNG as a proper alternative to silly TIFF files in camera!
I can't ever remeber anyone in the last 15 years having shot in TIFF format in camera!

It'd be interesting to hear back if your use of IPTC data into Olympus raw files is seen across the board by other software(and which does, and which don't).

[Tannin]

^ All of that granted, Bridge remains bloated, slow, and horrible to use. Why would anyone want fat, ugly, waddling Bridge when there are so many slim, fit, attractive beauties like XNVView and Faststone willing and eager to dance with you?

[John King]

Tony, I've replied twice now, and both times the site has lost my post/s. I won't try again.

Bridge handles IPTC keywords and searches better/faster than any other program I have come across. It also does this with complex searches on multiple keywords/image attributes.

[ameerat42]

JK. I can see 3 of your posts in this thread. Is that what you mean?
Do you use the autosave feature? You can restore that content before
making a new post. At worst, you have to switch into Go Advanced mode.

[John King]

JK. I can see 3 of your posts in this thread. Is that what you mean?
Do you use the autosave feature? You can restore that content before
making a new post. At worst, you have to switch into Go Advanced mode.

Thanks, Am. The posts disappeared, and auto-save hadn't saved them .

[Tannin]

I can't comment on keywords, John, IPTC or otherwise. I've never used them or even seriously considered them. Possibly this is a mistake, but I have always made the assumption that the time overhead for entering them for thousands upon thousands of images is prohibitive.

Now if my pictures were, say, a collection of news photographs, that might be different. It would take a hell of a lot of work, but I can imagine some politician (e.g.) dying in a car crash and wanting to see every picture I had of that person, and probably of his close associates, and see all of those before a given deadline. So I'm not saying it's useless, I just cannot imagine a realistic non-commercial reason compelling enough to justify such a massive time investment.

(And don't you hate losing posts?) Somewhere, far, far away in another part of the multiverse, there is a planet where all those millions upon millions of lost posts go; a planet where they wait patiently for some dimension-skipping space-time traveller to come along and appreciate them in all their delicately faded glory. As you know, all the very best posts are lost posts, and all those laboriously typed replacements which make it to the web are just B-grade copies of the lost magnificent originals.)

[John King]

I can't comment on keywords, John, IPTC or otherwise. I've never used them or even seriously considered them. Possibly this is a mistake, but I have always made the assumption that the time overhead for entering them for thousands upon thousands of images is prohibitive.

Tony, my brother has used your system for many years, since the early 2000s. He now has such a mess and huge duplication of images that he is undertaking basic keywording of all his images. This will make the identification of duplicates far easier and faster. It has taken him about 1-2 weeks so far, and now largely completed.

Now if my pictures were, say, a collection of news photographs, that might be different. It would take a hell of a lot of work, but I can imagine some politician (e.g.) dying in a car crash and wanting to see every picture I had of that person, and probably of his close associates, and see all of those before a given deadline. So I'm not saying it's useless, I just cannot imagine a realistic non-commercial reason compelling enough to justify such a massive time investment.

I incrementally upload files from a card to its own folder. Each card is labelled both physically and digitally so that I can easily identify possible card failures. With today's huge cards, I initially put the files into dated folders under the main card's folder. When the card becomes full, I move all the images into the main card folder, then audit them against the card to ensure that none have been missed. Once that process is done, I then put a text file into that folder with the filename "Audited against card". Moving files on the same NTFS volume doesn't actually move the file (as I'm sure you know), it just rewrites the MFT entry to refer to a new head entry in the volume's database. That is to say, there is no possibility of data change/loss doing this process, other than by catastrophic HDD failure.

I do not erase the card files until I have at least one verified backup of the files onto an offline HDD.

With my filing system, once uploaded, files never get moved again unless for some kind of major system change. All variations of the original file/s (RAW+JPEG) always contain the original camera filename. I am currently 'between computers'. I am building a "new" PC, which is nearly finished. Once complete, I will use Windows Easy Transfer, to make a checklist of all programs and settings on my old main w/s, then remove its enterprise level HDDs into the new PC, while preserving the drive mappings.

Files are backed up multiple times to external HDDs. I have a 2TB Toshiba portable HDD that lives in the front pocket of my main camera bag ... I use SYNCBACK for this. If you are particularly neurotic about the backup integrity (like me), SYNCBACK allows you to use checksums and the like to verify backups. Just takes (far) longer.

Keywording is actually fairly fast. Because you can select all the images that you want to apply a particular keyword to, then just tick that keyword. Bridge does not use a catalog system, it (now) uses a MySQL database for searching and manipulation, but also embeds the keywords directly into the file where possible (e.g. JPG, TIFF, DNG) and into sidecar (.XMP) files where impossible to embed into the RAW file. I do know of at least one program that will embed keywords directly into RAW files, but it is klutzy (and I consider this to be an undesirable practice ... ). Bridge automatically keeps RAW and XMP file pairs together if you use Bridge to move, copy, delete the RAW file. Otherwise, it's hardly difficult to keep them together using any normal file manager, like Windows Explorer.

How I do keywording is to avoid the IPTC classification system like the plague!! It doesn't even make much sense to me ... Over the years, I have built up a couple of thousand keywords and phrases that are organized under major categories of things, people, creatures, activities and the like. I backup all these keywords to a text file pretty regularly.

After a shoot, I will (theoretically ... ) upload the images to the computer, then apply a general descriptor keyword - e.g. "Beaumaris Concourse car show 2017" to each class of shots. This allows me to isolate these a bit later (either by using the search facility, or by ticking the keyword in the L/H pane, Keywords in Bridge), then apply other keywords such as "Aston Martin", "Elfin", etc to individual shots.

Ticking the "No keywords" selection in the L/H pane, Keywords section of Bridge makes these entries hide themselves after application of the first keyword. This tells you how much pain there is still to go!

(And don't you hate losing posts?)

YES !! Usually some carefully worded response that addresses specific issues in a specific order. Clear, concise, never to be repeated ... Usually done painfully on a tablet ... .

Somewhere, far, far away in another part of the multiverse, there is a planet where all those millions upon millions of lost posts go; a planet where they wait patiently for some dimension-skipping space-time traveller to come along and appreciate them in all their delicately faded glory. As you know, all the very best posts are lost posts, and all those laboriously typed replacements which make it to the web are just B-grade copies of the lost magnificent originals.)

Quite ...

- - - Updated - - -

Tony, as an example of the power of keywording, on my new PC (where all my E-M1 MkII images are stored/worked on), there are 10,694 image files. Bridge took 35 seconds to extract the 858 files with "cats" in their keywords. "Cats" images are sprinkled randomly throughout all my images.

I could just as easily have made this search to find all images with the keyword "cat", while excluding those that contained the keyword "cats". I use the keyword "cat" for cats other than our own pair, and "cats" exclusively for our own two villains. Bridge is very, very powerful in this way.

These files are currently located on a very slow hard disk, with a minimal cache (if any at all!). I would expect this process to be much faster after I have moved the enterprise class HDDs out of my current old w/s.

[Tannin]

^ Not bad for a B-grade copy.