storage and organization

[poider]

G'day, just wondering what are the better options for storage and organization of photos?
I have over 40,000 photos from my wife and I and our travels, usually I put the photos in folders with place names etc, but find it excrutiating when I want to look at said photos, any folder with 200 or more photos takes ages to load in and settle, as I am scrolling it continually updates and changes the pictures order etc.
Is there a way that I can put in a search in my main (parent folder) that will search for and tell me how many photos I have taken at 300mm or how many with centre weighted or whichever perimeter I Choose, I know if I put all my photos in one folder I can group them by many different perimeters but having 40,000 photos in one folder would probably crash??
Peter

[ameerat42]

This will do all or most of that.
Faststone Image Viewer.

It also does some basic editing. Lots of people use it on AP.

[paulheath]

Lightroom

[Tannin]

Ameerat42 is spot on.

What you need is an image viewer with decent performance. There are dozens of image viewers, many of them very good. Rule One is don't even consider anything from Adobe - Adobe products are a byword for bloated, slug-like performance which would put a sloth to sleep and bring a fast computer to its knees.

Faststone is fine. XNView is excellent. I still love the ancient PMView because of its outstanding ergonomics. Then there is ACDC, IrfanView, and various others I can't remember. Any one those mentioned will be around 1000% faster than any Adobe product this side of Pluto. None of them have the slightest trouble loading large folders.

(To be fair to Adobe, none of their products are actually designed-for-purpose in the way that the programs mentioned above are. Image viewers are not the same thing as image editors. Adobe doesn't make an image viewer (unless you count Bridge, which would be stretching a point), they only make image editors which can, at a pinch, be pressed into service as a second-rate viewer. But if they did make one, there is no reason to suspect that it would be any less bloated, glacial, and awkward in use than the likes of Lightroom, Acrobat, or Photoshop. Go with one of those I mentioned, or an equivalent of your choice.)

[poider]

Thank you all I will look at faststone, I have photoshop elements 14 for editing and have never really liked light room.
Thank you
Peter

[ricktas]

Thank you all I will look at faststone, I have photoshop elements 14 for editing and have never really liked light room.
Thank you
Peter

Photoshop (even elements) is an editing package. Lightroom is an asset manager with editing capabilities. Lightroom does have wonderful cataloguing, when used correctly.

But as with everything from camera brand to lens selection, it is personal choice.

[arthurking83]

For file viewing I have a preference for Fastone Viewer too.

BUT!! .. (big but here). If I need to view a lot of files in one go, nothing beats XNViewerMP.
What I mean by that is that as you store files in specific folders, most viewer software forces you to view the images from within each folder.
So if you have 20K image files, in 1000 folders with XNViewMP you can leave it to display only the files within each folder, OR you can set it to display all the files within the most senior folder recursively(it's your choice).

It does take a while to build up it's cache when viewing a lot of files in one go(ie. recursively) but it still displays a very high number as you view them whilst it build it's cache.
On my last usage it took 10mins to build the thumbnail cache for about 1000 80+ Mb NEF images.
If you only have jpg files saved, it's super duper much faster. Raw files take a bit of time to decode/read.

What I didn't like about Lightroom is that it forces you to make a catalog .. ie. that you can view all files at once once they're catalogged.
it's OK in what it does, but I much prefer the option to do so or not.
And it's catalog is not just a simple catalog .. it's a monstrous behemoth of a database.
Mine once ended up at over 6Gb with barely half my 200K images forcefully catalogged. I used to regularly delete it(only as I never wanted/needed another catalog of images).

On my primary and secondary archive locations I have about 200K images all up(jpgs, tifs and mostly NEFs).
XNViewMP will take about 20-30mins to rebuild the thumbnail cache(noting that i always clear it out, or delete it). So every time I use XNViewMP I basically start over from scratch.
For me this is important for one specific reason .. and why I use XNViewMP.
I once had a data corruption whilst moving images from one drive to another. 100+ K images to view in their entirety is too big to go through individually via each separate folder, so unknown to me I'd lost about 20 NEF images due to that data corruption.
Literally .. too many folders too many images to sift through each one.
By the time I'd found this out, I'd already updated my archives a few times over, but in that update I saved corrupted images where before the corruption they weren't.
So I was backing up corrupted images.

For me, XNViewer was the only practical way to locate every corrupted image I may have had.
Problem with Lightroom is that it builds it's own database, and relies on it's thumbnail which is stored separately as it's own file.
People may think this is good, but if the original file is corrupted, displaying the perfectly fine thumbnail doesn't show you that the actual raw file is corrupted.
That's where XNViewer comes into it.
Having to build the thumnail cache from scratch it showed a non image thumbnail. That is, for those corrupted raw files, it didn't display a thumnail of the raw file, it displayed a generic icon.
From that, I know there's an issue with the raw file data.
So that's my primary use for XNViewer. I set the thumbnail size to the smallest size, and open my primary and secondary photo archives recursively. I can sift through all 200K images in about 10 mins on that page.
Once I feel secure that they aren't corrupted I resave one of those those archives again to another location as a backup and then update those archives again once a year.

Hope that made sense.

Basically:
1a. For viewing and very basic edits of NEF files ; Nikons' ViewnNX2
1b. For general viewing of many image formats and device types ; FSViewer. it's the smallest fastest viewer that you don't even have to install(you can ruin it as a portable install).
3.. For the more complex viewing tasks ; XNViewerMP.


Question. Do you shoot in raw(NEF) mode or jpg?
I have to make dinner now, but a very important point to note for you is Keywording/Tagging, and IPTC.
I've found over the years, that catalogs the way that programs like Lr and other do them are pretty much useless unless you plan to lock yourself into those programs systems for ever.
Keyword raw files with embedded data, not thirdparty external data!
On a raw file this is not so easy to do.

[John King]

Keywords in Bridge are industry standards compliant. That's a big plus.

Keywording is a PITA!!

However, once done, retrieving any subset of files is very fast. Bridge can search using multiple keywords and logical associations between them.

Far be it from me to defend Adobe in any way, but this works.

Bridge CC is a free download, and can use any editor linked to it.

[Tannin]

It wouldn't work for everyone, but I store everything - absolutely everything - by date, with folders further labelled by location. I practically never have to spend more than a few moments searching, because all I have to do is remember the place, which leads in turn to the journey, which provides me with the approximate date - and I never, ever have to enter a keyword. Once in a long while I trip up over a place I've been to many times and have to look harder. When (to take an example that occurred just yesterday) did I see that vast flock of Flock Bronzewings near Cunnamullah? I've visited the district perhaps six times, and passed through on the way to somewhere else another dozen times, often camping there overnight. Took me about ten minutes to track the pictures down last night, which is very unusual. Are they good enough to publish? Hmmmm .... maybe - which is exactly the same conclusion I failed to come to the last five times I looked at them

Come to think of it, it's not quite as primitive a system as that. If I'm uncertain, provided that there was at least one picture I thought worth posting on that day or close to it, my website is database-driven and readily searchable by things like species, lens, time of day, habitat type, geographical location, and so on, so it's a simple task to find that picture, and thus the date, which leads directly to the other one I'm after. So having said "I don't use or need tagging", I actually do use a form of it (albeit for a very small subset of all pictures) and it is very useful. The point here is that if the task of tagging a gazillion pictures seems way, way too difficult, provided that your basic storage system is robust, you usually only need to tag a very few representative examples. 98% of the benefit for 1% of the effort.

[arthurking83]

Keywords in Bridge are industry standards compliant. That's a big plus.

....

When they set the standards, then yes. But interoperability with other software is next to zero(unless the other software has an import feature) .. even then, sometimes the imported data becomes garbled in some ways.

For true(er) 'industry standard' stick with ITPC data tagging into the raw file and you're safer for the future.

My history was similar to Tannin, been to many places multiple times, and it then becomes hard to differentiate some tiny insignificant detail difference between dates.
But I do the same.
Uppermost folder structure starts with dates, and then locations.

[John King]

Arthur, that's exactly what Bridge does.

Almost no other software supports IPTC keywords. Picmeta does, but it's glacially slow at searching compared with Bridge, and does not support multi-level searching or logical operators.

Credit where and when it's due ...

[Tannin]

^ All of that granted, Bridge remains bloated, slow, and horrible to use. Why would anyone want fat, ugly, waddling Bridge when there are so many slim, fit, attractive beauties like XNVView and Faststone willing and eager to dance with you?

[John King]

Tony, I've replied twice now, and both times the site has lost my post/s. I won't try again.

Bridge handles IPTC keywords and searches better/faster than any other program I have come across. It also does this with complex searches on multiple keywords/image attributes.

[ameerat42]

JK. I can see 3 of your posts in this thread. Is that what you mean?
Do you use the autosave feature? You can restore that content before
making a new post. At worst, you have to switch into Go Advanced mode.

[John King]

JK. I can see 3 of your posts in this thread. Is that what you mean?
Do you use the autosave feature? You can restore that content before
making a new post. At worst, you have to switch into Go Advanced mode.

Thanks, Am. The posts disappeared, and auto-save hadn't saved them .

[Tannin]

I can't comment on keywords, John, IPTC or otherwise. I've never used them or even seriously considered them. Possibly this is a mistake, but I have always made the assumption that the time overhead for entering them for thousands upon thousands of images is prohibitive.

Now if my pictures were, say, a collection of news photographs, that might be different. It would take a hell of a lot of work, but I can imagine some politician (e.g.) dying in a car crash and wanting to see every picture I had of that person, and probably of his close associates, and see all of those before a given deadline. So I'm not saying it's useless, I just cannot imagine a realistic non-commercial reason compelling enough to justify such a massive time investment.

(And don't you hate losing posts?) Somewhere, far, far away in another part of the multiverse, there is a planet where all those millions upon millions of lost posts go; a planet where they wait patiently for some dimension-skipping space-time traveller to come along and appreciate them in all their delicately faded glory. As you know, all the very best posts are lost posts, and all those laboriously typed replacements which make it to the web are just B-grade copies of the lost magnificent originals.)

[John King]

I can't comment on keywords, John, IPTC or otherwise. I've never used them or even seriously considered them. Possibly this is a mistake, but I have always made the assumption that the time overhead for entering them for thousands upon thousands of images is prohibitive.

Tony, my brother has used your system for many years, since the early 2000s. He now has such a mess and huge duplication of images that he is undertaking basic keywording of all his images. This will make the identification of duplicates far easier and faster. It has taken him about 1-2 weeks so far, and now largely completed.

Now if my pictures were, say, a collection of news photographs, that might be different. It would take a hell of a lot of work, but I can imagine some politician (e.g.) dying in a car crash and wanting to see every picture I had of that person, and probably of his close associates, and see all of those before a given deadline. So I'm not saying it's useless, I just cannot imagine a realistic non-commercial reason compelling enough to justify such a massive time investment.

I incrementally upload files from a card to its own folder. Each card is labelled both physically and digitally so that I can easily identify possible card failures. With today's huge cards, I initially put the files into dated folders under the main card's folder. When the card becomes full, I move all the images into the main card folder, then audit them against the card to ensure that none have been missed. Once that process is done, I then put a text file into that folder with the filename "Audited against card". Moving files on the same NTFS volume doesn't actually move the file (as I'm sure you know), it just rewrites the MFT entry to refer to a new head entry in the volume's database. That is to say, there is no possibility of data change/loss doing this process, other than by catastrophic HDD failure.

I do not erase the card files until I have at least one verified backup of the files onto an offline HDD.

With my filing system, once uploaded, files never get moved again unless for some kind of major system change. All variations of the original file/s (RAW+JPEG) always contain the original camera filename. I am currently 'between computers'. I am building a "new" PC, which is nearly finished. Once complete, I will use Windows Easy Transfer, to make a checklist of all programs and settings on my old main w/s, then remove its enterprise level HDDs into the new PC, while preserving the drive mappings.

Files are backed up multiple times to external HDDs. I have a 2TB Toshiba portable HDD that lives in the front pocket of my main camera bag ... I use SYNCBACK for this. If you are particularly neurotic about the backup integrity (like me), SYNCBACK allows you to use checksums and the like to verify backups. Just takes (far) longer.

Keywording is actually fairly fast. Because you can select all the images that you want to apply a particular keyword to, then just tick that keyword. Bridge does not use a catalog system, it (now) uses a MySQL database for searching and manipulation, but also embeds the keywords directly into the file where possible (e.g. JPG, TIFF, DNG) and into sidecar (.XMP) files where impossible to embed into the RAW file. I do know of at least one program that will embed keywords directly into RAW files, but it is klutzy (and I consider this to be an undesirable practice ... ). Bridge automatically keeps RAW and XMP file pairs together if you use Bridge to move, copy, delete the RAW file. Otherwise, it's hardly difficult to keep them together using any normal file manager, like Windows Explorer.

How I do keywording is to avoid the IPTC classification system like the plague!! It doesn't even make much sense to me ... Over the years, I have built up a couple of thousand keywords and phrases that are organized under major categories of things, people, creatures, activities and the like. I backup all these keywords to a text file pretty regularly.

After a shoot, I will (theoretically ... ) upload the images to the computer, then apply a general descriptor keyword - e.g. "Beaumaris Concourse car show 2017" to each class of shots. This allows me to isolate these a bit later (either by using the search facility, or by ticking the keyword in the L/H pane, Keywords in Bridge), then apply other keywords such as "Aston Martin", "Elfin", etc to individual shots.

Ticking the "No keywords" selection in the L/H pane, Keywords section of Bridge makes these entries hide themselves after application of the first keyword. This tells you how much pain there is still to go!

(And don't you hate losing posts?)

YES !! Usually some carefully worded response that addresses specific issues in a specific order. Clear, concise, never to be repeated ... Usually done painfully on a tablet ... .

Somewhere, far, far away in another part of the multiverse, there is a planet where all those millions upon millions of lost posts go; a planet where they wait patiently for some dimension-skipping space-time traveller to come along and appreciate them in all their delicately faded glory. As you know, all the very best posts are lost posts, and all those laboriously typed replacements which make it to the web are just B-grade copies of the lost magnificent originals.)

Quite ...

- - - Updated - - -

Tony, as an example of the power of keywording, on my new PC (where all my E-M1 MkII images are stored/worked on), there are 10,694 image files. Bridge took 35 seconds to extract the 858 files with "cats" in their keywords. "Cats" images are sprinkled randomly throughout all my images.

I could just as easily have made this search to find all images with the keyword "cat", while excluding those that contained the keyword "cats". I use the keyword "cat" for cats other than our own pair, and "cats" exclusively for our own two villains. Bridge is very, very powerful in this way.

These files are currently located on a very slow hard disk, with a minimal cache (if any at all!). I would expect this process to be much faster after I have moved the enterprise class HDDs out of my current old w/s.

[Tannin]

^ Not bad for a B-grade copy.

[John King]

Beats the heck out of the eyeball search algorithm. .. .

[John King]

Interestingly, I just did the same search on my old w/s. It took just under 60 seconds to extract 4305 files from about 92,693.

What's interesting about this is the apparent difference HDD speed makes here, along with general differences between the two PCs.

New PC:

Intel Core 2 Duo clocking at 2.93 GHz, SATA3 SSD 3Gb/s (running at SATA2 speed due to M/B limitations), separate old office/commercial PC 500GB HDD for image storage currently (with basic cache??), 2GB DDR5 RAM graphics card, 16 GB DDR3 main RAM, Win7Pro 64 bit.

Old PC:

Intel Core 2 Duo clocking at 2.13 GHz, SATA2 HDD for OS, separate Seagate Enterprise 3TB HDD for image storage (64MB cache), 512MB DDR2 RAM graphics card, 4 GB DDR2 main RAM, WinXPPro 32 bit.

Bridge is appallingly slow to initially cache images, but very fast to do anything else. FSV (for example) is pretty fast to do the initial caching process, but just cannot do most of the rest of what Bridge does, or can do. I never took to Lightroom, worst of all possible worlds!

The Olympus Viewer 3 program is much faster since upgrading the 512MB DDR2 graphics card to the 2GB DDR5 one in the "new" PC yesterday. Went from 50+ seconds to update changes to a RAW file to 7 seconds. Still tragic, but tolerable. I do not use OV3 for image editing or management. Apart from anything else, it can only use an aRGB colour space (16 bit), and I almost always use a ProPhotoRGB 16 colour space for editing. PS6 and Bridge are noticeably faster, but not by that sort of order.