Lightroom Classic: Multiple dng files from conversion

[DMEEPhD]

After conversion of tiffs to dngs, I have multiple dngs!!!

Here's the details...

After importing a series of tiff images, I converted the tiffs into Adobe dngs files.  After restasrting LR 7.5, I find that I have multiple dngs!?!   At least three (3)!



Really?  Seriously?

My Finder does not show these duplicates as actually existing.  So where in the heck did they come from?

[Victoria Bampton LR Queen]

Oooooooh, you might have just found a clue in a bug I've been tracking. Can you take us through the steps you took leading up to that state please?

[Just Shoot Me]

What is the Exact File Name of those 3 DNG files? Just because it ends with 2019021.dng doesn't mean they were created from the same TIF file. I see you have a JPG file that ends with that 2019021 group of characters. Could be one of the DNG files that were created was made from that JPG file.

Now the question I have to ask.
Why are you creating DNG files from TIF files? In my mind there is NO GOOD reason to do that.
TIFF is a multi platform/program recognized image format. The file can be read and displayed by many many different computer OS's and programs. DNG is not.
When I send an image over to Photoshop for further editing I save it as a TIF file. I would not then convert it back to a Digital NeGative (DNG) for any reason.

[thedigitaldog]

My question too; why convert a TIFF to what is basically another TIFF?

[JohanElzenga]

I see a tiff and a jpeg with the same name too. It could well be that you also have a raw file with that name. If all three were selected when you converted to DNG (and they were not in the same folder) then you could end up with three DNG files with the same name (in three different folders, you can't get that in the same folder).

There is a problem though that has been reported a few times, where people have multiple copies of one and the same image showing in their catalog. It seems to be some kind of catalog corruption, but it is unclear what causes it. That is what Victoria is referring to.

[DMEEPhD]

All five (5) files in LR have exactly the same file name.  However, the two (2) extra dngs do not actually exist in any folder on any of my drives.  This is confirmed.

The tiff file is a raw file from an old camera which produces a tiff and a jpeg.  My Photoshop guru prefers working with dngs, hence the GOOD reason for creating one (1) from the tiff.    Only the tiff was selected when the dng creation was executed, and the fact that the others do not really exist except in LR is proof.

[DMEEPhD]

Nope.  Please see response above.  I think it is a catalog corruption too, and the catalog is currently being checked for integrity and then optimization.

[thedigitaldog]

Your Photoshop guru is confused (or you are if indeed the original is a TIFF) and you really need to check up on what is rather silly “advice” IF so.
Verify THEN trust.
Ask him exactly what good it does you and report back. Raw, TIFF and DNG are all based on the same Adobe owned and controlled format.
A raw converted to DNG is not the same as a TIFF converted to DNG! One is rendered and the other is not.

[DMEEPhD]

Hi Victoria,

I was simply following my regular workflow when dealing with files from this old camera.  I download the files into a directory created by a photography management program named Light Blue.  The folder structure is created and named automatically; in this case the folder structure is:

Mexico's Mayan & Aztec Illumination
Mexico's Mayan & Aztec Illumination\processed
Mexico's Mayan & Aztec Illumination\unprocessed

The images are copied with file verification using Path Finder 8 into the 'unprocessed' subfolder.  They are then renamed as 'Mexico's Mayan & Aztec Illumination - 'sub location' - 'original file number'.

In this case, the files were named:

Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.tif
Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.jpg

After copying and renaming, the files are imported into LR.

After import, depending upon the number of files and their size in the import, I might optimize the catalog.  I might keyword fist, and then optimize.  There's no hard and fast rule; I typically see how LR is responding.  If it is flying along, then I continue the workflow without optimization.  If it is dogging it, I will optimize.  If the optimization takes a long time, I will restart LR and then continue.

In this case, LR was humming along nicely, so I keyworded the images and then selected the tiffs for conversion to dng.

One (1) dng was produced for each tiff.  I optimized and restarted and viola!  Three (3) dngs in the collection 'Previous Import,' faithfully aligned with the tiff and the jpeg, all with the very same file number.  Except, the duplicates actually do NOT exist in any folder on any drive.  I verified this with Path Finder 8.

I've been using LR since version 3 and have never seen this behavior previously.  In fact, I'm pretty sure I have encountered more oddball behavior and bugs in LR 7.5 than in any previous version, and my observations have been borne out by reading the blogs since the release of LR 7.

That's why I delayed for months before upgrading, and I did that all because of a piece of hardware — Loupedeck+.

Ah well.  What do you think, Victoria?  Should I remove the dngs from the catalog and start again?  Should I move the real dngs where LR can't find them and then see of LR cannot find them and then move them back?

I can see several permutations one could apply here.

[DMEEPhD]

My Photoshop guru is a she and Simone makes a very nice six-figure income in the UK providing Photoshop services to corporate clients.  They, and I, trust her implicitly, and they put their trust in her in the form of British Pounds.

I'm not completely aware of the history of the raw file, but back in the day there were several cameras which produced as their 'raw' file a tiff file.  However, I do know that raw files which are not tiffs are NOT an "Adobe owned and controlled format"—that is an idiotic premise.  The raw file format (e.g., .arw . CR2, etc.) is proprietary to the camera manufacturer.   I think Sony or Canon or Nikon would be surprised to know that they do not own their patented intellectual property.

You should also look up the definition of "rendered" as what you imply is grossly incorrect.

As to your claim about the basement of DNG and TIFF formats, once again you are incorrect.  The DNG format is based upon the international standard ISO 12234-2, titled "Electronic still-picture imaging – Removable memory – Part 2: TIFF/EP image data format".  This is different from the Tagged Image File Format, which is a standard administered by Adobe currently called "TIFF, Revision 6.0 Final – June 3, 1992."

In a nutshell, the major difference between DNG and the ISO 12234-2 standard is the increased capacity for the inclusion and handling of metadata.  That's why Simone—and many other Photoshop professionals working with images from professional photographers—prefers using the DNG format rather than a simple TIFF.

[DMEEPhD]

Update: After catalog optimization, the duplicate DNGs are still present.  However, when one right-clicks on the image in Library and selects 'Show in Finder', each DNG points right back to the single DNG file.

In essence, these duplicate DNGs exist only in Lightrooms imagination.

The question remains: "How to remove the extras?  Which ones should be selected, or does it make no difference?

[thedigitaldog]

A TIFF is a rendered image. A raw isn’t. Both are based on TIFF/EP. If your camera produces an actual TIFF (rendered), there is absolutely NO reason to convert it do DNG. If it is actually a proprietary raw, there are indeed many reasons to convert to DNG:
http://digitaldog.net/files/ThePowero...

You stated you are converting rendered TIFFS to DNG but hopefully you are confused between proprietary raws and rendered TIFFs.

[thedigitaldog]

BTW, this is raw and I'd hope you would agree, it's nothing like a rendered TIFF:

[DMEEPhD]

Where exactly did I state I was converting 'rendered' tiffs?  I am converting a raw file which happens to be in the tiff format into a dng.   There are high-end and speciality cameras which write tiff files as their raw format.

And I think you are still a bit confused about the tiff format.  There are two (2) diffierent ISO standards for this format; dated decades apart.  You cannot talk about a tiff file without making the distinction between which standard it is based upon.

[thedigitaldog]

Where exactly did I state I was converting 'rendered' tiffs

Where? In your VERY first post. Copy and paste:
 I converted the tiffs into Adobe dngs files 

A TIFF is rendered. I know of no digital camera that produces an actual TIFF that is raw data. I know of lots that produce proprietary raw data, not rendered that is based on a TIFF.
You should IMHO, follow your 'guru's' advise and convert proprietary raws to DNG. And I suspect you're not clear on the differences between TIFF and a proprietary raw data file or I and at least one other here would not have asked you specifically WHY you're converting TIFF to DNG which is something that can be done. You can convert a JPEG to DNG too. Pointless. And explained why. JPEGs, like TIFFs are rendered images, NOT raw data. Do you see the difference now? And why two of us asked why you're converting TIFF to DNG? 

What you really wanted to say was  I converted the raws into Adobe DNGs files. But you didn't so we had to read your writings as provided. 

[thedigitaldog]

Just re-sync the folder(s) or create a new catalog and import. Now what do you see? 

[DMEEPhD]

Andrew, despite you having a blog and publishing articles, you seem to not really understand what 'render', 'rendered', or 'rendering' is.

In essence, rendering is the application of the rendering equation—which is an integral equation in which the equilibrium radiance leaving a point is given as the sum of emitted plus reflected radiance under a geometric optics approximation—by a computer program.

ALL photorealistic or non-photorealistic images from a 2D or 3D model (or models in what collectively could be called a scene file)  are rendered by means of computer programs.

The results of displaying such a model can be called a render.  A scene file contains objects in a strictly defined language or data structure; it would contain geometry, viewpoint, texture, lighting, and shading information as a description of the virtual scene.  The data contained in the scene file is then passed to a rendering program to be processed and output to a digital image or raster graphics image file.

Your attempt to make a distinction by case of rendering is like saying there is a difference between puppies which are born and puppies which are whelped.

A difference which makes no difference is not a difference.

[thedigitaldog]

See the differences and the specific mention of TIFF below? I'm not making this stuff up sir:

https://en.wikipedia.org/wiki/Raw_image_format

A camera raw image file contains minimally processed data from the image sensor of either a digital camera, or motion picture film scanner, or other image scanner.[1][2] Raw files are named so because they are not yet processed and therefore are not ready to be printed or edited with a bitmap graphics editor. Normally, the image is processed by a raw converter in a wide-gamut internal color space where precise adjustments can be made before conversion to a "positive" file format such as TIFF or JPEG for storage, printing, or further manipulation. This often encodes the image in a device-dependent color space. There are dozens, if not hundreds, of raw formats in use by different models of digital equipment (like cameras or film scanners).[3]

Maybe you'll inform us of the actual camera you're using to produce the raw?

[thedigitaldog]

Andrew, despite you having a blog and publishing articles, you seem to not really understand what 'render', 'rendered', or 'rendering' is.

I'm not sure you really understand the difference between a rendered TIFF and a raw despite my efforts to show you (again, I'm not making this stuff up):

Raw vs. rendered 
https://www.dpbestflow.org/camera/raw-vs-rendered

https://www.strollswithmydog.com/raw-file-conversion-steps/
HOW DOES A RAW IMAGE GET RENDERED?

https://www.lynda.com/Maya-tutorials/Rendering-raw-files/370603/384936-4.html
Rendering raw files

Need more outside references? 

[DMEEPhD]

The article fails to point out that despite calling a tiff rendered and a raw file not rendered, the underlying mathematics of creating the file is based upon the very same equation introduced into computer graphics by David Immel and James Kajiya in 1986.  The various realistic rendering techniques in computer graphics attempt to solve this equation.

In short, all image files are rendered using this equation or a variant.  Therefore, they are all renedered from the data in the file.  What they look like to the naked eye is another matter, and I think that is where you are attemtping to draw the distinction.  However, from an engineering standpoint, it makes no difference.

I am not saying you are completely wrong; however you are playing a bit fast and loose with the underlying engineering whilst skipping the mathematics.  I see it as similiar to a book of popular science versus an article in a peer-reviewed science journal.  Both are essentially correct, but one is far more correct than the other.

[thedigitaldog]

All the articles speak of taking raw data, something you specifically did not mention, and rendering a TIFF. I don’t know if you are purposely trying not to understand this, or if you are really struggling with it.

EVERYTHING your computer does is math! It's all 1's and zero's. Your issue is language in defining what you're doing with what data. Which is why two of us asked you why you're converting TIFFS to DNG which we wouldn’t' have asked has you phrased your language ideally as I pointed out above ( to repeat: What you really wanted to say was  "I converted the raws into Adobe DNGs files". But you didn't so we had to read your writings as provided).

As the Chinese proverb says: The first step towards genius is calling things by their proper name.

[thedigitaldog]

Maybe you'll inform us of the actual camera you're using to produce the raw.

Or maybe you can't?

BTW, this is raw and I'd hope you would agree, it's nothing like a rendered TIFF:

Or maybe you can't? 

[DMEEPhD]

I am using a Hassalblad with a custom back made by Olympus Medical.  Actually, there are (or were) several camera manufacturers who produced digital cameras with their raw files having the .tif extension.  In the early days, Olympus was one of these.  Understood, as they are usually credited with developing the first mass market digital camara (though as an engineer and not a historian, I cannot state this for a certainty).

I am sure you are aware that many raw file formats, including IIQ (Phase One), 3FR (Hasselblad), DCR, K25, KDC (Kodak), CRW CR2 CR3 (Canon), ERF (Epson), MEF (Mamiya), MOS (Leaf), NEF (Nikon), ORF (Olympus), PEF (Pentax), RW2 (Panasonic) and ARW, SRF, SR2 (Sony), are based on the TIFF file format.  These files may deviate from the TIFF standard in a number of ways, including the use of a non-standard file header, the inclusion of additional image tags and the encryption of some of the tagged data.

So, I did not say I was converting 'rendered' tiffs.  I said I was converting raw files into Adobe dngs.  I did not use the term 'rendered' because it was superfluous if one understands the process of rendering.   I did not use the term 'raw' because that would also have been superfluous as that camera back produces tiff as its raw file.  You jumped all over this because you do not understand the definition or mathematical basis for render.

[thedigitaldog]

Please show me where Hasselblad states their raw file (.3FR) is a TIFF or why it doesn't have a .TIF extension if as you incorrectly assume, it's a TIFF? 

Please tell me that when you double click on a rendered TIFF vs. the 3FR, if Photoshop proper opens with a rendered image OR ACR opens with that 3FR? 

Please tell me why raw file formats, including IIQ (Phase One), 3FR (Hasselblad), DCR, K25, KDC (Kodak), CRW CR2 CR3 (Canon), ERF (Epson), MEF (Mamiya), MOS (Leaf), NEF (Nikon), ORF (Olympus), PEF (Pentax), RW2 (Panasonic) and ARW, SRF, SR2 (Sony), have unique extensions and again NEVER open rendered in Photoshop or similar but MUST be opened in a raw converter BECAUSE they are NOT TIFFs! 

You seem to believe that bread crumbs and a BLT sandwich are the same because both use bread. If I came onto a forum on cooking and asking about making a BLT with bread crumbs instead of bread, I suspect some who understand the differences would ask for clarity, and warn that using bread crumbs to build a sandwich isn't a good idea. Just like converting a TIFF to DNG is pointless. Two of us pointed this out to you due to your somewhat sloppy description of your workflow. No sir, you are not converting TIFFs to DNG. You are converting raw files, specifically a proprietary Hasselblad file to DNG. It may be based on TIFF but it ain't a TIFF and if you try treating it the same, the differences pointed out to should be obvious. Assuming you're here to learn or assistance which is questionable after so many posts trying to get you to clearly define what you're actually doing before we even get into the Lightroom 'issue' if it even exists. 

Now you came here for help despite your super guru who charges so much money which is a bit odd. You've asked a question in a way that is incorrect and confusing. OK, we understand now you're trying to convert a proprietary raw to DNG. It isn't a TIFF. No more than a BLT sandwich is the same as breadcrumbs. 

You wrote:  
I am sure you are aware that many raw file formats, are based on the TIFF file format.  
Yes as I told you this before you made that statement. Read then reply. 
DNG is based on TIFF too. So you converted from TIFF to TIFF using your not so well defined language of what's based on what? No, you converted from raw to DNG. Why confuse the issue and just form a question in a way we can attempt to aid you (for free unlike your guru). 

Had you as much experience with Hasselblad (Imacon) cameras as I sir, you'd recall the time their native raw format, written to disk WAS A DNG! Which isn't a TIFF. Nor is it today. 

[DMEEPhD]

Sweet Jesus take the wheel!  Do you deliberately misunderstand everything or are you simply simple?  The Hasselblad has a custom back on it made by Olympus Medical.  The 'blad body is actually a 500C film camera with the film back removed.

The raw files listed above are all based upon the tif standard.  These files may deviate from the TIFF standard in a number of ways, including the use of a non-standard file header, the inclusion of additional image tags and the encryption of some of the tagged data.  However, that does not mean they are tifs, but they are rendered using Immel and Kajiya's equation in the IC.  Clearly, you do not have an engineering degree—at least not from an accredited university.

Show me with equations and data structures how a tifand a raw file, both derived from an ISO standard.  Do not send me your opinions or layman articles.  Speak engineering or don't speak at all.

[thedigitaldog]

Sweet Jesus take the wheel!  Do you deliberately misunderstand everything or are you simply simple?  The Hasselblad has a custom back on it made by Olympus Medical.  The 'blad body is actually a 500C film camera with the film back removed.

Then you failed to answer the question:  WHAT raw format does the Olympus Medical back output? 

No, not all raw files are based on TIFF. And basing a raw on TIFF doesn’t make it a TIFF or it would behave like a TIFF which it will not assuming you actually try. 

You don't need equations, you need to understand how to define data being used in correct language. You didn't, hence you were asked by two people WTF you mean and why you are converting a TIFF to DNG. 

Now we know you're not doing that. You're just writing in a less then clear fashion even after being told is the correct terminology. Maybe your super guru can explain that to you and maybe you'll accept it.