why can't they contain all data? - because it would be not efficient *** generic formats would have to have all possible data points of all possible formats it supports (which would make the generic format inefficient, "bloated" and not so generic since every time new supported format was added new formats data points (the ones who you can't map at that time) would have to be added) - Generic formats just do not do that.
This is untrue. The design of a file format such as dng or tiff is specifically done to be easily extensible without breaking the format. There is no a priori reason why a file would become bloated by just including extra property tags or extra data points. If it does it is just badly designed.
Now to the example you used - PSD (as specific) and TIFF (as generic) - first of all - TIFF DOES NOT contain 100% of PSD data - just its rasterized image data - all layer, masking, effects, smart object, vector etc data IS LOST
Butch already corrected you on this, but you can use tiff as a complete replacement to psd. All photoshop layers, blending modes, etc. carry over. Also vector layers, text layers, layer style effect properties (i.e. drop shadows and such), and everything else that you can imagine inside a photoshop document transfer over. Tiff is actually a great example of a very versatile file format which is why Lightroom uses it as default for the "edit in Photoshop" command.
- now imagine RAW NEF being a PSD type format. - look at the example below (NOTE: this is a drastically simplified example which is designed to only act as an example and is NOT how the real generic/specific mapping actually work): Imagine a Specific format containing data points A, B, C, D - generic format had data points: 1, 2, 3, 4, 5, 6. Simple mapping of the specific format would be 1=A, 2=B, 3=C, 4=D, 5=NULL, 6=NULL - all works fine - generic has 100 % of the specific data.
This is not how data storage works in modern files. File structures are not rigid in this sense. What you do in a format such as tiff or typical raw formats is to have a header that says that what follows is a data block in 12-bit format that has N x M points, starts at position x in the file and goes to position y and is compressed using algorithm z. Then you just dump the block of data. If you convert that into another format, you will have a similar header with the same info and the data can be copied into it literally or by using a different compression technique. If you are talking about metadata such as camera settings, such stuff can simply be put in an extensible scheme such as xml. This is in fact what Adobe does in dng. This is in no way bloated.
But what if (we will disregard simple example as A,B,C,D,E,F,G,H - where H could not be mapped and would have to be stored as a dynamic field) the specific data has something like : A, A1 (layer data), A2, A2.1(masking data), B,C, D,D1? during conversion process (lets look at A) A data point gets captured (or sometime recalculated) and all sub-sets get discarded - just import image to PS, create some layers, masks etc and export it to TIFF - then open that TIFF in PS - do you have any of the layer data? No, it is gone - forever.
Have you ever used Photoshop? That is not how it works. Tiff contains all layers, masks and more. It can even contain smart objects, smart layers and such, so you can keep the raw data intact and change your raw conversion settings on the fly for a nondestructive workflow! All with tiff as the native file format!
If you want to now change something what use to be an easy fix in the original PSD (you want to add to a layer mask lets say) becomes an impossible task. Now imagine a D-lighting, NR, WB being treated as layers in NEF (or most likely nodes) and LOST once you convert to DNG. Makes me wonder if that's why "lifting" that example will always be better in Capture NX-D as during import of the NEF LR may just "flatten" all the layer/node data - I hope you can at least admit that TIFF DOES NOT hold entire PSD data set...
D-lighting (as is NR and WB) is written into the nef as a simple very short metadata tag. It is just an instruction to Nikon software to interpret the raw data using their HDR routines. There are no layers or nodes for it. The instruction is not lost when converting to dng. It is still there for any software to read it if it needs to and the conversion puts the relevant metadata (such as white balance info) in the new file structure where appropriate. However, non-Nikon software does not have the same HDR routines as Nikon software and so generally will ignore the tag. The same for noise reduction instructions. Other raw conversion software does not have the same noise reduction routines and so there is no point in respecting the tag.
... View more