Adding keywords in GUI & saving file adds to Subject, HierarchicalSubject NOT Keywords (exiftool)

"I tried with various formats DNG and also canon CR3 with xmp sidecar. For example: the keywords added in lightroom do in fact show up in the xmp file but under the Subject XML tag and when I query the file using exiftool --subject works but --keywords does not. I'm looking more but I think that there may be varying standards."

Summary: You are in a maze of twisty little passages, all alike. The industry has created a mess with EXIF, IPTC, and XMP containing overlapping metadata field definitions. LR has long followed the Metadata Working Group's specifications for handling EXIF, IPTC, and XMP, which imposes some order and consistency. LR's behavior with keywords conforms to that spec.

Details

LR writes keywords in up to to three fields, XMP:Subject, XMP:HierarchicalSubject, and IPTC:Keywords (I'm using Exiftool's notation here. When using Exiftool, it's best to use the options "-a -G" so you can see precisely which fields are involved.)

For JPEGs and TIFFs, LR writes XMP:Subject, XMP:HierarchicalSubject, and IPTC:Keywords.

For DNG, LR writes XMP:Subject and XMP:HierarchicalSubject and deletes IPTC:Keywords if it is present. 

For XMP sidecars, LR writes XMP:Subject and XMP:HierarchicalSubject (XMP sidecards don't support the IPTC format).

The term "IPTC" in this context can be ambiguous. I'm using it as Exiftool uses it, to refer to the legacy Information Interchange Model format, which can be stored in various file formats including JPEG and TIFF. The International Press Telecommunications Council has defined the newer IPTCCore specification that stores similarly named fields in the XMP format. With "-a -G", Exiftool will show these new IPTCCore fields labeled as XMP:, but if you use the "-a -G1" option, you can see which namespace within XMP is storing a field:

$ exiftool -subject DSC02572.xmp
Subject : abcd
$ exiftool -a -G -subject DSC02572.xmp
[XMP] Subject : abcd
$ exiftool -a -G1 -subject DSC02572.xmp
[XMP-dc] Subject : abcd

The MWG's rules for reading keywords are a little involved when more than one of the three keyword fields is present.  If you use Exiftool's composite tag MWG:Keywords, then Exiftool will do the heavy lifting of reading and writing meteadata fields in conformance with the MWG spec.

[Use the reply button under the first post to ensure replies sort properly.]

To build on Tony's reply, we'll need much more detail. I just tested this with Exiftool and JPEGs, and it behaved as expected: LR's Metadata > Save Metadata To File wrote the keywords to XMP:Subject, XMP:HierarchicalSubject, and IPTC:Keywords.  Metadata > Read Metadata From File read the keywords from IPTC:Keywords.

One thing to remember is that you must do Save Metadata To File and Read Metadata From File -- LR won't read or write the metadata automatically (there's an option for writing).

[Use the reply button under the first post to ensure replies sort properly.]

Glen:

Wondering what file format are you using in this process?

Or is it various formats?