Is anyone smart enough to know how to avoid duplicate sub-keywords?

[robirdman1]

Is anyone smart enough to know how to avoid duplicate sub-keywords? I am not. I had made phylogenetic keywords originally, in the form, Diptera (FLIES).  I found out, importing to my website, that the proper separation is with commas, and () causes problems.  So I decided to rename to Diptera, FLIES,

This is an Order of Insects, and there are Families, Subfamilies and Tribes below this. And there are parents, Arthropods and Insects above.  I was first just trying to work on one family of Bee Flies, Bombyllidae, whose format I thus changed.  But then I found when I checked one of the subkeywords, the same one in the older one would also check.  Likewise with uncheck.  I spent hours, trying to view all the subkeywords and starting over, unchecking all the levels but they still came duplicated.  that when I then went to the parent keyword Diptera and did the same thing, but it made no difference except creating another duplicate of Bee flies in the new format.  Then I selected all the images in the family folder and selected every level and uncheck all the way up to Arthropods. I deleted all levels of duplicate keywords. I I closed and reopened the Bridge and tried again, all to no avail. I had also tried deleting both instances of a subkeyword in the old and new keywords and then putting back in the new and checking.  The old came back with all the levels.. I changed the preference to not apply parents, but the subkeyword still duplicates.

[gregreser]

So you got the metadata into Excel and editied it?

Then you tried to save it as a tab-delimited .txt file. Is this where the error appeared or were trying to import the metadata back in using the script?

To be honest, I don't use OneDrive and I'm not sure it would work with my script. You might have to save the .txt file locally, like your desktop.

[robirdman1]

That was the problem.  I don't use onedrive either and didn't notice that Microsoft automatically switched to that rather than where I had originally put in my documents folder.  So it saved and I wasn't sure if I needed to make the same selection of spiders again or not before importing.  I did and got a spincircle for awhile as it looked throught whole folder.  seemed it was going to work but then I got.  

[gregreser]

The problem is all of the "Column1 Column2, extc. at the top. Try the attached version.

It worked for me on one test image.

It seems like "Aranae, SPIDERS Aranae" might need to be cleaned up in the spreadsheet, but the import worked.

Also, after import, check your NEF files to make sure they worked.

I think we still have some work to do to set up a clean hierchical keyword list for you, but if we can get the spreadsheet process working for cleaning up the embedded keywords, we will be in better shape.

[robirdman1]

As I looked at the attachment, as a txt file, it was hard to make sense as now data of a record is on 2 lines, an thus not so neatly ordered.  so I went to the original and see the problem about the columns heading.  In MSWorks this would be easy to remove the record, but in Excel I am looking all over and can't see how.  

When I do get to the stage of trying to import, should I first reselect the chosen thumbnails or will the program do that automatically. In this case I already moved all those to a new folder for easy reference so it won't matter. 

Also, after the initial export to Excel, can I delete all the fields that are empy or don't want changed to reduce the clutter.  Thanks.

[robirdman1]

I went back to the homoptera file.  I reopened to make sure I had the right version and got the offer to decide which columss to include so I skipped a lot of empty ones.  this file was imported into Excel differently as I had pasted the text file from notepad into a blank spreadsheet and so it has no first column with 'columns'. I saw that it had the desired changes and went to save and saw the problem here too it was automatically saving to onedrive also, so changed to document folder.  I then tried import and it said 55 successfuly imported. There were about 350 originally so I wondered where the rest were and then I realized I had since created a subfolder for Hemiptera and thought I had put all in there but there were a few Membracids missed but I didn't see any change.  I moved them into the Hemiptera folder and tried import again, but I was having a spincircle so long after it again said only 55 updated.  I tried closing Bridge and restarting with control, tab, shift to get the option to redo thumbnails, but it wasn't coming up.  I tired Bridge 2022, 2023 and Beta but they all just kept giving me a long time spincircle, so I finally rebooted.. I tried the import again, but it only said 55 updated, so maybe I have to do all over again because of the move.

I had also moved all the Arachnids into a subfolder.  I did such moves as the Sort folder had over 8000 files in and often took long times to do anything. I had saved the spider-greg folder in my downloads, and so now tried to use it with the Arachnid folder.  But though it clearly shows in the download folder, when I go to the import menu, it doesn't appear.

So then I tried the 

[gregreser]

Yes, after the initial export to Excel, you delete all the fields (spreadsheet columns) that are empy or don't want changed. The script can use just the columns you need. You must always have the "File Name" and "File Folder" columns.

The "File Folder" column is important. it must have the correct folder path. The script does work on subfolders as long as that option is selected and the folder paths are correct.

Can you send the import .txt file that only worked on 55 files?

[robirdman1]

I made a lot of progress, getting mor familiar with the automaticity resulting in bad results.  So I successfully did a selection of Orthoptera in the Sort folder and when I checked the results, I saw that in the filter column, some showed (0) which had beeen selected before.  this was because they had been corrected and no longer in a filter such as ().  but there were still a bunch unchanged so I reselected and ran again.  A few more may have changed but there were a lot that hadn't.  So I selected only those needing correction and exported.  there were 64 and it said 4 were successfully exports only.  So I thought, and then realized that it was the NEF raw files that weren't being exported.  I dcided to to a trial on 5 Mantids and in the view menu, selected sho hidden files, to get the xmp files.  But they didn't show.  I had to show all and then pick the xmps adjacent to the Mantids. So I exported the 10 files, but only 5 xmps exported.  I altered a comma with no space following to comma with space and imported back.  And they were all corrected.  I had also changed the default in Excel to stop saving to One Drive and to my documents and also to save as tab delineated instead of Excel workbood.  So now I am able to avoid the pitfalls and correct and save files as desired. GREAT!

[robirdman1]

So it's been working well. I did Arachnids and Beetles in 2 folders with good results.  One bad mistake is when I did a batch to replace , with , space.  It was going to produce in some double spaces, so then I replaced all double spaces with single space.  Bad because I didn't select the keyword column and so it did it for ones in the description and file name.  So I had to go all over each of the hundreds in those 2 columns and put back the double spaces which otherwise would have screwed up everything.  The reason those have double spaces is because it is a scientific name followed by a COMMON NAME.  When I import into MSWorks I convert all the double spaces to tab to get in a new field.  So that was tedious.  then I did the unnamed Flies in the new sort folder.  That was more complex and when I imported back, I found lots of unmodified ones.  I checked the table and they weren't corrected there.  Looked up each and corrected and reimported and still more uncorrected.  Again, I had missed in the table and looked each up and corrected and on the 3rd import everything was OK.  The first time though there was also a problem, in that when I started the import, I left the room and when I came back Bridge had closed.  I reopened and had a spinning circle and tried restarting several times with the same results.  Then I went out for hours and came back and still stuck there.  I was finally able to restart with resetting thumbnails and then I could see the results of the 1st import.

   so it's working when but requires great attention to find the many places where corrections are needed.

I am wondering if I am missing something in the way I do the export into Excel, as it doesn't give me the option to leave out fields and it comes with the columns at the top so I have to copy and paste all into another blank spreadsheet so I don't have that columns row.

I don't think that import that only had 55 return would be any use.  that was before I realized no NEF raw files would export unless I had the hidden files viewed, so they were all missing.

[gregreser]

Glad to hear you're making progress. Excel can be a pain, but once you have a system worked out, you'll be able to do a lot cleanup very quickly.

For the unwanted columns names at the top:

- select the entire row by clicking on the far right (the row will be highlighted)

- right click > Delete

Or you can do this:

- select the entire row by clicking on the far right (the row will be highlighted)

- right click > Delete

The same works for deleting columns you don't need:

- select the column(s)

- right click > Delete

If you export a lot of files to Excel and only want to work on some, you can sort or filter just those rows and copy and paste them in a new spreadsheet for clean up.

The important thing about sorting is to sort the entire sheet so that all rows move together. Otherwise, you will mix up the cells vertically which is not good, Always keep a copy of your spreadsheet before making a big change. If you mess up, undo before saving or closing.

The safest way to sort:

- slect the entire sheet

- select Sort and Filter > Custom Sort

- select My Data has Headers

- select the column to sort by

Filtering is useful to only show rows contating a word within a cell.

See these instructions:

https://support.microsoft.com/en-us/office/filter-data-in-a-range-or-table-01832226-31b5-4568-8806-3...

[gregreser]

@robirdman1 it is possible to modify my Bridge export-import script to only export certain fields, such as Keywords.

Let me know if you want to do this and I will explain it.

[robirdman1]

So. since the SORT folder was so large (~8000) files.  I had made subfolders there were 2 parents, one for ones who had IDs and name had been added to preceed their code number and the other not yet definitely ID'd.  Each of these parent folders then had subfolders for Orders.   Now I had gone through various unnamed files and rated 5 stars.  I just do this to mark them temporarily and then select all with 5 stars to make tiffs as the next stage.  I had about 46 from a search in the master SORT folder, which is including subfolders and I opened in groups of 12 to make tiffs.  Along the way, I had several that said data could not be read and will be ignored.  I was still able to make the tiffs. but when I then looked at the sort for 5 starts, about 4 didn't show the tiffs.  I went to the NEFs and find in Bridge and found tiffs next to them. I was looking at the metadata keywords to see if there was anything unusual and then I saw that those tiffs had no metadata.  I was trying to see what was different in the NEF and all the keywords looked all right and then I noticed the difference was that other files had a date followed by a time with AM or PM and these had no AM or PM.  Then I started looking in the subfolders and found that a whole lot have been affected thus, hundreds I assume in several of the subfolders.  I hadn't noticed so many are last backed up this way.  If I click on the date in the NEF to try to change it, it goes blank. 

Do you think I can get the exporter to get the dates to conform again to be recognized by Bridge?  I see that it is 24 hour time, on these. Also the seconds are gone,  Sorting by dates, I found one at the border between normal and changed dates and found I could paste the date in, but this would be extremely tedious.

[robirdman1]

Only the date metadata in the IPTC core has been affected by the change to military time.  the date under the thumbnail and the date in camera data (exif) are in the original Bridge form.  If I make a tiff of a RAW file it strips all the metadata, which must be entered manually.  I can copy and paste the date from the camera data (exif). with each individual changed file, but since there are hundreds, perhaps thousands so changed, can I use the exporter with Excel to change back?  I don't want to start moving files to their specific family folders or upload any to the website until corrected.

[gregreser]

I think we can restore the dates. First, I need to better understand which dates you are referring to.

The date that is wrong (is missing, has no seconds, or has no AM or PM) is the IPTC Core Date Created?

You are manually copying the camera data (exif) to IPTC Core Date Created?

Send a screen shot of the bad date field.

[robirdman1]

It's not looking good.  Meanwhile I posted to the Microsoft Community forum and suggestion to change the date format to have AM & PM.  I did a sample export/inport on 4 and it looked good, but when I went to make a tiff, I got the warning about unreadable data being ignored. I went back and saw that between date and time there is no comma.  Clicking in the box makes the date disappear as before.  Then I had the idea of going to exell and replacing a date such as 9/1/23 with 9/1/2023, (added comma) but it doesn't find anything to change, So in one picture, you can see the date created in the IPTC lacks the comma, while under the thumbnail and in the Camera data (exif) is still OK.  So I have pasted a few hundred from the latter to the former to make tiffs with metadata, but it is very time-consuming and maybe thousands were changed before I noticed the problem.  The other picture shows what I changed to in Excel that still didn't work.  It may be that the seconds are gone, so I don't get the choice that the MS forum person suggested. I wonder i

[gregreser]

Were these the dates you saw when you imported your metadata into Excel or did you enter new dates?

When I open exported metadata in Excel, Date created is in this form:

This is the form it should be. XMP requires this. Bridge will display the date in a prettier form, but the actual data needs to be "YYYY:MM:DD HH:MM:SS" with time in 24-hour format. 

If you change the Date Created in the spreadsheet, you have to use the YYYY:MM:DD HH:MM:SS form.

Do you still have your original export .txt file? If so, you get those original dates and put them in an import spreadsheet, making sure they are formatted correctly. How do you do that? When you open the .txt file in Excel, format the date column as text, not a date. Excel wants to reformat to it's own date. I usually do not see this happen, so it might that you are opening the Excel data differently.

It would also be possible to use a script or Exiftool to copy the Exif Date Time Original to the IPTC Date Created, saving you a lot of manual work.

Another option for future Excel imports: delete the entire Date Created column from the spreadsheet. This way no date will be imported back in, eliminating the date format problem.

[robirdman1]

I have a large folder which is for reject files.  I keep till I extract their numbers so I can also delete from my ACCESS database.  So I'm using it as a test.  I rated a few with 'reject', while everything else is one star.  I select the 10 with reject and export and import into Excel.  One thing I noticed is that it is back to tab delimiter, as it was on all the other imports - except when I did the last test to change dates and it showed comma delineator, which I changed back to tab.  So here is a picture of how the date format imports. 

[gregreser]

How are you creating tiff from RAW? You may have selected to not include metadata.

[robirdman1]

I just click to open, then after getting past camera raw I get this box.

It is unfortunate, because I'd gone through all the subfolders of the SORT folder and corrected anything that was still out of the desired keyword formats and everything in all the folders has the neat ordering without unwanted duplications. I though I was ready to begin looking at how the keyword list would be adjust to fit.  Many keywords are now orphans without parent, so I was wondering about that part the operation.  Fortunately I didn't do anything out of the SORT folder, so the tens of thousands in the phylogentic folders and unaffected and I definitely need to be able to exclude fields before doing any more.  But I have to rectify this before moving newly ID'd and named subjects to such folders and to the website. It seems that unless some macro could be created that would do what I do manually, copying and pasting date and then going to the next folder, this will take hours and days to rectify. 

[gregreser]

Is it just the date that was not copied to the tiffs? Are the tiff keywords okay?

[robirdman1]

No, in the tiffs all metadata is gone, so I have to correct the date to avoid losing all the info.

[robirdman1]

So the seconds are already gone on import to Excel.  My import procedure is to create new file in Excel.  Then click blank worksheet, click on data and select data, then data from text/cvsv.  I then choose the 65001 and except with the changed date one, tab is selected and I import. there was no choice to exclude fields and everything comes in.

[gregreser]

Do you mean that if you correct the RAW date, then the tiff is copied with the metadata? That's one task - make the RAW import work so tiffs can be made correctly in the future. The second task is to fix the tiffs that curretnly have no metadata. Right?

[gregreser]

One more task: fix the dates in the RAW files?

[robirdman1]

I had initially selected about 40 RAW files to make tiffs and on several I got that message but didn't know what it meant, till I went to the next stage of processing and saw that some weren't still selected and I found they had no rating or data.  I pasted all the fields from the raw file into the metadata boxes and then when I found the date format was the problem and I could paste to correct, I only made subsequent tiffs after correcting the dates.  So there are no tiffs lacking metadata.  Also many of the processed files were already tiffs or jpegs and the import export process did not lose other metadata fields, just changed date format.  Initially I didn't realize why some files remained unchanged until I realized that it was the RAW files and they wouldn't be processed unless the hidden xmp files were shown and included in the export.

As I processed everything, I'm not sure why a lot in the beetles subfolder retain correct dates.  There must have been something different about the import. In fact it looks like none in this subfolder have the wrong date format.  I rember that it was one of the biggest imports with something like 1500 or 2000.  In contrast the subfolder with named beetles has few correctly formatted dates. Those were probably moved from the unnamed folder to the named after the import that was OK. 

So I'm looking at saved Excel files and I see 3 versions of Beetles sort with dates within a few minutes of each other.  I think I may have opened one, had the extra columns row, and then tried a new one and then pasted the data into a previous spreadsheet that didn't have the extra columns.  I can see that reopening the latest one, I do get more options, about choice of fields to include and I can see that the date format here looks like nothing will be lost.

[robirdman1]

In contrast, reopening the Named beetle folder I can see that the dates are already gone.