A jpg. It worked until recently as I often use it but it would appear to have stopped working after a recent Microsoft security update. I checked with family who also have the same version running on Windows 10 v1709 and they have the same issue. I don't really want to have to go uninstalling MS updates to see if I can work out which is causing it. I'm sure I've used it since I upgraded belatedly to v1709 at the beginning of April so its most likely going to be caused by a more recent update, such as KB4134661, KB4103727 or KB4093112. Up until recently I could control when the updates installed as was on wireless with "metered connection" enabled but my wireless died about 5 weeks ago and have been cabled until now so the updates are installing when they like (feature update can't because I'm using Win10Pro and can stop those updates for 12 months). It somewhat frustrating that only this type of feature in PSE11 has stopped working, the enhance menu features all work OK or I would have noticed it before now as I use them even more often.
Interestingly the flip and straighten menu options are not greyed out, just transform, recompose and free rotate layer under rotate.
The last Windows update does appear to have caused a number of conflicts most notably with RAM. Requiring a tweak of the registry but I have not seen this transform/rotate issue.
Try duplicating the layer using Ctrl+J
Does that make any difference when using Layer 1
Otherwise try trashing the prefs by holding down the three keys Ctrl+Alt+Shift while starting the Editor. Release the keys after a couple of seconds and if you see the settings dialog, click Yes.