How can i switch the shortcut to "Select Next Layer"

If you can't find a function in any of the Menus/sub-menus or right click menus, then AFAIK, we can't change its shortcut (if it has one).  You can't use an action to control such a function ether (AFAIK).

There is a second problem. It looks like the Keyboard Shortcuts dialog box don’t accept a single-key shortcut as shown in the picture below, when I tried using [ as a shortcut for a random command. So the single-key shortcuts don’t seem to be editable by the user.

A possible alternative would be to use a third-party shortcuts app to remap [ for Photoshop only, so that it executes Alt + [ , but I’m not sure how well that would work depending on which app gets to intercept the keystroke first.

I cannot find that shortcut anywhere in the keyboard shortcuts dialog, so it seems that it is one of the shortcuts that cannot be changed.