It helps to understand how this all works.
When you install Photoshop, all the installation files go to the Program Files directory (on Windows; Mac OS works the same way). All of that is read-only and remains unchanged.
Now, the instant you open Photoshop and start using it, the configuration starts changing in all kinds of ways, some obvious, some hidden and under the hood. These changes are all recorded and stored in your system user account. That’s the Settings folder @AxelMatt points to, and this is rewritten on every application exit, to capture all changes.
Each major Photoshop version has its own Settings folder! That’s the crucial point that you need to be aware of. The reason for this is that new application code means the preferences (settings) may not mean the same thing. They may be inapplicable altogether, or they may cause problems because of minor differences. So you can’t just reuse them.
In addition, since they are constantly rewritten, preferences are very vulnerable to corruption. Small errors accumulate. Corrupt preferences can cause unpredictable behavior that usually looks like bugs, but that’s not what it is.
Custom actions and brushes aren’t easily re-creatable, so you should always save them out to .atn or .abr files, which you can reload in the new version.
All the rest is pretty easy to set up fresh. Considering that you do this once a year, ten minutes isn’t really unreasonable, and it’s a cheap insurance against problems and instability.