Is there a setting to temporarily select a tool by holding down it's key while you use it?

Ctrl calls the last used selection tool (Selection or Direct Selection);

Space (Alt if you are working with text) — Hand tool;

Ctrl+Space — Zoom tool.

BTW In Photoshop this also works 😉

Yes, if i understand correctly.

i.e.: I often use the 'Direct Selection Tool' and the 'Pen Tool'.

Press A for 'Direct Selection Tool', then press P for the 'Pen Tool'.

When you hold down the Ctrl key, you'll get the 'Direct Selection Tool' and when you release Ctrl you'll have the 'Pen Tool' again.

It depends on the Tool. As you discovered, some tools, like the Zoom tool, have that ability. Some others are the Spacebar to temporarily select the Hand tool, Control (Win) or Command (Mac) to temporarily select the Selection tool (whichever one was used last - Selection or Direct Selection)