Arrow keys move toolbar instead of moving document's page

In Acrobat Pro, when I have a page zoomed in I use arrow keys (left, right, up, down) to scroll the page or to go to the next/prev page. But sometimes when I switch from another window arrow keys move toolbar (the one on the left side of the window). So I have to click on the document’s page to set focus on it because it seems that focus gets set to the toolbar for some reason. How do I fix this?