To speed up removal of unwanted bits from the timeline, it should be possible to marquee around selected clips in order to select them and then delete rather than having to return to selection tool.
You can set In-Out points and use Lift or Extract, it's more precise than dragging by hand. If you want to delete whole clips just Delete or Ripple Delete them.
@ali Jaber is correct, in point, our point, lift / extract is the way. All of these have keyboard shortcuts. Keyboard shortcut for "add edit point" will let you avoid using the razor mouse tool altogether. A keyboard shortcut for "select clip under play head" will let you choose a clip you've cut and then lift/extract/delete key to get rid of it.
Point is: you will likely be faster and more accurate with keyboard shortcuts than you will be with a mouse.