Yes, I tried both round and square with the rest of the settings the same:


Round is the top one. These are screen shots of the results. I used 150 dpi, 70 frequency, 45 angle. The original circle was 30% gray. The size difference is only because I zoomed in differently before taking the screen shot.
I have Photoshop CC updated to the latest version.
The problem is your choices of resolution and line frequency. The size of a cell that can hold one dot, either a black one or a white one next to it, is the output resolution divided by the line frequency. In your case, 150 ppi / 70 lpi = 2.14, which get rounded down to 2. So, each dot is 4 × 4 pixels. Four of these arranged in a 2 × 2 group make one halftone cell. There are 5 black pixels in each 16 pixel halftone cell; 5 / 16 = 0.3125, which is very close to 30%.
Since your dots are only 2 × 2 pixels, they can't help but look square. You need to either raise your output resolution, or lower your screen frequency, or both.