The good folks over at creativepro might have what you're after. Two links in particular - an article by Mike Rankin on the topic; and a script that creates artistic style scalloped edges:
One way would be to make the frame in Illustrator and copy/paste into InDesign, but you aren't using a wavy stroke, exactly, so I'm not sure it would be faster than just making your series of circles as shown, combining them into a compound path and then using the Pathfinder panel to cut out the rectangular area by placing a rectangle on top.