Ungrouping issue

To make it easier to understand, I am referring  an object as a box.

I started with a box with strokes with some gaps. I added some interior marking in the box and change it to solid lines.  I grouped the solid lines.  Then I grouped the whole thing.  

When I go to edit the box.  I have to ungroup about 10 times if not more. I even have to ungroup quite a bit on the interior solid line too.Its very frustrating considering I should only have 2 groups.  What is going on.  Its taking forever to ungroup every box.