What am I doing wrong here?

I've created all my lines using a drawing tablet and brush tool. All the lines have been closed. I select all > expand appearance and then select all > merge. This is where it goes south. I created a background color and then I send it to back:

Next I select all to isolate:

It's solated:

Now I select the border so that I can delete everything outside the lines:

Now this is what I get:

I've checked all points to see if there are any openings and they are all closed. What am I doing wrong here...