It would be absolutely possible to do this one with a lookbehind, but that's not how I started figuring this one out. Can you go into more detail on what you're trying to do? It sounds like you simply want to replace all double chevron quotes with double quote marks. I'm not sure why you'd want to do it in multiple passes; it looks to me like it's best done in one pass.
The "followed by whatever except" bit is easy enough to do in GREP. If there are multiple nested quotes in a single paragraph, we'd need to have an expression that wasn't greedy; we want it to stop at the first double chevron close quote, right? The expression for that is
.+?
So my "Find" expression would be
(«)(.+?)(«)(.+?)(»)(.+?)(»)
and my "Change to" expression would be
"$2"$4"$6"
because enclosing an expression in parentheses allows you to handle each parenthetical expression separately in the "Change to" field by addressing each group with a dollar sign. So this one replaces all the double chevron quotes with typographer's quotes. I would assume that you'd want the nested quotes to be single quotes, not double quotes, so I'd suggest this "Change to" expression instead:
"$2'$4'$6"
which leads to this behavior:

If that's not what you're looking for, please do post again and fill us in. If you absolutely had to do one pass to replace the inner quotes, followed by another pass to replace the outer quotes, the first "Change to" expression would be more like
$1$2"$4"$6$7
(Also I'm sure that my method is a bit of a brute force method, and there are far more elegant ways to pull it off.)