Adjustment Layer over a clip that transitions to another - not smooth transition?

I am editing raw footage, and putting an adjustment layer over each clp.

I have a clip (with an adjustment layer over it) that does a crossfade to a new clip with it's own adjustment layer

The transition then is very hard, or abrupt when there is an adjustment layer over the clips - does that make sense?  

How can I avoid this?  Thank you!