Generative fill option to only reference layers underneath

There should be an option to have generative fill reference only the image/layers underneath the generative fill layer, as opposed to the full stack. For example, if I have a file with many adjustment layers on top (making overall brightness/contrast/color adjustments) and I need to generate something below all of them, generative fill creates an image that blends as if it should be the top-most layer. This creates a huge hinderance in workflow since all adjustment layers on top need to be turned off before using generative fill.