Pattern swatches using PS smart objects?

Is it possible to make a pattern swatch in Illustrator, using a linked file (smart object) from Photoshop in your design? When I try to do this, it doesn’t work…I suspect it’s similar to trying to make a new pattern with an already defined pattern. I really prefer making patterns in Illustrator (vs PS) but haven’t found a way to pull over any raster elements from PS without fully embedding them, which of course makes the Illustrator file more difficult to edit/change in the future.