You're probably running out of video RAM (VRAM). What GPU do you have, and how much VRAM does it have? If it's a dedicated card, this will be onboard VRAM - if it's an integrated Intel GPU it will use from the system RAM.
Without any system details or details of the file sizes etc, no-one can help you with this. Please go to Photoshop's Help > System info and press copy. Then paste the info here alomg with teh typical file sizes