I have done it and, depending on the final viewing screen, it works quite well! You will probably find that it works best with 1920x1080 HD project settings. Start your timeline with an HD clip and go from there. Put your 4K clips where you want them.
One option is to use the 4K in an intermediate 4K project and take advantage of the extra resolution for cropping or panning. Output the "edited" clips as HD for use with your HD clips in and HD project.