816
Replies
What do you think and is it possible for this super res zoom to be made to the full sensor not a crop and produce a 48mp image even on a 12mp bayer sensor like iphone 14 base?
By @powerful_Elixir5E29
We are experimenting with this. The biggest obstacle is performance - to get 48MP out of 12MP is roughly equivalent to getting 12MP our of 3MP 4 times. In other words, it is expected to be 4x slower. Once we are happy with the performance and image quality for such a use-case, we will work on adding a higher resolution mode to the app. Stay tuned.
@BorisTheBlade I took a night shot with hdr scene, people in motions as well, and noticed the skyline has halos and overly weird skies coloring (some blotches). Can this be addressed? I will attach the raw, sooc jpeg, and post processed.
By @nhan_8084
Will look into it. Thanks for reporting.
@BorisTheBlade There was clouds yes, the post processed pic is what my eyes saw.
By @nhan_8084
OK, thanks. I tried using the two JPEGs you provided and cranking up the brightness in the sky to see what happens. To me they look the same... i.e., I am not sure I see any extra "blotchiness" in one vs the other. Are you able to help me here and point out what exactly are the blotchy artifacts you are seeing? Thank you.
It does! You just need to ensure that the appropriate permissions are set.
@BorisTheBlade What is your take on this a quad bayer 48mp sensor like on the 16 pro at 2x if you could get the 12mp crop not like right now when apple doesnt allow the full 48mp thus the 2x crop even on 16 pro is limited to 3mp do you think if apple allowed the full 48mp input that if we crop the 48mp quad bayer to 2x and use super res zoom on the 2x crop it would get the same detail as a 12mp bayer at 2x crop because of the lens resolution to resolve detail or the 48mp quad bayer will produce more detailed 2x zoom? Thanks!
By @powerful_Elixir5E29
(What follows is my 2c, not the team's official position on the matter) Typically the spatial resolution of a quad bayer sensor is lower than that of the bayer sensor, because each 2x2 group of the same color is under the same microlens, so technically they see the same part of the scene. This is why quad bayer sensor are really intended for lower resolution capture, but using the 4 pixels in a group to either recover better dynamic range (by exposing each differently), or by getting lower noise (by exposing each the same and averaging). Having said that, it is possible that there is extra information in each 2x2 group due to imperfections of the microlenses which could be exploited - I am not sure how likely that is, and if so, how one could use it for better super-resolution algorithms. In any case, there are 3 possible ways for the center 12MP of a 48MP sensor to be exposed:
- without any modifications (we get 12MP quad bayer)
- with "remosaicing" (convert single frame 12MP quad bayer to 12MP bayer raw)
- with stacking (use computational photography and several frames to recover the 12MP bayer raw details).
None of these are perfect. For 1, we would need to develop new algorithms that deal with quad bayer. For 2, there would be quality issues. And for 3, it would be slow and it would have Apple's computational photography baked into the bayer raw, which we wouldn't want.
If I understood the original question, Indigo's SuperRes zoom is already similar if not better than the native 2x sensor crop due to bad processing from Apple with heavy denoising and oversharpening right @BorisTheBlade ? SuperRes operates on upscaling as well? Also in theory Indigo team "could" achieve 48mp jpeg similar to the Topaz AI Upscaler algos but definitely not the raw 48mp right?
By @nhan_8084
True "raw" 48MP would require a 48MP raw sensor. Anything else is some form of "processed raw". Nothing wrong with that, as all computational photography raws fall into that category, except that if the processed raw includes resolution increase then there is always potential for more artifacts than if the resolution is maintained (generally speaking; details would depend on the details of the processing algorithm). Apple also does computational photography of some kind to deliver a 12MP cropped ProRaw at 2x zoom, and how they do that we do not know. After they get the "processed raw" however there is definitely some difference in how much post-processing is done vs what Indigo does. Apple might as well update their processing algorithms in the next iOS version and improve on the 2x zoom quality, or they may want for a new SoC which has the updated processing baked into the ISP. Only Apple knows what the plan there is.
Regarding super-resolution used in Indigo, it relies on the lens not being diffraction limited (which wide camera isn't), such that when there is natural handshake, different details from the scene are captured by different frames and can be reconstructed back. So it is not the Topaz-style upsampling, which uses a single-image + pre-trained model to increase image resolution, often by hallucinating the missing detail. Indigo super-resolution may create wrong detail in the image, but that is by mistake (which we are trying to fix), not because the model itself has hallucination baked in.
@BorisTheBlade Ah ok, so let say SR at 2x uses 9 frames, all 9 will be slightly different due to handshaking, merge and re-align? Isn't this the same principle as the non super-res 1x and 5x? Or is the 1x and 5x taking let say 9 frames, discarding any frames that's blurry, merge and re-align?
By @nhan_8084
This blog from Google from way back can explain a bit more how extra detail can be recovered in presence of handhake: https://research.google/blog/see-better-and-further-with-super-res-zoom-on-the-pixel-3/
Multi-frame super-resolution and align&merge use different approaches to handling individual frames and to merging them. Generally, we do avoid using blurry frames, as they have obvious detail losses. But the impact of discarding blurred frames is different for super-resolution than it is for aligning&merging.
@BorisTheBlade Yes, I am aware of RAISR, awesome tech from Marc and team! I know Indigo is still troubleshooting the overheat issues from SR but is it possible to have SR starting at 1.1x and 5.1x to behave similar to ISZ crops?
By @nhan_8084
I am not sure what ISZ crops means. In principle it is possible to run our SR on any capture, but there would need to me some changes made to the underlying ML model. Our priority is performance and stability, so expanding the zoom range where SR is active will be secondary for the time being.
So if i understood correctly that means super res zoom for a quad bayer 2x crop wont deliver more detail than super res zoom 2x crop of a 12mp bayer given the same sensor size on both?
By @powerful_Elixir5E29
The true answer is that I don't really know. We'd need to develop a new super-res algorithm which uses the quad bayer input and then see what we get. AFAIK, it may as well be dependent on how quad bayer is implemented wrt physical microlenses and other sensor detail.
AdChoices
