mask drawing script sometimes not quite pixel perfect... is 3762 a problem number ?

Forum|Forum|4 years ago
October 25, 2021
3 replies
509 views

hi all

I have a simple script to draw rectangular masks on a layer, it works a treat, but this morning I noticed one mask that wasn't quite pixel perfect. All of the other masks are fine but one mask has 2 vertices with an X value of 3762 and when they are set it results in a vertex actually being placed at 3762.00024414062

The wierd thing is that if I try and fix it via the UI, the same thing happens ?

3762 set via UI results in 3762.0002

3761 works

3763 also wigs out to 3762.9998

3764 works

3765 works

what's going on, is this some kind of binary rounding error ?

visually, it's not really an issue but it throws a bit of spanner in the rest of the scripting oprations...

any suggestions greatly appreciated

// apply to a solid layer that is 6256 x 3600
// in a comp of the same dimensions

app.beginUndoGroup("add masks");

var C = app.project.activeItem;
var L = C.selectedLayers[0];

//A2  (4vertices) *** WORKS PERFECTLY
var myShape = new Shape()
myShape.vertices = [[1090,437],[1540,437],[1540,3297],[1090,3297]]
myShape.inTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.outTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.closed = true
var newMask = L.Masks.addProperty("mask");
newMask.maskShape.setValue(myShape);
newMask.name = "A2";
newMask.color = [0.16078431372549,0.43921568627451,0.70588235294118];
newMask.maskFeather.setValue([0,0]);
newMask.locked = false
newMask.maskMode  = MaskMode.ADD;

//B1  (4vertices) *** ALSO WORKS PERFECTLY
var myShape = new Shape()
myShape.vertices = [[1552,3297],[1552,1088],[1660,1088],[1660,3297]]
myShape.inTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.outTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.closed = true
var newMask = L.Masks.addProperty("mask");
newMask.maskShape.setValue(myShape);
newMask.name = "B1";
newMask.color = [0.1843137254902,0.71372549019608,0.16078431372549];
newMask.maskFeather.setValue([0,0]);
newMask.locked = false
newMask.maskMode  = MaskMode.ADD;

//B4  (4vertices) **** THIS ONE WIGS OUT on both vertices with an X value of 3762 ???
var myShape = new Shape()
myShape.vertices = [[3762,3297],[3762,2074],[4214,2074],[4214,3297]]
myShape.inTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.outTangents = [[0,0],[0,0],[0,0],[0,0]]
myShape.closed = true
var newMask = L.Masks.addProperty("mask");
newMask.maskShape.setValue(myShape);
newMask.name = "B4";
newMask.color = [0.70196078431373,0.71372549019608,0.16078431372549];
newMask.maskFeather.setValue([0,0]);
newMask.locked = false
newMask.maskMode  = MaskMode.ADD;


app.endUndoGroup();

Scripting

This topic has been closed for replies.

Mylenium

Legend

Math precision issues can come down to specific processors even, but typically are caused by different code and compiler behavior. Impossible to say much more than that, because apaprently we don't know which code Adobe are using under the hood. Quite possible that they forgot to update a variable to use a different data type and this isn't handled gracefully by the compiler in some situations, rippling down to the end user's build. Perhaps one of the devs might chime in, but for the time being it's likely best to report this on the Adobe Uservoice page as a bug.

Mylenium

Legend

Sounds like a display issue. Perhaps some internal buffer is off and the masks wrongly get "corrected" when actually being applied. As a start I would check with and without hardware acceleration as well as different resolutions, with the obvious point being to work at Full and 100% zoom. Unless there is another very exotic math issue dependent on a specific DLL or CPU type, this would be my closest guess, as the relation to power of two/ multiples of two math can usually best explained with GPU stuff. On that note it may also be worth trying with different source footage/ layer types. in such a scenario a wrongly decoded MP4 or such might also somehow have an effect I would imagine.

Mylenium

J

jeffbottomAuthor

Inspiring

hey Mylenium, thanks for the suggestions. As soon as I saw your reply I thought "of course it must be because I am using display scaling" (4K monitor 125% scaling). but no, dang and blast. I tried everything, all display settings at native res, all hardwear accleration on and off. 8,16 and 32 bit color spaces. multiple restarts. I also tried more standard comp resolutions i.e. 1080p. and 720p no joy.

Then I dug out my old Mac laptop and it does exactly the same thing there. so surely this means there is some kind of math precison bug... but how did it work fine on Dan's system ? I am so confused...

Dan Ebberts

Community Expert

When I use your code to create the masks and then use a text layer to report the path points created for B4, I get back exactly the vertices used to create the mask (including 3762).