Understanding Tolerance Calculation in Adobe Illustrator's Magic Wand Tool

[Ghulam Mohyuddin]

I understand that to select objects based on their fill color using Adobe Illustrator's Magic Wand tool, I can choose the Fill Color option and set a Tolerance value between 0 and 255  for RGB or 0 and 100  for CMYK. Lower tolerance values select objects very similar to the clicked object, while higher tolerance values select objects with a broader range of the selected property.

However, I would like to know the exact process Illustrator uses to calculate and apply tolerance for selecting objects based on fill color in both RGB and CMYK modes. For example, if I have an object with RGB values of (60, 90, 120) and set the tolerance to 20, what range of colors will be selected for each channel? Additionally, could you explain the process for a CMYK object as well? Thank you.

[Monika Gause]

I assume the fastest procedure would be that you find out what works for your artwork by experiment.

[Kurt Gold]

As for CMYK process colours the rule is what one may call a "common responsibility".

That is, to be considered and selected as objects with the "same" colour, none of the four inks (relative to the reference colour) are allowed to exceed or fall below the percentaged tolerance setting in the Magic Wand palette. Otherwise the Magic Wand won't select them.

Probably that's valid for RGB colours as well, but the calculation is obviously a bit more complicated.

[Ghulam Mohyuddin]

I understand your point, but I would like to know the precise method by which the Magic Wand tolerance is calculated.

[Doug A Roberts]

See if this thread helps:

https://www.photoshopgurus.com/forum/threads/tolerance.52555/post-1533711906

[Ghulam Mohyuddin]

"I have reviewed your reference material related to raster images. I am now seeking clarification on how the tolerance setting operates within Adobe Illustrator. Could you provide a detailed explanation of its function in Adobe Illustrator?"

[Doug A Roberts]

I don't see why it would operate differently.

Edit: It does appear different, strangely.

If I create a 255/0/0 to 0/0/0 gradient in Photoshop and select with a tolerance of 10, I get a selection from 255 to 245.

If I create a 255/0/0 to 0/0/0 object blend in Illustrator and select with a tolerance of 10, I get a selection from 255 to 230:

[Ghulam Mohyuddin]

"I believe the information regarding how tolerance is calculated in Adobe Illustrator should be accessible to users, as it is crucial for effectively using the tool. Specifically, I am interested in understanding how tolerance is applied in the Magic Wand tool. For instance, if we set a tolerance level of 20 in Adobe Illustrator for a CMYK color with values C=70, M=47, Y=55, K=60, does the tolerance apply as a ±20 range for each channel (e.g., C=50-90, M=27-67, Y=35-75, K=40-80)? Or is it based on the overall color difference between the sample color and the object you want to select? Your clarification on this would be greatly appreciated."

[Monika Gause]

---

Ghulam Mohyuddin  schrieb:

"I have reviewed your reference material related to raster images. I am now seeking clarification on how the tolerance setting operates within Adobe Illustrator. Could you provide a detailed explanation of its function in Adobe Illustrator?"

---

In this forum 95% of participants are users just like you. But even if they were engineers it's not very likely that they would give you that information. 

[Ghulam Mohyuddin]

"I believe the information regarding how tolerance is calculated in Adobe Illustrator should be accessible to users, as it is crucial for effectively using the tool. Specifically, I am interested in understanding how tolerance is applied in the Magic Wand tool. For instance, if we set a tolerance level of 20 in Adobe Illustrator for a CMYK color with values C=70, M=47, Y=55, K=60, does the tolerance apply as a ±20 range for each channel (e.g., C=50-90, M=27-67, Y=35-75, K=40-80)? Or is it based on the overall color difference between the sample color and the object you want to select? Your clarification on this would be greatly appreciated."

[Kurt Gold]

For instance, if we set a tolerance level of 20 in Adobe Illustrator for a CMYK color with values C=70, M=47, Y=55, K=60, does the tolerance apply as a ±20 range for each channel (e.g., C=50-90, M=27-67, Y=35-75, K=40-80)?

--------------------------------------------------------------------------------------------------------------------------------

Yes, exactly. That's the basic rule about CMYK colours I was trying to explain in my rather generalised statement above.

It gets a bit different when there are 0 % inks. Basically, the 0 % inks do not count, unless you change at least one of the non-zero % inks. Then the zero % inks kind of may get some bonus points.

A long time ago I created a comparative table that should explain the issue in detail. Will have to look at one of my archives and post it in case I can locate it.

[Ghulam Mohyuddin]

"Could you please share the comparison table? I am looking forward to reviewing it."

[Ghulam Mohyuddin]

Despite applying a tolerance of ±20 for each color channel, the expected selection did not occur. For example, consider a base object with a specific color and a set tolerance range of ±20. Another object with colors within this range should logically be selected, but it remains unselected. This discrepancy suggests that additional factors or constraints are influencing the selection process.Look at this example

[Ghulam Mohyuddin]

Despite applying a tolerance of ±20 for each color channel, the expected selection did not occur. For example, consider a base object with a specific color and a set tolerance range of ±20. Another object with colors within this range should logically be selected, but it remains unselected. This discrepancy suggests that additional factors or constraints are influencing the selection process.Look at this example

[Ghulam Mohyuddin]

Despite applying a tolerance of ±20 for each color channel, the expected selection did not occur. For example, consider a base object with a specific color and a set tolerance range of ±20. Another object with colors within this range should logically be selected, but it remains unselected. This discrepancy suggests that additional factors or constraints are influencing the selection process.Look at this example

[Ghulam Mohyuddin]

I am seeking a detailed understanding of how the tolerance settings work with the Magic Wand tool in Adobe Illustrator for both CMYK and RGB color modes.

CMYK Tolerance Calculation:

• Reference Color: CMYK (70, 0, 100, 0)
• Tolerance: 20%
• Could you please specify the range of color values for each channel (Cyan, Magenta, Yellow, Black) that would be selected with this tolerance setting?

  RGB Tolerance Calculation:

  • Reference Color: RGB (255, 81, 47)
  • Tolerance: 30
  • Could you provide the range of color values for each channel (Red, Green, Blue) that would be selected with this tolerance setting?

    Tolerance Application:

    • Does the Magic Wand tool apply tolerance settings to each color channel individually, or does it use a combined average of all channels to determine color similarity?

Your insights on these queries would greatly assist in understanding how the Magic Wand tool processes color selections with varying tolerance levels.

[Monika Gause]

Please continue in your other thread.

[Ghulam Mohyuddin]

I have previously asked several questions about how tolerance functions in Adobe Illustrator's Magic Wand Tool, but I have not received a satisfactory explanation. Could someone please provide a detailed explanation of how tolerance works? For instance, does a tolerance setting of 20 mean that it will select colors within a range of +20 and -20 shades from the specified color? Or does tolerance work differently? I am seeking a precise explanation of the behavior of tolerance in Adobe Illustrator. Thank you.

[Monika Gause]

You will not get other information than in your other threads: 

https://community.adobe.com/t5/illustrator-discussions/how-tolerance-works-in-magic-wand-tool-in-ado... 

and 

https://community.adobe.com/t5/illustrator-discussions/understanding-tolerance-calculation-in-adobe-...

So maybe you finally post an exact case you need a solution for.

[Ghulam Mohyuddin]

Despite applying a tolerance of ±20 for each color channel, the expected selection did not occur. For example, consider a base object with a specific color and a set tolerance range of ±20. Another object with colors within this range should logically be selected, but it remains unselected. This discrepancy suggests that additional factors or constraints are influencing the selection process.Look at this example

[Anshul_Saini]

Hi Ghulam Mohyuddin,

Thanks for reaching out. The Tolerance setting in the Magic Wand tool in Adobe Illustrator can indeed be a bit tricky. The tolerance setting controls the range of color variation that the Magic Wand tool will select. A tolerance of ±20 means that the tool will select colors that fall within a range of 20 units above or below the base color's RGB values.

If you’re finding that the tool isn’t selecting as expected despite setting the tolerance correctly, consider the following:

1. Ensure that the objects' color mode (RGB or CMYK) is consistent. If the color mode is inconsistent, the Magic Wand tool’s tolerance setting might not work as expected.

2. The Magic Wand tool doesn’t always account for opacity or specific effects applied to objects. Check if any transparency or effects might be affecting the selection.

3. Sometimes, color profiles or slight variations in color values can influence selection. Double-check that the colors are within the expected range.

4. Ensure that the objects are entirely within the selection bounds. The tool might not select them if they are partially outside or overlapping with other objects.

If you still encounter issues, please provide more details or a specific example, and I’d be happy to help further!

Best,

Anshul Saini

[Ghulam Mohyuddin]

"Hello Anshul_Saini,I have a base object with the following color values: C=70%, M=5%, Y=100%, K=10%. I also have another object with color values: C=90%, M=5%, Y=80%, K=10%. I set the tolerance to 20, and according to the logic you provided, the object should have been selected, but it was not. Could you please help me understand why this is happening?"

[Larry G. Schneider]

Does it work if you change the tolerance level to 21 instead of 20?

[Ghulam Mohyuddin]

No it does not work if I change the tolerance level to 21 instead of 20.

[Monika Gause]

20 in RGB does not necessarily transfer to 20 in CMYK.