Unwanted sharp edges with Magic Wand?

[Guest]

Hi guys,

I feel like a newbie even though ive been using Photoshop for about 15 years. But as of yesterday, when i use the magic wand tool on a selection and then fill it in, im not getting anti-aliasing! I think I accidentally hit a shortcut key that i don't know about. I'm used to Photoshop on a PC, but im now using mac and yesterday i was fumbling around for the new layer shortcut, so it probably happened then. The Anti-aliasing box is ticked on everything, tolerance at 0 (and 32 for the wand). Bicubic/smooth for gradients is on (sometimes i switch it to hard edges for Pixel art). Filling a selection through the marquee tool gives me smooth edges, but if i then select that same area with the magic wand it fills as a jagged pixelated edge. Im not using paths or anything like that, just basic selection and fill. What the heck have i done??

Here is where i first noticed it. Original selection made with the circular marquee on the left, nice smooth fill. Selected with the Magic wand and when i fill it again, it is suddenly jagged and hard.

[Bettina Di Virgilio]

Hi, I think the problem is that you set your Tolerance to 0. Increase the Tolerance and the Anti-Alias will be back.

[rcfs]

God bless you.  I've had this issue for awhile 

[Guest]

Thanks but i took this screengrab before I realised the tolerance for the Magic Wand was supposed to be 32 by default. When i changed it back to the 32 i still get the same problem. I tried turning the tolerance up on the fill tool too, but it also made no difference. I dont know what I could have pressed to change this.

[Bettina Di Virgilio]

Hello, try to reset the tool from the drop down menu of the tool preset and see if it get fixed.

One more idea: to check if the problem is the magic wand use the Refine edge tool to check the borders, I'm not sure if the problem can be the way you fill the area: p.e. you're using Dissolve bending mode.

[Guest]

Hmm, that didnt seem to work, and blending modes are all fine. But now i'm beginning to think its something to do with the block of colour i am trying to select, or maybe the file, or something to do with the resolution? I just tried using the wand in another picture that i drew as a doodle a few years ago (a lot smaller, and lower in resolution): Selected a single block of colour, filled that selection in on a new layer and sure enough it is completely smooth. I don't know what I could have done different. But every selection with the wand, on any layer, on this one particular file is giving me the hard edges.

I feel like this is something really simple and obvious and im just not seeing it!

[Terri_Stevens]

Try looking at your image at higher magnification and look particularly for the checkerboard pattern that jpeg compression causes. I can see that it is now a PSD , but if it started out as a jpeg the pattern will still be there and the edge might then get selected with a stepping pattern even if anti-alias is on. If you have a plugin like Topaz Dejpeg  which removes compression artifacts this might help verify if this is the problem or not.

[Bettina Di Virgilio]

Check the color space and send a screen of the layers panel

[Chuck Uebele]

The magic wand tool is a very crude selector. It doesn't take much to get harsh jagged lines.

[Guest]

That may be the case, but as you can see, its not a big complex selection i am trying to fill that has lots of colour gradients etc. It worked fine up until about an hour before i took this screenshot. I dont know what has changed. I tried resetting all the tools, but that didnt work

[Terri_Stevens]

It might be quicker here just to jump in and delete your preference returning the system back to how it was at install time.

Hold down cmd-option-shift and with the keys depressed start Photoshop. In the dialog that appears opt to delete the preferences.

[Guest]

Bettina Di Virgilio​

I reset the preferences as you suggested, but still seem to have the same problem. I am a very basic user in Photoshop. I rarely mess around with colour profiles, I rarely use paths and masks etc. Im often happy to use my tablet to draw/illustrate, and then use layers to colour and shade. I created this pic as a new psd file at 300 dpi in the default RGB. Ive screen-grabbed the Layers panel, but as you can see its, just a super basic (and messy) bunch of layers. The blending mode is all just 'Normal'.

Im starting to think maybe its just me. Maybe the Wand tool has always done this, and i never noticed until i tried to do something else with it. But probably not, because its so instinctive for me to try and fill the selection in the way I did, i must have done it this way all along...

[Bettina Di Virgilio]

Hi, so making a resume of what you do, you select with magic wand the layer of the hair, then you create a new layer an fill it with the ligher grey and the borders are crispy. You fill it with the paint bucket tool? In the screen shot you sent I don't see the layer with light grey that you have in your first screen shot. You tried to do the same in a different file and everything is ok? Try to see the selection with the refine edge before filling: once you make your selection, choose the button refine edge and choose black and white in the drop down menu view, check if the borders are smooth or crisp. Let me know.

[Guest]

Thanks Bettina. Thats all correct. Select smooth edge hair > fill the selection on the separate layer = Crispy edges. I tried it on two other older images and it worked fine, and i even tried dragging all the layers from this project into a new canvas with the same settings to see if it would for some reason change it. It appears to be something with the layer/s, as refine edges under the black and white also shows it as sharp edges. And I cant even smooth it out in refine edges, because just turning the smoothness up by 1 pixel just makes it look completely blurred.

[Bettina Di Virgilio]

Try to check if the new layer has as a bending mode "Dissolve".

As I see that the hair you're selecting is aleady on a layer with trasparence, to load the selection keep pressed Cmd/Ctrl and clic on the thumbnail of that layer: this way you will load the same selection and you can fill it in a new layer.

[Guest]

Well that worked! Thanks so much! So what does the Load Selection do exactly? I dont know if it will be a long-term solution, but it means that I can at least continue with this project.

[Bettina Di Virgilio · Accepted solution]

Pressing Ctrl/Cmd on the thumbnail of a layer you load a selection that is equal to the visible content of that layer. I think it was what you wanted to obtain with the Magic Wand, but using the technique I explained to you you will have a more precise final selection.