How to make reflect or shear transform for a art?

There are probably thousands of pages on reflection & shear transformations on the web, but they'll require you to understand a fair bit about matrices, which -- from your questions -- I'm guessing you don't.

Matrices are useful, but they take a bit to wrap your head around. They kind of defy 'normal' math a little bit. For example, if you want to flip a polygon without moving it, you basically need to do this in Illustrator:

SetTranslation(matrix, [back to origin])

ConcatScale(matrix, -1, -1)  // that will flip it in both  X & Y, so you'll need to make one '1' if you want to only flip in one direction

ConcatTranslation(matrix, [back to where you started])

Then you apply that matrix you've built up. If you just apply a matrix that you've simply flipped the sign on the scale, I believe it will also reflect the polygon's position relative to the origin of the page, and you probably don't want that. Also, the translation is simple for things like points, but it will be trickier for polygons -- you have to pick your 'origin of reflection' within the polygon, ad then pretend you're translation that point back to the origin (which is usually easy, since it should just be (-x, -y) where x & y are the coordinates of your reflection origin).

Shearing is much more complicated. You'll have to do some searching for 2D shear matrices and then plug in the values manually. I don't know off-hand how you convert Illustrator's shearing options into the relevant matrix, but if you find a good explanation of shearing somewhere their options will probably make sense w.r.t. to the math behind it.