Shaders How to detect an odd normal value pattern

[Misu]

Hi there, I am working on an experiment in 3D. I am trying to find a way to compare between two pixels to see the order on its colors and check if the order is oddly backwards (inverted) from the first to the other.

This may sound hard to comprehend but I'll explain...

I have a normal mapped deffered render passed into my shader
And for each pixel, Im iterating atleast through the pixels around the current one to follow the order it has on the image. If the order is unnusual to common normal map color format (or inverted), it fills it up with red.

Is there a way or algorithm to be able to accomplish this?

 

[Xor]

So if I'm understanding you right, you need to compare two normals and see if they are facing opposite directions along the x axis.

And just to be sure, your normal map should look more like this (image from a google search):

These normals are calculated with the view matrix and then converted from the range -1 to 1 to the range 0 to 1.

Then in a shader you can check two normals like this:

vec2 Pixel = vec2(.5/640.,0);//Change the 640 to the width of the normal map
vec3 Normal1 = texture2D(NormalMap,v_vTexcoord-Pixel).xyz*2.-1.;
vec3 Normal2 = texture2D(NormalMap,v_vTexcoord+Pixel).xyz*2.-1.;

Now you can compare the directions of the two normals and calculate the red value:

float Red = .5+.5*sign(Normal1.x)*(Normal1.x*Normal2.x);

The code above should make a gradient effect. If this doesn't help, I'll need more details as I don't know how you want the red value set (could be gradient or binary) or how you're calculated normals.
Also none of the code was tested so it may have issues. I hope this helps!
-Xor

 

[Misu]

@Xor Sorry for the late, reply, was updating GMS2.

Anyway it reached the point that I have to explain what I am really doing since this has led to many confusion already and really hate telling what I work on...

Well Im doing ss ambient occlusion... I have manage to get something working so far with a normal map and depth map (which I havent used it yet because of being stuck on this part).

This is the progress Ive gotten so far...

This is a normal map I have...

And what I cant seem to get though (I did tried depth map at the point I got stuck but didnt really resolved it) is how to not apply what is curved out and only apply to what is curved in.

curved in shapes tend to follow the inverted color shading of a normal. So I thought that will help me detect the curved-in shapes or deep corners. In reality, I dunno what to do now at this point and people have suggested me several stuff but I cant get them to work in the shader either. I am not sure if you have a brighter idea what I can do and I already read several articles on how to do so, none of them really helped me except saying I need to use normal map and a depth buffer map.

You think you can help me on this?

 

[hamdrax24]

@Misu If SSAO your trying to implement then you simply perform the dot product of the two normals.