Unity3D Raycast with LayerMask - How layermask works? Working example of Raycasthit and RaycastHit2D and LayerMasks , both in 2D and 3D.

You can select which layers to cast raycast in Unity3D. You can exclude some layers or you can select only the ones you want.

Unity3D uses LayerMask as an 32 bit integer. This is why you also have only 32 avaible layer slots. Each of this layers are marked as bits of that 32 bit integer. So you need to do some bit

With these operators below, you can do all your LayerMask operations.
1) bit shift operator (<<)
2) bit inverse operator (~)
3) bitwise (binary) OR operator (|)

dont worry it is much easier then it sounds :)


Here is how LayerMask represented really:
0000 0000 0000 0000 0000 0000 0000 0000
first on the right one is the first Layer you see on your unity editor.
second one comes from the second on the right.,
 so goes on like this.

You have to create LayerMask in unity like this:
int  ExitButtonLayerMask = 1 << ExitButtonLayer;
ExitButtonLayer is an integer. You get the layer numbers from unity editor.
then use the Mask on your raycast, there you get your ray which only hits only the objects which reside on your chosen layers.


Here is an example class:

public class Raycast2D : MonoBehaviour {

int ExitButtonLayer;

int RateButtonLayer;

int ExitButtonLayerMask;

int RateButtonLayerMask;

int CombinedMask;

void Awake() {

//open your unity editor, and look at layers, click "Add layer" button if you need to see the layer numbers.

ExitButtonLayer = 8;

RateButtonLayer = 9;

//ok, lets add our layers. these layers will be affected when a ray hits them:

ExitButtonLayerMask = 1 << ExitButtonLayer;

RateButtonLayerMask = 1 << RateButtonLayer;



// This is the binary OR operator, this is how we combine different masks.

CombinedMask = ExitButtonLayerMask | RateButtonLayerMask;

//btw if you want to "omit" these layers above, then you must combine them first then exclude them

// like this: ~((1 << ExitButtonLayerMask) | (1 << ExitButtonLayerMask))

}



void Update() {

if (Input.GetMouseButtonDown (0)) {

//RaycastHit2D hit = Physics2D.Raycast(Camera.main.ScreenToWorldPoint(Input.mousePosition), Vector2.zero);

Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);

//here we use our combined layer mask, so taht unity will cast a ray on the objects which are on these layer masks. Other objects wont get affected.

RaycastHit2D hit = Physics2D.GetRayIntersection (ray, Mathf.Infinity, CombinedMask);

if (hit.collider != null) { 

Debug.Log ("object hit: " + hit.collider.gameObject.name + " , Target Position: " + hit.collider.gameObject.transform.position); 

} /*else {

Debug.Log("missed");

}*/

}

}



}

examine the code above and ask me questions if you need.
also be sure to read http://forum.unity3d.com/threads/211708-Unity-2D-Raycast-from-mouse-to-screen , MelvMay MelvMay (Unity Developer) explained 2d ray hit in 3d space.


Related Unity3d Documentation Links:
1)  Physics2D.GetRayIntersection  : Cast a 3D ray against the colliders in the scene returning the first collider along the ray. (http://docs.unity3d.com/Documentation/ScriptReference/Physics2D.GetRayIntersection.html)
2) Physics2D.GetRayIntersectionAll  : Cast a 3D ray against the colliders in the scene returning all the colliders along the ray. (http://docs.unity3d.com/Documentation/ScriptReference/Physics2D.GetRayIntersectionAll.html)
3) Physics2D.GetRayIntersectionNonAlloc  : Cast a 3D ray against the colliders in the scene returning the colliders along the ray. (http://docs.unity3d.com/Documentation/ScriptReference/Physics2D.GetRayIntersectionNonAlloc.html)
4) Found a good reading on unity3d layermask issue with raycast ignore. try http://answers.unity3d.com/questions/8715/how-do-i-use-layermasks.html good article on different approaches, including bit values and operators.