If vector $\vec{\alpha}$ lie in the plane of $\vec{\beta}$ and $\vec{\gamma}$ then all three vectors $\vec{\alpha}, \vec{\beta}$ and $\vec{\gamma}$ are coplanar.
For three vectors to be coplanar, their scalar triple product must be zero
i.e. $\left[\vec{\alpha} \,\vec{\beta} \, \vec{\gamma} \right]=0$