The three vectors a,b and c are coplanar if and only if a⋅(b×c)=0.
Consider that the vectors a,b and c are coplanar. If b and c are parallel vectors, then b×c=0 and so a⋅(b×c)=0. If b and c are not parallel then, since a,b and c are coplanar, b×c is perpendicular to a
So, a⋅(b×c)=0.
Conversely, suppose that a .(b×c)=0. If a and b×c are both non-zero, then we conclude that a and b×c are perpendicular vectors. But b×c is perpendicular to both b and c. Therefore, a,b and c must lie in the plane, i.e., they are coplanar. If a=0, then a is coplanar with any two vectors, in particular with b and c. If (b×c)=0, then b and c are parallel vectors and so, a, b and c are coplanar since any two vectors always lie in a plane determined by them and a vector which is parallel to any one of it also lies in that plane.