If both AB and BA exists then number of columns of A = number of rows of B ⇒n+5=m ...(1)
and number of columns of B= number of row of A ⇒11−n=m ...(2)
Solving (1) and (2) we get n=3 and m=8
Hence, A has order 8×8 and B has order 8×8
Hence, both AB and BA are square matrix