As A is an orthogonal matrix, AAT=I ⇒31⎣⎡12a2122−2b⎦⎤.31⎣⎡12221−2a2b⎦⎤=⎣⎡100010001⎦⎤ ⇒91⎣⎡12a2122−2b⎦⎤⎣⎡12221−2a2b⎦⎤=⎣⎡100010001⎦⎤ ⇒⎣⎡90a+4+2b092a+2−2ba+4+2b2a+2−2ba2+4+b2⎦⎤=⎣⎡900090009⎦⎤ ⇒a+4+2b=0,2a+2−2b=0,a2+4+b2=9 ⇒a+2b+4=0,a−b+1=0a2+b2=5 ⇒a=−2,b=−1