c=(a×c)+b ⇒∣c∣2=∣a+c∣2+∣b∣2+2(a×c)⋅b ⇒2[abc]=∣a×c∣2+1−∣c∣2
Let angle between vectors a and c is ′θ′
So, [abc]=21[∣c∣2sin2θ−∣c∣2+1] =21[1−∣c∣2cos2θ] ...(i)
Now, c⋅(a⋅c)=(a×c)2+b⋅(a×c) ⇒[abc]=∣c∣2sin2θ ...(ii)
From Eq. (i) and (ii) 21{1−∣c∣2cos2θ}=∣c∣2sin2θ ⇒∣c∣2=cos2θ+2sin2θ1 ∵[abc]=∣c∣2sin2θ=cos2θ+2sin2θsin2θ=1+sin2θsin2θ =cosec2θ+11
For maximum value of [abc],cosec2θ=1,
so [abc]max=21