(1+x)m(1−x)n=[1+mx+2m(m−1)x2+...] [1−nx+2n(n−1)x2+...] =1+(m−n)x+[2m(m−1)+2n(n−1)−mn]x2+...
term containing power of x≥3.
Now, m−n=3....(i) [∵ coefficient of x=3, given]
and 21m(m−1)x+21n(n−1)−mn=−6 ⇒m(m−1)+n(n−1)−2mn=−12 ⇒m2−m+n2−n−2mn=−12 ⇒(m−n)2−(m+n)=−12 ⇒m+n=9+12=21.....(ii)
On solving Eqs. (i) and (ii), we get m=12