Suppose x−7 occurs in (r+1)th term.
we have Tr+1=nCrxn−rar in (x+a)n.
In the given question, n = 1, x = ax, a = bx2−1 ∴Tr+1=11Cr(ax)11−r(bx2−1)r =11Cra11−rb−rx11−3r(−1)r
This term contains x−7 if 11−3r=−7 ⇒r=6
Therefore, coefficient of x−7 is 11C6(a)5(b−1)6=b6462a5