Let \(n\) be a positive integer. The coefficient of \(x^3y^5\) in the expansion of
\[(1+xy+y^2)^n\]
equals
\[(a) \quad n,\quad(b) \quad 2^n,\quad(c) \quad \begin{pmatrix}n\\3\end{pmatrix} \begin{pmatrix}n\\5\end{pmatrix},\quad(d) \quad 4\begin{pmatrix}n\\4\end{pmatrix},\quad(e) \quad\begin{pmatrix}n\\8\end{pmatrix}.\]
We can write the expression as
\[(1+y(x+y))^n = 1 + \begin{pmatrix}n\\1\end{pmatrix}y(x+y)+\begin{pmatrix}n\\2\end{pmatrix}y^2(x+y)^2+\begin{pmatrix}n\\3\end{pmatrix}y^3(x+y)^3+\begin{pmatrix}n\\4\end{pmatrix}y^4(x+y)^4+\begin{pmatrix}n\\5\end{pmatrix}y^5(x+y)^5+...\]
The powers in \(x^3y^5\) add to 8, so the only term that interests us is \(\begin{pmatrix}n\\4\end{pmatrix}y^4(x+y)^4.\)
We need the \(x^3y\) term in \((x+y)^4\), which has coefficient \(\begin{pmatrix}4\\3\end{pmatrix} = 4\). Thus the coefficient of \(x^3y^5\) overall is \(4\begin{pmatrix}n\\4\end{pmatrix}\), and the answer is (d).
