The position vectors of the points \(A, B\) and \(C\) with respect to an origin \(O\) are the unit vectors \(\mathbf{a}, \mathbf{b}\) and \(\mathbf{c}\) respectively. It is given that \(OB\) is perpendicular to \(OC\), \(m = \cos \widehat{AOC}\), \(n = \cos \widehat{AOB}\), and \(m+n \neq 1\). The vector \(\mathbf{r}\), where \(\mathbf{r} = \mathbf{a} + \lambda \mathbf{b} + \mu \mathbf{c}\), is perpendicular to the plane \(ABC\). Show that

\[\lambda = \dfrac{(1-m)(1+m-n)}{1-m-n}\]

and find a similar expression for \(\mu\).

We know (where \(a\) is the magnitude of \(\mathbf{a}\)) that \(\mathbf{a}.\mathbf{b} = abn = n\), and similarly, \(\mathbf{a}.\mathbf{c} = acm=m.\)

Since \(\mathbf{r}\) is perpendicular to \(ABC\), we have that \(\mathbf{r}.(\mathbf{a}-\mathbf{b})=0\), and so \((\mathbf{a} + \lambda \mathbf{b} + \mu \mathbf{c}).(\mathbf{a}-\mathbf{b})=0,\) and since the distributive law holds for the scalar product, and since \(\mathbf{b}.\mathbf{c}=0\), \[\begin{align} 0 &= a^2 + \lambda \mathbf{a}.\mathbf{b} + \mu \mathbf{a}.\mathbf{c} -\mathbf{a}.\mathbf{b}-\lambda b^2 \notag \\ &= 1+\lambda n + \mu m -n-\lambda.\label{eq:1} \end{align}\] Similarly we know that \(\mathbf{r}.(\mathbf{a}-\mathbf{c}) = 0\), which gives us that \[\begin{equation} 0 = 1 + \lambda n + \mu m -m -\mu. \label{eq:2} \end{equation}\] Subtracting \(\eqref{eq:2}\) from \(\eqref{eq:1}\) gives \(n+\lambda=m+\mu\). Then substituting \(\mu\) in \(\eqref{eq:2}\) we get \[\begin{align*} 0 &= 1+\lambda n+(\lambda+n-m)m-m-(\lambda+n-m) \\ \implies\quad \lambda(n+m-1) &= m^2+n-mn-1 \\ \implies\quad \lambda &= \frac{(m-1)(m-n+1)}{n+m-1} = \frac{(1-m)(1+m-n)}{1-m-n} \end{align*}\] as required.

In dividing here, we have used the given fact that \(m+n \neq 1\).

Similarly, substituting \(\lambda\) gives us \[\begin{equation*} \mu = \frac{(1-n)(1+n-m)}{1-m-n}. \end{equation*}\]

Notice how the expressions for \(\lambda\) and \(\mu\) are symmetrical in \(m\) and \(n\).

It is further given that \(\widehat{AOB}=\widehat{AOC}=120^\circ\). The point \(N\) is the foot of the perpendicular from \(O\) on the plane \(ABC\). Find the position vector of \(N\), and deduce that the position vector of the reflection of \(O\) in the plane \(ABC\) is

\[\dfrac{4}{5}\mathbf{a}+ \dfrac{3}{5}\mathbf{b} + \dfrac{3}{5}\mathbf{c}.\]

We are given now that \(m = n = \cos 120^\circ = -\dfrac{1}{2}\), which means on substitution that \(\lambda = \mu = \dfrac{3}{4}\), giving \(\mathbf{r} = \mathbf{a} + \dfrac{3}{4}\mathbf{b}+\dfrac{3}{4}\mathbf{c}\).

Let \(\mathbf{n}\) be the position vector of \(N\) (note that \(\mathbf{n}\) is a vector, which distinguishes it from \(n\)).

Since \(\mathbf{r}\) is perpendicular to the plane, we know that \(\mathbf{n}\) is a multiple of \(\mathbf{r}\), so \(\mathbf{n} = \gamma \mathbf{a}+ \dfrac{3}{4}\gamma \mathbf{b} + \dfrac{3}{4}\gamma \mathbf{c}\) for some \(\gamma\).

But \(N\) is also on the plane \(ABC\), which means that \(\mathbf{n} = \mathbf{a} + \alpha(\mathbf{b}-\mathbf{a}) + \beta (\mathbf{c}-\mathbf{a})\), and so

\[\mathbf{n} = (1-\alpha-\beta)\mathbf{a}+\alpha \mathbf{b} + \beta \mathbf{c}.\]

Now \(\mathbf{a}\), \(\mathbf{b}\) and \(\mathbf{c}\) are independent vectors (we cannot write one as the sum of multiples of the other two) which means that we can compare coefficients here.

So the equation \((1-\alpha-\beta)\mathbf{a}+\alpha \mathbf{b} + \beta \mathbf{c} = \gamma \mathbf{a}+ \dfrac{3}{4}\gamma \mathbf{b} + \dfrac{3}{4}\gamma \mathbf{c}\) leads to \((1-\alpha-\beta)=\gamma\), \(\alpha = \dfrac{3}{4}\gamma\), \(\beta = \dfrac{3}{4}\gamma\).

This gives us that \(\gamma = \dfrac{2}{5}\), and so \(\alpha = \beta = \dfrac{3}{10}\).

Thus \(\mathbf{n} = \dfrac{2}{5}\mathbf{a}+ \dfrac{3}{10}\mathbf{b} + \dfrac{3}{10}\mathbf{c}\) and if we reflect the point \(O\) in the plane \(ABC\), it takes on the position vector \[2\mathbf{n}= \dfrac{4}{5}\mathbf{a}+ \dfrac{3}{5}\mathbf{b} + \dfrac{3}{5}\mathbf{c},\] as required.