Let

\[f(x) = \int_0^1(xt)^2\:dt, \quad \text{and} \quad g(x) = \int_0^xt^2\:dt.\]

Let \(A>0\). Which of the following statements is true?

\(g(f(A))\) is always bigger than \(f(g(A))\).

\(f(g(A))\) is always bigger than \(g(f(A))\).

They are always equal.

\(f(g(A))\) is bigger if \(A < 1\), and \(g(f(A))\) is bigger if \(A > 1\).

\(g(f(A))\) is bigger if \(A < 1\), and \(f(g(A))\) is bigger if \(A > 1\).

We have \[f(x) = \int_0^1(xt)^2\:dt= x^2\int_0^1 t^2 \:dt = x^2\left[\dfrac{t^3}{3}\right]_0^1= \dfrac{x^2}{3}\] and \[g(x) = \int_0^xt^2\:dt= \left[\dfrac{t^3}{3}\right]_0^x= \dfrac{x^3}{3}.\]

Thus \[f(g(A)) = f\left(\dfrac{A^3}{3}\right) = \dfrac{A^6}{3^3}\] and \[g(f(A)) = g\left(\dfrac{A^2}{3}\right) = \dfrac{A^6}{3^4}.\]

Now it is always true that \(\dfrac{A^6}{3^3}>\dfrac{A^6}{3^4}\), and so (b) is our answer.