during the last part ("vector notation using the hessian") I do not understand why is it necessary to transpose that vector in the quadratic term. I mean.. You can expand the quadratic term exacly in the same manner without transposing that vector right?? As it is done in the exercise you end up with 2 vectors, why would you need to have the vector on the left transposed??

The dimensions must be right for matrix multiplication.

So, could these sorts of things be used to generalise the taylor series to higher dimension?

Yep it is a generalisation, higher order terms consist of tensorlike operations (3. order fijk(x1,x2)*xi*xj*xk, while 2. order terms can be written as a matrix multiplication).

At the top, in your definition of Qf(x), I think the partial derivatives of Q are not the same as the partial derivatives of f, due to the presence of the quadratic term. Only the second partials match. I suppose we could modify the "coefficients" on the first-order term to include the negative of the value of the partial derivatives of the quadratic term. Would this improve the approximation? Hmm.

When you evaluate at the particular point (x_0, y_0), the partial derivatives of the quadratic term go to zero.

Main content

Multivariable calculus

Course: Multivariable calculus > Unit 3

Lesson 2: Quadratic approximations

Quadratic approximation

Google Classroom

Quadratic approximations extend the notion of a local linearization, giving an even closer approximation of a function.

Background:

What we're building to

The goal, as with a local linearization, is to approximate a potentially complicated multivariable function

f

near some input, which I'll write as the vector

x_{0}

. A quadratic approximation does this more tightly than a local linearization, using the information given by second partial derivatives.

Non-vector form

In the specific case where the input of

f

is two dimensional, and you are approximating near a point

(x_{0}, y_{0})

, you will see below that the quadratic approximation ends up looking like this:

\begin{aligned} Q_{f} (x, y) & = f (x_{0}, y_{0}) + \\ f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0}) + \\ \frac{1}{2} f_{x x} (x_{0}, y_{0}) (x - x_{0})^{2} + \\ f_{x y} (x_{0}, y_{0}) (x - x_{0}) (y - y_{0}) + \\ \frac{1}{2} f_{y y} (x_{0}, y_{0}) (y - y_{0})^{2} \end{aligned}

Vector form:

The general form of this, for a scalar-valued function

f

with any kind of multidimensional input, here's what that approximation looks like:

Q_{f} (x) = \underset{Constant}{\underset{⏟}{f (x_{0})}} + \underset{Linear term}{\underset{⏟}{\nabla f (x_{0}) \cdot (x - x_{0})}} + \underset{Quadratic term}{\underset{⏟}{\frac{1}{2} (x - x_{0})^{T} H_{f} (x_{0}) (x - x_{0})}}

I know it looks a bit complicated, but I'll step through it piece by piece later on. Here's a brief outline of each term.

$f$ ‍ is a function with multi-dimensional input and a scalar output.
$\nabla f (x_{0})$ ‍ is the gradient of $f$ ‍ evaluated at $x_{0}$ ‍.
$H_{f} (x_{0})$ ‍ is the Hessian matrix of $f$ ‍ evaluated at $x_{0}$ ‍.
The vector $x_{0}$ ‍ is a specific input, the one we are approximating near.
The vector $x$ ‍ represents the variable input.
The approximation function, $Q_{f}$ ‍, has the same value as $f$ ‍ at the point $x_{0}$ ‍, all its partial derivatives have the same value as those of $f$ ‍ at this point, and all its second partial derivatives have the same value as those of $f$ ‍ at this point.

Tighter and tighter approximations

Imagine you are given some function

f (x, y)

with two inputs and one output, such as

$f (x, y) = \sin (x) \cos (y)$ ‍

Khan Academy video wrapper

See video transcript

The goal is to find a simpler function that approximates

f (x, y)

near some particular point

(x_{0}, y_{0})

. For example,

$(x_{0}, y_{0}) = (\frac{π}{3}, \frac{π}{6})$ ‍

Zero-order approximation

The most naive approximation would be a constant function which equals the value of

f

(x_{0}, y_{0})

everywhere. We call this a "

0

-order approximation".

In the example:

\begin{aligned} C (x, y) & = \sin (\frac{π}{3}) \cos (\frac{π}{6}) \\ = (\frac{\sqrt{3}}{2}) \frac{\sqrt{3}}{2} \\ = \frac{3}{4} \end{aligned}

Written in the abstract:

$C (x, y) = f (x_{0}, y_{0}) \leftarrow Constant function$ ‍

Graphically:

The graph of this approximation function

C (x, y)

is a flat plane passing through the graph of our function at the point

(x_{0}, y_{0}, f (x_{0}, y_{0}))

. Below is a video showing how this approximation changes as we move the point

(x_{0}, y_{0})

around.

Khan Academy video wrapper

See video transcript

The graph of

f

is pictured in blue, the graph of the approximation is white, and the point

(x_{0}, y_{0}, f (x_{0}, y_{0}))

is pictured as a red dot.

First-order approximation

The constant function zero-order approximation is pretty lousy. Sure, it is guaranteed to equal

f (x, y)

at the point

(x_{0}, y_{0})

, but that's about it. One step better is to use a local linearization, also known as a "First-order approximation".

In the example:

$L_{f} (x, y) = \frac{3}{4} + \frac{\sqrt{3}}{4} (x - \frac{π}{3}) + \frac{- \sqrt{3}}{4} (y - \frac{π}{6})$ ‍

Written in the abstract:

$L_{f} (x, y) = f (x_{0}, y_{0}) + f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})$ ‍

Here,

f_{x}

and

f_{y}

denote the partial derivatives of

f

Graphically:

The graph of a local linearization is the plane tangent to the graph of

f

at the point

(x_{0}, y_{0}, f (x_{0}, y_{0}))

. Here is a video showing how this approximation changes as we move around the point

(x_{0}, y_{0})

Khan Academy video wrapper

See video transcript

Second-order approximation.

Better still is a quadratic approximation, also called a "second-order approximation".

The remainder of this article is devoted to finding and understanding the analytic form of such an approximation, but before diving in, let's see what such approximations look like graphically. You can think of these approximations as nestling into the curves the graph at the point

(x_{0}, y_{0}, f (x_{0}, y_{0}))

, giving it a sort of mathematical hug.

Khan Academy video wrapper

See video transcript

"Quadratic" means product of two variables

In single variable functions, the word "quadratic" refers to any situation where a variable is squared as in the term

x^{2}

. With multiple variables, "quadratic" refers not only to square terms, like

x^{2}

and

y^{2}

, but also terms that involve the product of two separate variables, such as

x y

In general, the "order" of a term which is the product of several things, such as

3 x^{2} y^{3}

, is the total number of variables multiplied into that term. In this case, the order would be

5

: Two

x

's, three

y

's, and the constant doesn't matter.

Graphs of quadratic functions

One way to think of quadratic functions is in terms of their concavity, which might depend on which direction you are moving in.

If the function has an upward concavity, as is the case, for example, with

f (x, y) = x^{2} + y^{2}

, the graph will look something like this:

This shape, which is a three-dimensional parabola, goes by the name paraboloid.

If the function is concave up in one direction and linear in another, the graph looks like a parabolic curve has been dragged through space to trace out a surface. For example this happens in the case of

f (x, y) = x^{2} + y

Finally, if the graph is concave up when traveling in one direction, but concave down when traveling in another direction, as is the case for

f (x, y) = x^{2} - y^{2}

, the graph looks a bit like a saddle. Here's what such a graph looks like:

Khan Academy video wrapper

See video transcript

Reminder on the local linearization recipe

To actually write down a quadratic approximation of a function

f

near the point

(x_{0}, y_{0})

, we build up from the local linearization:

$L_{f} (x, y) = \underset{Constant term}{\underset{⏟}{f (x_{0}, y_{0})}} + \underset{Linear terms}{\underset{⏟}{f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})}}$ ‍

It's worth walking through the recipe for finding the local linearization one more time since the recipe for finding a quadratic approximation is very similar.

Start with the constant term $f (x_{0}, y_{0})$ ‍, so that our approximation at least matches $f$ ‍ at the point $(x_{0}, y_{0})$ ‍.
Add on linear terms $f_{x} (x_{0}, y_{0}) (x - x_{0})$ ‍ and $f_{y} (x_{0}, y_{0}) (y - y_{0})$ ‍.
Use the constants $f_{x} (x_{0}, y_{0})$ ‍ and $f_{y} (x_{0}, y_{0})$ ‍ to ensure that our approximation has the same partial derivatives as $f$ ‍ at the point $(x_{0}, y_{0})$ ‍.
Use the terms $(x - x_{0})$ ‍ and $(y - y_{0})$ ‍ instead of simply $x$ ‍ and $y$ ‍ so that we don't mess up the fact that our approximation equals $f (x_{0}, y_{0})$ ‍ at the point $(x_{0}, y_{0})$ ‍.

Finding the quadratic approximation

For the quadratic approximation, we add on the quadratic terms

(x - x_{0})^{2}

(x - x_{0}) (y - y_{0})

, and

(y - y_{0})^{2}

, and for now we write their coefficients as the constants

a

b

and

c

which we will solve for in a moment:

\begin{aligned} Q_{f} (x, y) & = \underset{Order 0 part}{\underset{⏟}{f (x_{0}, y_{0})}} + \\ \underset{Order 1 part}{\underset{⏟}{f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})}} + \\ \underset{Quadratic part}{\underset{⏟}{a (x - x_{0})^{2} + b (x - x_{0}) (y - y_{0}) + c (y - y_{0})^{2}}} \end{aligned}

In the same way that we made sure that the local linearization has the same partial derivatives as

f

(x_{0}, y_{0})

, we want the quadratic approximation to have the same second partial derivatives as

f

at this point.

The really nice thing about the way I wrote

Q_{f}

above is that the second partial derivative

\frac{\partial^{2} Q_{f}}{\partial x^{2}}

depends only on the

a (x - x_{0})^{2}

term.

Try it! Take the second partial derivative with respect to $x$ ‍ of every term in the expression of $Q_{f} (x, y)$ ‍ above, and notice that they all go to zero except for the $a (x - x_{0})^{2}$ ‍ term.

Did you really try it? I'm serious, take a moment to reason through it. It really helps in understanding why

Q_{f}

is expressed the way it is.

This fact is nice because rather than taking the second partial derivative of the entire monstrous expression, you can view it like this:

\begin{aligned} \frac{\partial^{2} Q_{f}}{\partial x^{2}} (x, y) & = (A bunch of 0 ’s) + \frac{\partial^{2}}{\partial x^{2}} a (x - x_{0})^{2} + (more 0 ’s) \\ = \frac{\partial}{\partial x} 2 a (x - x_{0}) \\ = 2 a \end{aligned}

Since the goal is for this to match

f_{x x} (x, y)

at the point

(x_{0}, y_{0})

, you can solve for

a

like this:

$a = \frac{1}{2} f_{x x} (x_{0}, y_{0})$ ‍

Test yourself: Use similar reasoning to figure out what the constants

b

and

c

should be.

[Solution]

We can now write our final quadratic approximation, with all six of its terms working in harmony to mimic the behavior of

f

(x_{0}, y_{0})

\begin{aligned} Q_{f} (x, y) & = f (x_{0}, y_{0}) + \\ f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0}) + \\ \frac{1}{2} f_{x x} (x_{0}, y_{0}) (x - x_{0})^{2} + \\ f_{x y} (x_{0}, y_{0}) (x - x_{0}) (y - y_{0}) + \\ \frac{1}{2} f_{y y} (x_{0}, y_{0}) (y - y_{0})^{2} \end{aligned}

Example: Approximating $\sin (x) \cos (y)$ ‍

To see this beast in action, let's try it out on the function from the introduction.

Problem: Find the quadratic approximation of

\begin{array}{r} f (x, y) = \sin (x) \cos (y) \end{array}

about the point

(x, y) = (\frac{π}{3}, \frac{π}{6})

Solution:

To collect all the necessary information, you need to evaluate

f (x, y) = \sin (x) \cos (y)

and all if its partial derivatives and all of its second partial derivatives at the point

(\frac{π}{3}, \frac{π}{6})

$f (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

$f_{x} (x, y) =$ ‍
$f_{x} (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

$f_{y} (x, y) =$ ‍
$f_{y} (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

$f_{x x} (x, y) =$ ‍
$f_{x x} (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

$f_{x y} (x, y) =$ ‍
$f_{x y} (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

$f_{y y} (x, y) =$ ‍
$f_{y y} (\frac{π}{3}, \frac{π}{6}) =$ ‍

[Answer]

Almost there! As a final step, apply all these values to the formula for a quadratic approximation.

[Answer]

So for example, to generate the animation of quadratic approximations, this is the formula I had to plug into the graphing software.

Khan Academy video wrapper

See video transcript

Vector notation using the Hessian

Perhaps it goes without saying that the expression for the quadratic approximation is long. Now imagine if

f

had three inputs,

x

y

and

z

. In principle you can imagine how this might go, adding terms involving

f_{z}

f_{x z}

f_{z z}

, on and on with all

3

partial derivatives and all

9

second partial derivative. But this would be a total nightmare!

Now imagine you were writing a program to find the quadratic approximation of a function with

100

inputs. Madness!

It actually doesn't have to be that bad. When something is not that complicated in principle, it shouldn't be that complicated in notation. Quadratic approximations are a little complicated, sure, but they're not absurd.

Using vectors and matrices, specifically the gradient and Hessian of

f

, we can write the quadratic approximation

Q_{f}

as follows:

\begin{aligned} Q_{f} (x) & = \underset{Constant}{\underset{⏟}{f (x_{0})}} + \underset{Linear term}{\underset{⏟}{\nabla f (x_{0}) \cdot (x - x_{0})}} + \underset{Quadratic term}{\underset{⏟}{\frac{1}{2} (x - x_{0})^{T} H_{f} (x_{0}) (x - x_{0})}} \end{aligned}

Let's break this down:

The boldfaced $x$ ‍ represents the input variable(s) as a vector,
$\begin{aligned} x & = [\begin{array}{c} x \\ y \\ ⋮ \end{array}] \end{aligned}$ ‍
Moreover, $x_{0}$ ‍ is a particular vector in the input space. If this has two components, this formula for $Q_{f}$ ‍ is just a different way to write the one we derived before, but it could also represent a vector with any other dimension.
The dot product $\nabla f (x_{0}) \cdot (x - x_{0})$ ‍ will expand into the sum of all terms of the form $f_{x} (x_{0}) (x - x_{0})$ ‍, $f_{y} (x_{0}) (y - y_{0})$ ‍, etc. if this is not familiar from the vector notation for local linearization, work it out for yourself in the case of $2$ ‍-dimensions to see!
[Solution]
The little superscript $T$ ‍ in the expression $(x - x_{0})^{T}$ ‍ indicates "transpose". This means you take the initial vector $(x - x_{0})$ ‍, which looks something like this:
$(x - x_{0}) = [\begin{matrix} x - x_{0} \\ y - y_{0} \end{matrix}]$ ‍
Then you flip it, to get something like this:
$(x - x_{0})^{T} = [\begin{matrix} x - x_{0} & y - y_{0} \end{matrix}]$ ‍
$H_{f} (x_{0})$ ‍ is the Hessian of $f$ ‍.
The expression $(x - x_{0})^{T} H_{f} (x_{0}) (x - x_{0})$ ‍ might seem complicated if you have never come across something like it before. This way of expressing quadratic terms is actually quite common in vector-calculus and vector-algebra, so it's worth expanding an expression like this at least a few times in your life. For example, try working it out in the case where $x$ ‍ is two-dimensional to see what it looks like.
[Solution]
You should find that it is exactly $2$ ‍ times the quadratic portion of the non-vectorized formula we derived above.

What's the point?

In truth, it is a real pain to compute a quadratic approximation by hand, and it requires staying very organized to do so without making a little mistake. In practice, people rarely work through a quadratic approximation like the example above, but knowing how they work is useful for at least two broad reasons:

Computation: Even if you never have to write out a quadratic approximation, you may one day need to program a computer to do it for a particular function. Or even if you are relying on someone else's program, you may need to analyze how and why the approximation is failing in some circumstance.
Theory: Being able to reference a second-order approximation helps us to reason about the behavior of general functions near a point. This will be useful later in figuring out if a point is a local maximum or minimum.

Want to join the conversation?

Sort by:

Philip Teisseyre
Posted 6 years ago. Direct link to Philip Teisseyre's post “In the worked example (Ap...”
In the worked example (Approximating sin(x)cos(y)) the very last term in the solution (fyy) is written in brown as 3/4 - this is missing a minus sign
Button navigates to signup pageComment on Philip Teisseyre's post “In the worked example (Ap...”
(10 votes)
Answer
gschex1112
Posted 8 years ago. Direct link to gschex1112's post “In the example using sin(...”
In the example using sin(x)cos(y), the second derivative with respect to y (the last one) is sin(x)cos(y), but shouldn't it be -sin(x)cos(y)? If you have the first partial as -sin(x)sin(y), and take the partial of that with respect to y, you get the derivative of sin(y) = cos(y), not -cos(y), right? Why did the sign change again?
Button navigates to signup pageComment on gschex1112's post “In the example using sin(...”
(9 votes)
Answer
Victor Gutierrez
Posted 5 years ago. Direct link to Victor Gutierrez's post “during the last part ("ve...”
during the last part ("vector notation using the hessian") I do not understand why is it necessary to transpose that vector in the quadratic term. I mean.. You can expand the quadratic term exacly in the same manner without transposing that vector right?? As it is done in the exercise you end up with 2 vectors, why would you need to have the vector on the left transposed??
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- Alexander Wu
  Posted 5 years ago. Direct link to Alexander Wu's post “The dimensions must be ri...”
  The dimensions must be right for matrix multiplication.
  Button navigates to signup page
  (6 votes)
Shubham
Posted 8 years ago. Direct link to Shubham's post “fyy(x,y) = -sin(x)cos(y) ...”
fyy(x,y) = -sin(x)cos(y) not sin(x)cos(y).
Button navigates to signup pageButton navigates to signup page
(4 votes)
Answer
Lee, N. M.
Posted 8 years ago. Direct link to Lee, N. M.'s post “So, could these sorts of ...”
So, could these sorts of things be used to generalise the taylor series to higher dimension?
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- Dean Smith
  Posted 8 years ago. Direct link to Dean Smith's post “Yep it is a generalisatio...”
  Yep it is a generalisation, higher order terms consist of tensorlike operations (3. order fijk(x1,x2)*xi*xj*xk, while 2. order terms can be written as a matrix multiplication).
  Button navigates to signup page
  (3 votes)
Devansh Sehta
Posted 6 years ago. Direct link to Devansh Sehta's post “What about cubic approxim...”
What about cubic approximations? Would we need a cubical "Hessian Matrix" analogue?
And how would we define the multiplication?
Button navigates to signup pageButton navigates to signup page
(2 votes)
Answer
Harrison
Posted 6 years ago. Direct link to Harrison's post “What is the formula (not ...”
What is the formula (not in the vector/matrix form) for a quadratic approximation when z is added to the input of the function f, making it f(x,y,z)?
Button navigates to signup pageButton navigates to signup page
(2 votes)
Answer
Richard Kopcke
Posted 8 years ago. Direct link to Richard Kopcke's post “At the top, in your defin...”
At the top, in your definition of Qf(x), I think the partial derivatives of Q are not the same as the partial derivatives of f, due to the presence of the quadratic term. Only the second partials match. I suppose we could modify the "coefficients" on the first-order term to include the negative of the value of the partial derivatives of the quadratic term. Would this improve the approximation? Hmm.
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
- Grant
  Posted 8 years ago. Direct link to Grant's post “When you evaluate at the ...”
  When you evaluate at the particular point (x_0, y_0), the partial derivatives of the quadratic term go to zero.
  Button navigates to signup page
  (2 votes)
sauj123
Posted 8 years ago. Direct link to sauj123's post “For the solution of findi...”
For the solution of finding the b constant, finding the first partial derivative with respect to y does not make c(y - y0)^2 zero. It would actually be 2c(y - y0). Nevertheless, this has no effect in the final answer as applying the partial derivative respect to x makes that term zero.
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
Oconnoran17
Posted 4 years ago. Direct link to Oconnoran17's post “Would it be possible to f...”
Would it be possible to find f given Q and the input vector? so like finding a best for a particular set of data
Button navigates to signup pageComment on Oconnoran17's post “Would it be possible to f...”
(1 vote)
Answer

Multivariable calculus

Course: Multivariable calculus > Unit 3

Background:

What we're building to

Tighter and tighter approximations

Zero-order approximation

First-order approximation

Second-order approximation.

"Quadratic" means product of two variables

Graphs of quadratic functions

Reminder on the local linearization recipe

Finding the quadratic approximation

Example: Approximating sin⁡(x)cos⁡(y)‍

Vector notation using the Hessian

What's the point?

Want to join the conversation?

Example: Approximating $\sin (x) \cos (y)$ ‍