I'm having trouble understanding the 3rd step under the formal argument. If we move hv1 in the x direction, how does this imply that the output will be hv1*fx(x0,y0)? (Sorry for the notation - I'm on my phone).

That is the definition of the derivative. Remember: `fₓ(x₀,y₀) = lim_Δx→0 [(f(x₀+Δx,y₀)-f(x₀,y₀))/Δx]` Then, we can replace Δx with hv₁ because both Δx and h are very small, so we get: `fₓ(x₀,y₀) = (f(x₀+hv₁,y₀)-f(x₀,y₀))/hv₁` We can then rearrange this equation to get: `f(x₀+hv₁,y₀) = hv₁ × fₓ(x₀,y₀) + f(x₀,y₀)`

Can anyone help me understand how to solve the puzzle at the end of the article? I am having trouble understanding it.

I had trouble with this puzzle too, but then I thought about it in terms of vectors. We need to maximize 100A + 20B + 2C, right? By definition of the dot product, this expression is equal to the dot product of two vectors [100, 20, 2] * [A, B, C]. So we want to maximize the dot product. When does the dot product have the maximum value? It is maximum when two vectors are parallel, or, in other words, one vector is multiple of the other (this can be understood from the graphical interpretation of the dot product). Therefore, our vector [A, B, C] should be [100x, 20x, 2x], where x is some number. The second insight is to express A^2 + B^2 + C^2 = 10404 equation in vector notation. Expression A^2 + B^2 + C^2 is equal to the dot product of vector [A,B,C] with itself: [A,B,C]*[A,B,C] = 10404. Here instead of [A,B,C] we substitute [100x, 20x, 2x] vector and solve for x: [100x, 20x, 2x]* [100x, 20x, 2x] = 10000x^2 + 400x^2 + 4x^2 = 10404x^4 = 10404. x = 1. Therefore, our vector [A,B,C] is [100 * 1, 20 * 1, 2 * 1] = [100, 20, 2]. So, we have our answer: A = 100 B = 20 C = 2. I hope it wasn't confusing. :)

step 4. how does adding the change in the x direction and y direction give us the total change in the function? we are adding "perpendicular numbers", not vectors, so the adding should look more like pythagoras, shouldn't it?

I think The explanation in this article is not precise, as the reason for we to be able to ignore hv0 to add the 2 nadge is: The function must be locally be essentially linear, i.e., there must be a linear approximation L(x)=f(a)+Df(a)(x−a) That ignorance and directly add make sense for L(x). Detailed info, you can refer to : http://mathinsight.org/directional_derivative_gradient_derivation

Isn't it good to color "h" in black in the figure just below the subtitle of "Breaking down the nudge" for the consistency? Just a suggestion.

If you meant that we should bold h like 𝐡, then this is my answer: No, because h is a scalar value, and we're taking the limit as h approaches the (scalar value) zero. This corresponds to the nudge in the input, which is the vector h𝐯, approaching the vector value 𝟎. h𝐯 is equal to the input nudge direction 𝐯 scaled (multiplied) by the step size h.

I don't understand this sentence: The famous triangle inequality tells us that this will be maximized by the unit vector in the direction (nabla) f (x0, y0) To me, this doesn't seem obvious at all, can I find explanation perhaps somewhere else on the khan academy? I have looked at triangle inequality `||x + y|| <= ||x||+||y||` , but I don't understand how the two things are related, other than that both somehow talk about vectors. Directional derivative however works with dot products and not adding vectors together, as in the triangle inequality, so I don't immediately see the connection between the two.

I'm in the same boat and don't see how the triangle inequality can be applied, however using the slightly different Cauchy-Schwarz inequality works. The Cauchy-Scwarz inequality states that ```x·y <= ||x|| * ||y||``` for any two vectors x and y. Cauchy-Schwarz also says that the inequality can be turned to an equality ```x · y = ||x|| * ||y||``` if x and y are parallel.

Hi, i've a question on the following point "Computing the directional derivative involves a dot product between the gradient and the vector v". when i look at the definition of dot product, it says "|a|.|b|.cosine theeta". but in this definition no cosine theeta is involved. is that mean, the gradient (which has the partial derivatives of x,y) not considered as a vector here? or is it? if yes,why cosine theeta is not involved here?

The gradient is vector here, see http://mathworld.wolfram.com/DotProduct.html You can figure out why `cosine` is omitted.

There's an error at the formal definition part. You say dx goes to zero. That means df/dx is undefined?

It is not undefined because the definition says that dx approaches zero, so it is never really zero. That is why you take the limit.

Main content

Multivariable calculus

Course: Multivariable calculus > Unit 2

Lesson 3: Partial derivative and gradient (articles)

Directional derivatives (going deeper)

Q: I don't understand this sentence: The famous triangle inequality tells us that this will be maximized by the unit vector in the direction (nabla) f (x0, y0) To me, this doesn't seem obvious at all, can I find explanation perhaps somewhere else on the khan academy? I have looked at triangle inequality `||x + y|| <= ||x||+||y||` , but I don't understand how the two things are related, other than that both somehow talk about vectors. Directional derivative however works with dot products and not adding vectors together, as in the triangle inequality, so I don't immediately see the connection between the two.

I'm in the same boat and don't see how the triangle inequality can be applied, however using the slightly different Cauchy-Schwarz inequality works. The Cauchy-Scwarz inequality states that ```x·y <= ||x|| * ||y||``` for any two vectors x and y. Cauchy-Schwarz also says that the inequality can be turned to an equality ```x · y = ||x|| * ||y||``` if x and y are parallel.

Q: Hi, i've a question on the following point "Computing the directional derivative involves a dot product between the gradient and the vector v". when i look at the definition of dot product, it says "|a|.|b|.cosine theeta". but in this definition no cosine theeta is involved. is that mean, the gradient (which has the partial derivatives of x,y) not considered as a vector here? or is it? if yes,why cosine theeta is not involved here?

The gradient is vector here, see http://mathworld.wolfram.com/DotProduct.html You can figure out why `cosine` is omitted.

Q: There's an error at the formal definition part. You say dx goes to zero. That means df/dx is undefined?

It is not undefined because the definition says that dx approaches zero, so it is never really zero. That is why you take the limit.

Google Classroom

A more thorough look at the formula for directional derivatives, along with an explanation for why the gradient gives the slope of steepest ascent.

Background:

This article is targetted for those who want a deeper understanding of the directional derivative and its formula.

Formal definition of the directional derivative

There are a couple reasons you might care about a formal definition. For one thing, really understanding the formal definition of a new concept can make clear what it is really going on. But more importantly than that, I think the main benefit is that it gives you the confidence to recognize when such a concept can and cannot be applied.

As a warm up, let's review the formal definition of the partial derivative, say with respect to

x

\frac{\partial f}{\partial x} (x_{0}, y_{0}) = lim_{h \to 0} \frac{f (x_{0} + h, y_{0}) - f (x_{0}, y_{0})}{h}

The connection between the informal way to read

\frac{\partial f}{\partial x}

and the formal way to read the right-hand side is as follows:

Symbol	Informal understanding	Formal understanding
$\partial x$ ‍	A tiny nudge in the $x$ ‍ direction.	A limiting variable $h$ ‍ which goes to $0$ ‍, and will be added to the first component of the function's input.
$\partial f$ ‍	The resulting change in the output of $f$ ‍ after the nudge.	The difference between $f (x_{0} + h, y_{0})$ ‍ and $f (x_{0}, y_{0})$ ‍, taken in the same limit as $h \to 0$ ‍.

We could instead write this in vector notation, viewing the input point

(x_{0}, y_{0})

as a two-dimensional vector

\begin{array}{r} x_{0} = [\begin{array}{c} x_{0} \\ y_{0} \end{array}] \end{array}

Here

x_{0}

is written in bold to emphasize its vectoriness. It's a bit confusing to use a bold

x

for the entire input rather than some other letter, since the letter

x

is already used in an un-bolded form to denote the first component of the input. But hey, that's convention, so we go with it.

Instead of writing the "nudged" input as

(x_{0} + h, y_{0})

, we write it as

x_{0} + h \hat{i}

, where

\hat{i}

is the unit vector in the

x

-direction:

\begin{array}{r} \frac{\partial f}{\partial x} (x_{0}) = lim_{h \to 0} \frac{f (x_{0} + h \hat{i}) - f (x_{0})}{h} \end{array}

In this notation, it's much easier to see how to generalize the partial derivative with respect to

x

to the directional derivative along any vector

\vec{v}

\nabla_{\vec{v}} f (x_{0}) = lim_{h \to 0} \frac{f (x_{0} + h \vec{v}) - f (x_{0})}{h}

In this case, adding

h \vec{v}

to the input for a limiting variable

h \to 0

formalizes the idea of a tiny nudge in the direction of

\vec{v}

Seeking connection between the definition and computation

Computing the directional derivative involves a dot product between the gradient

\nabla f

and the vector

\vec{v}

. For example, in two dimensions, here's what this would look like:

\begin{aligned} \nabla_{\vec{v}} f (x, y) & = \nabla f \cdot \vec{v} \\ = [\begin{array}{c} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{array}] \cdot [\begin{array}{c} v_{1} \\ v_{2} \end{array}] \\ = v_{1} \frac{\partial f}{\partial x} (x, y) + v_{2} \frac{\partial f}{\partial y} (x, y) \end{aligned}

Here,

v_{1}

and

v_{2}

are the components of

\vec{v}

\begin{array}{r} \vec{v} = [\begin{array}{c} v_{1} \\ v_{2} \end{array}] \end{array}

The central question is, what does this formula have to do with the definition given above?

Breaking down the nudge

The computation for

\nabla_{v} f

can be seen as a way to break down a tiny step in the direction of

v

into its

x

and

y

components.

Specifically, you can imagine the following procedure:

Start at some point $(x_{0}, y_{0})$ ‍.
Choose a tiny value $h$ ‍.
Add $h v_{1}$ ‍ to $x_{0}$ ‍, which means stepping to the point $(x_{0} + h v_{1}, y_{0})$ ‍. From what we know of partial derivatives, this will change the output of the function by about

\begin{array}{r} h v_{1} (\frac{\partial f}{\partial x} (x_{0}, y_{0})) \end{array}

Now add $h v_{2}$ ‍ to $y_{0}$ ‍ to bring us up/down to the point $(x_{0} + h v_{1}, y_{0} + h v_{2})$ ‍. The resulting change to $f$ ‍ is now about

\begin{array}{r} h v_{2} (\frac{\partial f}{\partial y} (x_{0} + h v_{1}, y_{0})) \end{array}

Adding the results of steps

3

and

4

, the total change to the function upon moving from the input

(x_{0}, y_{0})

to the input

(x_{0} + h v_{1}, y_{0} + h v_{2})

has been about

\begin{array}{r} h v_{1} (\frac{\partial f}{\partial x} (x_{0}, y_{0})) + h v_{2} (\frac{\partial f}{\partial y} (x_{0} + h v_{1}, y_{0})) \end{array}

This is very close to the expression for the directional derivative, which says the change in

f

due to this step

h \vec{v}

should be about

\begin{aligned} h \nabla_{\vec{v}} f (x_{0}, y_{0}) \\ = h \vec{v} \cdot \nabla f (x_{0}, y_{0}) \\ = h v_{1} \frac{\partial f}{\partial x} (x_{0}, y_{0}) + h v_{2} \frac{\partial f}{\partial y} (x_{0}, y_{0}) \end{aligned}

However, this differs slightly from the result of our step-by-step argument, in which the partial derivative with respect to

y

is taken at the point

(x_{0} + h v_{1}, y_{0})

, not at the point

(x_{0}, y_{0})

Luckily we are considering very, very small values of

h

. In fact, more technically, we should be talking about the limit as

h \to 0

. Therefore evaluating

\frac{\partial f}{\partial y}

(x_{0} + h v_{1}, y_{0})

will be almost the same as evaluating it at

(x_{0}, y_{0})

. Moreover, as

h

approaches

0

, so does the difference between these two, but we have to assume that

f

is continuous.

Why does the gradient point in the direction of steepest ascent?

Having learned about the directional derivatives, we can now understand why the direction of the gradient is the direction of steepest ascent.

Specifically, here's the question at hand.

Setup:

Let $f$ ‍ be some scalar-valued multivariable function, such as $f (x, y) = x^{2} + y^{2}$ ‍.
Let $(x_{0}, y_{0})$ ‍ be a particular input point
Consider all possible directions, i.e. all unit vectors $\hat{u}$ ‍ in the input space of $f$ ‍.

Question (informal): If we start at

(x_{0}, y_{0})

, which direction should we walk so that the output of

f

increases most quickly?

Question (formal): Which unit vector

\hat{u}

maximizes the directional derivative along

\hat{u}

\begin{array}{r} \nabla_{\hat{u}} f (x_{0}, y_{0}) = \underset{Maximize this quantity}{\underset{⏟}{\hat{u} \cdot \nabla f (x_{0}, y_{0})}} \end{array}

The famous triangle inequality tells us that this will be maximized by the unit vector in the direction

\nabla f (x_{0}, y_{0})

Notice, the fact that the gradient points in the direction of steepest ascent is a consequence of the more fundamental fact that all directional derivatives require taking the dot product with

\nabla f

[A seemingly unrelated puzzle to help with the intuition]

Want to join the conversation?

Sort by:

Rich Rasta
Posted 8 years ago. Direct link to Rich Rasta's post “I'm having trouble unders...”
I'm having trouble understanding the 3rd step under the formal argument. If we move hv1 in the x direction, how does this imply that the output will be hv1*fx(x0,y0)? (Sorry for the notation - I'm on my phone).
Button navigates to signup pageButton navigates to signup page
(23 votes)
Answer
- Tejas
  Posted 8 years ago. Direct link to Tejas's post “That is the definition of...”
  That is the definition of the derivative. Remember:
  fₓ(x₀,y₀) = lim_Δx→0 [(f(x₀+Δx,y₀)-f(x₀,y₀))/Δx]
  Then, we can replace Δx with hv₁ because both Δx and h are very small, so we get:
  fₓ(x₀,y₀) = (f(x₀+hv₁,y₀)-f(x₀,y₀))/hv₁
  We can then rearrange this equation to get:
  f(x₀+hv₁,y₀) = hv₁ × fₓ(x₀,y₀) + f(x₀,y₀)
  Comment on Tejas's post “That is the definition of...”
  (28 votes)
98angelpadilla
Posted 6 years ago. Direct link to 98angelpadilla's post “Can anyone help me unders...”
Can anyone help me understand how to solve the puzzle at the end of the article? I am having trouble understanding it.
Button navigates to signup pageButton navigates to signup page
(4 votes)
Answer
- Evgenii Neumerzhitckii
  Posted 6 years ago. Direct link to Evgenii Neumerzhitckii's post “I had trouble with this p...”
  I had trouble with this puzzle too, but then I thought about it in terms of vectors. We need to maximize 100A + 20B + 2C, right? By definition of the dot product, this expression is equal to the dot product of two vectors [100, 20, 2] * [A, B, C]. So we want to maximize the dot product. When does the dot product have the maximum value? It is maximum when two vectors are parallel, or, in other words, one vector is multiple of the other (this can be understood from the graphical interpretation of the dot product). Therefore, our vector [A, B, C] should be [100x, 20x, 2x], where x is some number.
  
  The second insight is to express A^2 + B^2 + C^2 = 10404 equation in vector notation. Expression A^2 + B^2 + C^2 is equal to the dot product of vector [A,B,C] with itself:
  
  [A,B,C]*[A,B,C] = 10404.
  
  Here instead of [A,B,C] we substitute [100x, 20x, 2x] vector and solve for x:
  
  [100x, 20x, 2x]* [100x, 20x, 2x] = 10000x^2 + 400x^2 + 4x^2 = 10404x^4 = 10404.
  x = 1.
  
  Therefore, our vector [A,B,C] is [100 * 1, 20 * 1, 2 * 1] = [100, 20, 2].
  
  So, we have our answer:
  A = 100
  B = 20
  C = 2.
  
  I hope it wasn't confusing. :)
  Comment on Evgenii Neumerzhitckii's post “I had trouble with this p...”
  (59 votes)
Taras.Pokalchuk
Posted 8 years ago. Direct link to Taras.Pokalchuk's post “step 4. how does adding t...”
step 4. how does adding the change in the x direction and y direction give us the total change in the function? we are adding "perpendicular numbers", not vectors, so the adding should look more like pythagoras, shouldn't it?
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- xfcjscn
  Posted 7 years ago. Direct link to xfcjscn's post “I think The explanation i...”
  I think The explanation in this article is not precise, as the reason for we to be able to ignore hv0 to add the 2 nadge is:
  The function must be locally be essentially linear, i.e., there must be a linear approximation
  L(x)=f(a)+Df(a)(x−a)
  That ignorance and directly add make sense for L(x).
  Detailed info, you can refer to :
  http://mathinsight.org/directional_derivative_gradient_derivation
  Button navigates to signup page
  (2 votes)
jaeyungkim12
Posted 5 years ago. Direct link to jaeyungkim12's post “Isn't it good to color "h...”
Isn't it good to color "h" in black in the figure just below the subtitle of "Breaking down the nudge" for the consistency? Just a suggestion.
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- saajidchowdhury
  Posted 3 years ago. Direct link to saajidchowdhury's post “If you meant that we shou...”
  If you meant that we should bold h like 𝐡, then this is my answer: No, because h is a scalar value, and we're taking the limit as h approaches the (scalar value) zero. This corresponds to the nudge in the input, which is the vector h𝐯, approaching the vector value 𝟎. h𝐯 is equal to the input nudge direction 𝐯 scaled (multiplied) by the step size h.
  Button navigates to signup page
  (2 votes)
vyletel
Posted 6 years ago. Direct link to vyletel's post “I don't understand this s...”
I don't understand this sentence: The famous triangle inequality tells us that this will be maximized by the unit vector in the direction (nabla) f (x0, y0)

To me, this doesn't seem obvious at all, can I find explanation perhaps somewhere else on the khan academy? I have looked at triangle inequality ||x + y|| <= ||x||+||y|| , but I don't understand how the two things are related, other than that both somehow talk about vectors. Directional derivative however works with dot products and not adding vectors together, as in the triangle inequality, so I don't immediately see the connection between the two.
Button navigates to signup pageComment on vyletel's post “I don't understand this s...”
(2 votes)
Answer
- JamesCagalawan
  Posted 6 years ago. Direct link to JamesCagalawan's post “I'm in the same boat and ...”
  I'm in the same boat and don't see how the triangle inequality can be applied, however using the slightly different Cauchy-Schwarz inequality works. The Cauchy-Scwarz inequality states that
  x·y <= ||x|| * ||y||
  for any two vectors x and y. Cauchy-Schwarz also says that the inequality can be turned to an equality
  x · y = ||x|| * ||y||
  if x and y are parallel.
  Comment on JamesCagalawan's post “I'm in the same boat and ...”
  (4 votes)
prashantcomsc
Posted 4 months ago. Direct link to prashantcomsc's post “For a given gradient vect...”
For a given gradient vector, I can understand that any unit vector in the direction of gradient vector will give maximum value of dot product between itself and gradient vector.
But how does it proves the gradient vector is itself the direction of maximum ascent.
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- sm2701
  Posted 2 months ago. Direct link to sm2701's post “I don’t think that quite ...”
  I don’t think that quite matters. If a unit vector in the direction of the gradient vector is the direction of greatest ascent, then moving in the direction of the gradient vector is also in the direction of the greatest ascent. One is just a multiple of another—they still pointin the same direction.
  Button navigates to signup page
  (1 vote)
Yixuan Liu
Posted 3 years ago. Direct link to Yixuan Liu's post “I have more of a conceptu...”
I have more of a conceptual question. If we think a partial derivative as a little nudge to a certain direction and that nudge (h) is approaching 0, why does that concept not transfer to directional derivative? Namely, why would the directional derivative of 2v twice as big as that of v? If we think of the nudge in v as so tiny that goes to 0, why would 2v versus v even matter?
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- Victrix
  Posted 5 months ago. Direct link to Victrix's post “To gain an intuitive unde...”
  To gain an intuitive understanding of that particular concept, you'd need to consider how the single variable calculus differentiation process deals with constant multipliers. For example, dy/dx (y = c*f(x)) gives dy = c*f'(x)dx for all constants c.
  
  In the same respect, the scaling factor of the directional vector v is taken into consideration. Note that while we're dealing with something approaching 0, it doesn't ever quite get there.
  Button navigates to signup page
  (1 vote)
roger.llrt
Posted 5 years ago. Direct link to roger.llrt's post “Regarding directional der...”
Regarding directional derivative I can not understand why the vector v has not to be a unit vector, namely, it can has an arbitrary magnitude.

In the one hand, the formal definition of the directional derivative supplied in the article lies upon unit vectors i and j. From my point of view, it is easy to understand this definition will hold for any direction other than i and j always that we treat with unit vectors. Another thing, and this is what I can not figure out clearly, is wether the magnitud of these vectors can also be generalized. To illustrate what it shocks me, if we set a limit from 0 to infinity for the magnitud of v, the formula will ends up with an indetermination of the form h·||v||=0·Inf, because h goes to 0 and ||v|| goes to Inf. In contrast, if we restrict ourselves into unit vectors this would not happen.

On the other hand, as far as I understood we are trying to get a measure that tells us how f(x, y) changes in the direction of v, which has nothing to do with how far we move in that direction. Therefore the only way I can figure out to capture just the variation due to change in direction (and nothing else) is "playing" with number 1 (i.e. unit vectors) whenever we multiply.
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
Adrien Zabat
Posted 7 years ago. Direct link to Adrien Zabat's post “There's an error at the f...”
There's an error at the formal definition part. You say dx goes to zero. That means df/dx is undefined?
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
- 98angelpadilla
  Posted 6 years ago. Direct link to 98angelpadilla's post “It is not undefined becau...”
  It is not undefined because the definition says that dx approaches zero, so it is never really zero. That is why you take the limit.
  Button navigates to signup page
  (4 votes)
stallionsri
Posted 7 years ago. Direct link to stallionsri's post “Hi, i've a question on th...”
Hi, i've a question on the following point "Computing the directional derivative involves a dot product between the gradient and the vector v". when i look at the definition of dot product, it says "|a|.|b|.cosine theeta".
but in this definition no cosine theeta is involved. is that mean, the gradient (which has the partial derivatives of x,y) not considered as a vector here? or is it? if yes,why cosine theeta is not involved here?
Button navigates to signup pageComment on stallionsri's post “Hi, i've a question on th...”
(2 votes)
Answer
- Water Hui
  Posted 7 years ago. Direct link to Water Hui's post “The gradient is vector he...”
  The gradient is vector here, see http://mathworld.wolfram.com/DotProduct.html
  You can figure out why cosine is omitted.
  Comment on Water Hui's post “The gradient is vector he...”
  (2 votes)