If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

# Least squares examples

An example using the least squares solution to an unsolvable system. Created by Sal Khan.

## Want to join the conversation?

• Is this least squares solution equivalent to the orthocenter of the triangle?
If the shortest distance is the projection, then the solution point should be on the perpendicular to each line.
...or did I miss something?
• Nope, it's not the orthocenter. The least squares solution is always inside the triangle whereas the orthocenter can be outside. Nor does the least squares solution minimize the sum of the perpendicular distances to each line as some commenters claim, nor the distances to the corners of the triangle which was my second guess. In fact, the least squares solution has no geometric significance in R^2 because it's not unique! If you take the equation for one of the lines and multiply through by a constant then the least squares solution changes, even though it's the exact same line. It turns out every point in the triangle is a valid least squares solution if you use the right coefficients.

The way to see what's going on geometrically in this problem is to look at the codomain in R^3 and the solution vector b = [2, 1, 4]. Applying transformation A to the original triangle gets you a new triangle in R^3 whose corners are [2, 1, 1] [2, 6, 4] and [17, 1, 4]. If we subtract b from each corner vector we get [0, 0, -3] [0, 5, 0] and [15, 0, 0]. And now, if we multiply the original linear equations by the constants c1, c2, and c3 respectively then those difference vectors become [0, 0, -3*c1] [0, 5*c2, 0] and [15*c3, 0, 0]. So without changing the original triangle we can stretch the transformed triangle relative to b along all 3 axes at will. And if we stretch it the right way we can move the shortest vector from b to the transformed triangle wherever we like, and thus we can move the least squares point of the original triangle wherever we like.
• I'm having trouble figuring out why I'm getting two different answers. The second line written in slope intercept form has a slope of -1/2 and an intercept of 1/2. So I would think the line could also be written as (1/2)x + y = 1/2. However, when I then go through the process of finding the approximate intersection using these values in my matrices (using for the second rows of A and b [1/2 1] and [1/2] instead of [1 2] and [1] as they are in the video) I end up with a different approximate intersection (x: 52/31 y: 69/62). I've gone back through the math a couple times and don't seem to be finding a place where I'm making an error. Am I missing something? Or is such an approximate solution not unique and I'll find different ones depending on which numbers I use? Thanks.
• You're absolutely right, equivalent forms of the same lines will yield different least squares solutions.
(1 vote)
• At around 9 minutes, when he swaps the rows (1,6 and 6,1), why does it not make the matrix negative?
• At around 9 minutes, when he swaps the rows (1,6 and 6,1), why does not swap x1 and x2, so the result it [3/7; 10/7]; if you don swap x1 and x2, [6 1; 1 6]*[10/7; 3/7] is not equal [9; 4]
• If he didn't swap the rows, he would've gotten the same answer:
`[6 1 | 9][1 6 | 4]`
divide R1 by 6:
`[1 1/6 | 3/2][1 6 | 4]`
R2-R1:
`[1 1/6 | 3/2][0 35/6 | 5/2]`(4-3/2 = 8/2-3/2 = 5/2)
R2*6/35:
`[1 1/6 | 3/2][0 1 | 3/7]`(5/2*6/35 = 30/70 = 3/7)
R1-R2/6:
`[1 0 | 10/7]`(3/2-(3/7)/6 = 3/2-3/42 = 21/14-1/14 = 20/14 = 10/7)
`[0 1 | 3/7]`
This is the same thing he got in the video, but I never swapped the 2 rows.
• Shouldn't it be x ~= [(A^T * A)^-1] * [A^T * b]?
(the inverse of A^T * A)
• That's basically what he did, except he solved the system by reducing to reduced row echelon form instead of multiplying by the inverse.

1st he multiplied A^T*A to get a matrix, we'll call it C
2nd he multiplied A^T*b to get a vectord we'll call it d
Lastly he solved Cx=d
which you could solve by reducing to reduce row echelon form or you could solve by C^-1*d=x
which is the same as (A^T*A)^-1*(A^T*b)=x

Hope this makes sense
• Is there a way to associate a confidence with the final intersection point in this least squares example? I've been looking at confidence videos and lessons and not finding a direct way to apply that to this example.