In this video, I’ll talk about Hessian matrix,

positive semidefinite matrix, negative semidefinite matrix, and convex and concave functions.

Let f be a function of n variables x1 through xn.

Assume f has continuous second-order partial derivatives.

The function f(x1, x2, …, xn) is convex if and only if its n × n Hessian matrix is

positive semidefinite for all possible values of x1 through xn.

f(x1, x2, …, xn) is concave if and only if its n × n Hessian matrix is negative semidefinite

for all possible values of x1 through xn. What is a Hessian matrix? What is positive

semidefinite and what is negative semidefinite? The Hessian matrix is a square matrix of second-order

partial derivatives of a function. For an n variable function f(x1, x2, …, xn),

its Hessian matrix is n row by n column. For the element in the ith row and jth column,

its value is the second order partial derivative of f with respect to xi and xj.

Therefore, this element is the second order partial derivative with respect to x1 and

x1 again; this element is the second order partial derivative with respect to x1 and

x2; and so on. If the second-order partial derivatives are

all continuous, then the Hessian is a symmetric matrix. That means, Hij=Hji.

Let’s calculate the Hessian matrix of this example.

x1 minus x2 squared plus 2 times x3 squared. It has three variables, x1, x2, and x3.

Let’s first calculate the first order partial derivative with respect x1. It’s 2 times x1

minus x2. Then we calculate the second order partial

derivative with respect to x1 and x1, the second order partial derivative with respect

to x1 and x2, and the second order partial derivative with respect to x1 and x3.

They are 2, -2, and 0, respectively. This is the first row of the Hessian matrix.

Similarly, we can calculate the second and third row of the Hessian matrix.

Finally, this is the Hessian matrix of this function.

Now, let’s check the concepts like positive semidefinite and negative semidefinite matrices.

The n x n Hessian matrix H is positive semidefinite if the scalar resulted from this matrix multiplication

z transpose times H times z is non-negative for every non-zero column vector z of n real

numbers. The n x n Hessian matrix H is negative semidefinite

if the scalar resulted from this matrix multiplication is non-positive for every non-zero column

vector z of n real numbers. This explanation still seems complex and difficult

to understand. Let’s try to see an example.

This is the Hessian we got from the previous example.

We will determine whether it is positive semi-definite or negative semi-definite.

Now, let’s define a non-zero column vector z. It contains three numbers, z1, z2, and

z3. For the matrix multiplication, we can expand

it in this form. Then we calculate the product of the first

two matrices. This is a 1 by 3 matrix, and this is a 3 by

3 matrix. So the result will be a 1 by 3 matrix. The first element will be the sum of this

row times the first column. It is 2z1 minus 2z2.

The second element will be the sum of this row times the second column. It is -2z1 plus

2z2. The third element will be the sum of this

row times the third column. It is 4z3. Now let’s multiply this row by this column.

This is the result. We find that this item must be greater than

or equal to 0, so must this item. Therefore, their sum should also be greater

than or equal to 0. And this Hessian matrix is positive semidefinite

based on the definition. Finally, we can conclude that this function

is a convex function. The method I just introduced can be used to

determine whether a function is convex or concave. It is applicable to functions with

any number of variables. For a function with only two variables, there

is an easier way to tell whether it is convex or concave

We need to calculate the second order partial derivative with respect to x1 only, the second

order partial derivative with respect to x2 only, and their product minus the second order

partial derivative with respect to x1 and x2 squared.

If they are all greater than or equal to 0 for all possible values of x1 and x2, then

the two-variable function is convex. If they are all greater than 0 but not equal

to, then the function is strictly convex. If the first two are less than or equal to

0 and the last one is greater than or equal to 0, then the function is concave.

If the first two are less than 0 and the last one is greater than 0, then the function is

strictly concave. With this table in mind, let’s check this

example. This is a two-variable function.

The second order partial derivative with respect to x1 only is 2.

The second order partial derivative with respect to x2 only is 2.

Their product minus the second order partial derivative with respect to x1 and x2 is 0.

Because the last one is 0, so the function is impossible to be strictly convex or strictly

concave. Also, because the first two items 2 and 2

are not less than or equal to 0, so it cannot be concave.

So, we have eliminated the three possibilities. We can conclude that f(x1, x2) is convex,

but not strictly convex. Here is another example with two variables.

It is just the opposite of the previous function. After calculation, we find these three items

are -2, -2, and 0, respectively. Because the last one is 0, so the function

is impossible to be strictly convex or strictly concave.

Also, because the first two items -2 and -2 are not greater than or equal to 0, so it

cannot be convex. We can conclude that g(x1, x2) is concave,

but not strictly concave. Actually, we don’t have to go through all

this trouble because if a function is convex, then its opposite is concave, and vice versa.

Okay, I just talked about Hessian matrix, positive semidefinite matrix, negative semidefinite

matrix, and convex and concave functions. Thanks for watching.

Very helpful. Thankyou Mr. Wang!

Very clear explanation of the hessian matrix and some examples. Thank you Mr. Wang

very simple ,very good explanation ,excellent examples . thank you Dr wang

Thank you for sharing Mr. Wang!

Can I know the source or the literature?

Thanku very much sir…that was easy

thanks a lot sir

Excellent video. Do you have a video on the KKT conditions?

Thanks alot

Thnaks alot sir

Very helpful! Thanks a lot for putting the time and effort in for this video!

Thanks a lot for your video, I have an optimisation exam tomorrow morning this was very helpful!

Thank you so much for uploading, the given examples were sooo clear and easy to understand!! May I ask how to determine the function whether its quasiconvex or quasiconcave? Thanks!

excellent

Very well explained, thanks!

Very much appreciated! Had a hard time understanding my professor.

Super

thanx sir…very nice.

Very useful sir…

thank you! I have a question: how to prove if the function is non convex (or even nonsmooth) function?

There is a mistake at 2:49 df/dx3=4×3

d2f/dx1x2 = 0 no?

Hi Guys, please comment and let me know what you think about this Operations Research Open Course. Your feedback is really appreciated. If you enjoy the video, please subscribe and share. All my replies here are only related to the content in my own videos. I am afraid I won't be able to answer other questions. Thanks for your understanding.

Thank you for the clear explanation. One precision: on the screen around 2 m 49 s the third partial derivative appears as df/dx1 = 4×3 when in fact it's the derivative with respect to x3, so it should be df/dx3 = 4×3. Otherwise all good!

All the theorems and examples are clearly explained. Thank you for your sharing. There is a tiny mistake in video at 2:48. The beginning of the fourth line, it should be partial derivative of X3. No big deal.

Please, I'd appreciate if you do something on Marquardt Method of forcing the Hessian Matrix to be positive definite. It's something I really need help on

On the 3×3 hessian example, you knew the final z equation was all greater than zero because the coefficients were all positive and the z’s were squared, what if my z’s aren’t all squared but all my coefficients are positive? Is it still positive semi definite?

you you are the best

Awesome