#LyX 1.3 created this file. For more info see http://www.lyx.org/
\lyxformat 221
\textclass article
\begin_preamble
\usepackage{ragged2e}
\RaggedRight
\setlength{\parindent}{1 em}
\end_preamble
\language english
\inputencoding auto
\fontscheme pslatex
\graphics default
\paperfontsize 12
\spacing single
\papersize Default
\paperpackage a4
\use_geometry 1
\use_amsmath 0
\use_natbib 0
\use_numerical_citations 0
\paperorientation portrait
\leftmargin 1in
\topmargin 1in
\rightmargin 1in
\bottommargin 1in
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default
\layout Title
Derivatives and Minimization
\layout Author
Paul E.
Johnson
\layout Section
Functions of one variable.
\layout Subsection
Review definitions.
\layout Standard
Recall the derivative is the slope of a line that is tangent to a curve.
Suppose we have a function
\begin_inset Formula \begin{eqnarray*}
y & = & f(x)\end{eqnarray*}
\end_inset
\newline
Suppose that you begin at a point,
\begin_inset Formula $x_{0}$
\end_inset
(pronounced
\begin_inset Quotes eld
\end_inset
x naught
\begin_inset Quotes erd
\end_inset
).
And you go a little bit up the X axis to a point
\begin_inset Formula $x.$
\end_inset
The change is
\begin_inset Formula $\Delta x=x-x_{0}$
\end_inset
.
The value of
\begin_inset Formula $y$
\end_inset
changes as a result, from
\begin_inset Formula $f(x_{0})$
\end_inset
to
\begin_inset Formula $f(x)$
\end_inset
.
The change is referred to as
\begin_inset Formula $\Delta y=f(x)-f(x_{0}).$
\end_inset
\layout Standard
The derivative is defined as the ratio of
\begin_inset Formula $\Delta y$
\end_inset
to
\begin_inset Formula $\Delta x$
\end_inset
when
\begin_inset Formula $\Delta x$
\end_inset
shrinks to 0.
\layout Standard
\added_space_bottom vfill
I'm leaving room here for you to draw a picture:
\layout Standard
\added_space_top vfill
People use many kinds of notation for the derivative, but the most commonly
used are
\begin_inset Formula \[
\frac{dy}{dx}\]
\end_inset
\layout Standard
or
\begin_inset Formula \[
f'(x)\]
\end_inset
\layout Standard
or
\begin_inset Formula \[
D(f(x))\]
\end_inset
\layout Standard
or
\layout Standard
\begin_inset Formula \[
\dot{y}\]
\end_inset
\layout Standard
You get the idea, right? For any thing you read, it may be the author makes
up notation.
By far the most common are the first two.
\layout Subsection
Maxima and Minima
\layout Standard
\added_space_bottom vfill
If we want to find the maximum point of a continuous function, we look for
the spot at which the derivative equals 0.
Again, make a picture.
\layout Subsection
The Second Derivative
\layout Standard
\added_space_bottom vfill
What if you think of the derivative as a funciton of x.
You can make a plot of the derivative in the vicinity of a maximum or minimum,
right?
\layout Standard
Now take the derivative of the derivative.
Various notations for that:
\begin_inset Formula \begin{equation}
\frac{d^{2}y}{dx^{2}}\label{eq:secondderiv}\end{equation}
\end_inset
\layout Standard
or
\begin_inset Formula \[
f''(x)\]
\end_inset
\layout Standard
or
\layout Standard
\begin_inset Formula \[
D^{2}(f(x))\]
\end_inset
\layout Standard
The second derivative is the change in the change.
Humph!
\layout Standard
Think of the first derivative like
\begin_inset Quotes eld
\end_inset
velocity
\begin_inset Quotes erd
\end_inset
.
\layout Standard
The second derivative is the
\begin_inset Quotes eld
\end_inset
acceleration
\begin_inset Quotes erd
\end_inset
or
\begin_inset Quotes eld
\end_inset
decelleration
\begin_inset Quotes erd
\end_inset
.
\layout Standard
If a curve is straight, the derivative is the slope and the second derivative
would be 0 because slope is not changing.
\layout Subsection
Second order conditions.
\layout Standard
If you find the value of x for which
\begin_inset Formula $f'(x)=0$
\end_inset
, then you are either at a maximum or a minimum.
(I mean
\begin_inset Quotes eld
\end_inset
locally
\begin_inset Quotes erd
\end_inset
, in the immediate vicinity of x.)
\layout Standard
If f''(x) >0, then you are at a minimum.
The curve f is
\begin_inset Quotes eld
\end_inset
concave up.
\begin_inset Quotes erd
\end_inset
\layout Standard
\added_space_bottom vfill
if f''(x) < 0, then you are at a maximum.
The curve f is
\begin_inset Quotes eld
\end_inset
concave down.
\begin_inset Quotes erd
\end_inset
\layout Section
\added_space_top vfill
Functions of several variables.
\layout Subsection
Functions of several variables.
\layout Standard
Suppose you have 3 input variables,
\begin_inset Formula $x_{1}$
\end_inset
,
\begin_inset Formula $x_{2}$
\end_inset
, and
\begin_inset Formula $x_{3}$
\end_inset
.
\begin_inset Formula \[
y=f(x_{1},x_{2},x_{3})\]
\end_inset
\layout Standard
The calculus of many variables can get really complicated, but most of the
time it is really simple.
\layout Standard
A
\series bold
partial derivative
\series default
is the change in
\begin_inset Formula $f(x_{1},x_{2},x_{3})$
\end_inset
that results when all of the variables are being held constant except one.
The most common notation for the partial derivative is
\begin_inset Formula \begin{equation}
\frac{\partial y}{\partial x_{1}}\label{eq:partial}\end{equation}
\end_inset
\layout Standard
Because it is tedious to type that fraction all of the time, you sometimes
see authors inventing convenient notation for partial derivatives.
My personal favorite is to use a subscript to tell which variable I'm allowing
to change:
\begin_inset Formula \[
f_{1}(x_{1},x_{2},x_{3})\]
\end_inset
\newline
That is supposed to be the same as
\begin_inset LatexCommand \ref{eq:partial}
\end_inset
\layout Subsection
Finding Optima
\layout Standard
If you are given a function like
\begin_inset Formula $f(x_{1},x_{2},x_{3})$
\end_inset
and you are instructed to find the maximum or mimimum, the first thing
you do is find the place where ALL OF THE PARTIAL DERIVATIVES equal 0.
That is, solve this system of equations:
\begin_inset Formula \[
\frac{\partial y}{\partial x_{1}}=0\]
\end_inset
\begin_inset Formula \[
\frac{\partial y}{\partial x_{2}}=0\]
\end_inset
\begin_inset Formula \[
\frac{\partial y}{\partial x_{3}}=0\]
\end_inset
\layout Standard
Note, you could as well think of this as a matrix of derivatives:
\begin_inset Formula \begin{equation}
D=\left[\begin{array}{c}
\frac{\partial y}{\partial x_{1}}\\
\frac{\partial y}{\partial x_{2}}\\
\frac{\partial y}{\partial x_{3}}\end{array}\right]=0\label{eq:FOC}\end{equation}
\end_inset
\layout Standard
One tires quickly of writing down 3 rows of derivatives over and over, so
one often just refers to this condition for a maximum or minimum as
\begin_inset Formula $D=0$
\end_inset
.
\layout Subsection
Second order conditions
\layout Standard
In the calculus of one variable, it is easy to tell if one has found a maximum
or a minimum from the second derivative.
\layout Standard
It is not so simple in the calculus of several variables.
It is easy to calculate a second partial derivative of
\begin_inset Formula $f()$
\end_inset
with respect to
\begin_inset Formula $x_{1}$
\end_inset
\begin_inset Formula \[
\frac{\partial^{2}y}{\partial x_{1}\partial x_{1}}=\frac{\partial^{2}y}{\partial^{2}x_{1}}\]
\end_inset
\newline
And one can also find the partial of
\begin_inset Formula $\frac{\partial y}{\partial x_{1}}$
\end_inset
with respect to another variable, say
\begin_inset Formula $x_{2}$
\end_inset
.
\begin_inset Formula \[
\frac{\partial^{2}y}{\partial x_{1}\partial x_{2}}\]
\end_inset
\layout Standard
I prefer short hand notation like
\begin_inset Formula $f_{11}()$
\end_inset
or
\begin_inset Formula $f_{12}()$
\end_inset
for these.
\layout Standard
Anyway, suppose you begin with the matrix of first partials.
You can differentiate each item by each of the 3 variables, so that means
you can build up a 3x3 matrix of second partial derivatives like so:
\begin_inset Formula \begin{equation}
D'=\left[\begin{array}{ccc}
f_{11} & f_{12} & f_{13}\\
f_{21} & f_{22} & f_{23}\\
f_{31} & f_{32} & f_{33}\end{array}\right]=\left[\begin{array}{ccc}
\frac{\partial^{2}y}{\partial x_{1}\partial x_{1}} & \frac{\partial^{2}y}{\partial x_{1}\partial x_{2}} & \frac{\partial^{2}y}{\partial x_{1}\partial x_{3}}\\
\frac{\partial^{2}y}{\partial x_{1}\partial x_{2}} & \frac{\partial^{2}y}{\partial x_{2}\partial x_{2}} & \frac{\partial^{2}y}{\partial x_{2}\partial x_{3}}\\
\frac{\partial^{2}y}{\partial x_{3}\partial x_{1}} & \frac{\partial^{2}y}{\partial x_{3}\partial x_{2}} & \frac{\partial^{2}y}{\partial x_{3}\partial x_{3}}\end{array}\right]\label{eq:dprime}\end{equation}
\end_inset
\layout Standard
This is the so-called
\series bold
Hessian matrix.
\series default
There are various conditions that can be set so that one can diagnose the
question of whether a maximum, a minimum, or neither, has been found.
\layout Standard
The one that sticks in my mind is the idea of a
\begin_inset Quotes eld
\end_inset
positive definite
\begin_inset Quotes erd
\end_inset
matrix.
Take a
\begin_inset Formula $3x1$
\end_inset
column vector z.
It is supposed to represent a small change from the location where one
currently is,
\begin_inset Formula $(x_{1},x_{2},x_{3}).$
\end_inset
Calculate the quantity:
\begin_inset Formula \begin{equation}
z'\cdot D'\cdot z\label{eq:zdpz}\end{equation}
\end_inset
\newline
or, more verbosely,
\layout Standard
\begin_inset Formula \begin{equation}
\left[z_{1},z_{2},z_{3}\right]\left[\begin{array}{ccc}
f_{11} & f_{12} & f_{13}\\
f_{21} & f_{22} & f_{23}\\
f_{31} & f_{32} & f_{33}\end{array}\right]\left[\begin{array}{c}
z_{1}\\
z_{2}\\
z_{3}\end{array}\right]\label{eq:zffz}\end{equation}
\end_inset
\layout Standard
There's a theorem that says:
\layout Standard
If z'D'z > 0, then D' is a positive definite matrix.
\layout Standard
If z'D'z < 0, then D' is a negative definite matrix.
\layout Standard
Use these ideas to check if you have found a maximum or a minimum.
Find the point where all the partials are 0, call it
\begin_inset Formula $x*$
\end_inset
and then evaluate the Hessian at that point.
\layout Standard
If the Hessian is positive definite, you have found a minimum point.
\layout Standard
If the Hessian is negative definite, you have found a maximum point.
\layout Standard
Note this is entirely similar to the univariate case, where
\begin_inset Formula $f''(x)>0$
\end_inset
means you have a minimum and
\begin_inset Formula $f''(x)<0$
\end_inset
means you have a maximum.
\the_end