Week 8 - HackMD

:::success During next 5 weeks, we plan to discuss the following key concepts 1. Functions of several variables 2. Partial derivatives 3. Applications of partial derivatives ::: ```markmap # Partial Derivatives ## Taylor Series - Linearisation - [Laplace Approximation](https://bookdown.org/rdpeng/advstatcomp/laplace-approximation.html) - [Extended Kalman Filter](https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/11-Extended-Kalman-Filters.ipynb) - *Real world*: Navigation systems - GPS Statistical computing ## Gradients - Maxima/Minima - Directional Derivatives - [Gradient Descent](https://www.geeksforgeeks.org/gradient-descent-algorithm-and-its-variants/) - *Real world*: Optimisation - OR Machine Learning ``` ## 3.1 Functions of Several Variables We have already seen functions of a single variable, their derivatives and integrals. **Example 1** consider the function $f: \mathbb{R} \to \mathbb{R}$ given by $$f(x)=x \sin ( x^2 )\,.$$ Find its derivative and the integral on $[0,\sqrt{\pi}]$. :::spoiler Answer $$ \frac{d}{dx} f(x) = \sin (x^2) + 2x^2 \cos (x^2)\,. $$ $$\int_0^\sqrt{\pi} f(x) \, dx = -\frac{1}{2}\cos(x^2) \bigg|^{\sqrt\pi}_0 = 1\,.$$ ::: *From now on, we will focus on functions of multiple variables.* **Example 2** Consider a box of length $x$, width $y$, and height $z$. What is the volume of the box? :::spoiler Answer The volume $V$ of the box is given by $$V(x,y,z) = xyz\,.$$ $V: \mathbb R_+ \times\mathbb R_+ \times\mathbb R_+ \to \mathbb R$ is a function of three variables. ::: :::info **Definition** A **function $f : D \to \mathbb R$ of $n$ variables** is a rule that assigns to every ordered $n-$tuple of real numbers, $(x_1,x_2,\dots,x_n) \in D \subseteq \mathbb R^n$, a unique real number denoted by $f(x_1,x_2,\dots,x_n)$. The set $D$ is the *domain* of $f$ and is denoted by $Domain(f)$. The *range* of $f$ is the set of values $f$ takes, i.e., $$Range(f):=\left\{f(x_1,x_2,\dots,x_n)\, \big| (x_1,x_2,\dots,x_n) \in D \right\}.$$ ::: **Example 3** What is the domain and the range of the function $V: \mathbb R_+ \times\mathbb R_+ \times\mathbb R_+ \to \mathbb R$ in the Example 2 given by $V(x,y,z)=xyz$. :::spoiler Answer Length, width and height can be any positive real number. So, the domain of $V$ is $\mathbb R_+ \times\mathbb R_+ \times\mathbb R_+$. Since we can obtain any positive real number as a multiplication of three positive real numbers, e.g., $r = r \times 1 \times 1$, the range is $\mathbb R_+$. ::: **Example 4** What is the domain and the range of the function $f$ given by $$f(x,y) = \sqrt{16-x^2-y^2}$$ :::spoiler Answer For the square root to be defined $16-x^2-y^2 \geq 0$. That is the domain of $f$ is $\{(x,y)| x^2+y^2 \leq 16\}$ the closed disc centred at $(0,0)$ with radius $4$. Note that $\sqrt{16-x^2-y^2} \leq 4$. Also, $\sqrt{16-x^2-y^2} \geq 0$. If $z \in [0,4]$, then $f(0, \sqrt{16-z^2}) = \sqrt{16-16+z^2} = z\,.$ So the range is $[0,4]$. ::: ### 3.1.1 Graphing Functions :::info **Definition** If $f$ is a function of $n$ variables with domain $D \subseteq \mathbb R^n$, then the **graph of $f$** is the set of all points $(x_1,x_2,\dots, x_n, z) \in \mathbb R^{n+1}$ such that $z = f(x_1,x_2,\dots,x_n)$ for all $(x_1,x_2\dots,x_n) \in D$. ::: **Example 5** Sketch the graph of $f:\mathbb R_+ \times \mathbb R_+ \to \mathbb R$ given by $$f(x,y) = 6-3x-2y\,.$$ :::spoiler Answer Note that, if $z=f(x,y) = 6-3x-2y$, then $3x+2y+z=6$ is the equation of a plane passing through $(0,0,6), (0,3,0)$ and $(2,0,0)$ restricted to $x > 0$ and $y > 0$. ![plane1](https://hackmd.io/_uploads/Sk212lqgye.png) ::: Below is a python code example to visualise this. Run it and cofirm that the sketch you have drawn is correct. :::spoiler 3D Graphs in Python ```=python import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D import numpy as np %matplotlib inline def f(x, y): return 6-3*x-2*y x = np.linspace(0, 6, 100) y = np.linspace(0, 6, 100) X, Y = np.meshgrid(x, y) Z = f(X, Y) fig = plt.figure(figsize=(12, 10)) ax = plt.axes(projection='3d') ax.contour3D(X, Y, Z, 150, cmap='binary') ax.set_xlabel('x') ax.set_ylabel('y') ax.set_zlabel('z'); ``` Reference: [3D plotting in Matplotlib.](https://jakevdp.github.io/PythonDataScienceHandbook/04.12-three-dimensional-plotting.html) ::: **Example 6** Sketch the graph of $$f(x,y) = \sqrt{16-x^2-y^2}$$ :::spoiler Answer Note that $z = \sqrt{16-x^2-y^2}$ if and only if $x^2+y^2+z^2 = 16$. This is the equation of a sphere centred at $(0,0,0)$ of radius $4$. Since we are taking the positive square root for $z$, the graph is the upper hemisphere of this sphere. ![image](https://hackmd.io/_uploads/HkBpaxcekl.png) ::: Modify the python code example for plotting in 3D to confirm your answer. ### 3.1.2 Level Sets :::info **Definition** The **level sets** of a function of several variables, $f$, are the sets with equations $f(x_1,\dots,x_n) = k$ where $k \in Range(f)$. ::: Below left: *level curves* of a function of two variables. Below right: *isolines* or *contours* on a topographical map. ![image](https://hackmd.io/_uploads/HkMqFW5xJg.png) (Figure taken from [1].) **Example 6** Sketch the level sets of the function of two variables $f(x,y) = \sqrt{16-x^2-y^2}$ for $k \in \{2, 2.5, 3, 3.5, 4\}$. :::spoiler Answer Note that $2 = \sqrt{16-x^2-y^2}$ if and only if $x^2+y^2 = 12$. This is the equation of a circle centred at $(0,0)$ of radius $\sqrt{12}$. In fact, all the level curves are circles except the last one, which is just the origin because $4 = \sqrt{16-x^2-y^2} \iff x^2+y^2 = 0 \iff x=0 \land y = 0$. ![image](https://hackmd.io/_uploads/S1pIHW5e1e.png) ::: Use [this link](https://jakevdp.github.io/PythonDataScienceHandbook/04.04-density-and-contour-plots.html) to see a python code to plot level curves (= *contours*) of a function of two variables. **Example 7** What are the level sets of the function $f(x,y,z) = x^2+y^2+z^2$? :::spoiler Answer Note that $x^2+y^2+z^2 = k$ if and only if $k \geq 0$, and when $k>0$, the level set is the surface of a sphere with radius $\sqrt{k}$ and when $k=0$ it is just a point: $(0,0,0)$. ![image](https://hackmd.io/_uploads/HybNkG5gJx.png) ::: --- **References** 1. *Chapter 14.1* : Stewart, J. (2012). Calculus (8th ed.). Boston: Cengage Learning. 2. *Chapter 13.1* : Strang G., Calculus (3rd ed.). [https://ocw.mit.edu/](https://ocw.mit.edu/courses/res-18-001-calculus-fall-2023/pages/textbook/) --- ## 3.2 Limits and Continuity We recall the notions of limits and continuity of a function of a single variable: :::info Let $D\subset \mathbb R$ and $f: D \to \mathbb R$. Let $a$ be a point such that $D$ contains points that are arbitrarily close to $a$. We say that the limit of $f$ as $x$ approaches $a$ is $L$, i.e., $$\lim_{x \to a} f(x) = L$$ if for all $\varepsilon >0$ there exists $\delta>0$ such that $$x \in D, \,\,\,0<|x-a|<\delta \implies |f(x)-L|<\varepsilon.$$ ::: ![Limit](https://hackmd.io/_uploads/H1cbhWlbkg.gif =325x275) :::info If $a \in D$ and $\lim_{x \to a} f(x) = f(a)$, then we say that $f$ is *continuous at* $a$. ::: :::info If $f$ is continuous at $a$ for all $a \in D$, we say $f$ is *continuous on* $D$. ::: **Example 1** consider the function $f: \mathbb{R}\setminus \{0\} \to \mathbb{R}$ given by $$f(x)=x \sin \left(\frac{1}{x}\right)\,.$$ 1. Find the limit $\lim_{x \to 0} f(x)$. 2. Extend $f$ to a function on $\mathbb R$ by defining $f(0)$ so that $f$ is continuous on $\mathbb R$. :::spoiler Answer 1. For $x\neq 0$, we have $0\leq |f(x)|\leq |x|$. Since $\lim_{x \to 0} |x| =0$, by the Sandwich Theorem, $\lim_{x\to 0}f(x) =0$. 2. For the extended $f$ to be continuous on $\mathbb R$, we should have $\lim_{x \to 0} f(x) = f(0)$. So, $f(0)=0$ and when $x \neq 0$, $f(x)$ is as before. ::: ### 3.2.1 Limits in higher dimensions Now, we would like to extend these notions to higher dimensions. However, in higher dimensions there are infinitely many ways to approach a point, not just left and right. See below. ![Limit directions](https://hackmd.io/_uploads/BkSyT-g-1g.png =300x200) (Figure taken from [1].) One solution is to approach a point using squares as shown below. ![Limit squares](https://hackmd.io/_uploads/B1UETbg-kg.png =300x300) Using this approach, we have the following definition. :::info **Definition** Let $f: D \to \mathbb R$ be a function of $n$ variables such that $D$ contains points arbitrarily close to $(a_1,\dots,a_n)$. Then we say that **the limit of $f$ as $(x_1,\dots, x_n)$ approaches $(a_1,\dots,a_n)$ is $L$**, i.e., $$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} f(x_1,\dots, x_n) = L\,,$$ if for all $\varepsilon >0$ there exists $\delta>0$ such that $$(x_1,\dots,x_n) \in D, \,0<|x_i-a_i|<\delta,\, i=1,\dots,n\implies |f(x_1,\dots,x_n)-L|<\varepsilon.$$ ::: - All the common properties about limits in one-dimension extend to higher dimensions: :::success **Theorem** If $f, g : D \to \mathbb R$ are functions of $n$ variables and $$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} f(x_1,\dots, x_n)= L\,,\,\text{and}\,\,\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} g(x_1,\dots, x_n)=M\,.$$Then $$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} (f \pm g)(x_1,\dots, x_n)= L \pm M\,,\,\,\,\,\text{and}$$$$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} (f \cdot g)(x_1,\dots, x_n)= LM\,.$$ ::: What about $f/g\,$? Can you come up with the corresponding theorem statement? :::success :::spoiler **Theorem** If the function $f/g$ is well-defined on $D$ and $M\neq 0$, then $$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} \left(\frac{f}{g}\right)(x_1,\dots, x_n)= \frac{L}{M}\,.$$ ::: **Example 2** Find these limits. $$(a) \lim_{(x,y)\to (1,1)} \frac{2xy}{x^2+y^2}\,\,\,\text{and}\,\,\,(b) \lim_{(x,y,z)\to (0,0,0)} \frac{3x^2yz}{x^2+y^2}.$$ :::spoiler Answer $(a)$ Note that $\lim_{(x,y)\to (1,1)} 2xy = 2\,,\,\,\,\lim_{(x,y)\to (1,1)} x^2+y^2 = 2$ and $\frac{2xy}{x^2+y^2}$ is well-defined close to the point $(1,1)$. So, $$\lim_{(x,y)\to (1,1)} \frac{2xy}{x^2+y^2} = 2/2 = 1.$$ $(b)$ We cannot use the previous approach for this limit (why?). However, it is easy to see that for $(x,y)\neq (0,0)$,$$\left|\frac{3x^2y}{x^2+y^2}\right| = \frac{3|xz|}{2}\frac{2|x||y|}{x^2+y^2} \leq \frac{3|xz|}{2}\,$$Since, $\lim_{(x,y,z)\to (0,0,0)} \frac{3|xz|}{2} = 0$ by the Sandwich theorem we have that $$\lim_{(x,y,z)\to (0,0,0)} \frac{3x^2yz}{x^2+y^2} = 0\,.$$ ::: **Example 3** Show that the following limit does not exist. $$\lim_{(x,y)\to (0,0)} \frac{x^2-y^2}{x^2+y^2}$$ :::spoiler Answer Consider the two cases of approaching the origin along the $y=0$ line and along the $x=0$ line. When $(x,y)\neq (0,0)$, in the first case, $\frac{x^2-y^2}{x^2+y^2} = \frac{x^2-0}{x^2+0} = 1$ and in the second case, $\frac{x^2-y^2}{x^2+y^2}=\frac{-y^2}{y^2}=-1$. If the limit exists, it is be independent of the way we approach the origin. So, in this case, the limit cannot exist. ::: ### 3.2.2 Continuity The definition of continuity at a point follows immediately. :::info **Definition** Let $f:D \to \mathbb R$ be a function of $n$ variables and $f$ is **continuous at the point $(a_1,\dots,a_n) \in D$**, if $$\lim_{(x_1,\dots, x_n)\to (a_1,\dots,a_n)} f(x_1,\dots, x_n) = f(a_1,\dots,a_n).$$ ::: Similar to the one-dimensional case, we have the following. :::success Polynomials, rational functions, trigonometric functions, exponentials, logarithms are continuous *in their domains*. ::: **Example 4** What is the *maximal* set in which the following function is continuous. $$g(x,y)=\begin{cases}(x^2+y^2)\ln(x^2+y^2) & \text{if}\,\,(x,y)\neq (0,0)\\ 0 & \text{if}\,\,(x,y)= (0,0)\end{cases}$$ :::spoiler Answer Note that $x^2+y^2$ is continuous everywhere, and because $\ln(\cdot)$ is continuous except at $0$, we have that $\ln(x^2+y^2)$ is continuous except when $(x^2+y^2)=0$ which is true if and only if $(x,y)=(0,0)$. So, their product (and hence, $g$) is continuous when $(x,y)\neq (0,0)$. We are left to check whether $g$ is continuous at $(0,0)$. To this end, we have to check whether $$\lim_{(x,y)\to (0,0)}(x^2+y^2)\ln(x^2+y^2)=0=g(0,0).$$This is true because $\lim_{(x,y)\to(0,0)} x^2+y^2 = 0$ and $\lim_{s\to 0} s \ln s = 0\,.$. Therefore, $g$ is continuous on $\mathbb R^2\,.$ ::: --- **References** 1. *Chapter 14.2* : Stewart, J. (2012). Calculus (8th ed.). Boston: Cengage Learning. 2. *Chapter 2.1* : Corral, M. (2021). Vector Calculus. [https://www.mecmath.net/](https://www.mecmath.net/) --- ## 3.3 First Order Partial Derivatives We recall the following definition of derivative of a function of a single variable. :::info A function $f:(b,c) \to \mathbb R$ is *differentiable* at $a \in (b,c)$, if the following limit exists.$$\lim_{h \to 0} \frac{f(a+h)-f(a)}{h} =:f'(a).$$The limit, $f'(a)$ is the *derivative of $f$ at $a$*. ::: The derivative at $c$ can be intepreted as the rate of change of the function at $c$. Also, the derivative gives the slope of the tangent line to the graph of the fuction $f$ at c. ![image](https://hackmd.io/_uploads/H1yLBexZJg.png) :::info If $f$ is differentiable at $a$ for all $a \in (b,c)$, we say $f$ is *differentiable on $(b,c)$*. ::: :::info If $f$ is differentiable on $(b,c)$ and $f':(b,c) \to \mathbb R$ is continuous on (b,c), then we say $f$ is *continuosly differentiable on $(b,c)$.* ::: We can generalise this notion of derivative to higher dimensions by considering rates of changes along axis directions. Let $f$ be a function of two variables $(x,y)$ with its graph $z=f(x,y)$ as shown below. ![image](https://hackmd.io/_uploads/H1aTSge-Jl.png) Suppose we let only $x$ vary while keeping $y = b$ fixed. Then $g_1(x) := f(x, b)$ is a function of a single variable. If $g_1$ is differentiable at $a$, then we call it the partial derivative of $f$ with respect to $x$ at $(a,b)$ and denote it by $f_x(a,b)$. :::success $$f_x(a,b)=\lim_{h\to 0}\frac{f(a+h,b)-f(a,b)}{h}$$ ::: - The partial derivative with respect to $x$ represents the slope of the tangent lines to the curve that are parallel to the $xz-$plane. Simlarly, we let only $y$ vary while keeping $x = a$ fixed. Then $g_2(y) := f(a, y)$ is a function of a single variable. If $g_2$ is differentiable at $b$, then we call it the partial derivative of $g$ with respect to $y$ at $(a,b)$ and denote it by $f_y(a,b)$. :::success $$f_y(a,b)=\lim_{h\to 0}\frac{f(a,b+h)-f(a,b)}{h}$$ ::: - The partial derivative with respect to $y$ represents the slope of the tangent lines to the curve that are parallel to the $yz-$plane. **Example 1** Let $f:\mathbb R^2 \to \mathbb R$ be $f(x,y)=\sqrt[3]{x^3+y^3}$. Compute $f_x(0,0)$. Guess the value of $f_y(0,0)$. :::spoiler Answer $$f_x(0,0)=\lim_{h \to 0} \frac{\sqrt[3]{(0+h)^3+0^3}-\sqrt[3]{0^3+0^3}}{h}=\lim_{h \to 0} \frac{\sqrt[3]{h^3}}{h} = \lim_{h \to 0} \frac{h}{h} = 1\,.$$Since the function is symmetric in $x$ and $y$, $f_y(0,0)=1$. ::: In general, we have the following definition. :::info **Definition** Let $D \subseteq \mathbb R^n$ be a disc, let $f:D\to \mathbb R$ be a function, and let $(a_1,\dots,a_n) \in D$. We say the **first order partial derivative of $f$ with respect to $x_j$ at $(a_1,\dots,a_n)$** exists if the limit $$\lim_{h \to 0}\frac{f(x_1,\dots, x_{j-1}, x_j+h, x_{j+1},\dots, x_n) - f(x_1,\dots, x_{j-1}, x_j, x_{j+1},\dots, x_n) }{h}$$exists. This limit is denoted by $f_{x_j}(a_1,\dots,a_n)$ or $\frac{\partial f}{\partial x_j}(a_1,\dots,a_n)$, ::: - For $f_{x_j}$, we keep all the variables except $x_j$ fixed. So, the rule for finding the partial derivative of $f$ with respect to $x_j$ is to differentiate $f$ with respect of $x_j$ treating all other variables to be constants. :::success The partial derivative $f_{x_j}$ is commonly denoted by $\frac{\partial f}{\partial x_j}\,.$ ::: Similar to the one-dimensional case, we have the following. :::success Polynomials, rational functions, trigonometric functions, exponentials, logarithms have partial derivatives *inside their domains*. ::: **Example 2** Let $f:\mathbb R^3 \to \mathbb R$ be$$f(x,y,z)=x+y+z+xy+yz+xz+xyz\,.$$ Find $f_x(1,1,1)$, $f_y(0,1,0)$ and $f_z(0,0,0)$. :::spoiler Answer Note that $f$ is a polynomial in $x, y$ and $z$. So, we can take the partial derivatives at any point. Differentiating with respect to $x$ and treating $y$ and $z$ to be constant, we obtain $$f_x(x,y,z) = 1+0+0+y+0+z+yz = 1+y+z+yz\,.$$Similarly, $$f_y(x,y,z) = 1+x+z+xz\,\,\,\text{and}\,\,\,f_z(x,y,z) = 1+x+y+xy\,.$$Therefore,$$f_x(1,1,1)= 4\,,\,\,\,f_y(0,1,0)=1\,,\,\,\,\text{and}\,\,\, f_z(0,0,0)=1\,.$$ ::: **Example 3** Let $f$ be given by $f(x,y)=\sin\big(\frac{x}{1+y}\big)$. Find $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ along with their domains. :::spoiler Answer Note that the domain of the function is $D=\{(x,y)\in \mathbb{R}^2| y \neq -1\}$. Also, $\sin (\cdot)$ is differentiable everywhere and $\frac{x}{1+y}$ is a rational function defined everwhere except when $y=-1$. So, both $f_x$ and $f_y$ exist on $D$. Differentiating with respect to $x$ while treating $y$ to be a constant other than $-1$: $$f_x(x,y)=\cos\Big(\frac{x}{1+y}\Big)\cdot \frac{1}{1+y}\,.$$ Differentiating with respect to $y$ while treating $x$ to be a constant: $$f_y(x,y)=\cos\Big(\frac{x}{1+y}\Big)\cdot \frac{- x}{(1+y)^2}\,.$$ ::: **Example 4** Let $f: \mathbb R_+ \times \mathbb R$ be given by $f(x,y)=x^y$. Find $f_x$ and $f_y$ along with their domains. :::spoiler Answer Differentiating with respect to $x$ while treating $y$ to be a constant we have $$f_x(x,y)=y\,x^{y-1}.$$ Note that $f(x,y) = e^{\ln x^y} = e^{y \ln x}\,.$ So,$$f_y(x,y)=e^{y \ln x} \ln x = (\ln x) \,x^y\,.$$ ::: **Example 5** consider the function $f: \mathbb{R}^3 \to \mathbb{R}^3$ given by $$f(x,y,z)=(x+y^2+2z^3)^2.$$ Find $\frac{\partial}{\partial x}\big(\frac{\partial f}{\partial z}\big)\,.$ :::spoiler Answer Differentiating with respect to $z$ (by applying chain rule), $$\frac{\partial f}{\partial z}(x,y,z) = 2 (x+y^2+2z^3)\cdot 3 \cdot 2z^2 = 12z^2(x+y^2+2z^3)\,.$$ Next, differentiating with respect to $x$, $$\frac{\partial}{\partial x}\left(\frac{\partial f}{\partial z}\right)(x,y,z) = 12z^2\,.$$ ::: --- **Reference** 1. *Chapter 14.3* : Stewart, J. (2012). Calculus (8th ed.). Boston: Cengage Learning. 2. *Chapter 13.2* : Strang G., Calculus (3rd ed.). [https://ocw.mit.edu/](https://ocw.mit.edu/courses/res-18-001-calculus-fall-2023/pages/textbook/) 3. *Chapter 2.2* : Corral, M. (2021). Vector Calculus. [https://www.mecmath.net/](https://www.mecmath.net/) --- :::danger **Summary**: Now, we can - Sketch graphs of functions of two variables. - Sketch level sets of functions of two or three variables. - Describe (intuitively) the notion of limit of a function of several variables. - Determine whether a limit of functions of several variables exists or not. - Evaluate a limit of functions of several variables when the limit exists. - Identify points of continuity/discontinuity of functions of several variables. - Compute first order partial derivatives of functions of several variables. :::