chain_figure(annotate; width = 640, height = 360)Draw the forward, tangent and reverse chains of $f_K \circ \cdots \circ f_1$ with mathjax_figure. Additional labels are added with annotate(label, geometry) where label(text, x, y; color, fontsize, class) adds a label and geometry gives the position of the columns xs with their index ks.
Reverse mode in action
Reverse mode
Interpretation of tangents
n =
Forward on forward
Given $\text{Dual}(s_1, t_1)$ with $s_1 = \text{Dual}(f_1(x), \frac{\partial f_1}{\partial x_j})$ and $t_1 = \text{Dual}(\frac{\partial f_1}{\partial x_i}, \frac{\partial^2 f_1}{\partial x_i \partial x_j})$
Compute $s_2 = f_2(s_1) = (f_2(f_1(x)), J_2 \cdot \frac{\partial f_1}{\partial x_j}) = ((f_2 \circ f_1)(x), \partial (f_2 \circ f_1) / \partial x_j)$
Compute $J_{f_2}(s_1)$ which gives $\text{Dual}(J_2, H_{2j})$
Compute
$$\begin{align} J_{f_2}(s_1) \cdot t_1 & = \text{Dual}(J_2, H_{2j}) \cdot \text{Dual}(\frac{\partial f_1}{\partial x_i}, \frac{\partial^2 f_1}{\partial x_i \partial x_j})\\ & = \text{Dual}(J_2 \cdot \frac{\partial f_1}{\partial x_i}, J_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j} + H_{2j} \cdot \frac{\partial f_1}{\partial x_i})\\ & = \text{Dual}(\frac{\partial (f_2 \circ f_1)}{\partial x_i}, \frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j}) \end{align}$$
Second-order AD
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸخ!¶…°persist_js_state·has_pluto_hook_featuresÂÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65fŠ§running§runtimeÎ Ïg¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65f¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸد©Ž°persist_js_state·has_pluto_hook_featuresÂÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3aŠ§running§runtimeÎ üÙ¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3a¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙZ
Forward Differentiation
Chain rule
$$\begin{align} \frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j} & = \frac{\partial}{\partial x_j} \left(\frac{\partial (f_2 \circ f_1)}{\partial x_i} \right)\\ & = \frac{\partial}{\partial x_j} \left(\partial f_2 \cdot \frac{\partial f_1}{\partial x_i} \right)\\ & = \left(\partial^2 f_2 \cdot \frac{\partial f_1}{\partial x_j} \right) \cdot \frac{\partial f_1}{\partial x_i} + \partial f_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j} \end{align}$$
In terms of the matrices $J_k = \partial f_k$ and $H_{kj} = \frac{\partial}{\partial x_j} J_k = \partial^2 f_k \cdot \frac{\partial s_{k-1}}{\partial x_j}$, it becomes
$$\begin{align} \frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j} & = H_{2j} \cdot \frac{\partial f_1}{\partial x_i} + J_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j} \end{align}$$
Neural network
If AD in an interpreted language like Python, the bottleneck of scalar-valued AD becomes the Python interpreter. Solved by using compiled language like Julia/Rust/C/C++/...
If each node is heap allocated (e.g. with
mallocin C), not cache friendly. Solved by a centralized storage for all the values of all nodes, e.g.MathOptInterface.Nonlinear.ReverseAD's values vector (TODO hyperlink)Not using optimized BLAS routines or GPU kernels for matrix-matrix multiplications. Not solveable for scalar-valued AD, need vector-valued AD!
Spaces and scalar products
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ'ª‹°persist_js_state·has_pluto_hook_featuresÂÙ$383461cb-9f51-42b2-ab76-d9580b39409eŠ§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$383461cb-9f51-42b2-ab76-d9580b39409e¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¡°persist_js_state·has_pluto_hook_featuresÂÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092Š§running§runtimeÎ Þ¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙŸ$b_k$ is a column vector, $S_i, X, W_i, Y$ are matrices.
mathjax_figure
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸخXí°persist_js_state·has_pluto_hook_featuresÂÙ$cc2a09a1-c949-4b09-816b-b49ba7ca8983Š§running§runtimeÎ Êï¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$cc2a09a1-c949-4b09-816b-b49ba7ca8983¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÛ ªmathjax_figure(draw; width, height)Luxor only draws the shapes, the labels are overlaid in HTML so that they are typeset by the MathJax of Pluto. draw(label) draws the shapes with the origin at the center and adds labels with label(text, x, y; color, fontsize, class). Labels of class equal are centered between the labels of class inner-product once MathJax has set their width.
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ ’À°persist_js_state·has_pluto_hook_featuresÂÙ$9ca3b930-3d9d-460c-9f20-e3c135477b05Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$9ca3b930-3d9d-460c-9f20-e3c135477b05¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¢×°persist_js_state·has_pluto_hook_featuresÂÙ$ac52550e-3287-427f-b957-ac61bc850f4dŠ§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$ac52550e-3287-427f-b957-ac61bc850f4d¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¡õ°persist_js_state·has_pluto_hook_featuresÂÙ$bebe2bb8-cf74-4eaa-85d1-712b990b5985Š§running§runtimeÎ æè¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$bebe2bb8-cf74-4eaa-85d1-712b990b5985¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚ˜For our Jacobians:
$$\begin{align} \langle t_k, r_k \rangle_k & = \langle \partial f_k(t_{k-1}), r_k \rangle_k\\ & = \langle t_{k-1}, \partial f_k^*(r_k) \rangle_{k-1} \end{align}$$
so by defining the backward pass with $r_{k-1} = \partial f_k^*(r_k)$, we get
$$\langle t_k, r_k \rangle = \langle t_{k-1}, r_{k-1} \rangle$$
By induction, we get
$$\langle t_0, r_0 \rangle = \langle t_K, r_K \rangle$$
Broadcasting (Vectorized way)
Consider applying a scalar function $f$ (e.g. $\tanh$ to each entry of a matrix $X$.)
$$\begin{align} (F(X))_{ij} & = f(X_{ij}) = f.(X)\\ G(\text{vec}(X)) \triangleq \text{vec}(F(X)) & = \text{vec}(f.(X))\\ J_G & = \text{Diag}(\text{vec}(f'.(X)))\\ J_G \text{vec}(T) & = \text{Diag}(\text{vec}(f'.(X))) \text{vec}(T)\\ \partial F[T] = \text{mat}(J_G \text{vec}(T)) & = f'.(X) \odot T\\ J_G^\top \text{vec}(R) & = \text{Diag}(\text{vec}(f'.(X))) \text{vec}(R)\\ \partial F^*[R] = \text{mat}(J_G^\top \text{vec}(R)) & = f'.(X) \odot R\\ \end{align}$$
Matrix multiplication (Scalar product way)
For the scalar product
$$\langle X, Y \rangle = \sum_{i,j} X_{ij} Y_{ij} = \sum_{i,j} (X \odot Y)_{ij} = \langle \text{vec}(X), \text{vec}(Y) \rangle = \text{tr}(X Y^\top), \quad A^* = A^\top$$
Now, given a forward tangent $T$ and a reverse tangent $R$
$$\begin{align} \langle AT, R \rangle & = \langle T, A^\top R \rangle \end{align}$$
so the backward pass computes $A^\top R$.
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ!Á °persist_js_state·has_pluto_hook_featuresÂÙ$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4Š§running§runtimeÎ r-¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚôWe build the graph of
$$f(x_1, x_2) = x_2 e^{x_1} \sqrt{x_1 + x_2 e^{x_2}}$$
at $x_1 = 1$, $x_2 = 2$. The slider first reveals the forward evaluation of each node $v_i$, then accumulates the adjoints $\bar{v}_i = \partial f / \partial v_i$ backwards, one node at a time.
Disadvantages of scalar-valued nodes
Product of Jacobians
Suppose that we need to differentiate a composition of functions: $(f_n \circ f_{n-1} \circ \cdots \circ f_2 \circ f_1)(w)$. For each function, we can compute a jacobian given the value of its input. So, during a forward pass, we can compute all jacobians. We now just need to take the product of these jacobians:
$$J_n J_{n-1} \cdots J_2 J_1$$
While the product of matrices is associative, its computational complexity depends on the order of the multiplications! Let $d_i \times d_{i - 1}$ be the dimension of $J_i$.
Neural network
Acknowledgements and further readings
Dualis inspired from ForwardDiffNodeis inspired from microgradHere is a good intro to AD
Figures are from the The Elements of Differentiable Programming book
Matrix multiplication (Vectorized way)
$$\begin{align} \text{vec}(AXB) & = (B^\top \otimes A) \text{vec}(X) & (A \otimes B)^\top & = A^\top \otimes B^\top \end{align}$$
Deriving VJP and JVP for matrix product:
$$\begin{align} F(X) & = AX\\ G(\text{vec}(X)) \triangleq \text{vec}(F(X)) & = (I \otimes A) \text{vec}(X)\\ J_G & = (I \otimes A)\\ J_G^\top \text{vec}(R) & = (I \otimes A^\top) \text{vec}(R)\\ \partial F^*[R] = \text{mat}(J_G^\top \text{vec}(R)) & = A^\top R\\ \end{align}$$
Comparison
Forward mode of $f(x)$ with dual numbers
Dual.(x, v)computes Jacobian-Vector Product (JVP) $J_f(x) \cdot v$Reverse mode of $f(x)$ computes Vector-Jacobian Product (VJP) $v^\top J_f(x)$ or in other words $J_v(x)^\top v$
Second-order
Consider a function $f: \mathbb{R}^n \to \mathbb{R}$, we want to compute the Hessian $\nabla^2 f(x)$, defined by
$$(\nabla^2 f(x))_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$$
Application: Given the optimization problem:
$$\begin{align} \min_x f(x)\\ g_i(x) & = 0 \quad \forall i \in \{1, \ldots, m\} \end{align}$$
The Hessian of the Lagrangian $\mathcal{L}(x, \lambda) = f(x) - \lambda_1 g_1(x) - \cdots - \lambda_m g_m(x)$ is obtained as
$$\nabla_x^2 \mathcal{L}(x, \lambda) = \nabla^2 f(x) - \sum_{i=1}^m \lambda_i \nabla^2 g_i(x)$$
Forward mode
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ$'ª°persist_js_state·has_pluto_hook_featuresÂÙ$28df733e-7db9-4e78-9121-52d8e6ca7591Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$28df733e-7db9-4e78-9121-52d8e6ca7591¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¡j°persist_js_state·has_pluto_hook_featuresÂÙ$df1f29ce-026d-4935-83aa-0d597cb166ddŠ§running§runtimeÎ ·é¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$df1f29ce-026d-4935-83aa-0d597cb166dd¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸخ>^}°persist_js_state·has_pluto_hook_featuresÂÙ$af404768-0663-4bc3-81dd-6931b3a486beŠ§running§runtimeÎ ¼•¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$af404768-0663-4bc3-81dd-6931b3a486be¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚÍConsider $f(x) = f_3(f_2(f_1(x)))$. If we don't have the expression of $f_1$ but we can only evaluate $f_i(x)$ or $f'(x)$ for a given $x$ ? The chain rule gives
$$f'(x) = f_3'(f_2(f_1(x))) \cdot f_2'(f_1(x)) \cdot f_1'(x).$$
Let's define $s_0 = x$ and $s_{k} = f_k(s_{k-1})$, we now have:
$$f'(x) = f_3'(s_2) \cdot f_2'(s_1) \cdot f_1'(s_0).$$
Two choices here:
$$\begin{align*} & \text{Forward} & & \text{Reverse}\\ t_0 & = 1 & r_3 & = 1\\ t_{k} & = f_k'(s_{k-1}) \cdot t_{k-1} & r_k & = r_{k+1} \cdot f_{k+1}'(s_{k})\\ \end{align*}$$
$f_1$ and $f_2$ now contribute to different entries of the output of $\text{dup}$ so there is nothing to combine:
$$\partial g_2^* \begin{bmatrix} r_1 \\ r_2 \end{bmatrix} = \begin{bmatrix} \partial f_1^* & 0 \\ 0 & \partial f_2^* \end{bmatrix} \begin{bmatrix} r_1 \\ r_2 \end{bmatrix} = \begin{bmatrix} \partial f_1^* r_1 \\ \partial f_2^* r_2 \end{bmatrix}$$
What remains is the reverse pass of $\text{dup}$... but $\text{dup}$ is a function like any other: we can compute its Jacobian!
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ%S°persist_js_state·has_pluto_hook_featuresÂÙ$f010e781-f41e-4861-af1c-32cf5a76ce4dŠ§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$f010e781-f41e-4861-af1c-32cf5a76ce4d¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|£D°persist_js_state·has_pluto_hook_featuresÂÙ$8b8682e4-8adb-4fb1-a013-054b1d9750d7Š§running§runtimeΠ੸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$8b8682e4-8adb-4fb1-a013-054b1d9750d7¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙ$f_1 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¸Ø!¾s°persist_js_state·has_pluto_hook_featuresÂÙ$402f0d1d-0a3f-404a-8019-afc92d154c85Š§running§runtimeÎ ÆѸdepends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$402f0d1d-0a3f-404a-8019-afc92d154c85¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙHInner products
Utils
Spaces and scalar products
Reverse on forward
Forward pass: Given $s_1 = \text{Dual}(f_1(x), \frac{\partial f_1}{\partial x_{\textcolor{red}i}})$
Forward mode computes $s_2 = f_2(s_1) = (f_2(f_1(x)), J_2 \cdot \frac{\partial f_1}{\partial x_{\textcolor{red}i}}) = ((f_2 \circ f_1)(x), \partial (f_2 \circ f_1) / \partial x_{\textcolor{red}i})$
The reverse mode computes the local Jacobian of this operation : $\partial s_2 / \partial s_1$. The local Jacobian of $(s_1)_1 \mapsto f_2((s_1)_1)$ is $J_2$. The local Jacobian of $s_1 \mapsto \partial f_2((s_1)_1) (s_1)_2$ is $(\partial^2 f_2((s_1)_1) \cdot (s_1)_2, \partial f_2((s_1)_1)) = (\partial^2 f_2(f_1(x)) \cdot \frac{\partial f_1}{\partial x_{\textcolor{red}i}}, \partial f_2(f_1(x)) = (H_{2\color{red}i}, J_2)$
Reverse pass:
$$\begin{align} (r_1)_1 & = (r_2)_1 \cdot J_2 + (r_2)_2 \cdot H_{2{\color{red}i}}\\ (r_1)_2 & = (r_2)_2 \cdot J_2 \end{align}$$
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ ]’°persist_js_state·has_pluto_hook_featuresÂÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¤U°persist_js_state·has_pluto_hook_featuresÂÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4eŠ§running§runtimeÍDY¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4e¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙl1×178 Matrix{Float64}:
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 … 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0¤mimeªtext/plain¬rootassignee§targets²last_run_timestampËAڸذžC°persist_js_state·has_pluto_hook_featuresÂÙ$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08Š§running§runtimeΪ!@¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÛ Ë
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸدñ3°persist_js_state·has_pluto_hook_featuresÂÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¡º°persist_js_state·has_pluto_hook_featuresÂÙ$d492a165-3e70-4c18-b0d9-57fc2e8a1b96Š§running§runtimeÎœe¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$d492a165-3e70-4c18-b0d9-57fc2e8a1b96¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸدÎZÙ°persist_js_state·has_pluto_hook_featuresÂÙ$4174d41c-877b-4878-b76e-442991907af0Š§running§runtimeÍ"–¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$4174d41c-877b-4878-b76e-442991907af0¹depends_on_disabled_cells¦queued¤logs¦output†¤body…¦prefix«Dual{Int64}¨elements’’¥value’¢-3ªtext/plain’ªderivative’£-10ªtext/plain¤type¦struct¬prefix_short¤Dual¨objectid°6ec0b84851636e1e¤mimeÙ!application/vnd.pluto.tree+object¬rootassigneeÀ²last_run_timestampËAڸؽ°persist_js_state·has_pluto_hook_featuresÂÙ$9988fc4a-cedc-499b-a334-048cc13de000Š§running§runtimeΠظdepends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$9988fc4a-cedc-499b-a334-048cc13de000¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙ$abs (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¸Ølºb°persist_js_state·has_pluto_hook_featuresÂÙ$24d52ef8-927e-46c8-b455-74ccdb33d3caŠ§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$24d52ef8-927e-46c8-b455-74ccdb33d3ca¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|£ì°persist_js_state·has_pluto_hook_featuresÂÙ$fa886ef3-192f-4daa-b59b-aca0c6365111Š§running§runtimeÎ ¥\c¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$fa886ef3-192f-4daa-b59b-aca0c6365111¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚ
fanout_figure
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸخ‹;T°persist_js_state·has_pluto_hook_featuresÂÙ$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47Š§running§runtimeÎ
wjȸdepends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸخ0Wë°persist_js_state·has_pluto_hook_featuresÂÙ$81deb227-a822-4857-a584-a51cc8ff51f4Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$81deb227-a822-4857-a584-a51cc8ff51f4¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز| Ü°persist_js_state·has_pluto_hook_featuresÂÙ$c733ca7e-b57e-4218-9bd4-238ab5749143Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$c733ca7e-b57e-4218-9bd4-238ab5749143¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¢s°persist_js_state·has_pluto_hook_featuresÂÙ$88534196-9f4a-430c-a534-805177ba718dŠ§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$88534196-9f4a-430c-a534-805177ba718d¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¢D°persist_js_state·has_pluto_hook_featuresÂÙ$310d6a10-b887-4dd0-ba93-f901726ddbbcŠ§running§runtimeÎ wR¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$310d6a10-b887-4dd0-ba93-f901726ddbbc¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚSÂfanout_figure(; dup, width = 640, height = 260)Draw the forward and reverse passes for x used by both f_1 and f_2 with mathjax_figure. If dup is true, x is first duplicated with dup so that we get a chain.
The adjoint of a linear map $A : \mathcal{S}_1 \to \mathcal{S}_2$ for scalar products $\langle \cdot, \cdot \rangle_1$ for $\mathcal{S}_1$ and $\langle \cdot, \cdot \rangle_2$ for $\mathcal{S}_2$ is the linear map $A^*$ such that
$$\forall x \in \mathcal{S}_1, y \in \mathcal{S}_2, \qquad \langle A(x), y \rangle_2 = \langle x, A^*(y) \rangle_1.$$
Issues with scalar AD
Evaluation
Reverse on reverse
Forward pass (2nd):
Forward pass computes $s_2 = f_2(s_1)$ → Jacobian $\partial s_2 / \partial s_1 = J_2$
Local Jacobian $J_2 = \partial f_2(s_1)$ → The Jacobian is the 3D array $\partial J_2 / \partial s_1 = \partial^2 f_2$
Backward pass computes $r_1 = r_2 \cdot \partial f_2(s_1)$ → Jacobian of $(s_1, r_2) \mapsto r_2 \cdot \partial f_2(s_1)$ is $(r_2 \cdot \partial^2 f_2(s_1), \partial f_2(s_1)) = (r_2 \cdot \partial^2 f_2, J_2)$. Note that here $r_2 \in \mathbb{R}^{d_k}$ is multiplying the first dimension of the tensor $\partial^2 f_2(s_1) \in \mathbb{R}^{d_k \times d_{k-1} \times d_{k-1}}$ so the result is a symmetric matrix of dimension $\mathbb{R}^{d_{k-1} \times d_{k-1}}$
Reverse pass (2nd): The result is $r_0$, let $\dot{r}_k$ be the second-order reverse tangent for $r_k$ and $\dot{s}_k$ be the second-order reverse tangent of $s_k$. We have
$$\begin{align} \dot{r}_2 & = J_2 \cdot \dot{r}_1\\ \dot{s}_1 & = (r_2 \cdot \partial^2 f_2(s_1)) \cdot \dot{r}_1 + \dot{s}_2 \cdot J_2 \end{align}$$
Forward mode
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ&},°persist_js_state·has_pluto_hook_featuresÂÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633aŠ§running§runtimeÎ ¶ˆ¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633a¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚ/Let $(\mathcal{S}_k)_k$ be the spaces for which $s_k, t_k, r_k \in \mathcal{S}_k$ and $f_k : \mathcal{S}_{k-1} \to \mathcal{S}_{k}$.
In the vectorized case, the Jacobian $\partial f_k$ may be expressed more naturally as a linear map $\partial f_k : \mathcal{S}_{k-1} \to \mathcal{S}_k$ and $t_k = \partial f_k(t_{k-1})$. To obtain the matrix-form of the Jacobian, we need to fix a basis for $\mathcal{S}_{k-1}$ and $\mathcal{S}_k$ and vectorize $t_k$ and $t_{k-1}$ according to that basis which is often cumbersome.
Instead, we choose a scalar products $\langle \cdot, \cdot \rangle_k$ for $\mathcal{S}_k$ and the adjoint of the Jacobians according to these bases in the backward pass. For instance, if $\mathcal{S}_K = \mathcal{R}$, $t_K = \langle t_K, 1 \rangle_K$ for the canonical scalar product of $\mathcal{R}$ so we pick $r_K = 1$. We then search for a way to obtain $r_0(r_K)$ so that $\langle t_0, r_0(r_K) \rangle_0 = \langle \partial f_K(\cdots\partial f_1(t_0), r_K\rangle_K$. Then, for instance if $\mathcal{S}_0 = \mathbb{R}^n$, with the canonical scalar product $\langle x, y \rangle_0 = x^\top y$, we have $\partial_i f(s_0) = \langle e_i, r_0(r_K) \rangle_0 = (r_0(r_K))_i$.
From forward to reverse mode
Concatenating the two columns gives the full Jacobian. We can then factor $\partial f_2(s_1)$ and then $\partial f_1(s_0)$ out of the concatenation:
$$\begin{align} \partial f(s_0) & = \begin{bmatrix} 2 & 4 \end{bmatrix}\\ & = \begin{bmatrix} \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \xcancel{\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}} \end{align}$$
The concatenation of the one-hot tangents is the identity so it disappears! We are left with $\partial f_2(s_1) \cdot \partial f_1(s_0)$ that we can compute from left to right in a single pass: this is reverse mode!
$$\begin{align} r_2 & = 1 & r_1 & = r_2 \cdot \partial f_2(s_1) = \begin{bmatrix} 1 & 1 \end{bmatrix} & r_0 & = r_1 \cdot \partial f_1(s_0) = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} = \begin{bmatrix} 2 & 4 \end{bmatrix} \end{align}$$
Forward tangents
Nodes with vector value
Batching
Applying a neural network to 2 inputs can be done simultaneously so that matrix-vector products become matrix-matrix product, which better exploit the parallelization capabilities of CPUs and GPUs. Let $X = \begin{bmatrix}x^{(1)} & x^{(2)}\end{bmatrix}$:
$$\begin{align} \begin{bmatrix} \sigma(W_1 x^{(1)}) & \sigma(W_1 x^{(2)}) \end{bmatrix} &= \sigma(\begin{bmatrix} W_1 x^{(1)} & W_1 x^{(2)} \end{bmatrix})\\ & = \sigma(W_1 \begin{bmatrix} x^{(1)} & x^{(2)} \end{bmatrix})\\ & = \sigma(W_1X) \end{align}$$
With bias, let $\mathbf{1}^\top = \begin{bmatrix} 1 & 1 \end{bmatrix}$:
$$\begin{bmatrix} W_1 x^{(1)} + b_1 & W_1 x^{(2)} + b_1 \end{bmatrix} = \begin{bmatrix} W_1 x^{(1)} & W_1 x^{(2)} \end{bmatrix} + \begin{bmatrix} b_1 & b_1 \end{bmatrix} = W_1 X + b_1 \mathbf{1}^\top$$
Automatic Differentiation
Benoît Legat
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸجú¸€°persist_js_state·has_pluto_hook_featuresÂÙ$906e5199-f2d2-4816-a195-6d2b1dee9403Š§running§runtimeÎ ½Ü¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$906e5199-f2d2-4816-a195-6d2b1dee9403¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙNWine example ðŸ·
🚀 GPU acceleration ⚡
Two equivalent approaches
Memory usage of forward mode
Reverse Differentiation
Reverse mode on a DAG
Forward mode: we know how to propagate $t$ through any graph.
Reverse mode on a chain: we know that $r_{k-1} = \partial f_k^* r_k$.
Reverse mode on a DAG: â“
Consider the simplest DAG that is not a chain: $x$ is used by both $f_1$ and $f_2$. The reverse pass brings two contributions $\partial f_1^* r_1$ and $\partial f_2^* r_2$ back to the same $x$.
Forward on reverse
Forward pass: Given $s_1 = \text{Dual}(f_1(x), \frac{\partial f_1}{\partial x_j})$
Compute $s_2 = f_2(s_1)$ → same as forward on forward
Compute $J_{f_2}(s_1)$ → same as forward on forward
Reverse pass: Given $r_2 = \text{Dual}((r_2)_1, (r_2)_2)$, compute
$$\begin{align} r_2 \cdot J_2 & = \text{Dual}((r_2)_1, (r_2)_2) \cdot \text{Dual}(J_2, H_{2j}) \cdot \\ & = \text{Dual}( (r_2)_1 \cdot J_2, (r_2)_2 \cdot J_2 + (r_2)_1 \cdot H_{2j}) \end{align}$$
Two different takes on the multivariate chain rule
The chain rule gives us
$$\frac{\partial f_3}{\partial x} (f_1(x), f_2(x)) = \partial_1 f_3(s_1, s_2) \cdot \frac{\partial s_1}{\partial x} + \partial_2 f_3(s_1, s_2) \cdot \frac{\partial s_2}{\partial x}$$
To compute this expression, we need the values of $s_1(x)$ and $s_2(x)$ as well as the derivatives $\partial s_1 / \partial x$ and $\partial s_2 / \partial x$.
Common to forward and reverse: Given $s_1, s_2$, computes local derivatives $\partial_1 f_3(s_1, s_2)$ and $\partial_2 f_3(s_1, s_2)$, shortened $\partial_1 f_3, \partial_2 f_3$ for conciseness.
Putting everything together
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ"ð|°persist_js_state·has_pluto_hook_featuresÂÙ$e27db9a0-09ff-4ee5-a807-d98933b6bcf1Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$e27db9a0-09ff-4ee5-a807-d98933b6bcf1¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|¢ù°persist_js_state·has_pluto_hook_featuresÂÙ$4d76422e-711c-4f3e-87ab-0ce851bac064Š§running§runtimeÀ¸depends_on_skipped_cellsµpublished_object_keys§erroredçcell_idÙ$4d76422e-711c-4f3e-87ab-0ce851bac064¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyƒ£msg³InterruptException:ªstacktrace«plain_error³InterruptException:¤mimeÙ'application/vnd.pluto.stacktrace+object¬rootassigneeÀ²last_run_timestampËAڸز|£Ï°persist_js_state·has_pluto_hook_featuresÂÙ$6c441843-d6da-46b1-9463-d133ba74e66aŠ§running§runtimeÎ/{
¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$6c441843-d6da-46b1-9463-d133ba74e66a¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÚh =
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸذ(æÁ°persist_js_state·has_pluto_hook_featuresÂÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739Š§running§runtimeÎyiè¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸد·¯:°persist_js_state·has_pluto_hook_featuresÂÙ$73ba544c-616a-4db1-b91d-0b20a7b8924bŠ§running§runtimeÎ Ë«¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$73ba544c-616a-4db1-b91d-0b20a7b8924b¹depends_on_disabled_cells¦queued¤logs¦output†¤bodyÙdMemory usage of reverse mode
Broadcasting (Scalar product way)
$$\begin{align} \langle f'.(X) \odot T, R \rangle = \langle T, f'.(X) \odot R \rangle. \end{align}$$
From forward to reverse mode
Consider $f(x_1, x_2) = x_1^2 + x_2^2$ at $s_0 = (1, 2)$. We decompose it as $f = f_2 \circ f_1$ and compute the local Jacobians:
$$\begin{align} f_1(x_1, x_2) & = \begin{bmatrix} x_1^2 \\ x_2^2 \end{bmatrix} & s_1 & = f_1(s_0) = \begin{bmatrix} 1 \\ 4 \end{bmatrix} & \partial f_1(s_0) & = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix}\\ f_2(x_1, x_2) & = x_1 + x_2 & s_2 & = f_2(s_1) = 5 & \partial f_2(s_1) & = \begin{bmatrix} 1 & 1 \end{bmatrix} \end{align}$$
Forward mode computes one column of the Jacobian $\partial f(s_0)$ for each one-hot tangent $t_0$:
$$\begin{align} t_0 & = \begin{bmatrix} 1 \\ 0 \end{bmatrix} & t_1 & = \partial f_1(s_0) \cdot t_0 = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 0 \end{bmatrix} & t_2 & = \partial f_2(s_1) \cdot t_1 = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 \\ 0 \end{bmatrix} = 2\\ t_0 & = \begin{bmatrix} 0 \\ 1 \end{bmatrix} & t_1 & = \partial f_1(s_0) \cdot t_0 = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 4 \end{bmatrix} & t_2 & = \partial f_2(s_1) \cdot t_1 = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 0 \\ 4 \end{bmatrix} = 4 \end{align}$$
Notation
Let $f_k : \mathbb{R}^{d_{k-1}} \to \mathbb{R}^{d_k}$. $\partial f_k \triangleq \partial f_k(s_{k-1}) \in \mathbb{R}^{d_k \times d_{k-1}}$, $\partial^2 f_k \triangleq \partial^2 f_k(s_{k-1}) \in \mathbb{R}^{d_k \times d_{k-1} \times d_{k-1}}$ is a 3D array/tensor.
Given $v \in \mathbb{R}^{d_{k-1}}$, by the product $(\partial^2 f_k \cdot v) \in \mathbb{R}^{d_k \times d_{k-1}}$ , we denote the contraction of the 3rd (or 2nd since the tensor is symmetric over its last 2 dimensions) dimension:
$$(\partial^2 f_k \cdot v)_{ij} = \sum_{l = 1}^{d_{k-1}} (\partial^2 f_k)_{ijl} \cdot v_l$$
Given $u \in \mathbb{R}^{d_k}$, by the product $(u \cdot \partial^2 f_k) \in \mathbb{R}^{d_{k-1} \times d_{k-1}}$ , we denote the contraction of the 1st dimension.
$$(u \cdot \partial^2 f_k)_{ij} = \sum_{l = 1}^{d_k} u_l \cdot (\partial^2 f_k)_{lij}$$
Both $\partial^2 f_k \cdot v$ and $u \cdot \partial^2 f_k$ are matrices so then we're back to matrix notations.
$$f(x_1, x_2) = x_2 e^{x_1} \sqrt{x_1 + x_2 e^{x_2}}$$
The End
¤mime©text/html¬rootassigneeÀ²last_run_timestampËAڸةšÒJ°persist_js_state·has_pluto_hook_featuresÂÙ$895f4b81-51fe-4a1c-b958-1294743e3a9bŠ§running§runtimeÎ 0Ž¸depends_on_skipped_cellsµpublished_object_keys§errored§cell_idÙ$895f4b81-51fe-4a1c-b958-1294743e3a9b¹depends_on_disabled_cells¦queued¤logs¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAڸخœ’´°persist_js_state·has_pluto_hook_features±published_objects€¥bonds€´cell_execution_orderÜ ªÙ$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6Ù$40baa108-eb68-433f-9917-ac334334f198Ù$77a7de14-87d2-11ef-21ef-937b8239db5bÙ$e46fb3ff-b26f-4efb-aaa1-760e80017797Ù$af404768-0663-4bc3-81dd-6931b3a486beÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3aÙ$586c2c6b-4a53-46d5-924b-c843e4c09859Ù$1c44706b-fd7d-4826-84a3-73e2db5adaddÙ$45051873-3f0d-49b0-a9a6-bfde240594aaÙ$3803804a-f44d-4f56-bd59-fb1401d8fb9eÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6Ù$4174d41c-877b-4878-b76e-442991907af0Ù$8b8682e4-8adb-4fb1-a013-054b1d9750d7Ù$e211118d-eba8-467e-ae4a-665f1df02934Ù$bc93fb96-1097-4b22-a077-558f5662efecÙ$85fc455c-36cf-4a4c-aa64-84a827884693Ù$277bd2ce-fa7f-4288-be8a-0ddd8f23635cÙ$fa5dba01-a3f7-452c-877e-352d578ecf51Ù$69c08fab-c317-462c-817c-3f841a8a0941Ù$66bd8159-581d-471c-841d-8e42e0de0496Ù$37bfd8ed-8983-4a89-9ba5-a90a35151729Ù$7aba4481-bed6-44b7-85e3-3b2deb2b2fd7Ù$885bc5c9-aefc-4d8a-a4da-6062c64eaa41Ù$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0Ù$90850509-463d-44c7-88ae-4406aebd4be1Ù$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4Ù$1d56075c-e28d-46c9-9a0a-210079172388Ù$19578219-c6dd-4322-a2f4-44088ef640afÙ$7f75e3f3-c4e2-402d-be7b-336a4f65042aÙ$e88ada81-d37f-4811-aa17-17eb7dd9d8a7Ù$32a0bec0-4f1d-4dba-a5db-422741f88808Ù$1a0026be-7975-478c-a447-f66ea95f551dÙ$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0fÙ$73ba544c-616a-4db1-b91d-0b20a7b8924bÙ$dc4feb58-d2cf-4a97-aaed-7f4593fc9732Ù$2b631fcd-2703-42df-8a75-2fdff64b3311Ù$3556d366-0bc7-4239-b4f6-3f9bd28780e0Ù$69ae57b4-4e4c-44a2-aca7-d0fff89b9566Ù$9988fc4a-cedc-499b-a334-048cc13de000Ù$ceaeb177-7a6a-4062-9659-56bebce0e77bÙ$e50f8f52-a73f-4186-af5e-b4ca2c021142Ù$9862c791-31e8-4d59-8610-a929d72ea9c3Ù$6bd5ea51-54c4-46ba-8eb3-6427225e5249Ù$4d4e11a2-b6a1-490f-9c7b-ad617dbb395bÙ$533bf4bc-494b-47c2-91b6-7b30221765f5Ù$3f1068e0-97e2-444a-961d-2af55a4d797bÙ$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2Ù$e121f72b-fe6d-491a-ab03-ef92154c61caÙ$54457653-f029-4368-8bc0-15d69f22bfcdÙ$8194cce9-115f-44b4-9f28-caa88b764185Ù$292b182f-b8a4-4ac6-89ad-ef71d40eca1fÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092Ù$b92d17a9-8481-458a-bc0a-efb7333cbc6eÙ$9527686f-24e1-40bb-9a5d-22575aafec9bÙ$0980ef52-526d-421d-8c63-04272f16cbddÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633aÙ$c1ae773e-0f2d-4c1c-bae2-25f45bb28fbbÙ$068ef840-04cb-4de4-9512-721f57e5750eÙ$7331c447-c0a6-4a47-af00-36efc449d9d9Ù$bebe2bb8-cf74-4eaa-85d1-712b990b5985Ù$29287c62-e892-448f-a9d5-12785ae4a02fÙ$5f6529a1-4ace-4dd0-a7e2-f51070eab695Ù$402f0d1d-0a3f-404a-8019-afc92d154c85Ù$e9d6af05-63f7-4d31-85e2-b0233df23608Ù$802edb3a-4809-4c50-920b-25f7bdc255ddÙ$98db9022-f8ff-4af3-9c81-89cf09771928Ù$8c202da6-1e13-43b8-a22b-94badcef2934Ù$1994bf51-adf1-4b07-ab4c-f47552d90826Ù$906e5199-f2d2-4816-a195-6d2b1dee9403Ù$f5d3714d-3900-4dbe-9079-978a44584d1dÙ$6ddc06c0-3f5d-4cc9-8060-dd6997e0f662Ù$0e13e63d-fd08-4cc1-aa37-851c537afbefÙ$2adc9595-8829-4d35-be90-a7718c2e7ce7Ù$53b21ec0-28e9-46cd-a92e-8afc189c3a11Ù$778c40ff-4c9e-42fb-92a6-1e376837f6efÙ$43d2559f-8902-4c54-8fdf-cb268b6f868cÙ$35f8cf4f-3fcb-4e27-9462-244406d7800eÙ$17c91ea8-acb7-4bbd-b0b0-0f8193f45303Ù$9bbbda1f-74a6-458b-a084-9d034d6c291fÙ$03b6aa6d-7517-4906-9430-302516d0653bÙ$7d79ff81-59e0-41f0-b2fe-70b41f44591fÙ$da5895e7-af99-46ff-9f53-36529d1ca456Ù$9415a6ed-c05e-4487-b0be-f342ec7424cdÙ$3a2132e7-7d69-42d7-89d3-b2d7679ad74fÙ$5a79c09e-2a33-43d1-a5dc-caba3db467ddÙ$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6Ù$c1da4130-5936-499f-bb9b-574e01136ecaÙ$5b85a063-cf0f-4afa-89f4-420d7350ecc3Ù$b16f6225-1949-4b6d-a4b0-c5c230eb4c7fÙ$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47Ù$607000ef-fb7f-4204-b543-3cb6bb75ed71Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47Ù$df1f29ce-026d-4935-83aa-0d597cb166ddÙ$79df35ff-3718-4a9c-a384-4e8c12175ab8Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8eÙ$310d6a10-b887-4dd0-ba93-f901726ddbbcÙ$fa886ef3-192f-4daa-b59b-aca0c6365111Ù$6c441843-d6da-46b1-9463-d133ba74e66aÙ$a9a4841d-18cc-4386-aa38-017771650afbÙ$7f459370-eb6f-4d82-bf99-1528e136ca5cÙ$895f4b81-51fe-4a1c-b958-1294743e3a9bÙ$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72bÙ$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56Ù$ee3299e2-3367-4c68-a050-c26650a97c8cÙ$9a07a949-4ade-4325-8ae6-b05bdbbfae00Ù$9661a3b9-ae42-466e-967d-69230b9769b3Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccefÙ$722ad63a-c2ac-4ed6-b268-41d0f8b745f1Ù$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3Ù$f9a37c14-5b62-4e8d-96a0-2c41db73e65fÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16Ù$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08Ù$cbfc0129-9361-4edb-a467-1456a1f3aeaeÙ$fa20b8db-9ac7-490d-b8d8-8d57469d24e4Ù$cc2a09a1-c949-4b09-816b-b49ba7ca8983Ù$d2b8fa5c-c604-4093-a2dd-5c95f2eaa676Ù$d1dbdd3f-9782-4fba-8c4e-819f152e6c30Ù$673c3acc-0009-416a-91bb-f57c1fe8eefcÙ$e8c60922-5bbf-45b5-8311-18c8f8525623Ù$2f8baccc-19d1-44d6-b71f-0243fd8696baÙ$cd6d807d-6238-44ce-9267-1614679f527aÙ$40ed6c94-d2f9-4225-80b3-9060f04f8971Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6abÙ$db7aaa4c-5568-4619-a9c4-00c7916c321aÙ$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3Ù$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4eÙ$2fcf25d2-fd51-4c13-b57c-86236aceead2Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16Ù$a580ef44-234a-4ed1-b007-920651415427Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282Ù$81deb227-a822-4857-a584-a51cc8ff51f4Ù$8deca676-8a0b-41eb-b7a0-4d65e1158b0bÙ$383461cb-9f51-42b2-ab76-d9580b39409eÙ$10e5c137-64a9-4282-b3fe-26e625625ae4Ù$83ef86e0-bcfb-42ee-a574-16758606423aÙ$28df733e-7db9-4e78-9121-52d8e6ca7591Ù$626abc7c-87ef-4838-9f0a-294cf0a4be6aÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853bÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496Ù$c73f79c6-a28f-4c7a-89e5-8d70a245a210Ù$ac52550e-3287-427f-b957-ac61bc850f4dÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779Ù$74063eb5-be06-466a-a2f1-e266c35295eaÙ$88534196-9f4a-430c-a534-805177ba718dÙ$347a1500-43aa-4078-9ea3-480bc8a4c291Ù$c733ca7e-b57e-4218-9bd4-238ab5749143Ù$33bdaa23-707e-4227-b936-c5d7aaf2c48eÙ$7921c4c6-56b8-4c6f-a9be-cd1d9984680bÙ$edad5700-9b04-47c6-90f8-b6bac0897340Ù$9ca3b930-3d9d-460c-9f20-e3c135477b05Ù$e27db9a0-09ff-4ee5-a807-d98933b6bcf1Ù$95eb9960-89f0-4edc-8943-77a75bce2b80Ù$9afd31c9-e938-417a-8c3f-e0d1ba88f95bÙ$f010e781-f41e-4861-af1c-32cf5a76ce4dÙ$4317bba0-723f-4cdc-9d52-67033540a8d2Ù$6ded46e6-1c89-4875-addb-8c709e949bb1Ù$78d8c8c9-568d-472a-9f06-a50b1cf2384bÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3fÙ$4d76422e-711c-4f3e-87ab-0ce851bac064Ù$24d52ef8-927e-46c8-b455-74ccdb33d3caÙ$4de9baec-f444-47b8-b9e6-7f3d9e9609b1Ù$16f611e8-8e41-43f3-830c-9976cb720b9fÙ$200e2b1b-065a-4d60-b5fd-86700e4c811aÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361®process_statusªno_processjulia_version§v1.13.3ªcell_orderÜ ªÙ$40baa108-eb68-433f-9917-ac334334f198Ù$77a7de14-87d2-11ef-21ef-937b8239db5bÙ$e46fb3ff-b26f-4efb-aaa1-760e80017797Ù$af404768-0663-4bc3-81dd-6931b3a486beÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3aÙ$fa20b8db-9ac7-490d-b8d8-8d57469d24e4Ù$586c2c6b-4a53-46d5-924b-c843e4c09859Ù$1c44706b-fd7d-4826-84a3-73e2db5adaddÙ$45051873-3f0d-49b0-a9a6-bfde240594aaÙ$3803804a-f44d-4f56-bd59-fb1401d8fb9eÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6Ù$4174d41c-877b-4878-b76e-442991907af0Ù$8b8682e4-8adb-4fb1-a013-054b1d9750d7Ù$e211118d-eba8-467e-ae4a-665f1df02934Ù$bc93fb96-1097-4b22-a077-558f5662efecÙ$85fc455c-36cf-4a4c-aa64-84a827884693Ù$cc2a09a1-c949-4b09-816b-b49ba7ca8983Ù$277bd2ce-fa7f-4288-be8a-0ddd8f23635cÙ$fa5dba01-a3f7-452c-877e-352d578ecf51Ù$69c08fab-c317-462c-817c-3f841a8a0941Ù$8deca676-8a0b-41eb-b7a0-4d65e1158b0bÙ$66bd8159-581d-471c-841d-8e42e0de0496Ù$6c441843-d6da-46b1-9463-d133ba74e66aÙ$383461cb-9f51-42b2-ab76-d9580b39409eÙ$37bfd8ed-8983-4a89-9ba5-a90a35151729Ù$a9a4841d-18cc-4386-aa38-017771650afbÙ$7aba4481-bed6-44b7-85e3-3b2deb2b2fd7Ù$10e5c137-64a9-4282-b3fe-26e625625ae4Ù$885bc5c9-aefc-4d8a-a4da-6062c64eaa41Ù$d2b8fa5c-c604-4093-a2dd-5c95f2eaa676Ù$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0Ù$d1dbdd3f-9782-4fba-8c4e-819f152e6c30Ù$83ef86e0-bcfb-42ee-a574-16758606423aÙ$90850509-463d-44c7-88ae-4406aebd4be1Ù$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4Ù$673c3acc-0009-416a-91bb-f57c1fe8eefcÙ$28df733e-7db9-4e78-9121-52d8e6ca7591Ù$626abc7c-87ef-4838-9f0a-294cf0a4be6aÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853bÙ$1d56075c-e28d-46c9-9a0a-210079172388Ù$19578219-c6dd-4322-a2f4-44088ef640afÙ$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16Ù$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08Ù$f9a37c14-5b62-4e8d-96a0-2c41db73e65fÙ$7f75e3f3-c4e2-402d-be7b-336a4f65042aÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496Ù$c73f79c6-a28f-4c7a-89e5-8d70a245a210Ù$ac52550e-3287-427f-b957-ac61bc850f4dÙ$e88ada81-d37f-4811-aa17-17eb7dd9d8a7Ù$32a0bec0-4f1d-4dba-a5db-422741f88808Ù$1a0026be-7975-478c-a447-f66ea95f551dÙ$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8eÙ$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0fÙ$e8c60922-5bbf-45b5-8311-18c8f8525623Ù$73ba544c-616a-4db1-b91d-0b20a7b8924bÙ$2f8baccc-19d1-44d6-b71f-0243fd8696baÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779Ù$dc4feb58-d2cf-4a97-aaed-7f4593fc9732Ù$74063eb5-be06-466a-a2f1-e266c35295eaÙ$607000ef-fb7f-4204-b543-3cb6bb75ed71Ù$88534196-9f4a-430c-a534-805177ba718dÙ$2b631fcd-2703-42df-8a75-2fdff64b3311Ù$9988fc4a-cedc-499b-a334-048cc13de000Ù$ceaeb177-7a6a-4062-9659-56bebce0e77bÙ$3556d366-0bc7-4239-b4f6-3f9bd28780e0Ù$69ae57b4-4e4c-44a2-aca7-d0fff89b9566Ù$e50f8f52-a73f-4186-af5e-b4ca2c021142Ù$9862c791-31e8-4d59-8610-a929d72ea9c3Ù$6bd5ea51-54c4-46ba-8eb3-6427225e5249Ù$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56Ù$ee3299e2-3367-4c68-a050-c26650a97c8cÙ$4d4e11a2-b6a1-490f-9c7b-ad617dbb395bÙ$df1f29ce-026d-4935-83aa-0d597cb166ddÙ$533bf4bc-494b-47c2-91b6-7b30221765f5Ù$9a07a949-4ade-4325-8ae6-b05bdbbfae00Ù$9661a3b9-ae42-466e-967d-69230b9769b3Ù$895f4b81-51fe-4a1c-b958-1294743e3a9bÙ$3f1068e0-97e2-444a-961d-2af55a4d797bÙ$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2Ù$e121f72b-fe6d-491a-ab03-ef92154c61caÙ$54457653-f029-4368-8bc0-15d69f22bfcdÙ$8194cce9-115f-44b4-9f28-caa88b764185Ù$292b182f-b8a4-4ac6-89ad-ef71d40eca1fÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092Ù$b92d17a9-8481-458a-bc0a-efb7333cbc6eÙ$9527686f-24e1-40bb-9a5d-22575aafec9bÙ$cd6d807d-6238-44ce-9267-1614679f527aÙ$0980ef52-526d-421d-8c63-04272f16cbddÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633aÙ$c1ae773e-0f2d-4c1c-bae2-25f45bb28fbbÙ$310d6a10-b887-4dd0-ba93-f901726ddbbcÙ$068ef840-04cb-4de4-9512-721f57e5750eÙ$7331c447-c0a6-4a47-af00-36efc449d9d9Ù$bebe2bb8-cf74-4eaa-85d1-712b990b5985Ù$347a1500-43aa-4078-9ea3-480bc8a4c291Ù$29287c62-e892-448f-a9d5-12785ae4a02fÙ$c733ca7e-b57e-4218-9bd4-238ab5749143Ù$5f6529a1-4ace-4dd0-a7e2-f51070eab695Ù$33bdaa23-707e-4227-b936-c5d7aaf2c48eÙ$402f0d1d-0a3f-404a-8019-afc92d154c85Ù$e9d6af05-63f7-4d31-85e2-b0233df23608Ù$802edb3a-4809-4c50-920b-25f7bdc255ddÙ$98db9022-f8ff-4af3-9c81-89cf09771928Ù$7921c4c6-56b8-4c6f-a9be-cd1d9984680bÙ$edad5700-9b04-47c6-90f8-b6bac0897340Ù$8c202da6-1e13-43b8-a22b-94badcef2934Ù$40ed6c94-d2f9-4225-80b3-9060f04f8971Ù$1994bf51-adf1-4b07-ab4c-f47552d90826Ù$9ca3b930-3d9d-460c-9f20-e3c135477b05Ù$e27db9a0-09ff-4ee5-a807-d98933b6bcf1Ù$95eb9960-89f0-4edc-8943-77a75bce2b80Ù$9afd31c9-e938-417a-8c3f-e0d1ba88f95bÙ$f010e781-f41e-4861-af1c-32cf5a76ce4dÙ$906e5199-f2d2-4816-a195-6d2b1dee9403Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6abÙ$db7aaa4c-5568-4619-a9c4-00c7916c321aÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4eÙ$f5d3714d-3900-4dbe-9079-978a44584d1dÙ$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2Ù$6ddc06c0-3f5d-4cc9-8060-dd6997e0f662Ù$722ad63a-c2ac-4ed6-b268-41d0f8b745f1Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16Ù$a580ef44-234a-4ed1-b007-920651415427Ù$0e13e63d-fd08-4cc1-aa37-851c537afbefÙ$2adc9595-8829-4d35-be90-a7718c2e7ce7Ù$53b21ec0-28e9-46cd-a92e-8afc189c3a11Ù$778c40ff-4c9e-42fb-92a6-1e376837f6efÙ$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$43d2559f-8902-4c54-8fdf-cb268b6f868cÙ$4317bba0-723f-4cdc-9d52-67033540a8d2Ù$35f8cf4f-3fcb-4e27-9462-244406d7800eÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$17c91ea8-acb7-4bbd-b0b0-0f8193f45303Ù$85303791-bdc4-468a-bc40-48ef2a186282Ù$6ded46e6-1c89-4875-addb-8c709e949bb1Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccefÙ$9bbbda1f-74a6-458b-a084-9d034d6c291fÙ$03b6aa6d-7517-4906-9430-302516d0653bÙ$78d8c8c9-568d-472a-9f06-a50b1cf2384bÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3fÙ$7d79ff81-59e0-41f0-b2fe-70b41f44591fÙ$da5895e7-af99-46ff-9f53-36529d1ca456Ù$9415a6ed-c05e-4487-b0be-f342ec7424cdÙ$4d76422e-711c-4f3e-87ab-0ce851bac064Ù$3a2132e7-7d69-42d7-89d3-b2d7679ad74fÙ$24d52ef8-927e-46c8-b455-74ccdb33d3caÙ$4de9baec-f444-47b8-b9e6-7f3d9e9609b1Ù$5a79c09e-2a33-43d1-a5dc-caba3db467ddÙ$16f611e8-8e41-43f3-830c-9976cb720b9fÙ$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6Ù$200e2b1b-065a-4d60-b5fd-86700e4c811aÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361Ù$c1da4130-5936-499f-bb9b-574e01136ecaÙ$5b85a063-cf0f-4afa-89f4-420d7350ecc3Ù$b16f6225-1949-4b6d-a4b0-c5c230eb4c7fÙ$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6Ù$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47Ù$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5Ù$79df35ff-3718-4a9c-a384-4e8c12175ab8Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$fa886ef3-192f-4daa-b59b-aca0c6365111Ù$7f459370-eb6f-4d82-bf99-1528e136ca5cÙ$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72bÙ$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96Ù$cbfc0129-9361-4edb-a467-1456a1f3aeaeÙ$81deb227-a822-4857-a584-a51cc8ff51f4«cell_inputsÞ ªÙ$90850509-463d-44c7-88ae-4406aebd4be1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$90850509-463d-44c7-88ae-4406aebd4be1«code_foldedäcode¸md"## Computation graph"Ù$77a7de14-87d2-11ef-21ef-937b8239db5b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$77a7de14-87d2-11ef-21ef-937b8239db5b«code_foldedäcodeÚ”md""" # Differentiation approaches We can compute partial derivatives in different ways: 1. **Symbolically**, by fixing one of the variables and differentiating with respect to the others, either manually or using a computer. 2. **Numerically**, using the formula ``f'(x) \approx (f(x + h) - f(x)) / h``. 3. **Algorithmically**, either forward or reverse : this is what we will explore here. """Ù$e46fb3ff-b26f-4efb-aaa1-760e80017797„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e46fb3ff-b26f-4efb-aaa1-760e80017797«code_foldedäcode°md"# Chain rule"Ù$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8e«code_foldedäcodeÚWchain_figure() do label, geometry (; ks, xs, Δy, colors) = geometry dy_down = 30 for (k, x) in zip(ks, xs) label("\\dfrac{\\partial s_{$k}}{\\partial s_0}", x, dy_down; color = colors.t, fontsize = 12) label("\\dfrac{\\partial s_K}{\\partial s_{$k}}", x, Δy + dy_down; color = colors.r, fontsize = 12) end endÙ$78d8c8c9-568d-472a-9f06-a50b1cf2384b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$78d8c8c9-568d-472a-9f06-a50b1cf2384b«code_foldedäcodeÙüqa(md"How can the Hessian of ``f`` be computed given an AD for Jacobian and gradient.", md""" Define the function ``g(x) = \nabla f(x)``, the Hessian of ``f`` is then the Jacobian of ``g``: ``\nabla^2 f(x) = J_g(x)``. See the solutions of `LabAD`! """)Ù$9661a3b9-ae42-466e-967d-69230b9769b3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9661a3b9-ae42-466e-967d-69230b9769b3«code_foldedäcodeÚlet x = VectNode(rand(-9:9, vector_n)) y = VectNode(rand(-9:9, vector_n)) names = IdDict(x => "x", y => "y") graph = CGE.Graph(x' * y; names) HTML(CGE.render_svg(graph, CGE.capture_frame(graph, "x' * y"), responsive = true, typeset_arrays = false)) endÙ$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56«code_foldedäcodeÙGmd"`n` = $(@bind scalar_n Slider(1:4, default = 2, show_value = true))"Ù$7921c4c6-56b8-4c6f-a9be-cd1d9984680b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7921c4c6-56b8-4c6f-a9be-cd1d9984680b«code_foldedäcodeÚGqa(md"Let ``A(X) = B \odot X``, what is the adjoint ``A^*`` ?", md""" ```math \begin{align} \langle B \odot X, Y \rangle & = \sum_{ij} ((B \odot X) \odot Y)_{ij}\\ & = \sum_{ij} (X \odot (B \odot Y))_{ij}\\ & = \langle X, B \odot Y \rangle. \end{align} ``` So ``A`` is self-adjoint, i.e., ``A^* = A``. """)Ù$54457653-f029-4368-8bc0-15d69f22bfcd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$54457653-f029-4368-8bc0-15d69f22bfcd«code_foldedäcodeÚÃmd""" 1. Start with an input vector ``x``. 2. Apply a matrix product ``y = W_1 x`` (or ``y = W_1 x + b_1`` with bias). 3. Apply an activation function, say `tanh`, independently to each entry: ``z_i = \tanh(y_i)``. Using the broadcasting syntax of Julia, we can write ``z = \tanh.(y)``. Usually denoted ``\sigma(y)``. 4. Repeat with ``W_2`` for next layer etc... ```math \hat{y} = W_\ell \tanh.(W_{\ell-1} \tanh.(\cdots W_2 \tanh.(W_1 x))) ``` """Ù$37bfd8ed-8983-4a89-9ba5-a90a35151729„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$37bfd8ed-8983-4a89-9ba5-a90a35151729«code_foldedäcodeÙÉmd""" ## Let's turn it into a chain! Key idea: introduce ``\text{dup}(x) = (x, x)`` so that ``f = g_2 \circ g_1`` is a chain with ``g_1 = \text{dup}`` and ``g_2(x_1, x_2) = (f_1(x_1), f_2(x_2))``. """Ù$a580ef44-234a-4ed1-b007-920651415427„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a580ef44-234a-4ed1-b007-920651415427«code_folded¤codeÙ1sum((W[2] * tanh.(W[1] * X) - y).^2) / size(y, 2)Ù$e9d6af05-63f7-4d31-85e2-b0233df23608„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e9d6af05-63f7-4d31-85e2-b0233df23608«code_foldedäcodeÚmd""" ``\langle\rangle_i``, the last one should be just product of scalars to use ``r_K = 1``, the first one should be canonical to be one-hot. The inner ones can be anything. Multiplying it by 2 makes one adjoint be multiplied by 2, the other by 1/2, it just cancels out. """Ù$edad5700-9b04-47c6-90f8-b6bac0897340„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$edad5700-9b04-47c6-90f8-b6bac0897340«code_foldedäcodeÙëqa(md"What should be saved for the backward pass ?", md"The Jacobian is diagonal, we just need to compute and store the values on the diagonal: ``f.(X)`` or we can also just store ``X`` and compute ``f.(X)`` during the backward pass.")Ù$11eadcc1-68bf-4ca7-9603-ef46468b4779„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779«code_foldedäcodeÚqa(md"Why don't we show the local Jacobian ``\partial J_k`` in the memory usage ?", md"For a scalar AD, it is best to compute the local Jacobian ``\partial J_k`` and ``s_k`` from ``s_{k-1}`` during the forward pass at the same time because part of the computation for one can be reused for the other one. For instance, if ``f_k(x) = \sin(x)``, it is more efficient to compute ``\sin(x)`` and ``\cos(x)`` at the same time with the Julia function `sincos`. However, as we will see, for a vectorized AD, we usually don't compute a local Jacobian and just use ``s_{k-1}`` to do the backward propagation of ``r_k`` towards ``r_{k-1}``. This is why this picture, which is taken from the book which focuses on vectorized AD doesn't show a memory usage for the local Jacobian.")Ù$dc4feb58-d2cf-4a97-aaed-7f4593fc9732„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$dc4feb58-d2cf-4a97-aaed-7f4593fc9732«code_foldedäcode¹md""" # Discontinuity """Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$cbfc0129-9361-4edb-a467-1456a1f3aeae«code_foldedäcodeÚsbegin struct Path path::String end function imgpath(path::Path) file = path.path if !('.' in file) file = file * ".png" end return joinpath(joinpath(@__DIR__, "images", file)) end function img(path::Path, args...; kws...) return PlutoUI.LocalResource(imgpath(path), args...) end struct URL url::String end function save_image(url::URL, html_attributes...; name = split(url.url, '/')[end], kws...) path = joinpath("cache", name) return PlutoTeachingTools.RobustLocalResource(url.url, path, html_attributes...), path end function img(url::URL, args...; kws...) r, _ = save_image(url, args...; kws...) return @htl("$r") end function img(file::String, args...; kws...) if startswith(file, "http") img(URL(file), args...; kws...) else img(Path(file), args...; kws...) end end endÙ$b92d17a9-8481-458a-bc0a-efb7333cbc6e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b92d17a9-8481-458a-bc0a-efb7333cbc6e«code_foldedäcodeÚÕTwoColumn(md""" ### Right-to-left ```math \begin{align*} S_{0} & = X\\ S_{2k-1} & = W_k S_{2k-2} + b_{k} \mathbf{1}^\top\\ S_{2k} & = \sigma(S_{2k-1})\\ S_{2H+1} & = W_{k+1} S_{2H}\\ S_{2H+2} & = \ell(S_{2H+1}; Y)\\ \end{align*} ``` """, md""" ### Left-to-right ```math \begin{align*} S_{0} & = X\\ S_{2k-1} & = S_{2k-2} W_k + \mathbf{1} b_{k}^\top\\ S_{2k} & = \sigma(S_{2k-1})\\ S_{2H+1} & = S_{2H} W_{k+1}\\ S_{2H+2} & = \ell(S_{2H+1}; Y)\\ \end{align*} ``` """)Ù$ee3299e2-3367-4c68-a050-c26650a97c8c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$ee3299e2-3367-4c68-a050-c26650a97c8c«code_foldedäcodeÚ)let x = ScalarNode.(rand(-9:9, scalar_n)) y = ScalarNode.(rand(-9:9, scalar_n)) names = IdDict() for i in eachindex(x) names[x[i]] = "x[$i]" names[y[i]] = "y[$i]" end graph = CGE.Graph(x' * y; names) HTML(CGE.render_svg(graph, CGE.capture_frame(graph, "x' * y"), responsive = true)) endÙ$db7aaa4c-5568-4619-a9c4-00c7916c321a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$db7aaa4c-5568-4619-a9c4-00c7916c321a«code_folded¤codeÙ&features = permutedims(data[:, 2:end])Ù$068ef840-04cb-4de4-9512-721f57e5750e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$068ef840-04cb-4de4-9512-721f57e5750e«code_foldedäcode®md"## Adjoint"Ù$95eb9960-89f0-4edc-8943-77a75bce2b80„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$95eb9960-89f0-4edc-8943-77a75bce2b80«code_foldedäcodeÚ–qa(md"What about the complexity of meeting in the middle between ``k`` and ``k+1``?", md""" We can also write ```math \langle J_n J_{n-1} \cdots J_2 J_1 \partial w, \partial y \rangle = \langle J_k J_{k-1} \cdots J_2 J_1\partial w, J_{k+1}^\top J_{k+2}^\top \cdots J_{n-1}^\top J_n^\top \partial y \rangle. ``` This corresponds to multiplying starting from some ``d_k`` where ``1 < k < n``. We would then first compute the left side: ```math \begin{align} J_{k+1,k+2} & = J_{k+2} J_{k+1} && \Omega(d_{k+2}d_{k+1}d_{k})\\ J_{k+1,k+3} & = J_{k+3} J_{k+1,k+2} && \Omega(d_{k+3}d_{k+2}d_{k})\\ \vdots & \quad \vdots\\ J_{k+1,n} & = J_{n} J_{k+1,n-1} && \Omega(d_nd_{n-1}d_k) \end{align} ``` then the right side: ```math \begin{align} J_{k-1,k} & = J_k J_{k-1} && \Omega(d_kd_{k-1}d_{k-2})\\ J_{k-2,k} & = J_{k-1,k} J_{k-2} && \Omega(d_kd_{k-2}d_{k-3})\\ \vdots & \quad \vdots\\ J_{1,k} & = J_{2,k} J_1 && \Omega(d_kd_1d_0)\\ \end{align} ``` and then combine both sides: ```math J_{1,n} = J_{k+1,n} J_{1,k} \qquad \Omega(d_nd_kd_0) ``` we have a complexity of ```math \Omega(d_nd_kd_0 + \sum_{i=1}^{k-1} d_kd_id_{i-1} + \sum_{i=k+2}^{n} d_id_{i-1}d_k). ``` """)Ù$fa5dba01-a3f7-452c-877e-352d578ecf51„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$fa5dba01-a3f7-452c-877e-352d578ecf51«code_foldedäcodeÚéhbox([ md""" #### Forward ```math \begin{align} t_3 & = \partial_1 f_3 \cdot t_1 + \partial_2 f_3 \cdot t_2\\ & = \begin{bmatrix} \partial_1 f_3 & \partial_2 f_3 \end{bmatrix} \cdot \begin{bmatrix} t_1\\ t_2 \end{bmatrix}\\ & = \partial f_3 \cdot \begin{bmatrix} t_1\\ t_2 \end{bmatrix} \end{align} ```""", Div(html" ", style = Dict("flex-grow" => "1")), md""" #### Reverse ```math \begin{align} \begin{bmatrix} r_1 & r_2 \end{bmatrix} & \mathrel{\raise{0.19ex}{\scriptstyle+}} = r_3 \cdot \partial f_3\\ & \mathrel{\raise{0.19ex}{\scriptstyle+}} = r_3 \cdot \begin{bmatrix} \partial_1 f_3 & \partial_2 f_3 \end{bmatrix}\\ & \mathrel{\raise{0.19ex}{\scriptstyle+}} = \begin{bmatrix} r_3 \cdot\partial_1 f_3 & r_3 \cdot\partial_2 f_3 \end{bmatrix} \end{align} ``` """, Div(html" ", style = Dict("flex-grow" => "1")), md""" #### Reverse* ```math \begin{align} \begin{bmatrix} r_1\\ r_2 \end{bmatrix} & \mathrel{\raise{0.19ex}{\scriptstyle+}} = \partial f_3^* \cdot r_3\\ & \mathrel{\raise{0.19ex}{\scriptstyle+}} = \begin{bmatrix} \partial_1 f_3\\ \partial_2 f_3 \end{bmatrix} \cdot r_3\\ & \mathrel{\raise{0.19ex}{\scriptstyle+}} = \begin{bmatrix} \partial_1 f_3 \cdot r_3 \\ \partial_2 f_3 \cdot r_3 \end{bmatrix} \end{align} ``` """ ])Ù$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0«code_foldedäcode·md"## Reverse tangents"Ù$69c08fab-c317-462c-817c-3f841a8a0941„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$69c08fab-c317-462c-817c-3f841a8a0941«code_foldedäcodeÙ¦md"""When using automatic differentiation, don't forget that we must always evaluate the derivatives. For the following example we choose to evaluate it in ``x=3``"""Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b«code_foldedäcodeÚbegin """ StepSlider(range::AbstractRange; default = first(range), show_value = true) A `Slider` flanked by `â—€` and `â–¶` buttons, so that a range can be walked one step at a time instead of by dragging. """ struct StepSlider range::AbstractRange default::Real show_value::Bool end StepSlider(range::AbstractRange; default = first(range), show_value = true) = StepSlider(range, default, show_value) function Base.show(io::IO, m::MIME"text/html", slider::StepSlider) show(io, m, @htl("""
$(slider.show_value ?
@htl("") :
nothing)
"""))
end
# Pluto's `@bind` prefers `Base.get` over `Bonds.initial_value`, so this is
# all that is needed for `graph_step` to be defined before the browser
# renders the widget.
Base.get(slider::StepSlider) = slider.default
StepSlider
end;Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16«code_foldedäcodeÙ4@bind graph_step StepSlider(eachindex(graph_frames))Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2fcf25d2-fd51-4c13-b57c-86236aceead2«code_folded¤codeºy = Float32.(targets .- 2)Ù$1c44706b-fd7d-4826-84a3-73e2db5adadd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1c44706b-fd7d-4826-84a3-73e2db5adadd«code_foldedäcodeµmd"## Implementation"Ù$e50f8f52-a73f-4186-af5e-b4ca2c021142„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e50f8f52-a73f-4186-af5e-b4ca2c021142«code_folded¤code¯abs(Dual(0, 1))Ù$626abc7c-87ef-4838-9f0a-294cf0a4be6a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$626abc7c-87ef-4838-9f0a-294cf0a4be6a«code_foldedäcodeÙôqa(md"What happens if ``f_4`` is handled before ``f_5`` in the backward pass ?",
md"""
``\partial f / \partial s_4`` is the sum of its contribution from ``f_5`` and ``f_7``. We must wait for ``f_5`` to be handled before we turn to ``f_4``.
""")Ù$157c413e-b166-48ff-b28e-58b88f018a94„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$157c413e-b166-48ff-b28e-58b88f018a94«code_folded¤codeÚ ’"""
chain_figure(annotate; width = 640, height = 360)
Draw the forward, tangent and reverse chains of ``f_K \\circ \\cdots \\circ f_1``
with [`mathjax_figure`](@ref).
Additional labels are added with `annotate(label, geometry)` where
`label(text, x, y; color, fontsize, class)` adds a label and `geometry`
gives the position of the columns `xs` with their index `ks`.
"""
function chain_figure(annotate; width = 640, height = 360)
mathjax_figure(; width, height) do _text
Luxor = CGE.Luxor
box_width = 45
dy_up = 3box_width/8
Δx = 90
Δy = 90
colors = (s = "#7ea6e0", t = "#67ab9f", r = "#ffb570")
function _arrow(start, finish; rev, color)
if rev
start, finish = finish, start
end
Luxor.sethue(color)
Luxor.arrow(start, finish; linewidth = 1.5, arrowheadlength = 10)
end
function draw_node(text, x, y; fill_color = "white", stroke_color = "#f79009", linewidth = 2, width = box_width, height = box_width)
center = Luxor.Point(x, y)
Luxor.sethue(fill_color)
Luxor.box(center, width, height, 12, :fill)
Luxor.sethue(stroke_color)
Luxor.setline(linewidth)
Luxor.box(center, width, height, 12, :stroke)
_text(text, x, y)
end
function draw_line(f, v, y; rev, color = "black", stroke_color)
dx_dots = box_width/4
_text("$(v)_0", -3Δx, y; color)
_arrow(Luxor.Point(-3Δx + box_width/2, y), Luxor.Point(-2Δx - box_width/2, y); rev, color)
draw_node("$(f)_1", -2Δx, y; stroke_color)
_arrow(Luxor.Point(-2Δx + box_width/2, y), Luxor.Point(-Δx - dx_dots, y); rev, color)
_text("$(v)_1", -2Δx + box_width/2 + dy_up, y - dy_up; color)
_text("\\cdots", -Δx, y)
_arrow(Luxor.Point(-Δx + dx_dots, y), Luxor.Point(-box_width/2, y); rev, color)
_text("$(v)_{k-1}", -box_width/2 - dy_up, y - dy_up; color)
draw_node("$(f)_k", 0, y; stroke_color)
_text("$(v)_k", box_width/2 + dy_up, y - dy_up; color)
_arrow(Luxor.Point(box_width/2, y), Luxor.Point(Δx - dx_dots, y); rev, color)
_text("\\cdots", Δx, y)
_arrow(Luxor.Point(Δx + dx_dots, y), Luxor.Point(2Δx - box_width/2, y); rev, color)
_text("$(v)_{K-1}", 2Δx - box_width/2 - dy_up, y - dy_up; color)
draw_node("$(f)_K", 2Δx, y; stroke_color)
_text("$(v)_K", 3Δx, y; color)
_arrow(Luxor.Point(2Δx + box_width/2, y), Luxor.Point(3Δx - box_width/2, y); rev, color)
end
draw_line("f", "s", -Δy, rev = false, color = colors.s, stroke_color = "#546b8c")
draw_line("\\partial f", "t", 0, rev = false; color = colors.t, stroke_color = "#40635c")
draw_line("\\partial f^*", "r", Δy, rev = true; color = colors.r, stroke_color = "#b58b63")
ks = ["0", "1", "k-1", "k", "K-1", "K"]
xs = [-3Δx, -2Δx + box_width/2 + dy_up, -box_width/2 - dy_up, box_width/2 + dy_up, 2Δx - box_width/2 - dy_up, 3Δx]
annotate(_text, (; ks, xs, Δy, dy_up, colors))
end
endÙ$1d56075c-e28d-46c9-9a0a-210079172388„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1d56075c-e28d-46c9-9a0a-210079172388«code_foldedäcode½md"## Reverse mode in action"Ù$ceaeb177-7a6a-4062-9659-56bebce0e77b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$ceaeb177-7a6a-4062-9659-56bebce0e77b«code_folded¤codeÙ!abs_bis(x) = ifelse(x > 0, x, -x)Ù$a9a4841d-18cc-4386-aa38-017771650afb„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a9a4841d-18cc-4386-aa38-017771650afb«code_foldedäcode¹fanout_figure(dup = true)Ù$43d2559f-8902-4c54-8fdf-cb268b6f868c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$43d2559f-8902-4c54-8fdf-cb268b6f868c«code_foldedäcode³md"## Reverse mode"Ù$63bdc7f5-89a5-4061-9e42-6588e4cd96c6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6«code_folded¤codeÙtBase.:*(x::Dual{T}, y::Dual{T}) where {T} = Dual(x.value * y.value, x.value * y.derivative + x.derivative * y.value)Ù$1a0026be-7975-478c-a447-f66ea95f551d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1a0026be-7975-478c-a447-f66ea95f551d«code_foldedäcodeÙ!md"## Interpretation of tangents"Ù$c73f79c6-a28f-4c7a-89e5-8d70a245a210„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c73f79c6-a28f-4c7a-89e5-8d70a245a210«code_foldedäcodeÚQqa(md"When is each mode faster than the other one to compute the full Jacobian ?", md"If ``f: \mathbb{R}^n \to \mathbb{R}^m``, then computing the full Jacobian requires ``n`` JVP or ``m`` VJP. So
* if ``n \gg m``, then reverse mode is faster;
* if ``m \gg n``, then forward mode is faster;
* if ``m \approx n``, then it's a close call.")Ù$3556d366-0bc7-4239-b4f6-3f9bd28780e0„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$3556d366-0bc7-4239-b4f6-3f9bd28780e0«code_folded¤codeÙ2Base.isless(x::Dual, y::Real) = isless(x.value, y)Ù$9a07a949-4ade-4325-8ae6-b05bdbbfae00„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9a07a949-4ade-4325-8ae6-b05bdbbfae00«code_foldedäcodeÙGmd"`n` = $(@bind vector_n Slider(1:4, default = 2, show_value = true))"Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b5c3e2ef-3d47-4f44-b968-d04734be2f16«code_folded¤codeÙ@W = [rand(Float32, h, size(X, 1)), rand(Float32, size(y, 1), h)]Ù$9415a6ed-c05e-4487-b0be-f342ec7424cd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9415a6ed-c05e-4487-b0be-f342ec7424cd«code_foldedäcodeÚÌmd"""
## Forward on forward
Given ``\text{Dual}(s_1, t_1)`` with ``s_1 = \text{Dual}(f_1(x), \frac{\partial f_1}{\partial x_j})`` and ``t_1 = \text{Dual}(\frac{\partial f_1}{\partial x_i}, \frac{\partial^2 f_1}{\partial x_i \partial x_j})``
1. Compute ``s_2 = f_2(s_1) = (f_2(f_1(x)), J_2 \cdot \frac{\partial f_1}{\partial x_j}) = ((f_2 \circ f_1)(x), \partial (f_2 \circ f_1) / \partial x_j)``
2. Compute ``J_{f_2}(s_1)`` which gives ``\text{Dual}(J_2, H_{2j})``
3. Compute
```math
\begin{align}
J_{f_2}(s_1) \cdot t_1
& =
\text{Dual}(J_2, H_{2j}) \cdot
\text{Dual}(\frac{\partial f_1}{\partial x_i}, \frac{\partial^2 f_1}{\partial x_i \partial x_j})\\
& =
\text{Dual}(J_2 \cdot \frac{\partial f_1}{\partial x_i}, J_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j} +
H_{2j} \cdot \frac{\partial f_1}{\partial x_i})\\
& =
\text{Dual}(\frac{\partial (f_2 \circ f_1)}{\partial x_i}, \frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j})
\end{align}
```
"""Ù$f5d3714d-3900-4dbe-9079-978a44584d1d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f5d3714d-3900-4dbe-9079-978a44584d1d«code_folded¤codeÙ~function normalise(x)
μ = Statistics.mean(x, dims=2)
σ = Statistics.std(x, dims=2, mean=μ)
return (x .- μ) ./ σ
endÙ$3803804a-f44d-4f56-bd59-fb1401d8fb9e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$3803804a-f44d-4f56-bd59-fb1401d8fb9e«code_folded¤codeÙ=Base.:-(x::Dual{T}) where {T} = Dual(-x.value, -x.derivative)Ù$03b6aa6d-7517-4906-9430-302516d0653b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$03b6aa6d-7517-4906-9430-302516d0653b«code_foldedäcode¼md"""
## Second-order AD
"""Ù$607000ef-fb7f-4204-b543-3cb6bb75ed71„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$607000ef-fb7f-4204-b543-3cb6bb75ed71«code_foldedäcodeÚlet
x = range(-1, stop = 1, length = 11)
PlotlyLight.Plot(
PlotlyLight.Config(
x = collect(x),
y = abs.(x),
mode = "lines",
name = "|x|",
),
PlotlyLight.Config(
annotations = [
PlotlyLight.Config(
x = λ/2 - (1 - λ)/2,
y = -1/2,
ax = 0,
ay = 0,
xref = "x",
yref = "y",
axref = "x",
ayref = "y",
showarrow = true,
arrowhead = 2,
arrowcolor = "orange",
text = "",
)
for λ in range(0, stop = 1, length = 11)
],
),
)
endÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65f«code_foldedäcodeÚ
begin
# One pullback per operation: given the adjoint of the output node,
# accumulate the adjoint of each input node.
function CGE.pullback!(::typeof(+), node::GraphNode, args::GraphNode...)
for arg in args
arg.metadata.derivative += node.metadata.derivative
end
end
function CGE.pullback!(::typeof(*), node::GraphNode, x::GraphNode, y::GraphNode)
x.metadata.derivative += node.metadata.derivative * y.value
y.metadata.derivative += node.metadata.derivative * x.value
end
function CGE.pullback!(::typeof(exp), node::GraphNode, x::GraphNode)
x.metadata.derivative += node.metadata.derivative * node.value
end
function CGE.pullback!(::typeof(sqrt), node::GraphNode, x::GraphNode)
x.metadata.derivative += node.metadata.derivative / (2 * node.value)
end
end;Ù$2ae277d5-8be4-4ea9-a3fc-8ad601577c3a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3a«code_foldedäcode½md"# Forward Differentiation"Ù$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47«code_folded¤code²import PlotlyLightÙ$da5895e7-af99-46ff-9f53-36529d1ca456„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$da5895e7-af99-46ff-9f53-36529d1ca456«code_foldedäcodeÚ{md"""
## Chain rule
```math
\begin{align}
\frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j}
& =
\frac{\partial}{\partial x_j} \left(\frac{\partial (f_2 \circ f_1)}{\partial x_i} \right)\\
& =
\frac{\partial}{\partial x_j} \left(\partial f_2 \cdot \frac{\partial f_1}{\partial x_i} \right)\\
& =
\left(\partial^2 f_2 \cdot \frac{\partial f_1}{\partial x_j} \right) \cdot \frac{\partial f_1}{\partial x_i} +
\partial f_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j}
\end{align}
```
In terms of the matrices ``J_k = \partial f_k`` and ``H_{kj} = \frac{\partial}{\partial x_j} J_k = \partial^2 f_k \cdot \frac{\partial s_{k-1}}{\partial x_j}``, it becomes
```math
\begin{align}
\frac{\partial^2 (f_2 \circ f_1)}{\partial x_i \partial x_j}
& =
H_{2j} \cdot \frac{\partial f_1}{\partial x_i} +
J_2 \cdot \frac{\partial^2 f_1}{\partial x_i \partial x_j}
\end{align}
```
"""Ù$e121f72b-fe6d-491a-ab03-ef92154c61ca„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e121f72b-fe6d-491a-ab03-ef92154c61ca«code_foldedäcodeºmd"""
# Neural network
"""Ù$347a1500-43aa-4078-9ea3-480bc8a4c291„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$347a1500-43aa-4078-9ea3-480bc8a4c291«code_foldedäcodeÚßqa(md"We are free to choose any scalar product ``\langle \cdot, \cdot \rangle_k``. Does this choice affect the Jacobian values as well as gradient of ``f`` ?",
md"""
If we change the scalar product ``\langle \cdot, \cdot \rangle_k``, it changes both the adjoint ``\partial f^*_k`` and ``\partial f^*_{k+1}``. But it does not affect the gradient of ``f`` as the changes made to both these Jacobian cancel out. The composition ``(\partial f^*_k) \circ (\partial f^*_{k+1})`` is unaffected by the choice of the scalar product ``\langle \cdot, \cdot \rangle_k``.
For example, suppose ``\mathcal{S}_0 = \mathcal{S}_1 = \mathcal{S}_2 = \mathbb{R}`` and ``f_1(x) = x^2`` and ``f_2(x) = \exp(x)``. For ``s_0 = 1``, we have ``s_1 = 1``, ``s_2 = e``, ``\partial f_1(1) = 2``, ``\partial f_2(1) = e``.
So forward math computes ``(f_2 \circ f_1)'(1) = \partial f_1(1) \cdot \partial f_2(1) = 2e``.
For ``\langle x, y \rangle_0 = \rangle x, y \rangle_1 = \rangle x, y \rangle_2 = xy``, we have ``\partial f_1^*(1) = 2``, ``\partial f_2^*(1) = e``. So the reverse mode computes ````(f_2 \circ f_1)'(1) = \partial f_1^*(1) \cdot \partial f_2^*(1) = 2e``.
For ``\rangle x, y \rangle_1 = 2xy``, we have ``\partial f_1^*(1) = 1``, ``\partial f_2^*(1) = 2e``. So the reverse mode computes ````(f_2 \circ f_1)'(1) = \partial f_1^*(1) \cdot \partial f_2^*(1) = 2e``. As we can see, by scaling the scalar product by 2, one jacobian gets scaled by 2 and one by 1/2. When we multiply them together, they cancel out.
""")Ù$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b«code_foldedäcodeÙLbegin
struct DummyMetadata end
DummyMetadata(value) = DummyMetadata()
end;Ù$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2«code_foldedäcodeÚ'md"""
* If AD in an interpreted language like Python, the bottleneck of scalar-valued AD becomes the Python interpreter. Solved by using compiled language like Julia/Rust/C/C++/...
* If each node is heap allocated (e.g. with `malloc` in C), not cache friendly. Solved by a centralized storage for all the values of all nodes, e.g. `MathOptInterface.Nonlinear.ReverseAD`'s values vector (TODO hyperlink)
* Not using optimized BLAS routines or GPU kernels for matrix-matrix multiplications. Not solveable for scalar-valued AD, need vector-valued AD!
"""Ù$0980ef52-526d-421d-8c63-04272f16cbdd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0980ef52-526d-421d-8c63-04272f16cbdd«code_foldedäcodeÙ!md"## Spaces and scalar products"Ù$cd6d807d-6238-44ce-9267-1614679f527a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$cd6d807d-6238-44ce-9267-1614679f527a«code_foldedäcodeÙ0img("Blondel_Rouvet_Figure_8_5", :height => 150)Ù$383461cb-9f51-42b2-ab76-d9580b39409e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$383461cb-9f51-42b2-ab76-d9580b39409e«code_foldedäcodeÚKqa(md"How should we combine ``\partial f_1^* r_1`` and ``\partial f_2^* r_2`` into ``r_0`` ? Sum them ? Multiply them ?", md"We don't know yet! 😱 The only way to find out is to make sure that reverse mode computes the same derivative as forward mode. But we only know how to do this for a **chain**... and this is not a chain.")Ù$e9c14738-2c1f-43d8-94f9-fb5440d74092„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092«code_foldedäcodeÙDmd"``b_k`` is a **column** vector, ``S_i, X, W_i, Y`` are matrices."Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3«code_folded¤codeÙ!X = Float32.(normalise(features))Ù$79df35ff-3718-4a9c-a384-4e8c12175ab8„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$79df35ff-3718-4a9c-a384-4e8c12175ab8«code_folded¤codeÚÃ"""
mathjax_figure(draw; width, height)
Luxor only draws the shapes, the labels are overlaid in HTML so that
they are typeset by the MathJax of Pluto.
`draw(label)` draws the shapes with the origin at the center and adds labels
with `label(text, x, y; color, fontsize, class)`.
Labels of class `equal` are centered between the labels of class
`inner-product` once MathJax has set their width.
"""
function mathjax_figure(draw; width, height)
Luxor = CGE.Luxor
labels = []
function label(text, x, y; color = "black", fontsize = 14, class = "")
push!(labels, @htl("""\$$text\$"""))
end
Luxor.Drawing(width, height, :svg)
Luxor.origin()
Luxor.background("white")
draw(label)
Luxor.finish()
svg = Luxor.svgstring()
@htl("""$(HTML(svg))
$labels
$question
$answer` if `question.content[]` is `Markdown.Paragraph` # This will print the question on a new line and we don't want that: h = HTML(_inline_html(question.content[])) return qa(h, answer) end endÙ$c733ca7e-b57e-4218-9bd4-238ab5749143„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c733ca7e-b57e-4218-9bd4-238ab5749143«code_foldedäcodeÙïqa(md"How should we store the Jacobian in the forward pass to save it for the backward pass ?", md"The matrix ``I \otimes A`` is block diagonal with the same block repeated so the structure is crucial to exploit. Storing ``A`` is enough.")Ù$88534196-9f4a-430c-a534-805177ba718d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$88534196-9f4a-430c-a534-805177ba718d«code_foldedäcodeÚ€qa(md"What about returning a convex combination of the derivative from the left and right ?", md"Any number between ``-1`` and ``1`` is a valid **subgradient**! Whereas the gradient is the normal to the **unique** tangent, the subgradient is an element of the **tangent cone**, depicted below. For convex functions, the notion of subgradient appropriately generalizes the notion of gradient for nonsmooth functions. Note that the notion of subgradient is not defined for nonconvex functions. So instead we use [Clarke differential](https://en.wikipedia.org/wiki/Clarke_generalized_derivative) but this is out of the scope of this course.")Ù$310d6a10-b887-4dd0-ba93-f901726ddbbc„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$310d6a10-b887-4dd0-ba93-f901726ddbbc«code_foldedäcodeÚ¤chain_figure() do label, geometry (; ks, xs, Δy) = geometry for i in eachindex(ks) k = ks[i] x = xs[i] label("\\mathcal{S}_{$k}", x, -3Δy/2) fontsize = 12 label("\\langle t_{$k}, r_{$k} \\rangle_{$k}", x, 3Δy/2; fontsize, class = "inner-product") if i != 1 label("=", (xs[i-1] + xs[i]) / 2, 3Δy/2; fontsize, class = "equal") end end endÙ$7331c447-c0a6-4a47-af00-36efc449d9d9„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7331c447-c0a6-4a47-af00-36efc449d9d9«code_foldedäcodeÚ{md""" The *adjoint* of a linear map ``A : \mathcal{S}_1 \to \mathcal{S}_2`` for scalar products ``\langle \cdot, \cdot \rangle_1`` for ``\mathcal{S}_1`` and ``\langle \cdot, \cdot \rangle_2`` for ``\mathcal{S}_2`` is the linear map ``A^*`` such that ```math \forall x \in \mathcal{S}_1, y \in \mathcal{S}_2, \qquad \langle A(x), y \rangle_2 = \langle x, A^*(y) \rangle_1. ``` """Ù$4317bba0-723f-4cdc-9d52-67033540a8d2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4317bba0-723f-4cdc-9d52-67033540a8d2«code_foldedäcodeÚ?qa(md"How to deduce the backward pass for reverse mode from the forward mode ?", md""" `W2 * (J_1 .* (T_1 * X))) * J_2'` → The broadcasted `*` is an Hadamard product, denoted $\odot$. So forward mode is: ```math W_2 (J_1 \odot (T_1 X)) J_2^\top ``` As this is scalar, it is trivially equal to its scalar product with ``1``. That is, we start with a reverse tangent ``R = 1``. We can then move everything to the right-hand side with the adjoint (here, it is the transpose): ```math \begin{align} \langle W_2 (J_1 \odot (\partial W_1 X)) J_2^\top, 1 \rangle & = \langle J_1 \odot (\partial W_1 X), W_2^\top J_2 \rangle\\ & = \langle \partial W_1 X, J_1 \odot (W_2^\top J_2) \rangle\\ & = \langle \partial W_1, (J_1 \odot (W_2^\top J_2))X^\top \rangle\\ \end{align} ``` Now, for the derivative with respect to the entry $W_{i,j}$, we use $\partial W_1 = e_ie_j^\top$ and ```math \langle e_ie_j^\top, (J_1 \odot (W_2^\top J_2))X^\top \rangle = ((J_1 \odot (W_2^\top J_2))X^\top)_{ij} ``` So the gradient with respect to $W_1$ is exactly the matrix $(J_1 \odot (W_2^\top J_2))X^\top$ ! """)Ù$6bd5ea51-54c4-46ba-8eb3-6427225e5249„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6bd5ea51-54c4-46ba-8eb3-6427225e5249«code_foldedäcode¼md"## Issues with scalar AD"Ù$9527686f-24e1-40bb-9a5d-22575aafec9b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9527686f-24e1-40bb-9a5d-22575aafec9b«code_foldedäcode±md"## Evaluation"Ù$f23ca90f-b567-4257-ae41-ec15c57c1f3f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3f«code_foldedäcodeÙîqa(md"Does the AD need to be the same for the gradient and the Jacobian ?", md""" No. Given an implementation of reverse mode and forward mode, this gives 4 possibilities depending on which mode is used for the gradient and Jacobian. """)Ù$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6«code_foldedäcodeÚVmd""" ## Reverse on reverse **Forward pass (2nd)**: 1. Forward pass computes ``s_2 = f_2(s_1)`` → Jacobian ``\partial s_2 / \partial s_1 = J_2`` 2. Local Jacobian ``J_2 = \partial f_2(s_1)`` → The Jacobian is the 3D array ``\partial J_2 / \partial s_1 = \partial^2 f_2`` 3. Backward pass computes ``r_1 = r_2 \cdot \partial f_2(s_1)`` → Jacobian of ``(s_1, r_2) \mapsto r_2 \cdot \partial f_2(s_1)`` is ``(r_2 \cdot \partial^2 f_2(s_1), \partial f_2(s_1)) = (r_2 \cdot \partial^2 f_2, J_2)``. Note that here ``r_2 \in \mathbb{R}^{d_k}`` is multiplying the first dimension of the tensor ``\partial^2 f_2(s_1) \in \mathbb{R}^{d_k \times d_{k-1} \times d_{k-1}}`` so the result is a symmetric matrix of dimension ``\mathbb{R}^{d_{k-1} \times d_{k-1}}`` **Reverse pass (2nd)**: The result is ``r_0``, let ``\dot{r}_k`` be the second-order reverse tangent for ``r_k`` and ``\dot{s}_k`` be the second-order reverse tangent of ``s_k``. We have ```math \begin{align} \dot{r}_2 & = J_2 \cdot \dot{r}_1\\ \dot{s}_1 & = (r_2 \cdot \partial^2 f_2(s_1)) \cdot \dot{r}_1 + \dot{s}_2 \cdot J_2 \end{align} ``` """Ù$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5«code_folded¤codeÚÙbegin using LaTeXStrings import MathTeXEngine # Luxor asks Cairo for MathTeXEngine's fonts by name so we register them # to avoid having to install them on the system let fontconfig = Base.require(Base.PkgId(Base.UUID("a3f928ae-7b40-5064-980b-68af3947d34b"), "Fontconfig_jll")) add_font = Base.Libc.Libdl.dlsym(fontconfig.libfontconfig_handle, :FcConfigAppFontAddFile) dir = joinpath(pkgdir(MathTeXEngine), "assets", "fonts", "NewComputerModern") for file in readdir(dir; join = true) if endswith(file, ".otf") ccall(add_font, Cint, (Ptr{Cvoid}, Cstring), C_NULL, file) end end end # Luxor draws the italic glyphs of letters with the upright font so we # replace them with their Unicode mathematical italic counterpart. # The arguments of font commands such as `\mathrm{...}` are left untouched # and `\mathcal{X}` is replaced by the Unicode script letter since # MathTeXEngine would use the bold script one. _math_italic(c) = c == 'h' ? 'â„Ž' : islowercase(c) ? Char(0x1D44E + (c - 'a')) : Char(0x1D434 + (c - 'A')) _math_script(c) = get(Dict('B' => 'ℬ', 'E' => 'â„°', 'F' => 'ℱ', 'H' => 'â„‹', 'I' => 'â„', 'L' => 'â„’', 'M' => 'ℳ', 'R' => 'â„›'), c, Char(0x1D49C + (c - 'A'))) function _latex(s) s = replace(s, r"\\mathcal\{([A-Z])\}" => m -> string(_math_script(m[end-1]))) s = replace(s, r"\\(math[a-z]+|text[a-z]*|operatorname)\{[^}]*\}|\\[a-zA-Z]+|[a-zA-Z]" => m -> length(m) == 1 ? string(_math_italic(m[1])) : m) return latexstring(s) end endÙ$0e13e63d-fd08-4cc1-aa37-851c537afbef„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0e13e63d-fd08-4cc1-aa37-851c537afbef«code_foldedäcode³md"## Forward mode"Ù$2f8baccc-19d1-44d6-b71f-0243fd8696ba„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2f8baccc-19d1-44d6-b71f-0243fd8696ba«code_foldedäcodeÙ0img("Blondel_Rouvet_Figure_8_4", :height => 300)Ù$191f33ab-b535-4cb1-96b3-6b2f10fd633a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633a«code_foldedäcodeÚámd""" Let ``(\mathcal{S}_k)_k`` be the spaces for which ``s_k, t_k, r_k \in \mathcal{S}_k`` and ``f_k : \mathcal{S}_{k-1} \to \mathcal{S}_{k}``. In the vectorized case, the Jacobian ``\partial f_k`` may be expressed more naturally as a linear map ``\partial f_k : \mathcal{S}_{k-1} \to \mathcal{S}_k`` and ``t_k = \partial f_k(t_{k-1})``. To obtain the matrix-form of the Jacobian, we need to fix a basis for ``\mathcal{S}_{k-1}`` and ``\mathcal{S}_k`` and vectorize ``t_k`` and ``t_{k-1}`` according to that basis which is often cumbersome. Instead, we choose a scalar products ``\langle \cdot, \cdot \rangle_k`` for ``\mathcal{S}_k`` and the **adjoint** of the Jacobians according to these bases in the backward pass. For instance, if ``\mathcal{S}_K = \mathcal{R}``, ``t_K = \langle t_K, 1 \rangle_K`` for the canonical scalar product of ``\mathcal{R}`` so we pick ``r_K = 1``. We then search for a way to obtain ``r_0(r_K)`` so that ``\langle t_0, r_0(r_K) \rangle_0 = \langle \partial f_K(\cdots\partial f_1(t_0), r_K\rangle_K``. Then, for instance if ``\mathcal{S}_0 = \mathbb{R}^n``, with the canonical scalar product ``\langle x, y \rangle_0 = x^\top y``, we have ``\partial_i f(s_0) = \langle e_i, r_0(r_K) \rangle_0 = (r_0(r_K))_i``. """Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9b4a78d8-e6da-41dd-b922-b35c895eee1a«code_folded¤codeÙ]if h < 200 # Forward Diff start being too slow for `h > 200` @time forward_diff(W, X, y) endÙ$32a0bec0-4f1d-4dba-a5db-422741f88808„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$32a0bec0-4f1d-4dba-a5db-422741f88808«code_foldedäcodeÚemd""" ## From forward to reverse mode Concatenating the two columns gives the full Jacobian. We can then factor ``\partial f_2(s_1)`` and then ``\partial f_1(s_0)`` out of the concatenation: ```math \begin{align} \partial f(s_0) & = \begin{bmatrix} 2 & 4 \end{bmatrix}\\ & = \begin{bmatrix} \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{bmatrix}\\ & = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \xcancel{\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}} \end{align} ``` The concatenation of the one-hot tangents is the identity so it disappears! We are left with ``\partial f_2(s_1) \cdot \partial f_1(s_0)`` that we can compute from left to right in a single pass: this is reverse mode! ```math \begin{align} r_2 & = 1 & r_1 & = r_2 \cdot \partial f_2(s_1) = \begin{bmatrix} 1 & 1 \end{bmatrix} & r_0 & = r_1 \cdot \partial f_1(s_0) = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} = \begin{bmatrix} 2 & 4 \end{bmatrix} \end{align} ``` """Ù$35f8cf4f-3fcb-4e27-9462-244406d7800e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$35f8cf4f-3fcb-4e27-9462-244406d7800e«code_folded¤codeÙgfunction reverse_diff(W, X, y) J_1, J_2 = forward_pass(W, X, y) (J_1 .* (W[2]' * J_2)) * X' endÙ$885bc5c9-aefc-4d8a-a4da-6062c64eaa41„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$885bc5c9-aefc-4d8a-a4da-6062c64eaa41«code_foldedäcode·md"## Forward tangents"Ù$83ef86e0-bcfb-42ee-a574-16758606423a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$83ef86e0-bcfb-42ee-a574-16758606423a«code_foldedäcodeÚyqa(md"Why is ``\partial\text{dup}^*`` a sum ?", md"The Jacobian is ``\partial\text{dup} = \begin{bmatrix} 1\\1\\1\end{bmatrix}``. In reverse mode, we multiply by the adjoint (why ? See next lecture!) of the Jacobian (here the transpose) ``\partial\text{dup}^* = \begin{bmatrix} 1 & 1 & 1\end{bmatrix}``. Left-multiplying a vector with a row vector of ones results in its sum.")Ù$bc93fb96-1097-4b22-a077-558f5662efec„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$bc93fb96-1097-4b22-a077-558f5662efec«code_folded¤codeÙ%(f_2 ∘ f_1)(Dual(1, 2), Dual(3, 4))Ù$533bf4bc-494b-47c2-91b6-7b30221765f5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$533bf4bc-494b-47c2-91b6-7b30221765f5«code_foldedäcode¾md"## Nodes with vector value"Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6«code_folded¤codeÙousing PlutoUI, PlutoUI.ExperimentalLayout, HypertextLiteral, PlutoTeachingTools, Statistics, CUDA, OneHotArraysÙ$8194cce9-115f-44b4-9f28-caa88b764185„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8194cce9-115f-44b4-9f28-caa88b764185«code_foldedäcodeÚAmd""" ## Batching Applying a neural network to 2 inputs can be done simultaneously so that **matrix-vector** products become **matrix-matrix** product, which better exploit the parallelization capabilities of CPUs and GPUs. Let ``X = \begin{bmatrix}x^{(1)} & x^{(2)}\end{bmatrix}``: ```math \begin{align} \begin{bmatrix} \sigma(W_1 x^{(1)}) & \sigma(W_1 x^{(2)}) \end{bmatrix} &= \sigma(\begin{bmatrix} W_1 x^{(1)} & W_1 x^{(2)} \end{bmatrix})\\ & = \sigma(W_1 \begin{bmatrix} x^{(1)} & x^{(2)} \end{bmatrix})\\ & = \sigma(W_1X) \end{align} ``` With bias, let ``\mathbf{1}^\top = \begin{bmatrix} 1 & 1 \end{bmatrix}``: ```math \begin{bmatrix} W_1 x^{(1)} + b_1 & W_1 x^{(2)} + b_1 \end{bmatrix} = \begin{bmatrix} W_1 x^{(1)} & W_1 x^{(2)} \end{bmatrix} + \begin{bmatrix} b_1 & b_1 \end{bmatrix} = W_1 X + b_1 \mathbf{1}^\top ``` """Ù$40baa108-eb68-433f-9917-ac334334f198„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$40baa108-eb68-433f-9917-ac334334f198«code_foldedäcodeÙË@htl("""
Automatic Differentiation
Benoît Legat
$(PlutoTeachingTools.ChooseDisplayMode()) $(PlutoUI.TableOfContents(depth=1)) """)Ù$906e5199-f2d2-4816-a195-6d2b1dee9403„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$906e5199-f2d2-4816-a195-6d2b1dee9403«code_foldedäcode·md"# Wine example ðŸ·"Ù$7f459370-eb6f-4d82-bf99-1528e136ca5c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7f459370-eb6f-4d82-bf99-1528e136ca5c«code_folded¤codeÙ$import LinearAlgebra, DelimitedFilesÙ$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3«code_foldedäcodeÚ‹begin # The graph carries the primal value; the adjoint is user metadata. mutable struct Adjoint derivative::Float64 end CGE.metadata(::Type{Adjoint}, ::Float64) = Adjoint(0.0) CGE.metadata_rows(data::Adjoint) = ["adjoint" => data.derivative] CGE.seed_metadata!(data::Adjoint, is_output::Bool) = data.derivative = is_output ? 1.0 : 0.0 const GraphNode = CGE.Node{Float64,Adjoint} end;Ù$9afd31c9-e938-417a-8c3f-e0d1ba88f95b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9afd31c9-e938-417a-8c3f-e0d1ba88f95b«code_foldedäcodeÚÏqa(md"Which mode should be used depending on the ``d_i`` ?", md""" We see that we should find the minimum ``d_k`` and start from there. If the minimum is attained at ``k = n``, this corresponds mutliplying from left to right, this is reverse differentiation. If the minimum is attained at ``k = 0``, we should multiply from right to left, this is forward mode. Otherwise, we should start from the middle, this would mean mixing both forward and reverse mode. """)Ù$17c91ea8-acb7-4bbd-b0b0-0f8193f45303„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$17c91ea8-acb7-4bbd-b0b0-0f8193f45303«code_foldedäcodeÙ md"## 🚀 GPU acceleration âš¡"Ù$33bdaa23-707e-4227-b936-c5d7aaf2c48e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$33bdaa23-707e-4227-b936-c5d7aaf2c48e«code_foldedäcodeÚ¿qa(md"How to prove that ``A^* = A^\top`` ?", md""" We have ``\langle X, Y \rangle = \text{tr}(XY^\top)`` ([why ?](https://en.wikipedia.org/wiki/Trace_(linear_algebra)#Trace_of_a_product)) so, using the [cyclic property of the trace](https://en.wikipedia.org/wiki/Trace_(linear_algebra)#Cyclic_property) ```math \langle AX, Y \rangle = \text{tr}(AXY^\top) = \text{tr}(XY^\top A) = \text{tr}(X(A^\top Y)^\top) = \langle X, A^\top Y \rangle. ``` """)Ù$292b182f-b8a4-4ac6-89ad-ef71d40eca1f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$292b182f-b8a4-4ac6-89ad-ef71d40eca1f«code_foldedäcodeÙ md"## Two equivalent approaches"Ù$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0f«code_foldedäcodeÙ#md"## Memory usage of forward mode"Ù$85fc455c-36cf-4a4c-aa64-84a827884693„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$85fc455c-36cf-4a4c-aa64-84a827884693«code_foldedäcode½md"# Reverse Differentiation"Ù$66bd8159-581d-471c-841d-8e42e0de0496„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$66bd8159-581d-471c-841d-8e42e0de0496«code_foldedäcodeÚ›md""" ## Reverse mode on a DAG * Forward mode: we know how to propagate ``t`` through any graph. * Reverse mode on a chain: we know that ``r_{k-1} = \partial f_k^* r_k``. * Reverse mode on a DAG: â“ Consider the simplest DAG that is not a chain: ``x`` is used by both ``f_1`` and ``f_2``. The reverse pass brings two contributions ``\partial f_1^* r_1`` and ``\partial f_2^* r_2`` back to the same ``x``. """Ù$3a2132e7-7d69-42d7-89d3-b2d7679ad74f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$3a2132e7-7d69-42d7-89d3-b2d7679ad74f«code_foldedäcodeÚmd""" ## Forward on reverse **Forward pass**: Given ``s_1 = \text{Dual}(f_1(x), \frac{\partial f_1}{\partial x_j})`` 1. Compute ``s_2 = f_2(s_1)`` → same as forward on forward 2. Compute ``J_{f_2}(s_1)`` → same as forward on forward **Reverse pass**: Given ``r_2 = \text{Dual}((r_2)_1, (r_2)_2)``, compute ```math \begin{align} r_2 \cdot J_2 & = \text{Dual}((r_2)_1, (r_2)_2) \cdot \text{Dual}(J_2, H_{2j}) \cdot \\ & = \text{Dual}( (r_2)_1 \cdot J_2, (r_2)_2 \cdot J_2 + (r_2)_1 \cdot H_{2j}) \end{align} ``` """Ù$85303791-bdc4-468a-bc40-48ef2a186282„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$85303791-bdc4-468a-bc40-48ef2a186282«code_folded¤codeÙif CUDA.functional() X_gpu = CUDA.CuArray(X) y_gpu = CUDA.CuArray(y) W_gpu = CUDA.CuArray.(W) @time reverse_diff(W_gpu, X_gpu, y_gpu) endÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0«code_folded¤code»@time reverse_diff(W, X, y)Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab«code_folded¤codeÙšdata = DelimitedFiles.readdlm( last(save_image(URL("https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data"))), ',', Float64, );Ù$277bd2ce-fa7f-4288-be8a-0ddd8f23635c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$277bd2ce-fa7f-4288-be8a-0ddd8f23635c«code_foldedäcodeÚ‘md""" ## Two different takes on the multivariate chain rule The chain rule gives us ```math \frac{\partial f_3}{\partial x} (f_1(x), f_2(x)) = \partial_1 f_3(s_1, s_2) \cdot \frac{\partial s_1}{\partial x} + \partial_2 f_3(s_1, s_2) \cdot \frac{\partial s_2}{\partial x} ``` To compute this expression, we need the values of ``s_1(x)`` and ``s_2(x)`` as well as the derivatives ``\partial s_1 / \partial x`` and ``\partial s_2 / \partial x``. Common to forward and reverse: Given ``s_1, s_2``, computes **local** derivatives ``\partial_1 f_3(s_1, s_2)`` and ``\partial_2 f_3(s_1, s_2)``, shortened ``\partial_1 f_3, \partial_2 f_3`` for conciseness. """Ù$6c60f9ca-ba04-41e2-9625-c9e10f1a853b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853b«code_foldedäcodeÙ³qa(md"How to prevent this from happening ?", md"We should first compute a [*topological ordering*](https://en.wikipedia.org/wiki/Topological_sorting) and then follow this order.")Ù$778c40ff-4c9e-42fb-92a6-1e376837f6ef„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$778c40ff-4c9e-42fb-92a6-1e376837f6ef«code_folded¤codeÙlfunction forward_diff(W, X, y) [forward_diff(W, X, y, i, j) for i in axes(W[1], 1), j in axes(W[1], 2)] endÙ$16f611e8-8e41-43f3-830c-9976cb720b9f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$16f611e8-8e41-43f3-830c-9976cb720b9f«code_foldedäcodeÙÝqa(md"Which value of ``r_k`` is solution for this recurrence equation ?", md""" We find ``r_k = \text{Dual}(\frac{\partial f}{\partial s_k}, \frac{\partial^2 f}{\partial s_k \partial x_{\color{red}j}})`` as solution. """)Ù$8c202da6-1e13-43b8-a22b-94badcef2934„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8c202da6-1e13-43b8-a22b-94badcef2934«code_foldedäcodeÙ"md"## Putting everything together"Ù$53b21ec0-28e9-46cd-a92e-8afc189c3a11„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$53b21ec0-28e9-46cd-a92e-8afc189c3a11«code_folded¤codeÙÜfunction forward_diff(W, X, y, j, k) W1, W2 = W T_1 = onehot(j, axes(W1, 1)) * onehot(k, axes(W1, 2))' J_1, J_2 = forward_pass(W, X, y) only((W2 * (J_1 .* (T_1 * X))) * J_2') # only: 1x1 matrix -> scalar endÙ$45051873-3f0d-49b0-a9a6-bfde240594aa„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$45051873-3f0d-49b0-a9a6-bfde240594aa«code_folded¤codeÙ7struct Dual{T} value::T # s_k derivative::T # t_k endÙ$d1dbdd3f-9782-4fba-8c4e-819f152e6c30„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d1dbdd3f-9782-4fba-8c4e-819f152e6c30«code_foldedäcodeÙ0img("Blondel_Rouvet_Figure_8_8", :height => 200)Ù$e27db9a0-09ff-4ee5-a807-d98933b6bcf1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e27db9a0-09ff-4ee5-a807-d98933b6bcf1«code_foldedäcodeÚCqa(md"What is the complexity of reverse mode", md""" The adjoint trick gives ```math \langle J_n J_{n-1} \cdots J_2 J_1 \partial w, \partial y \rangle = \langle \partial w, J_1^\top J_2^\top \cdots J_{n-1}^\top J_n^\top \partial y \rangle. ``` So reverse differentation corresponds to multiplying the adjoint from right to left or equivalently the original matrices from left to right. This means computing the product in the following order: ```math \begin{align} J_{(n-1),n} & = J_n J_{n-1} && \Omega(d_nd_{n-1}d_{n-2})\\ J_{(n-2),n} & = J_{(n-1),n} J_{n-2} && \Omega(d_nd_{n-2}d_{n-3})\\ J_{(n-3),n} & = J_{(n-2),n} J_{n-3} && \Omega(d_nd_{n-3}d_{n-4})\\ \vdots & \quad \vdots\\ J_{1,n} & = J_{2,n} J_1 && \Omega(d_nd_1d_0)\\ \end{align} ``` We have a complexity of ```math \Omega(\sum_{i=1}^{n-1} d_nd_id_{i-1}). ``` """)Ù$4d76422e-711c-4f3e-87ab-0ce851bac064„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4d76422e-711c-4f3e-87ab-0ce851bac064«code_foldedäcodeÚZqa(md"What is the closed form expression for ``t_k`` in terms of the matrices ``J_k`` and ``H_{kj}`` ?", md""" We can prove by induction that the second part of the dual number ``t_k`` is: ```math \frac{\partial^2 f_k}{\partial x_i \partial x_j} = J_k \cdots J_2 H_{1j} e_i + J_k \cdots J_3 H_{2j} J_1 e_i + H_{kj} J_{k-1} \cdots J_1 e_i ``` """)Ù$6c441843-d6da-46b1-9463-d133ba74e66a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6c441843-d6da-46b1-9463-d133ba74e66a«code_foldedäcodeºfanout_figure(dup = false)Ù$722ad63a-c2ac-4ed6-b268-41d0f8b745f1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$722ad63a-c2ac-4ed6-b268-41d0f8b745f1«code_foldedäcodeµmd"`h` = $(h_slider)"Ù$40ed6c94-d2f9-4225-80b3-9060f04f8971„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$40ed6c94-d2f9-4225-80b3-9060f04f8971«code_foldedäcodeÙ0img("Blondel_Rouvet_Figure_8_6", :height => 350)Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739«code_foldedäcodeÚ…graph_example = let xâ‚ = GraphNode(1.0) xâ‚‚ = GraphNode(2.0) vâ‚ = exp(xâ‚) vâ‚‚ = exp(xâ‚‚) v₃ = xâ‚‚ * vâ‚‚ vâ‚„ = xâ‚ + v₃ vâ‚… = sqrt(vâ‚„) v₆ = xâ‚‚ * vâ‚ f = v₆ * vâ‚… names = IdDict( xâ‚ => "xâ‚", xâ‚‚ => "xâ‚‚", vâ‚ => "vâ‚", vâ‚‚ => "vâ‚‚", v₃ => "v₃", vâ‚„ => "vâ‚„", vâ‚… => "vâ‚…", v₆ => "v₆", f => "f", ) CGE.Graph(f; names) end;Ù$73ba544c-616a-4db1-b91d-0b20a7b8924b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$73ba544c-616a-4db1-b91d-0b20a7b8924b«code_foldedäcodeÙ#md"## Memory usage of reverse mode"Ù$586c2c6b-4a53-46d5-924b-c843e4c09859„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$586c2c6b-4a53-46d5-924b-c843e4c09859«code_foldedäcodeÙ&aside(md"Figure 8.1", v_offset = -150)Ù$98db9022-f8ff-4af3-9c81-89cf09771928„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$98db9022-f8ff-4af3-9c81-89cf09771928«code_foldedäcodeÙœmd""" ## Broadcasting (Scalar product way) ```math \begin{align} \langle f'.(X) \odot T, R \rangle = \langle T, f'.(X) \odot R \rangle. \end{align} ``` """Ù$e88ada81-d37f-4811-aa17-17eb7dd9d8a7„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e88ada81-d37f-4811-aa17-17eb7dd9d8a7«code_foldedäcodeÚ!md""" ## From forward to reverse mode Consider ``f(x_1, x_2) = x_1^2 + x_2^2`` at ``s_0 = (1, 2)``. We decompose it as ``f = f_2 \circ f_1`` and compute the **local** Jacobians: ```math \begin{align} f_1(x_1, x_2) & = \begin{bmatrix} x_1^2 \\ x_2^2 \end{bmatrix} & s_1 & = f_1(s_0) = \begin{bmatrix} 1 \\ 4 \end{bmatrix} & \partial f_1(s_0) & = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix}\\ f_2(x_1, x_2) & = x_1 + x_2 & s_2 & = f_2(s_1) = 5 & \partial f_2(s_1) & = \begin{bmatrix} 1 & 1 \end{bmatrix} \end{align} ``` Forward mode computes one column of the Jacobian ``\partial f(s_0)`` for each one-hot tangent ``t_0``: ```math \begin{align} t_0 & = \begin{bmatrix} 1 \\ 0 \end{bmatrix} & t_1 & = \partial f_1(s_0) \cdot t_0 = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 0 \end{bmatrix} & t_2 & = \partial f_2(s_1) \cdot t_1 = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 2 \\ 0 \end{bmatrix} = 2\\ t_0 & = \begin{bmatrix} 0 \\ 1 \end{bmatrix} & t_1 & = \partial f_1(s_0) \cdot t_0 = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 4 \end{bmatrix} & t_2 & = \partial f_2(s_1) \cdot t_1 = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} 0 \\ 4 \end{bmatrix} = 4 \end{align} ``` """Ù$7d79ff81-59e0-41f0-b2fe-70b41f44591f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7d79ff81-59e0-41f0-b2fe-70b41f44591f«code_foldedäcodeÚömd""" ## Notation * Let ``f_k : \mathbb{R}^{d_{k-1}} \to \mathbb{R}^{d_k}``. ``\partial f_k \triangleq \partial f_k(s_{k-1}) \in \mathbb{R}^{d_k \times d_{k-1}}``, ``\partial^2 f_k \triangleq \partial^2 f_k(s_{k-1}) \in \mathbb{R}^{d_k \times d_{k-1} \times d_{k-1}}`` is a 3D array/tensor. * Given ``v \in \mathbb{R}^{d_{k-1}}``, by the product ``(\partial^2 f_k \cdot v) \in \mathbb{R}^{d_k \times d_{k-1}}`` , we denote the contraction of the 3rd (or 2nd since the tensor is symmetric over its last 2 dimensions) dimension: ```math (\partial^2 f_k \cdot v)_{ij} = \sum_{l = 1}^{d_{k-1}} (\partial^2 f_k)_{ijl} \cdot v_l ``` * Given ``u \in \mathbb{R}^{d_k}``, by the product ``(u \cdot \partial^2 f_k) \in \mathbb{R}^{d_{k-1} \times d_{k-1}}`` , we denote the contraction of the 1st dimension. ```math (u \cdot \partial^2 f_k)_{ij} = \sum_{l = 1}^{d_k} u_l \cdot (\partial^2 f_k)_{lij} ``` * Both ``\partial^2 f_k \cdot v`` and ``u \cdot \partial^2 f_k`` are matrices so then we're back to matrix notations. """Ù$10e5c137-64a9-4282-b3fe-26e625625ae4„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$10e5c137-64a9-4282-b3fe-26e625625ae4«code_foldedäcodeÚžqa(md"What is the reverse pass of ``\text{dup}`` ?", md""" In forward mode, ``\partial \text{dup} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}`` maps ``t`` to ``(t, t)``. For the reverse pass to compute the same derivative, it must multiply by its adjoint ``\partial \text{dup}^* = \begin{bmatrix} 1 & 1 \end{bmatrix}``: ```math r_0 = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} \partial f_1^* r_1 \\ \partial f_2^* r_2 \end{bmatrix} = \partial f_1^* r_1 + \partial f_2^* r_2 ``` It's a **sum**! 🎉 So we initialize ``r_0 = 0`` and each function using ``x`` accumulates its contribution with ``r_0 \mathrel{\raise{0.19ex}{\scriptstyle+}} = \partial f_i^* r_i``. """)Ù$19578219-c6dd-4322-a2f4-44088ef640af„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$19578219-c6dd-4322-a2f4-44088ef640af«code_foldedäcodeÙHmd""" ```math f(x_1, x_2) = x_2 e^{x_1} \sqrt{x_1 + x_2 e^{x_2}} ``` """Ù$5b85a063-cf0f-4afa-89f4-420d7350ecc3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5b85a063-cf0f-4afa-89f4-420d7350ecc3«code_foldedäcodeÙahtml"The End
"Ù$895f4b81-51fe-4a1c-b958-1294743e3a9b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$895f4b81-51fe-4a1c-b958-1294743e3a9b«code_foldedäcodeÙ†VectNode = let T = Int Value = Union{ T, Vector{T}, LinearAlgebra.Adjoint{T,Vector{T}}, } CGE.Node{Value,DummyMetadata} end;´last_hot_reload_timeË ±cell_dependenciesÞ ªÙ$90850509-463d-44c7-88ae-4406aebd4be1„´precedence_heuristic §cell_idÙ$90850509-463d-44c7-88ae-4406aebd4be1´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$77a7de14-87d2-11ef-21ef-937b8239db5b„´precedence_heuristic §cell_idÙ$77a7de14-87d2-11ef-21ef-937b8239db5b´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e46fb3ff-b26f-4efb-aaa1-760e80017797„´precedence_heuristic §cell_idÙ$e46fb3ff-b26f-4efb-aaa1-760e80017797´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8e„´precedence_heuristic §cell_idÙ$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8e´downstream_cells_map€²upstream_cells_mapƒ¡+£zip¬chain_figure‘Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$78d8c8c9-568d-472a-9f06-a50b1cf2384b„´precedence_heuristic §cell_idÙ$78d8c8c9-568d-472a-9f06-a50b1cf2384b´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$9661a3b9-ae42-466e-967d-69230b9769b3„´precedence_heuristic §cell_idÙ$9661a3b9-ae42-466e-967d-69230b9769b3´downstream_cells_map€²upstream_cells_mapŽ¨VectNode‘Ù$895f4b81-51fe-4a1c-b958-1294743e3a9b¡:¨vector_n‘Ù$9a07a949-4ade-4325-8ae6-b05bdbbfae00§adjoint¤HTML®CGE.render_svg¢=>¤conj©CGE.Graph¤rand£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¦IdDict±CGE.capture_frame¡*Ù$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56„´precedence_heuristic §cell_idÙ$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56´downstream_cells_map¨scalar_n‘Ù$ee3299e2-3367-4c68-a050-c26650a97c8c²upstream_cells_map‹¡:«PlutoRunner¯Core.applicable¤Core¤Base¨Base.get‘Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b§@md_str¨getindex¦Slider¥@bind·PlutoRunner.create_bondÙ$7921c4c6-56b8-4c6f-a9be-cd1d9984680b„´precedence_heuristic §cell_idÙ$7921c4c6-56b8-4c6f-a9be-cd1d9984680b´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$54457653-f029-4368-8bc0-15d69f22bfcd„´precedence_heuristic §cell_idÙ$54457653-f029-4368-8bc0-15d69f22bfcd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$37bfd8ed-8983-4a89-9ba5-a90a35151729„´precedence_heuristic §cell_idÙ$37bfd8ed-8983-4a89-9ba5-a90a35151729´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$a580ef44-234a-4ed1-b007-920651415427„´precedence_heuristic §cell_idÙ$a580ef44-234a-4ed1-b007-920651415427´downstream_cells_map€²upstream_cells_mapŠ¡/¡^¡X‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3¡-¡y‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2¤size¤tanh¡W‘Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16£sum¡*Ù$e9d6af05-63f7-4d31-85e2-b0233df23608„´precedence_heuristic §cell_idÙ$e9d6af05-63f7-4d31-85e2-b0233df23608´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$edad5700-9b04-47c6-90f8-b6bac0897340„´precedence_heuristic §cell_idÙ$edad5700-9b04-47c6-90f8-b6bac0897340´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779„´precedence_heuristic §cell_idÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$dc4feb58-d2cf-4a97-aaed-7f4593fc9732„´precedence_heuristic §cell_idÙ$dc4feb58-d2cf-4a97-aaed-7f4593fc9732´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$cbfc0129-9361-4edb-a467-1456a1f3aeae„´precedence_heuristic §cell_idÙ$cbfc0129-9361-4edb-a467-1456a1f3aeae´downstream_cells_map…£img™Ù$fa20b8db-9ac7-490d-b8d8-8d57469d24e4Ù$cc2a09a1-c949-4b09-816b-b49ba7ca8983Ù$d2b8fa5c-c604-4093-a2dd-5c95f2eaa676Ù$d1dbdd3f-9782-4fba-8c4e-819f152e6c30Ù$673c3acc-0009-416a-91bb-f57c1fe8eefcÙ$e8c60922-5bbf-45b5-8311-18c8f8525623Ù$2f8baccc-19d1-44d6-b71f-0243fd8696baÙ$cd6d807d-6238-44ce-9267-1614679f527aÙ$40ed6c94-d2f9-4225-80b3-9060f04f8971£URL’Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6abÙ$cbfc0129-9361-4edb-a467-1456a1f3aeaeªsave_image‘Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab§imgpath¤Path‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae²upstream_cells_mapÞ °HypertextLiteral‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¸HypertextLiteral.content¦String·HypertextLiteral.Result¤@htlµPlutoUI.LocalResource§PlutoUI‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6£URL‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¨joinpath¡!·HypertextLiteral.Bypass¿HypertextLiteral.attribute_pair¢inªstartswithÙ&PlutoTeachingTools.RobustLocalResource¥split¨@__DIR__¤Path‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¡*²PlutoTeachingTools‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6Ù$b92d17a9-8481-458a-bc0a-efb7333cbc6e„´precedence_heuristic §cell_idÙ$b92d17a9-8481-458a-bc0a-efb7333cbc6e´downstream_cells_map€²upstream_cells_mapƒ©TwoColumn§@md_str¨getindexÙ$ee3299e2-3367-4c68-a050-c26650a97c8c„´precedence_heuristic §cell_idÙ$ee3299e2-3367-4c68-a050-c26650a97c8c´downstream_cells_map€²upstream_cells_mapŽ¡:§adjoint¤HTML®CGE.render_svg©eachindexªScalarNode‘Ù$df1f29ce-026d-4935-83aa-0d597cb166dd¤conj©CGE.Graph¨scalar_n‘Ù$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56¤rand£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¦IdDict±CGE.capture_frame¡*Ù$db7aaa4c-5568-4619-a9c4-00c7916c321a„´precedence_heuristic §cell_idÙ$db7aaa4c-5568-4619-a9c4-00c7916c321a´downstream_cells_map¨features‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3²upstream_cells_mapƒ¡:¤data‘Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab«permutedimsÙ$068ef840-04cb-4de4-9512-721f57e5750e„´precedence_heuristic §cell_idÙ$068ef840-04cb-4de4-9512-721f57e5750e´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$95eb9960-89f0-4edc-8943-77a75bce2b80„´precedence_heuristic §cell_idÙ$95eb9960-89f0-4edc-8943-77a75bce2b80´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$fa5dba01-a3f7-452c-877e-352d578ecf51„´precedence_heuristic §cell_idÙ$fa5dba01-a3f7-452c-877e-352d578ecf51´downstream_cells_map€²upstream_cells_map‰¤BaseBase.getindex©@html_str¤hbox£Div§@md_str¢=>¤Dict®Base.Docs.HTMLÙ$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0„´precedence_heuristic §cell_idÙ$5aff8e66-787d-4dc5-a9b1-0fdec25ce0f0´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$69c08fab-c317-462c-817c-3f841a8a0941„´precedence_heuristic §cell_idÙ$69c08fab-c317-462c-817c-3f841a8a0941´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b„´precedence_heuristic §cell_idÙ$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b´downstream_cells_map„ªStepSlider‘Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16¨Base.get”Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16Ù$2e8dd18a-b784-4f7d-ac7d-e1f88ba2ca56Ù$9a07a949-4ade-4325-8ae6-b05bdbbfae00Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccef¨#467#val©Base.show²upstream_cells_mapÞ ¥Union°HypertextLiteral‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¥first¸HypertextLiteral.content·HypertextLiteral.Result¤step¤@htlº#___this_pluto_module_name¤Bool¤Real¤last·HypertextLiteral.BypassAbstractRange¿HypertextLiteral.attribute_pair¤Base¢IO¤@doc§nothing©@MIME_strÙ$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16„´precedence_heuristic §cell_idÙ$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16´downstream_cells_mapªgraph_step‘Ù$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08²upstream_cells_mapŠ¬graph_frames‘Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96¤BaseªStepSlider‘Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b¥@bind¨Base.get‘Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b«PlutoRunner¯Core.applicable©eachindex·PlutoRunner.create_bond¤CoreÙ$2fcf25d2-fd51-4c13-b57c-86236aceead2„´precedence_heuristic §cell_idÙ$2fcf25d2-fd51-4c13-b57c-86236aceead2´downstream_cells_map¡y•Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16Ù$a580ef44-234a-4ed1-b007-920651415427Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_mapƒ§Float32¡-§targets‘Ù$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4eÙ$1c44706b-fd7d-4826-84a3-73e2db5adadd„´precedence_heuristic §cell_idÙ$1c44706b-fd7d-4826-84a3-73e2db5adadd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e50f8f52-a73f-4186-af5e-b4ca2c021142„´precedence_heuristic §cell_idÙ$e50f8f52-a73f-4186-af5e-b4ca2c021142´downstream_cells_map€²upstream_cells_map‚¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa£abs‘Ù$9988fc4a-cedc-499b-a334-048cc13de000Ù$626abc7c-87ef-4838-9f0a-294cf0a4be6a„´precedence_heuristic §cell_idÙ$626abc7c-87ef-4838-9f0a-294cf0a4be6a´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$157c413e-b166-48ff-b28e-58b88f018a94„´precedence_heuristic §cell_idÙ$157c413e-b166-48ff-b28e-58b88f018a94´downstream_cells_map‚¬chain_figure’Ù$a3ba6cd3-b2c8-42c0-9483-cb0c0c88ed8eÙ$310d6a10-b887-4dd0-ba93-f901726ddbbc¨#463#val²upstream_cells_map‹¥Union¡/º#___this_pluto_module_name¡-£Any®mathjax_figure‘Ù$79df35ff-3718-4a9c-a384-4e8c12175ab8¥Tuple¡+¤@doc£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¡*Ù$1d56075c-e28d-46c9-9a0a-210079172388„´precedence_heuristic §cell_idÙ$1d56075c-e28d-46c9-9a0a-210079172388´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$ceaeb177-7a6a-4062-9659-56bebce0e77b„´precedence_heuristic §cell_idÙ$ceaeb177-7a6a-4062-9659-56bebce0e77b´downstream_cells_map§abs_bis‘Ù$9862c791-31e8-4d59-8610-a929d72ea9c3²upstream_cells_map…¦isless¡<¡-¡>¦ifelseÙ$a9a4841d-18cc-4386-aa38-017771650afb„´precedence_heuristic §cell_idÙ$a9a4841d-18cc-4386-aa38-017771650afb´downstream_cells_map€²upstream_cells_mapfanout_figure‘Ù$fa886ef3-192f-4daa-b59b-aca0c6365111Ù$43d2559f-8902-4c54-8fdf-cb268b6f868c„´precedence_heuristic §cell_idÙ$43d2559f-8902-4c54-8fdf-cb268b6f868c´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6„´precedence_heuristic §cell_idÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6´downstream_cells_map¦Base.*“Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_mapƒ¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa¤Base¡+Ù$1a0026be-7975-478c-a447-f66ea95f551d„´precedence_heuristic §cell_idÙ$1a0026be-7975-478c-a447-f66ea95f551d´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$c73f79c6-a28f-4c7a-89e5-8d70a245a210„´precedence_heuristic §cell_idÙ$c73f79c6-a28f-4c7a-89e5-8d70a245a210´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$3556d366-0bc7-4239-b4f6-3f9bd28780e0„´precedence_heuristic §cell_idÙ$3556d366-0bc7-4239-b4f6-3f9bd28780e0´downstream_cells_map«Base.isless²upstream_cells_mapƒ¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa¤Base¤RealÙ$9a07a949-4ade-4325-8ae6-b05bdbbfae00„´precedence_heuristic §cell_idÙ$9a07a949-4ade-4325-8ae6-b05bdbbfae00´downstream_cells_map¨vector_n‘Ù$9661a3b9-ae42-466e-967d-69230b9769b3²upstream_cells_map‹¡:«PlutoRunner¯Core.applicable¤Core¤Base¨Base.get‘Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b§@md_str¨getindex¦Slider¥@bind·PlutoRunner.create_bondÙ$b5c3e2ef-3d47-4f44-b968-d04734be2f16„´precedence_heuristic §cell_idÙ$b5c3e2ef-3d47-4f44-b968-d04734be2f16´downstream_cells_map¡W”Ù$a580ef44-234a-4ed1-b007-920651415427Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_map†§Float32¡h‘Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccef¡X‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3¡y‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2¤size¤randÙ$9415a6ed-c05e-4487-b0be-f342ec7424cd„´precedence_heuristic §cell_idÙ$9415a6ed-c05e-4487-b0be-f342ec7424cd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f5d3714d-3900-4dbe-9079-978a44584d1d„´precedence_heuristic §cell_idÙ$f5d3714d-3900-4dbe-9079-978a44584d1d´downstream_cells_map©normalise‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3²upstream_cells_map…®Statistics.std¡/¯Statistics.mean¡-ªStatistics‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6Ù$3803804a-f44d-4f56-bd59-fb1401d8fb9e„´precedence_heuristic §cell_idÙ$3803804a-f44d-4f56-bd59-fb1401d8fb9e´downstream_cells_map¦Base.-“Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_map‚¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa¤BaseÙ$03b6aa6d-7517-4906-9430-302516d0653b„´precedence_heuristic §cell_idÙ$03b6aa6d-7517-4906-9430-302516d0653b´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$607000ef-fb7f-4204-b543-3cb6bb75ed71„´precedence_heuristic §cell_idÙ$607000ef-fb7f-4204-b543-3cb6bb75ed71´downstream_cells_map€²upstream_cells_mapˆ£abs‘Ù$9988fc4a-cedc-499b-a334-048cc13de000¡/°PlotlyLight.Plot§collect¡-²PlotlyLight.Config«PlotlyLight‘Ù$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47¥rangeÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65f„´precedence_heuristic §cell_idÙ$f9a37c14-5b62-4e8d-96a0-2c41db73e65f´downstream_cells_mapCGE.pullback!‘Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96²upstream_cells_mapˆ¡/¦typeof©GraphNode‘Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3¡+£exp¤sqrt£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¡*Ù$2ae277d5-8be4-4ea9-a3fc-8ad601577c3a„´precedence_heuristic §cell_idÙ$2ae277d5-8be4-4ea9-a3fc-8ad601577c3a´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47„´precedence_heuristic §cell_idÙ$b7d4a91e-3c52-4f86-9a0d-5e8c1f2b6d47´downstream_cells_map«PlotlyLight‘Ù$607000ef-fb7f-4204-b543-3cb6bb75ed71²upstream_cells_map€Ù$da5895e7-af99-46ff-9f53-36529d1ca456„´precedence_heuristic §cell_idÙ$da5895e7-af99-46ff-9f53-36529d1ca456´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e121f72b-fe6d-491a-ab03-ef92154c61ca„´precedence_heuristic §cell_idÙ$e121f72b-fe6d-491a-ab03-ef92154c61ca´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$347a1500-43aa-4078-9ea3-480bc8a4c291„´precedence_heuristic §cell_idÙ$347a1500-43aa-4078-9ea3-480bc8a4c291´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b„´precedence_heuristic §cell_idÙ$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b´downstream_cells_mapDummyMetadata’Ù$df1f29ce-026d-4935-83aa-0d597cb166ddÙ$895f4b81-51fe-4a1c-b958-1294743e3a9b²upstream_cells_map€Ù$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2„´precedence_heuristic §cell_idÙ$221a9c57-0072-4fb6-b61f-d7e3ab9c4be2´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$0980ef52-526d-421d-8c63-04272f16cbdd„´precedence_heuristic §cell_idÙ$0980ef52-526d-421d-8c63-04272f16cbdd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$cd6d807d-6238-44ce-9267-1614679f527a„´precedence_heuristic §cell_idÙ$cd6d807d-6238-44ce-9267-1614679f527a´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$383461cb-9f51-42b2-ab76-d9580b39409e„´precedence_heuristic §cell_idÙ$383461cb-9f51-42b2-ab76-d9580b39409e´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092„´precedence_heuristic §cell_idÙ$e9c14738-2c1f-43d8-94f9-fb5440d74092´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3„´precedence_heuristic §cell_idÙ$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3´downstream_cells_map¡X•Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16Ù$a580ef44-234a-4ed1-b007-920651415427Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1aÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_mapƒ§Float32©normalise‘Ù$f5d3714d-3900-4dbe-9079-978a44584d1d¨features‘Ù$db7aaa4c-5568-4619-a9c4-00c7916c321aÙ$79df35ff-3718-4a9c-a384-4e8c12175ab8„´precedence_heuristic §cell_idÙ$79df35ff-3718-4a9c-a384-4e8c12175ab8´downstream_cells_map‚¨#468#val®mathjax_figure’Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$fa886ef3-192f-4daa-b59b-aca0c6365111²upstream_cells_mapÞ ¥Union¡/°HypertextLiteral‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¸HypertextLiteral.content¥push!·HypertextLiteral.Result¤@htlº#___this_pluto_module_name£Any¤HTML·HypertextLiteral.Bypass¥Tuple¡+¤@doc£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47Ù HypertextLiteral.attribute_valueÙ$cc2a09a1-c949-4b09-816b-b49ba7ca8983„´precedence_heuristic §cell_idÙ$cc2a09a1-c949-4b09-816b-b49ba7ca8983´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$9ca3b930-3d9d-460c-9f20-e3c135477b05„´precedence_heuristic §cell_idÙ$9ca3b930-3d9d-460c-9f20-e3c135477b05´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$ac52550e-3287-427f-b957-ac61bc850f4d„´precedence_heuristic §cell_idÙ$ac52550e-3287-427f-b957-ac61bc850f4d´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$bebe2bb8-cf74-4eaa-85d1-712b990b5985„´precedence_heuristic §cell_idÙ$bebe2bb8-cf74-4eaa-85d1-712b990b5985´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$4de9baec-f444-47b8-b9e6-7f3d9e9609b1„´precedence_heuristic §cell_idÙ$4de9baec-f444-47b8-b9e6-7f3d9e9609b1´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$802edb3a-4809-4c50-920b-25f7bdc255dd„´precedence_heuristic §cell_idÙ$802edb3a-4809-4c50-920b-25f7bdc255dd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$2ca19ff6-ec22-4327-aea2-80bdca55ccef„´precedence_heuristic §cell_idÙ$2ca19ff6-ec22-4327-aea2-80bdca55ccef´downstream_cells_map‚¨h_slider’Ù$722ad63a-c2ac-4ed6-b268-41d0f8b745f1Ù$6ded46e6-1c89-4875-addb-8c709e949bb1¡h’Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1a²upstream_cells_map‰¤Base¡:¥@bind¨Base.get‘Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72b«PlutoRunner¦Slider¯Core.applicable¤Core·PlutoRunner.create_bondÙ$5f6529a1-4ace-4dd0-a7e2-f51070eab695„´precedence_heuristic §cell_idÙ$5f6529a1-4ace-4dd0-a7e2-f51070eab695´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$d2b8fa5c-c604-4093-a2dd-5c95f2eaa676„´precedence_heuristic §cell_idÙ$d2b8fa5c-c604-4093-a2dd-5c95f2eaa676´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4„´precedence_heuristic §cell_idÙ$c4e91a07-2b8d-4f1e-9a63-7d05be21c8f4´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$3f1068e0-97e2-444a-961d-2af55a4d797b„´precedence_heuristic §cell_idÙ$3f1068e0-97e2-444a-961d-2af55a4d797b´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$69ae57b4-4e4c-44a2-aca7-d0fff89b9566„´precedence_heuristic §cell_idÙ$69ae57b4-4e4c-44a2-aca7-d0fff89b9566´downstream_cells_map«Base.isless²upstream_cells_mapƒ¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa¤Base¤RealÙ$1994bf51-adf1-4b07-ab4c-f47552d90826„´precedence_heuristic §cell_idÙ$1994bf51-adf1-4b07-ab4c-f47552d90826´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$6ded46e6-1c89-4875-addb-8c709e949bb1„´precedence_heuristic §cell_idÙ$6ded46e6-1c89-4875-addb-8c709e949bb1´downstream_cells_map€²upstream_cells_map„¨h_slider‘Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccef¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$6ddc06c0-3f5d-4cc9-8060-dd6997e0f662„´precedence_heuristic §cell_idÙ$6ddc06c0-3f5d-4cc9-8060-dd6997e0f662´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$c1da4130-5936-499f-bb9b-574e01136eca„´precedence_heuristic §cell_idÙ$c1da4130-5936-499f-bb9b-574e01136eca´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$29287c62-e892-448f-a9d5-12785ae4a02f„´precedence_heuristic §cell_idÙ$29287c62-e892-448f-a9d5-12785ae4a02f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$7f75e3f3-c4e2-402d-be7b-336a4f65042a„´precedence_heuristic §cell_idÙ$7f75e3f3-c4e2-402d-be7b-336a4f65042a´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9bbbda1f-74a6-458b-a084-9d034d6c291f„´precedence_heuristic §cell_idÙ$9bbbda1f-74a6-458b-a084-9d034d6c291f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$200e2b1b-065a-4d60-b5fd-86700e4c811a„´precedence_heuristic §cell_idÙ$200e2b1b-065a-4d60-b5fd-86700e4c811a´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$2b631fcd-2703-42df-8a75-2fdff64b3311„´precedence_heuristic §cell_idÙ$2b631fcd-2703-42df-8a75-2fdff64b3311´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$74063eb5-be06-466a-a2f1-e266c35295ea„´precedence_heuristic §cell_idÙ$74063eb5-be06-466a-a2f1-e266c35295ea´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$673c3acc-0009-416a-91bb-f57c1fe8eefc„´precedence_heuristic §cell_idÙ$673c3acc-0009-416a-91bb-f57c1fe8eefc´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$28df733e-7db9-4e78-9121-52d8e6ca7591„´precedence_heuristic §cell_idÙ$28df733e-7db9-4e78-9121-52d8e6ca7591´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$df1f29ce-026d-4935-83aa-0d597cb166dd„´precedence_heuristic §cell_idÙ$df1f29ce-026d-4935-83aa-0d597cb166dd´downstream_cells_mapªScalarNode‘Ù$ee3299e2-3367-4c68-a050-c26650a97c8c²upstream_cells_mapƒ£IntDummyMetadata‘Ù$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47Ù$af404768-0663-4bc3-81dd-6931b3a486be„´precedence_heuristic §cell_idÙ$af404768-0663-4bc3-81dd-6931b3a486be´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9862c791-31e8-4d59-8610-a929d72ea9c3„´precedence_heuristic §cell_idÙ$9862c791-31e8-4d59-8610-a929d72ea9c3´downstream_cells_map€²upstream_cells_map‚¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa§abs_bis‘Ù$ceaeb177-7a6a-4062-9659-56bebce0e77bÙ$8deca676-8a0b-41eb-b7a0-4d65e1158b0b„´precedence_heuristic §cell_idÙ$8deca676-8a0b-41eb-b7a0-4d65e1158b0b´downstream_cells_map€²upstream_cells_mapŠ¤BaseBase.getindex©@html_str¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4¤hbox£Div§@md_str¢=>¤Dict®Base.Docs.HTMLÙ$7aba4481-bed6-44b7-85e3-3b2deb2b2fd7„´precedence_heuristic §cell_idÙ$7aba4481-bed6-44b7-85e3-3b2deb2b2fd7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e8c60922-5bbf-45b5-8311-18c8f8525623„´precedence_heuristic §cell_idÙ$e8c60922-5bbf-45b5-8311-18c8f8525623´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$f010e781-f41e-4861-af1c-32cf5a76ce4d„´precedence_heuristic §cell_idÙ$f010e781-f41e-4861-af1c-32cf5a76ce4d´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$8b8682e4-8adb-4fb1-a013-054b1d9750d7„´precedence_heuristic §cell_idÙ$8b8682e4-8adb-4fb1-a013-054b1d9750d7´downstream_cells_map£f_1‘Ù$bc93fb96-1097-4b22-a077-558f5662efec²upstream_cells_map¡*Ù$402f0d1d-0a3f-404a-8019-afc92d154c85„´precedence_heuristic §cell_idÙ$402f0d1d-0a3f-404a-8019-afc92d154c85´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e211118d-eba8-467e-ae4a-665f1df02934„´precedence_heuristic §cell_idÙ$e211118d-eba8-467e-ae4a-665f1df02934´downstream_cells_map£f_2‘Ù$bc93fb96-1097-4b22-a077-558f5662efec²upstream_cells_map¡-Ù$b16f6225-1949-4b6d-a4b0-c5c230eb4c7f„´precedence_heuristic §cell_idÙ$b16f6225-1949-4b6d-a4b0-c5c230eb4c7f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$c1ae773e-0f2d-4c1c-bae2-25f45bb28fbb„´precedence_heuristic §cell_idÙ$c1ae773e-0f2d-4c1c-bae2-25f45bb28fbb´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$5a79c09e-2a33-43d1-a5dc-caba3db467dd„´precedence_heuristic §cell_idÙ$5a79c09e-2a33-43d1-a5dc-caba3db467dd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$2adc9595-8829-4d35-be90-a7718c2e7ce7„´precedence_heuristic §cell_idÙ$2adc9595-8829-4d35-be90-a7718c2e7ce7´downstream_cells_map¬forward_pass’Ù$53b21ec0-28e9-46cd-a92e-8afc189c3a11Ù$35f8cf4f-3fcb-4e27-9462-244406d7800e²upstream_cells_map†¡/¡^¡-¤tanh¤size¡*Ù$fa20b8db-9ac7-490d-b8d8-8d57469d24e4„´precedence_heuristic §cell_idÙ$fa20b8db-9ac7-490d-b8d8-8d57469d24e4´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$e4a9c57b-c811-428b-a6b8-191b78d5f361„´precedence_heuristic §cell_idÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4e„´precedence_heuristic §cell_idÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4e´downstream_cells_map§targets‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2²upstream_cells_mapƒ¡:¤data‘Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab«permutedimsÙ$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08„´precedence_heuristic §cell_idÙ$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08´downstream_cells_map€²upstream_cells_map†¬graph_frames‘Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96ªgraph_step‘Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16®CGE.render_svg¤HTML£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47graph_example‘Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739Ù$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496„´precedence_heuristic §cell_idÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$d492a165-3e70-4c18-b0d9-57fc2e8a1b96„´precedence_heuristic §cell_idÙ$d492a165-3e70-4c18-b0d9-57fc2e8a1b96´downstream_cells_map¬graph_frames’Ù$d18fb5c2-6e47-4a90-b3d1-90c7af4e2b16Ù$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08²upstream_cells_map‰±CGE.capture_frame¥push!µCGE.topological_order²CGE.forward_framesCGE.pullback!‘Ù$f9a37c14-5b62-4e8d-96a0-2c41db73e65f£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47§isemptygraph_example‘Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739§reverseÙ$4174d41c-877b-4878-b76e-442991907af0„´precedence_heuristic §cell_idÙ$4174d41c-877b-4878-b76e-442991907af0´downstream_cells_map€²upstream_cells_mapƒ¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa¡-¡*Ù$9988fc4a-cedc-499b-a334-048cc13de000„´precedence_heuristic §cell_idÙ$9988fc4a-cedc-499b-a334-048cc13de000´downstream_cells_map£abs’Ù$607000ef-fb7f-4204-b543-3cb6bb75ed71Ù$e50f8f52-a73f-4186-af5e-b4ca2c021142²upstream_cells_map„¦isless¡<¡-¦ifelseÙ$24d52ef8-927e-46c8-b455-74ccdb33d3ca„´precedence_heuristic §cell_idÙ$24d52ef8-927e-46c8-b455-74ccdb33d3ca´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$fa886ef3-192f-4daa-b59b-aca0c6365111„´precedence_heuristic §cell_idÙ$fa886ef3-192f-4daa-b59b-aca0c6365111´downstream_cells_map‚¨#464#valfanout_figure’Ù$6c441843-d6da-46b1-9463-d133ba74e66aÙ$a9a4841d-18cc-4386-aa38-017771650afb²upstream_cells_mapŠ¥Union¡/¥Tupleº#___this_pluto_module_name¡+¤@doc®mathjax_figure‘Ù$79df35ff-3718-4a9c-a384-4e8c12175ab8¡-£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¡*Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47„´precedence_heuristic §cell_idÙ$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47´downstream_cells_map£CGEœÙ$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08Ù$f9a37c14-5b62-4e8d-96a0-2c41db73e65fÙ$ee3299e2-3367-4c68-a050-c26650a97c8cÙ$df1f29ce-026d-4935-83aa-0d597cb166ddÙ$9661a3b9-ae42-466e-967d-69230b9769b3Ù$895f4b81-51fe-4a1c-b958-1294743e3a9bÙ$79df35ff-3718-4a9c-a384-4e8c12175ab8Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$fa886ef3-192f-4daa-b59b-aca0c6365111Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96²upstream_cells_map€Ù$81deb227-a822-4857-a584-a51cc8ff51f4„´precedence_heuristic §cell_idÙ$81deb227-a822-4857-a584-a51cc8ff51f4´downstream_cells_map‚¢qaÜ !Ù$8deca676-8a0b-41eb-b7a0-4d65e1158b0bÙ$383461cb-9f51-42b2-ab76-d9580b39409eÙ$10e5c137-64a9-4282-b3fe-26e625625ae4Ù$83ef86e0-bcfb-42ee-a574-16758606423aÙ$28df733e-7db9-4e78-9121-52d8e6ca7591Ù$626abc7c-87ef-4838-9f0a-294cf0a4be6aÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853bÙ$bab3a3cb-0ad2-4ea5-a15c-6593fc22e496Ù$c73f79c6-a28f-4c7a-89e5-8d70a245a210Ù$ac52550e-3287-427f-b957-ac61bc850f4dÙ$11eadcc1-68bf-4ca7-9603-ef46468b4779Ù$74063eb5-be06-466a-a2f1-e266c35295eaÙ$88534196-9f4a-430c-a534-805177ba718dÙ$347a1500-43aa-4078-9ea3-480bc8a4c291Ù$c733ca7e-b57e-4218-9bd4-238ab5749143Ù$33bdaa23-707e-4227-b936-c5d7aaf2c48eÙ$7921c4c6-56b8-4c6f-a9be-cd1d9984680bÙ$edad5700-9b04-47c6-90f8-b6bac0897340Ù$9ca3b930-3d9d-460c-9f20-e3c135477b05Ù$e27db9a0-09ff-4ee5-a807-d98933b6bcf1Ù$95eb9960-89f0-4edc-8943-77a75bce2b80Ù$9afd31c9-e938-417a-8c3f-e0d1ba88f95bÙ$f010e781-f41e-4861-af1c-32cf5a76ce4dÙ$4317bba0-723f-4cdc-9d52-67033540a8d2Ù$6ded46e6-1c89-4875-addb-8c709e949bb1Ù$78d8c8c9-568d-472a-9f06-a50b1cf2384bÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3fÙ$4d76422e-711c-4f3e-87ab-0ce851bac064Ù$24d52ef8-927e-46c8-b455-74ccdb33d3caÙ$4de9baec-f444-47b8-b9e6-7f3d9e9609b1Ù$16f611e8-8e41-43f3-830c-9976cb720b9fÙ$200e2b1b-065a-4d60-b5fd-86700e4c811aÙ$e4a9c57b-c811-428b-a6b8-191b78d5f361¬_inline_html²upstream_cells_mapˆ°HypertextLiteral‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¸HypertextLiteral.content¤@htl·HypertextLiteral.Result¨Markdown¦sprint¤HTML·HypertextLiteral.BypassÙ$c733ca7e-b57e-4218-9bd4-238ab5749143„´precedence_heuristic §cell_idÙ$c733ca7e-b57e-4218-9bd4-238ab5749143´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$88534196-9f4a-430c-a534-805177ba718d„´precedence_heuristic §cell_idÙ$88534196-9f4a-430c-a534-805177ba718d´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$310d6a10-b887-4dd0-ba93-f901726ddbbc„´precedence_heuristic §cell_idÙ$310d6a10-b887-4dd0-ba93-f901726ddbbc´downstream_cells_map€²upstream_cells_mapˆ¢==¡/¡+¡-¢!=©eachindex¡*¬chain_figure‘Ù$157c413e-b166-48ff-b28e-58b88f018a94Ù$7331c447-c0a6-4a47-af00-36efc449d9d9„´precedence_heuristic §cell_idÙ$7331c447-c0a6-4a47-af00-36efc449d9d9´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$4317bba0-723f-4cdc-9d52-67033540a8d2„´precedence_heuristic §cell_idÙ$4317bba0-723f-4cdc-9d52-67033540a8d2´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$6bd5ea51-54c4-46ba-8eb3-6427225e5249„´precedence_heuristic §cell_idÙ$6bd5ea51-54c4-46ba-8eb3-6427225e5249´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9527686f-24e1-40bb-9a5d-22575aafec9b„´precedence_heuristic §cell_idÙ$9527686f-24e1-40bb-9a5d-22575aafec9b´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3f„´precedence_heuristic §cell_idÙ$f23ca90f-b567-4257-ae41-ec15c57c1f3f´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6„´precedence_heuristic §cell_idÙ$fa6dd3f7-7b57-483b-ba0f-90c9bb7bb6a6´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5„´precedence_heuristic§cell_idÙ$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5´downstream_cells_map…MathTeXEngine‘Ù$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5¬LaTeXStrings¦_latex¬_math_script¬_math_italic²upstream_cells_mapÞ ¬Base.requireMathTeXEngine‘Ù$d26cba1c-27d8-4ee7-99e5-6c0e0fc6b6e5§Cstring«islowercase«latexstring¦C_NULL¡-§replace¨joinpath£get§readdir¦length©Base.UUID¡+¥ccall£Ptr¤Dict¥Cvoid¨endswith¤CharªBase.PkgId¢=>¤Cint¢==¤Base¦@r_str¦stringµBase.Libc.Libdl.dlsym¦pkgdirÙ$0e13e63d-fd08-4cc1-aa37-851c537afbef„´precedence_heuristic §cell_idÙ$0e13e63d-fd08-4cc1-aa37-851c537afbef´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$2f8baccc-19d1-44d6-b71f-0243fd8696ba„´precedence_heuristic §cell_idÙ$2f8baccc-19d1-44d6-b71f-0243fd8696ba´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$191f33ab-b535-4cb1-96b3-6b2f10fd633a„´precedence_heuristic §cell_idÙ$191f33ab-b535-4cb1-96b3-6b2f10fd633a´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9b4a78d8-e6da-41dd-b922-b35c895eee1a„´precedence_heuristic §cell_idÙ$9b4a78d8-e6da-41dd-b922-b35c895eee1a´downstream_cells_map€²upstream_cells_mapÞ ¦isless¡h‘Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccef¡X‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3«Base.gc_num»Base.Threads.lock_profiling¯Base.time_print«Base.string³Base.gc_alloc_count¡W‘Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16¬forward_diff’Ù$53b21ec0-28e9-46cd-a92e-8afc189c3a11Ù$778c40ff-4c9e-42fb-92a6-1e376837f6ef¦Base./¡y‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2¬Base.GC_Diff¥@time¡<¿Base.cumulative_compile_time_ns¦Base.-‘Ù$3803804a-f44d-4f56-bd59-fb1401d8fb9e¤Base¬Base.time_ns¨Base.===¦Base.*‘Ù$63bdc7f5-89a5-4061-9e42-6588e4cd96c6¾Base.cumulative_compile_timingÙ$32a0bec0-4f1d-4dba-a5db-422741f88808„´precedence_heuristic §cell_idÙ$32a0bec0-4f1d-4dba-a5db-422741f88808´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$35f8cf4f-3fcb-4e27-9462-244406d7800e„´precedence_heuristic §cell_idÙ$35f8cf4f-3fcb-4e27-9462-244406d7800e´downstream_cells_map¬reverse_diff’Ù$87c6a5bc-82bf-44a5-b4d6-6d50285348c0Ù$85303791-bdc4-468a-bc40-48ef2a186282²upstream_cells_map„¤conj§adjoint¡*¬forward_pass‘Ù$2adc9595-8829-4d35-be90-a7718c2e7ce7Ù$885bc5c9-aefc-4d8a-a4da-6062c64eaa41„´precedence_heuristic §cell_idÙ$885bc5c9-aefc-4d8a-a4da-6062c64eaa41´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$83ef86e0-bcfb-42ee-a574-16758606423a„´precedence_heuristic §cell_idÙ$83ef86e0-bcfb-42ee-a574-16758606423a´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$bc93fb96-1097-4b22-a077-558f5662efec„´precedence_heuristic §cell_idÙ$bc93fb96-1097-4b22-a077-558f5662efec´downstream_cells_map€²upstream_cells_map„¤Dual‘Ù$45051873-3f0d-49b0-a9a6-bfde240594aa£f_2‘Ù$e211118d-eba8-467e-ae4a-665f1df02934£âˆ˜£f_1‘Ù$8b8682e4-8adb-4fb1-a013-054b1d9750d7Ù$533bf4bc-494b-47c2-91b6-7b30221765f5„´precedence_heuristic §cell_idÙ$533bf4bc-494b-47c2-91b6-7b30221765f5´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6„´precedence_heuristic§cell_idÙ$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6´downstream_cells_map‡°HypertextLiteral•Ù$40baa108-eb68-433f-9917-ac334334f198Ù$79df35ff-3718-4a9c-a384-4e8c12175ab8Ù$e8b1c40d-27a6-4f39-b95e-6d3a0f81c72bÙ$cbfc0129-9361-4edb-a467-1456a1f3aeaeÙ$81deb227-a822-4857-a584-a51cc8ff51f4¤CUDA‘Ù$85303791-bdc4-468a-bc40-48ef2a186282§PlutoUI’Ù$40baa108-eb68-433f-9917-ac334334f198Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¬OneHotArrays²ExperimentalLayout²PlutoTeachingTools’Ù$40baa108-eb68-433f-9917-ac334334f198Ù$cbfc0129-9361-4edb-a467-1456a1f3aeaeªStatistics‘Ù$f5d3714d-3900-4dbe-9079-978a44584d1d²upstream_cells_map€Ù$8194cce9-115f-44b4-9f28-caa88b764185„´precedence_heuristic §cell_idÙ$8194cce9-115f-44b4-9f28-caa88b764185´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$40baa108-eb68-433f-9917-ac334334f198„´precedence_heuristic §cell_idÙ$40baa108-eb68-433f-9917-ac334334f198´downstream_cells_map€²upstream_cells_map‰°HypertextLiteral‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¸HypertextLiteral.contentÙ$PlutoTeachingTools.ChooseDisplayMode·HypertextLiteral.Result¤@htl·PlutoUI.TableOfContents§PlutoUI‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6²PlutoTeachingTools‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6·HypertextLiteral.BypassÙ$906e5199-f2d2-4816-a195-6d2b1dee9403„´precedence_heuristic §cell_idÙ$906e5199-f2d2-4816-a195-6d2b1dee9403´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$7f459370-eb6f-4d82-bf99-1528e136ca5c„´precedence_heuristic §cell_idÙ$7f459370-eb6f-4d82-bf99-1528e136ca5c´downstream_cells_map‚LinearAlgebra‘Ù$895f4b81-51fe-4a1c-b958-1294743e3a9b®DelimitedFiles‘Ù$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab²upstream_cells_map€Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3„´precedence_heuristic §cell_idÙ$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3´downstream_cells_map…¬CGE.metadata§Adjoint‘Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3©GraphNode’Ù$f9a37c14-5b62-4e8d-96a0-2c41db73e65fÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739²CGE.seed_metadata!±CGE.metadata_rows²upstream_cells_map†§Adjoint‘Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3¤Bool§Float64£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¢=>¤TypeÙ$9afd31c9-e938-417a-8c3f-e0d1ba88f95b„´precedence_heuristic §cell_idÙ$9afd31c9-e938-417a-8c3f-e0d1ba88f95b´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$17c91ea8-acb7-4bbd-b0b0-0f8193f45303„´precedence_heuristic §cell_idÙ$17c91ea8-acb7-4bbd-b0b0-0f8193f45303´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$33bdaa23-707e-4227-b936-c5d7aaf2c48e„´precedence_heuristic §cell_idÙ$33bdaa23-707e-4227-b936-c5d7aaf2c48e´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$292b182f-b8a4-4ac6-89ad-ef71d40eca1f„´precedence_heuristic §cell_idÙ$292b182f-b8a4-4ac6-89ad-ef71d40eca1f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0f„´precedence_heuristic §cell_idÙ$f4d1ee7c-4a01-4b2d-aa9b-ec41ceb0ad0f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$85fc455c-36cf-4a4c-aa64-84a827884693„´precedence_heuristic §cell_idÙ$85fc455c-36cf-4a4c-aa64-84a827884693´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$66bd8159-581d-471c-841d-8e42e0de0496„´precedence_heuristic §cell_idÙ$66bd8159-581d-471c-841d-8e42e0de0496´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$3a2132e7-7d69-42d7-89d3-b2d7679ad74f„´precedence_heuristic §cell_idÙ$3a2132e7-7d69-42d7-89d3-b2d7679ad74f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$85303791-bdc4-468a-bc40-48ef2a186282„´precedence_heuristic §cell_idÙ$85303791-bdc4-468a-bc40-48ef2a186282´downstream_cells_mapƒ¥X_gpu¥y_gpu¥W_gpu²upstream_cells_mapÞ ¡X‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3«Base.gc_num»Base.Threads.lock_profiling¯Base.time_print¤CUDA‘Ù$f1ba3d3c-d0a5-4290-ab73-9ce34bd5e5f6¬CUDA.CuArray¯CUDA.functional«Base.string¡W‘Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16³Base.gc_alloc_count¦Base./¡y‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2¬Base.GC_Diff¥@time¬reverse_diff‘Ù$35f8cf4f-3fcb-4e27-9462-244406d7800e¿Base.cumulative_compile_time_ns¦Base.-‘Ù$3803804a-f44d-4f56-bd59-fb1401d8fb9e¤Base¬Base.time_ns¨Base.===¦Base.*‘Ù$63bdc7f5-89a5-4061-9e42-6588e4cd96c6¾Base.cumulative_compile_timingÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0„´precedence_heuristic §cell_idÙ$87c6a5bc-82bf-44a5-b4d6-6d50285348c0´downstream_cells_map€²upstream_cells_mapÞ ¬Base.GC_Diff¥@time¬reverse_diff‘Ù$35f8cf4f-3fcb-4e27-9462-244406d7800e¡X‘Ù$0bcadb3a-4880-4e6c-bccb-b09df8ad8fa3¯Base.time_print«Base.gc_num»Base.Threads.lock_profiling³Base.gc_alloc_count«Base.string¡W‘Ù$b5c3e2ef-3d47-4f44-b968-d04734be2f16¦Base.-‘Ù$3803804a-f44d-4f56-bd59-fb1401d8fb9e¿Base.cumulative_compile_time_ns¤Base¬Base.time_ns¦Base./¦Base.*‘Ù$63bdc7f5-89a5-4061-9e42-6588e4cd96c6¡y‘Ù$2fcf25d2-fd51-4c13-b57c-86236aceead2¨Base.===¾Base.cumulative_compile_timingÙ$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab„´precedence_heuristic §cell_idÙ$0a9d0913-a1bb-4f47-8b85-db8714d0f6ab´downstream_cells_map¤data’Ù$db7aaa4c-5568-4619-a9c4-00c7916c321aÙ$6cfc019e-7b97-4bf1-b1f5-de9468b9ab4e²upstream_cells_map†£URL‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¶DelimitedFiles.readdlmªsave_image‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae§Float64¤last®DelimitedFiles‘Ù$7f459370-eb6f-4d82-bf99-1528e136ca5cÙ$277bd2ce-fa7f-4288-be8a-0ddd8f23635c„´precedence_heuristic §cell_idÙ$277bd2ce-fa7f-4288-be8a-0ddd8f23635c´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853b„´precedence_heuristic §cell_idÙ$6c60f9ca-ba04-41e2-9625-c9e10f1a853b´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$778c40ff-4c9e-42fb-92a6-1e376837f6ef„´precedence_heuristic §cell_idÙ$778c40ff-4c9e-42fb-92a6-1e376837f6ef´downstream_cells_map¬forward_diff‘Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1a²upstream_cells_map¤axesÙ$16f611e8-8e41-43f3-830c-9976cb720b9f„´precedence_heuristic §cell_idÙ$16f611e8-8e41-43f3-830c-9976cb720b9f´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$8c202da6-1e13-43b8-a22b-94badcef2934„´precedence_heuristic §cell_idÙ$8c202da6-1e13-43b8-a22b-94badcef2934´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$53b21ec0-28e9-46cd-a92e-8afc189c3a11„´precedence_heuristic §cell_idÙ$53b21ec0-28e9-46cd-a92e-8afc189c3a11´downstream_cells_map¬forward_diff‘Ù$9b4a78d8-e6da-41dd-b922-b35c895eee1a²upstream_cells_map‡¤conj¤only¦onehot¤axes§adjoint¡*¬forward_pass‘Ù$2adc9595-8829-4d35-be90-a7718c2e7ce7Ù$45051873-3f0d-49b0-a9a6-bfde240594aa„´precedence_heuristic §cell_idÙ$45051873-3f0d-49b0-a9a6-bfde240594aa´downstream_cells_map¤Dual˜Ù$3803804a-f44d-4f56-bd59-fb1401d8fb9eÙ$63bdc7f5-89a5-4061-9e42-6588e4cd96c6Ù$4174d41c-877b-4878-b76e-442991907af0Ù$bc93fb96-1097-4b22-a077-558f5662efecÙ$3556d366-0bc7-4239-b4f6-3f9bd28780e0Ù$69ae57b4-4e4c-44a2-aca7-d0fff89b9566Ù$e50f8f52-a73f-4186-af5e-b4ca2c021142Ù$9862c791-31e8-4d59-8610-a929d72ea9c3²upstream_cells_map€Ù$d1dbdd3f-9782-4fba-8c4e-819f152e6c30„´precedence_heuristic §cell_idÙ$d1dbdd3f-9782-4fba-8c4e-819f152e6c30´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$e27db9a0-09ff-4ee5-a807-d98933b6bcf1„´precedence_heuristic §cell_idÙ$e27db9a0-09ff-4ee5-a807-d98933b6bcf1´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$4d76422e-711c-4f3e-87ab-0ce851bac064„´precedence_heuristic §cell_idÙ$4d76422e-711c-4f3e-87ab-0ce851bac064´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$6c441843-d6da-46b1-9463-d133ba74e66a„´precedence_heuristic §cell_idÙ$6c441843-d6da-46b1-9463-d133ba74e66a´downstream_cells_map€²upstream_cells_mapfanout_figure‘Ù$fa886ef3-192f-4daa-b59b-aca0c6365111Ù$722ad63a-c2ac-4ed6-b268-41d0f8b745f1„´precedence_heuristic §cell_idÙ$722ad63a-c2ac-4ed6-b268-41d0f8b745f1´downstream_cells_map€²upstream_cells_mapƒ¨h_slider‘Ù$2ca19ff6-ec22-4327-aea2-80bdca55ccef§@md_str¨getindexÙ$40ed6c94-d2f9-4225-80b3-9060f04f8971„´precedence_heuristic §cell_idÙ$40ed6c94-d2f9-4225-80b3-9060f04f8971´downstream_cells_map€²upstream_cells_map‚£img‘Ù$cbfc0129-9361-4edb-a467-1456a1f3aeae¢=>Ù$c71e4f82-0d35-49ba-97c6-84a1bd50e739„´precedence_heuristic §cell_idÙ$c71e4f82-0d35-49ba-97c6-84a1bd50e739´downstream_cells_mapgraph_example’Ù$e52c7a3b-8d19-4c60-a7f2-31b6ec9d5a08Ù$d492a165-3e70-4c18-b0d9-57fc2e8a1b96²upstream_cells_map‰©CGE.Graph©GraphNode‘Ù$b3c8d70e-9a41-4d26-85fb-6e02f19ca4d3¡+£exp¤sqrt£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47¦IdDict¡*¢=>Ù$73ba544c-616a-4db1-b91d-0b20a7b8924b„´precedence_heuristic §cell_idÙ$73ba544c-616a-4db1-b91d-0b20a7b8924b´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$586c2c6b-4a53-46d5-924b-c843e4c09859„´precedence_heuristic §cell_idÙ$586c2c6b-4a53-46d5-924b-c843e4c09859´downstream_cells_map€²upstream_cells_mapƒ§@md_str¨getindex¥asideÙ$98db9022-f8ff-4af3-9c81-89cf09771928„´precedence_heuristic §cell_idÙ$98db9022-f8ff-4af3-9c81-89cf09771928´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e88ada81-d37f-4811-aa17-17eb7dd9d8a7„´precedence_heuristic §cell_idÙ$e88ada81-d37f-4811-aa17-17eb7dd9d8a7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$7d79ff81-59e0-41f0-b2fe-70b41f44591f„´precedence_heuristic §cell_idÙ$7d79ff81-59e0-41f0-b2fe-70b41f44591f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$10e5c137-64a9-4282-b3fe-26e625625ae4„´precedence_heuristic §cell_idÙ$10e5c137-64a9-4282-b3fe-26e625625ae4´downstream_cells_map€²upstream_cells_mapƒ¢qa‘Ù$81deb227-a822-4857-a584-a51cc8ff51f4§@md_str¨getindexÙ$19578219-c6dd-4322-a2f4-44088ef640af„´precedence_heuristic §cell_idÙ$19578219-c6dd-4322-a2f4-44088ef640af´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$5b85a063-cf0f-4afa-89f4-420d7350ecc3„´precedence_heuristic §cell_idÙ$5b85a063-cf0f-4afa-89f4-420d7350ecc3´downstream_cells_map€²upstream_cells_mapƒ¤Base©@html_str®Base.Docs.HTMLÙ$895f4b81-51fe-4a1c-b958-1294743e3a9b„´precedence_heuristic §cell_idÙ$895f4b81-51fe-4a1c-b958-1294743e3a9b´downstream_cells_map¨VectNode‘Ù$9661a3b9-ae42-466e-967d-69230b9769b3²upstream_cells_map†LinearAlgebra‘Ù$7f459370-eb6f-4d82-bf99-1528e136ca5c¥Union¦Vector£IntDummyMetadata‘Ù$4d4e11a2-b6a1-490f-9c7b-ad617dbb395b£CGE‘Ù$a06be2d9-73c1-4f85-b2e7-18d3c5a90f47©shortpath«autodiff.jl®last_save_timeËAڸة‚Ñö«notebook_idÙ$539281e4-d3ab-11f1-9208-43f90e1aa5da¥nbpkgŠbusy_packagesÙ,waiting_for_permission_but_probably_disabled§enabledðterminal_outputs°HypertextLiteralÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...MathTeXEngineÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...¤CUDAÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...¬LaTeXStringsÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...«PlotlyLightÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...ªStatisticsÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...ªnbpkg_syncÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...¬OneHotArraysÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...¸ComputationGraphExplorerÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...®DelimitedFilesÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...§PlutoUIÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...LinearAlgebraÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...²PlutoTeachingToolsÚ€ [0m[1mResolving...[22m [90m===[39m [36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Project.toml` [32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_iafuemnmnk/Manifest.toml` [90m[ce32ce21] [39m[92m+ MSVCRuntime_jll v14.51.36247+0[39m [90m[deac9b47] [39m[93m↑ LibCURL_jll v8.18.0+1 ⇒ v8.22.0+0[39m [90m[e37daf67] [39m[93m↑ LibGit2_jll v1.9.1+0 ⇒ v1.9.7+0[39m [90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m [90m[458c3c95] [39m[93m↑ OpenSSL_jll v3.5.6+0 ⇒ v3.5.8+0[39m [90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m [90m[83775a58] [39m[93m↑ Zlib_jll v1.3.1+2 ⇒ v1.3.2+0[39m [90m[8e850ede] [39m[93m↑ nghttp2_jll v1.67.1+0 ⇒ v1.70.0+0[39m [90m[3f19e933] [39m[93m↑ p7zip_jll v17.8.2+0 ⇒ v17.8.3+0[39m [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m Waiting for notebook process to start... Done. Starting precompilation...¶waiting_for_permission·restart_recommended_msgÀ´restart_required_msgÀ²installed_versionsŽ°HypertextLiteral¥1.0.0MathTeXEngine¥0.6.9¤CUDA¥6.4.2§PlutoUI¦0.7.83¬LaTeXStrings¥1.4.1«PlotlyLight¦0.13.1ªStatistics¦stdlibLinearAlgebra¦stdlibÙ!__internal_julia_manifest_version¦1.13.3¸__internal_julia_version¦1.13.3¬OneHotArrays¦0.2.11¸ComputationGraphExplorer¥0.2.2²PlutoTeachingTools¥0.4.7®DelimitedFiles¦stdlib¯install_time_nsεÝÉk¬instantiatedÃpluto_version¦v1.1.0«in_temp_dirÂ