Þ ¤pathÙ>/home/runner/work/LINMA2472/LINMA2472/Lectures/transformers.jl¬cell_resultsÞ “Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3Чrunning§runtimeÏ ¨™q¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°œVkã·has_pluto_hook_features¤bodyÚß
softmax is then applied to each column:
$$\text{softmax}(K^\top Q/\sqrt{d_k})$$
Division by $\sqrt{d_k}$ scales the input of softmax to preferable regions [VSP+17; Secton 3.2.1].
Illustrated on the right from [BCB16; Figure 3(a)].
$$\text{Attention}(V, K, Q) = V\text{softmax}(K^\top Q/\sqrt{d_k})$$
°persist_js_stateÂÙ$8c27b182-0c3c-4c19-9619-df62b7dd6bf0Чrunning§runtimeÎ
�–&¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$8c27b182-0c3c-4c19-9619-df62b7dd6bf0¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°–Á4ð·has_pluto_hook_features¤bodyڟ💡 Key idea In the model for $\hat{p}(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}})$, incorporate sub-models
$$\begin{align}
\bar{p}(&x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}})\\
\bar{p}(&x_{-1} | x_{-2}, \ldots, x_{-n_\text{ctx}})\\
& \quad\qquad\vdots\\
\bar{p}(&x_{-n_\text{ctx}+1} | x_{-n_\text{ctx}}).
\end{align}$$
Mask prevents $\hat{p}$ to look at inputs in the future:
$$M
=
\begin{bmatrix}
0 & 0 & \cdots & 0\\
-\infty & 0 & \ddots & \vdots\\
\vdots & \ddots & \ddots & 0\\
-\infty & \cdots & -\infty & 0
\end{bmatrix}$$
°persist_js_stateÂÙ$a3380780-ec5e-46c1-a0e0-af64b3e1da1cЧrunning§runtimeÎ ³Ä¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$a3380780-ec5e-46c1-a0e0-af64b3e1da1c¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°—+*o·has_pluto_hook_features¤bodyÚRWhy is the GPU utilization so low when generating tokens ?
Generating one token requires a matrix-vector product with each weight matrix so each weight read from the GPU memory is only used for 2 FLOPs. Decoding is therefore limited by the memory bandwidth, not by the FLOPs. Data centres batch the queries of many users together so that each weight read from memory is used for a matrix-matrix product. A local GPU serving a single user cannot do that.
°persist_js_stateÂÙ$9a8eef1b-27c0-4d57-a389-53708ade9058Чrunning§runtimeÎ ¾a¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$9a8eef1b-27c0-4d57-a389-53708ade9058¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�œø·has_pluto_hook_features¤bodyÚ¥Once $x_0$ is generated, the context window moves to $x_{-n_\text{ctx}+1}, \ldots, x_0$. 💡 For each layer $i < N$, the keys and values of $\hat{y}_{i,j}$ for $j < 0$ were already computed for the previous token, let's cache them! Since the positions are independent given these keys and values, we only need to compute the new column $\hat{y}_{i,0}$ of each layer $i$.
°persist_js_stateÂÙ$f39305ea-f7f5-440e-ac55-c83e27f6e7fcЧrunning§runtimeÎlRHí¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f39305ea-f7f5-440e-ac55-c83e27f6e7fc¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeÙ"application/vnd.pluto.table+object¬rootassignee¤llms²last_run_timestampËAÚ¼°žbª·has_pluto_hook_features¤bodyƒ¨objectid°7cba9af84f5df7b7¦schema‚¥types˜ªDataString«DataString?«DataString?ªDataString«DataString?«DataString?«DataString?«DataString?¥names˜¤NameªNum params£Ref°``n_\text{voc}``°``d_\text{emb}``°``n_\text{ctx}``¯``d_\text{ff}``©Tokenizer¤rowsœ’˜’¬"Gemini-1.5"ªtext/plain’§missingªtext/plain’ª"[TGLB24]"ªtext/plain’¦"256k"ªtext/plain’§missingªtext/plain’¥"10M"ªtext/plain’§missingªtext/plain’¯"SentencePiece"ªtext/plain’˜’ª"Gemini-1"ªtext/plain’ÙX"1.[8B/3.25B](https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf)"ªtext/plain’ª"[TABA24]"ªtext/plain’¦"256k"ªtext/plain’§missingªtext/plain’¥"10M"ªtext/plain’§missingªtext/plain’¯"SentencePiece"ªtext/plain’˜’©"Gemma-2"ªtext/plain’¥"27B"ªtext/plain’ª"[TRPS24]"ªtext/plain’¦"256k"ªtext/plain’¦"4608"ªtext/plain’¦"8192"ªtext/plain’§"73728"ªtext/plain’¯"SentencePiece"ªtext/plain’˜’©"Gemma-2"ªtext/plain’¤"9B"ªtext/plain’ª"[TRPS24]"ªtext/plain’¦"256k"ªtext/plain’¦"3584"ªtext/plain’¦"8192"ªtext/plain’§"28672"ªtext/plain’¯"SentencePiece"ªtext/plain’˜’©"Gemma-2"ªtext/plain’¤"2B"ªtext/plain’ª"[TRPS24]"ªtext/plain’¦"256k"ªtext/plain’¦"2304"ªtext/plain’¦"8192"ªtext/plain’§"18432"ªtext/plain’¯"SentencePiece"ªtext/plain’˜’§"Gemma"ªtext/plain’¤"7B"ªtext/plain’ª"[TMHD24]"ªtext/plain’¦"256k"ªtext/plain’¦"3072"ªtext/plain’¦"8192"ªtext/plain’§"49152"ªtext/plain’¯"SentencePiece"ªtext/plain’˜’§"Gemma"ªtext/plain’¤"2B"ªtext/plain’ª"[TMHD24]"ªtext/plain’¦"256k"ªtext/plain’¦"2048"ªtext/plain’¦"8192"ªtext/plain’§"32768"ªtext/plain’¯"SentencePiece"ªtext/plain’˜’§"GPT-2"ªtext/plain’¦"1.5B"ªtext/plain’ª"[RWCL19]"ªtext/plain’Ù("[50k](https://github.com/openai/gpt-2)"ªtext/plain’Ùq"[768](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/configuration_gpt2.py)"ªtext/plain’¦"1024"ªtext/plain’¦"3072"ªtext/plain’ª"tiktoken"ªtext/plain’ ˜’©"Llama-2"ªtext/plain’¤"7B"ªtext/plain’ª"[TMSA23]"ªtext/plain’Ù3"[32k](https://github.com/meta-llama/llama-models)"ªtext/plain’ÙN"[4096](https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/llama2)"ªtext/plain’Ù2"[4k](https://github.com/meta-llama/llama-models)"ªtext/plain’§missingªtext/plain’¯"SentencePiece"ªtext/plain’
˜’¨"GPT-4o"ªtext/plain’§missingªtext/plain’§missingªtext/plain’ÙX"[200k](https://github.com/kaisugi/gpt4_vocab_list/blob/main/o200k_base_vocab_list.txt)"ªtext/plain’§missingªtext/plain’¦"128k"ªtext/plain’§missingªtext/plain’ª"tiktoken"ªtext/plain¤more’˜’¥"big"ªtext/plain’§missingªtext/plain’ª"[VSPU17]"ªtext/plain’¥"37k"ªtext/plain’¦"1024"ªtext/plain’§missingªtext/plain’¦"4096"ªtext/plain’§missingªtext/plain°persist_js_stateÂÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313Чrunning§runtimeÎ s'¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š\Ù·has_pluto_hook_features¤bodyÚ´
[SVL14] I. Sutskever, O. Vinyals and Q. V. Le. Sequence to Sequence Learning with Neural Networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14 (MIT Press, Cambridge, MA, USA, Dec 2014); pp. 3104–3112. Accessed on Oct 23, 2024.
°persist_js_stateÂÙ$c7f318b9-30e6-4b79-b7da-52f70904d246Чrunning§runtimeÎ P ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$c7f318b9-30e6-4b79-b7da-52f70904d246¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°œÛåÚ·has_pluto_hook_features¤bodyÙ+substitute (generic function with 1 method)°persist_js_stateÂÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfcЧrunning§runtimeÎ #Ƹdepends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfc¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š[!·has_pluto_hook_features¤bodyÚ
[HZRS16] K. He, X. Zhang, S. Ren and J. Sun. Identity Mappings in Deep Residual Networks. In: Computer Vision – ECCV 2016, edited by B. Leibe, J. Matas, N. Sebe and M. Welling (Springer International Publishing, Cham, 2016); pp. 630–645.
[RWC+19] A. Radford, J. Wu, R. Child et al. Language Models Are Unsupervised Multitask Learners (2019). Accessed on Oct 23, 2024.
[SLP+23] J. Su, Y. Lu, S. Pan et al. RoFormer: Enhanced Transformer with Rotary Position Embedding (Nov 2023), arXiv:2104.09864. Accessed on Nov 12, 2024.
°persist_js_stateÂÙ$8b78360a-21cb-4574-a84d-46ea4d0cedb1Чrunning§runtimeÎ –©¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$8b78360a-21cb-4574-a84d-46ea4d0cedb1¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°–âØ,·has_pluto_hook_features¤bodyÚWµ
°persist_js_stateÂÙ$95ec4140-9147-11ef-2af4-5528bad0e6f5Чrunning§runtimeÎ Á\¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$95ec4140-9147-11ef-2af4-5528bad0e6f5¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�†ZS·has_pluto_hook_features¤bodyÙtLarge Language Models (LLMs)
°persist_js_stateÂÙ$77f446ac-6030-48f2-9bea-93c427f9fcb9Чrunning§runtimeÎþS¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$77f446ac-6030-48f2-9bea-93c427f9fcb9¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°–xzc·has_pluto_hook_features¤bodyÙ/softmax_lookup (generic function with 1 method)°persist_js_stateÂÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4aЧrunning§runtimeÎ R¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4a¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�›Š‹·has_pluto_hook_features¤bodyÙ\
°persist_js_stateÂÙ$d54b5390-0ec0-4ff8-ab18-51726482ca46Чrunning§runtimeÎ ïë¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d54b5390-0ec0-4ff8-ab18-51726482ca46¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�šZ›·has_pluto_hook_features¤bodyÙPExtensions of RNNs
°persist_js_stateÂÙ$f572e113-b36b-4a6b-96c7-c26f100e1ad4Чrunning§runtimeÎ ó4¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f572e113-b36b-4a6b-96c7-c26f100e1ad4¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�¯þX·has_pluto_hook_features¤bodyÙ6Utils
°persist_js_stateÂÙ$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89Чrunning§runtimeÎ —ñ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�®›9·has_pluto_hook_features¤bodyÚ
21.6 GWh ≈ 6,000 Belgian households for a year.
Experiments and development add a significant amount on top of the final training run : ≈ 50% for OLMo 2 Morrison et al., 2025.
The COâ‚‚ depends a lot on the electricity grid : BLOOM was trained on the French grid with 57 gCOâ‚‚/kWh Luccioni et al., 2022.
°persist_js_stateÂÙ$a21fbc70-9137-4d0e-8c8c-cbdc5269778fЧrunning§runtimeÎ \ð¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$a21fbc70-9137-4d0e-8c8c-cbdc5269778f¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™<·has_pluto_hook_features¤bodyÚ Recently, Mamba suggests a solution to the complexity issue [GD24]. As it scales better with $n_\text{ctx}$, it is even suggested to get rid of the tokenizer : [WGYR24].
°persist_js_stateÂÙ$ba00b790-67ac-49b8-86ee-6df42107f245Чrunning§runtimeÎ^õ™¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$ba00b790-67ac-49b8-86ee-6df42107f245¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°—&†¥·has_pluto_hook_features¤bodyÚA dense transformer with $N$ parameters needs $\approx 2N$ FLOPs per generated token (one multiplication and one addition per weight), thanks to the KV cache. For a mixture of experts, $N$ is the number of active parameters. An NVIDIA H100 performs $\approx 989 \cdot 10^{12}$ FLOP/s (dense BF16) at 700 W so $\approx 1.4 \cdot 10^{12}$ FLOP/J at peak utilization.
°persist_js_stateÂÙ$c3db7eb2-356a-428f-9777-6369662d8b06Чrunning§runtimeÎ ¤Ã¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$c3db7eb2-356a-428f-9777-6369662d8b06¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�—Åv·has_pluto_hook_features¤bodyÙmNote that the new tokens can also be part of the most frequence pair!
°persist_js_stateÂÙ$9e898325-e9e2-45bd-af74-3dd86f00f7b5Чrunning§runtimeÎ ‰,¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$9e898325-e9e2-45bd-af74-3dd86f00f7b5¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�™Ô�·has_pluto_hook_features¤bodyÚ¡Consider one-hot encoding with vocabulary size $n_\text{voc}$ and a bigram model
$$\hat{p}(x_0 | x_{-1}) = \text{softmax}(W_d \tanh(\cdots\tanh(W_1 x_{-1})\cdots)$$
The matrix $W_d$ has $n_\text{voc}$ rows and $W_1$ has $n_\text{voc}$ columns → issue if $n_\text{voc}$ is large
Embedding : Use vectors $c_1, \ldots, c_{n_\text{voc}} \in \mathbb{R}^{d_\text{emb}}$ with embedding size (aka hidden size) $d_\text{emb} \ll n_\text{voc}$.
Equivalently, we still use one-hot encoding but we add an encoder $C \in \mathbb{R}^{d_\text{emb} \times n_\text{voc}}$ and decoder $D \in \mathbb{R}^{n_\text{voc} \times d_\text{emb}}$
$$\hat{p}(x_0 | x_{-1}) = \text{softmax}(D W_d \tanh(\cdots\tanh(W_1 C x_{-1})\cdots)$$
°persist_js_stateÂÙ$97463c54-7cc7-4497-a8a6-6422f5f582bdЧrunning§runtimeÎ “R¦¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$97463c54-7cc7-4497-a8a6-6422f5f582bd¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š 7e·has_pluto_hook_features¤bodyÚ.
[TGL+24] G. Team, P. Georgiev, V. I. Lei et al. Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context (Aug 2024), arXiv:2403.05530. Accessed on Nov 11, 2024.
[TAB+24] G. Team, R. Anil, S. Borgeaud et al. Gemini: A Family of Highly Capable Multimodal Models (Jun 2024), arXiv:2312.11805. Accessed on Nov 11, 2024.
[TMH+24] G. Team, T. Mesnard, C. Hardin et al. Gemma: Open Models Based on Gemini Research and Technology (Apr 2024), arXiv:2403.08295. Accessed on Nov 11, 2024.
[TRP+24] G. Team, M. Riviere, S. Pathak et al. Gemma 2: Improving Open Language Models at a Practical Size (Oct 2024), arXiv:2408.00118. Accessed on Nov 11, 2024.
[SHB16] R. Sennrich, B. Haddow and A. Birch. Neural Machine Translation of Rare Words with Subword Units (Jun 2016), arXiv:1508.07909. Accessed on Oct 23, 2024.
[RWC+19] A. Radford, J. Wu, R. Child et al. Language Models Are Unsupervised Multitask Learners (2019). Accessed on Oct 23, 2024.
[BMR+20] T. B. Brown, B. Mann, N. Ryder et al. Language Models Are Few-Shot Learners (Jul 2020), arXiv:2005.14165. Accessed on Nov 11, 2024.
[TMS+23] H. Touvron, L. Martin, K. Stone et al. Llama 2: Open Foundation and Fine-Tuned Chat Models (Jul 2023), arXiv:2307.09288. Accessed on Oct 23, 2024.
[YSF+23] L. Yu, D. Simig, C. Flaherty et al. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (May 2023), arXiv:2305.07185. Accessed on Oct 23, 2024.
°persist_js_stateÂÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012Чrunning§runtimeÎ !º¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š"·has_pluto_hook_features¤bodyÚ
[CvMBB14] K. Cho, B. van Merrienboer, D. Bahdanau and Y. Bengio. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches (Oct 2014), arXiv:1409.1259. Accessed on Nov 11, 2024.
[Gra14] A. Graves. Generating Sequences With Recurrent Neural Networks (Jun 2014), arXiv:1308.0850. Accessed on Nov 11, 2024.
[GBC16] I. Goodfellow, Y. Bengio and A. Courville. Deep Learning (MIT Press, 2016). Accessed on Aug 28, 2024.
[GD24] A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (May 2024), arXiv:2312.00752. Accessed on Nov 11, 2024.
[WGYR24] J. Wang, T. Gangavarapu, J. N. Yan and A. M. Rush. MambaByte: Token-free Selective State Space Model (Aug 2024), arXiv:2401.13660. Accessed on Nov 11, 2024.
°persist_js_stateÂÙ$89305cae-098f-4644-9109-d00f1e3bc04cЧrunning§runtimeÎ ]œ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$89305cae-098f-4644-9109-d00f1e3bc04c¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°œ±Ñö·has_pluto_hook_features¤bodyÙ+pair_stats (generic function with 1 method)°persist_js_stateÂÙ$bcf7667f-f99b-4d10-af84-5d3879f1db5dЧrunning§runtimeÎ 2® ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$bcf7667f-f99b-4d10-af84-5d3879f1db5d¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°•†ùø·has_pluto_hook_features¤bodyÚÎWhat difference do you expect with respect to the previous model ?
The products $W_1C$ and $DW_d$ have the same dimension as the matrices $W_1$ and $W_d$ of the previous model. So the expressive power of the model was not improved while we increased the number of parameters and we potentially made the loss function "even more nonconvex".
If the hidden dimension (i.e., the number of rows of $C$ / columns of $W_1$ or the number of rows of $W_d$ / columns of $D$) is much smaller than $n_\text{voc}$, then it's faster to compute $W_1(Cx)$. Moreover, we are forcing the matrix $W_1C$ to have a low rank compared the model without $C$. This means less expressivness but it might also prevent overfitting so the case isn't so clear.
The case become clearer when the input embedding $C$ is shared between more than one character, i.e., $n_\text{ctx} > 1$. Same for the output embedding, $D$ is useful when it is not preceded by a linear with which it can just be merged.
°persist_js_stateÂÙ$25b79953-fd7c-46c1-b760-d57c09910981Чrunning§runtimeÎ
úƸdepends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$25b79953-fd7c-46c1-b760-d57c09910981¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™C÷has_pluto_hook_features¤bodyÚ{How does the number of parameters of transformers compare with [BDV00] or RNNs for large $n_\text{ctx}$ ?
The number of parameters of the transformer does not depend on $n_\text{ctx}$.
The number of parameters of [BDV00] depends linearly with $n_\text{ctx}$. Assuming that the number of hidden neurons scales proportionally with $n_\text{ctx}$, the number of parameters even scales quadratically with $n_\text{ctx}$!
For RNNs, if the dimension of the internal state scales proportionally with $n_\text{ctx}$, the number of parameters is also proportional with $n_\text{ctx}$! If the dimension of the internal state is kept too small, increasing the context won't be so helpful, due to encoder bottleneck, see next slide.
°persist_js_stateÂÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076Чrunning§runtimeÎ ™/¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š~·has_pluto_hook_features¤bodyÚØ
[BCB16] D. Bahdanau, K. Cho and Y. Bengio. Neural Machine Translation by Jointly Learning to Align and Translate (May 2016), arXiv:1409.0473. Accessed on Oct 23, 2024.
[VSP+17] A. Vaswani, N. Shazeer, N. Parmar et al. Attention Is All You Need. In: Advances in Neural Information Processing Systems, Vol. 30 (Curran Associates, Inc., 2017). Accessed on Oct 11, 2024.
°persist_js_stateÂÙ$b9a962a6-4f4f-4676-95b6-0c2cfd536031Чrunning§runtimeÎ ;7¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$b9a962a6-4f4f-4676-95b6-0c2cfd536031¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°–ÙÙ×·has_pluto_hook_features¤bodyÚ�Why does it still make sense ?
Thanks to the mask, the transformer incorporates the sub-models (see Masked Attention)
$$\bar{p}(x_{j+1} | x_j, \ldots, x_{-n_\text{ctx}})$$
so it is trained to predict the next token from contexts of every length from $1$ to $n_\text{ctx}$. We can reasonably assume that the trained weights follow an auto-regressive logic in which one more token of context can only improve the prediction. The same holds for the intermediate layers if we think of the columns of $\text{softmax}(C^\top \hat{Y}_i)$ as intermediate probabilities $\hat{p}_i$.
So the KV cache is not equivalent but, from an auto-regressive point of view, it should be at least as good. Note that this is an assumption: the transformer is trained on each window from scratch, it never sees cached values during training!
°persist_js_stateÂÙ$cb6b040d-c213-4ecb-a047-fb66407752a3Чrunning§runtimeÎ ã�¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$cb6b040d-c213-4ecb-a047-fb66407752a3¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°—+_Ý·has_pluto_hook_features¤bodyÚSIs a small model running on a local GPU cheaper than a web search ?
A local GPU drawing 300 W and generating 100 tokens/s answers with 500 tokens in 5 s : $300 \times 5 = 1500$ J $\approx 0.4$ Wh. A laptop drawing 40 W at 30 tokens/s : $40 \times 17 \approx 700$ J $\approx 0.2$ Wh.
This is the same order of magnitude as a web search (0.3 Wh in 2009) plus the device used to browse it. There is no network and no data centre but a single user cannot batch the queries so the GPU is used less efficiently than in a data centre. In a sense, the expensive training compressed the web into the weights so that the answer can be computed offline.
°persist_js_stateÂÙ$45efc71d-d5f8-474e-9b89-e72fac7110fdЧrunning§runtimeÎ »A¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$45efc71d-d5f8-474e-9b89-e72fac7110fd¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°šà·has_pluto_hook_features¤bodyÚx
[BDV00] Y. Bengio, R. Ducharme and P. Vincent. A Neural Probabilistic Language Model. In: Advances in Neural Information Processing Systems, Vol. 13 (MIT Press, 2000). Accessed on Oct 11, 2024.
°persist_js_stateÂÙ$083fcca6-10e0-4483-a754-19d1f2720ec7Чrunning§runtimeÎ Bmޏdepends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$083fcca6-10e0-4483-a754-19d1f2720ec7¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°œ˜U?·has_pluto_hook_features¤bodyÚàÇ
$\hat{Y}_0$$\hat{Y}_1$$\hat{Y}_2$$\hat{Y}_N$$x_{-4}$$x_{-3}$$x_{-3}$$x_{-2}$$x_{-2}$$x_{-1}$$x_{-1}$$x_{0}$$\text{same position}$$\text{past positions through keys and values}$
°persist_js_stateÂÙ$d050a7ee-3aa7-4539-a236-5b6446599dedЧrunning§runtimeÎ Ø;¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d050a7ee-3aa7-4539-a236-5b6446599ded¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°œž®O·has_pluto_hook_features¤body °persist_js_stateÂÙ$579a203b-e6f7-4190-b874-18b00a5c3f77Чrunning§runtimeÎ Ó¯¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$579a203b-e6f7-4190-b874-18b00a5c3f77¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°�·has_pluto_hook_features¤bodyÙ*load_llms (generic function with 1 method)°persist_js_stateÂÙ$570fa160-3adb-463e-99b8-b7dd05076908Чrunning§runtimeÎ ¢º¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$570fa160-3adb-463e-99b8-b7dd05076908¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°–o†·has_pluto_hook_features¤bodyÙ(softmax (generic function with 1 method)°persist_js_stateÂÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904Чrunning§runtimeÎ ¾¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™+³·has_pluto_hook_features¤bodyÚ)With $n_\text{ctx} > 1$, the encoder $C$ is shared by all tokens: See for instance the network below taken from [BDV00; Figure 1], the first popular application of neural nets for languages:
°persist_js_stateÂÙ$75ca478c-916f-464a-9435-8208ee726d50Чrunning§runtimeÎ (æO¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$75ca478c-916f-464a-9435-8208ee726d50¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°šd$·has_pluto_hook_features¤bodyÚj
[TMH+24] G. Team, T. Mesnard, C. Hardin et al. Gemma: Open Models Based on Gemini Research and Technology (Apr 2024), arXiv:2403.08295. Accessed on Nov 11, 2024.
[TRP+24] G. Team, M. Riviere, S. Pathak et al. Gemma 2: Improving Open Language Models at a Practical Size (Oct 2024), arXiv:2408.00118. Accessed on Nov 11, 2024.
[RWC+19] A. Radford, J. Wu, R. Child et al. Language Models Are Unsupervised Multitask Learners (2019). Accessed on Oct 23, 2024.
[TMS+23] H. Touvron, L. Martin, K. Stone et al. Llama 2: Open Foundation and Fine-Tuned Chat Models (Jul 2023), arXiv:2307.09288. Accessed on Oct 23, 2024.
°persist_js_stateÂÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Чrunning§runtimeΡdܸdepends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š…¼ä·has_pluto_hook_features¤bodyÚÓáSelf-Attention with embedding $C$ is:
$$\text{Masked-MultiHead}(CX, CX, CX)$$
The embedding vectors $CX$ take then different projections for value, key, query and also for different heads!
$$\text{head}_j = \text{Masked-Attention}(W_j^VCX, W_j^KCX, W_j^QCX)$$
°persist_js_stateÂÙ$1d02cfbb-9d6b-41fd-9621-0103a9110813Чrunning§runtimeÎ «¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$1d02cfbb-9d6b-41fd-9621-0103a9110813¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�êæ·has_pluto_hook_features¤bodyÙhTraining : compressing the web
°persist_js_stateÂÙ$9ff95a9a-192b-4a12-8e2e-7acd6659c066Чrunning§runtimeÎ n#¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$9ff95a9a-192b-4a12-8e2e-7acd6659c066¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�šù{·has_pluto_hook_features¤bodyÚé$$\begin{align}
Q & = \begin{bmatrix}
q_1 & \cdots & q_{n_\text{ctx}}
\end{bmatrix} &
K & = \begin{bmatrix}
k_1 & \cdots & k_{n_\text{ctx}}
\end{bmatrix} &
K^\top Q & =
\begin{bmatrix}
\langle k_1, q_1 \rangle & \cdots & \langle k_1, q_{n_\text{ctx}} \rangle\\
\vdots & \ddots & \vdots\\
\langle k_{n_\text{ctx}}, q_1 \rangle & \cdots & \langle k_{n_\text{ctx}}, q_{n_\text{ctx}} \rangle
\end{bmatrix}
\end{align}$$
°persist_js_stateÂÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Чrunning§runtimeÎ"H³¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°Ÿ?öZ·has_pluto_hook_features¤bodyÚÛ:Different weights $W_1 \in \mathbb{R}^{d_\text{ff} \times d_\text{emb}}$, $W_2 \in \mathbb{R}^{d_\text{emb} \times d_\text{ff}}$ for each layer:
$$x \mapsto W_2\max(0, W_1x + b_1) + b_2$$
Expansion factor $d_\text{ff} / d_\text{emb}$ is typically 4× like suggested in [VSP+17] (but not for Gemma)
| Name | Ref | $d_\text{emb}$ | $d_\text{ff}$ |
|---|
| GPT-2 | [RWCL19] | 768 | 3072 |
| Gemma | [TMHD24] | 2048 | 32768 |
| Gemma | [TMHD24] | 3072 | 49152 |
| Gemma-2 | [TRPS24] | 2304 | 18432 |
| Gemma-2 | [TRPS24] | 3584 | 28672 |
| Gemma-2 | [TRPS24] | 4608 | 73728 |
| Llama-3 | | | 4096 |
| base | [VSPU17] | 512 | 2048 |
| big | [VSPU17] | 1024 | 4096 |
°persist_js_stateÂÙ$ed5b5702-4cca-4116-a70f-4a562178f490Чrunning§runtimeÎ ì¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$ed5b5702-4cca-4116-a70f-4a562178f490¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�™‚Œ·has_pluto_hook_features¤bodyÚì
Challenging tradeoff: Encode text to increase length of "past text" while keeping $n_\text{ctx}$ and $n$ small enough.
Length of "past text" increases with vocabulary size $n_\text{voc}$ and context window $n_\text{ctx}$.
°persist_js_stateÂÙ$736920df-e4bb-4535-b982-e397aa0a782dЧrunning§runtimeÍWP¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$736920df-e4bb-4535-b982-e397aa0a782d¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeÙ!application/vnd.pluto.tree+object¬rootassignee¦iter_2²last_run_timestampËAÚ¼°�¶c·has_pluto_hook_features¤body…¦prefix£BPE¨objectid°6a1235af2321cbb3¤type¦struct¬prefix_short£BPE¨elements’’¤text’©"ZYdZYac"ªtext/plain’¥pairs’…¦prefix½Dict{Tuple{Char, Char}, Char}¨objectid°f2ec897bbd52e758¤type¤Dict¬prefix_short¤Dict¨elements’’’ƒ¨objectid®8a2b367aab1f86¤type¥Tuple¨elements’’’£'a'ªtext/plain’’£'b'ªtext/plainÙ!application/vnd.pluto.tree+object’£'Y'ªtext/plain’’ƒ¨objectid°fb8c82017ecb7739¤type¥Tuple¨elements’’’£'a'ªtext/plain’’£'a'ªtext/plainÙ!application/vnd.pluto.tree+object’£'Z'ªtext/plainÙ!application/vnd.pluto.tree+object°persist_js_stateÂÙ$55a09acc-84da-491c-86ba-9a66f4ea52feЧrunning§runtimeÎ eÄì¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$55a09acc-84da-491c-86ba-9a66f4ea52fe¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™:¤¾·has_pluto_hook_features¤bodyÛ [$$\begin{align}
h^{(t+1)} & = \tanh(Wh^{(t)} + Ux^{(t+1)} + b)\\
o^{(t)} &= Vh^{(t)} + c\\
\hat{y}^{(t)} &= \text{softmax}(o^{(t)})
\end{align}$$
Illustrated on the right [GBC16; Figure 10.3].
RNNs as language model showcased in [MKB+10].
Issue: Training time and space complexity is proportional to $n_\text{ctx}$ and cannot parallelize to speed up.
°persist_js_stateÂÙ$52c97de1-3d75-4b51-8397-86ea67af6c77Чrunning§runtimeÎ PˆÚ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$52c97de1-3d75-4b51-8397-86ea67af6c77¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°œžn‡·has_pluto_hook_features¤bodyÛ –
$\hat{Y}_0$$\hat{Y}_1$$\hat{Y}_2$$\hat{Y}_N$$x_{-4}$$x_{-3}$$x_{-2}$$x_{-1}$$x_{0}$$x_{1}$$\text{context window}$$\text{cached}$$\text{computed for } x_1$$\text{not needed}$$x_{-4} \text{ in the context}$
°persist_js_stateÂÙ$a6a6a352-7479-4f3f-9abb-900eb901466cЧrunning§runtimeÎ i¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$a6a6a352-7479-4f3f-9abb-900eb901466c¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°—+’G·has_pluto_hook_features¤bodyÚ&After how many queries does the inference energy exceed the training energy ?
With 21.6 GWh for training and 0.3 Wh per query : $21.6 \cdot 10^9 / 0.3 \approx 7 \cdot 10^{10}$ queries. With $10^9$ queries per day, this takes about 72 days. For a widely used model, the energy of the inference dominates over the lifetime of the model. For a rarely used model, the training dominates.
°persist_js_stateÂÙ$d558636d-c714-4033-ae73-5b92c3cdedf3Чrunning§runtimeÍUñ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d558636d-c714-4033-ae73-5b92c3cdedf3¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeÙ!application/vnd.pluto.tree+object¬rootassignee¤dict²last_run_timestampËAÚ¼°–H±ž·has_pluto_hook_features¤body…¦prefixÙ"Dict{Vector{Int64}, Vector{Int64}}¨objectid°c935df48de240dbb¤type¤Dict¬prefix_short¤Dict¨elements’’’…¦prefix¥Int64¨objectid°fd4d0513c696d43c¤type¥Array¬prefix_short ¨elements’’’¡0ªtext/plain’’¡1ªtext/plainÙ!application/vnd.pluto.tree+object’…¦prefix¥Int64¨objectid°eaacf6bf2156b6e3¤type¥Array¬prefix_short ¨elements’’’¢-1ªtext/plain’’¡1ªtext/plainÙ!application/vnd.pluto.tree+object’’…¦prefix¥Int64¨objectid¯42fe541d460ba90¤type¥Array¬prefix_short ¨elements’’’¡1ªtext/plain’’¡0ªtext/plainÙ!application/vnd.pluto.tree+object’…¦prefix¥Int64¨objectid°7ace6821419dbfb5¤type¥Array¬prefix_short ¨elements’’’¡1ªtext/plain’’¡1ªtext/plainÙ!application/vnd.pluto.tree+object°persist_js_stateÂÙ$8d231f2c-4b0c-4c37-a746-16e98d4cafc8Чrunning§runtimeÎ 8[¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$8d231f2c-4b0c-4c37-a746-16e98d4cafc8¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�šÅø·has_pluto_hook_features¤bodyÙHAttention head
°persist_js_stateÂÙ$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbeЧrunning§runtimeÎ
a˜¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbe¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�žCB·has_pluto_hook_features¤bodyÚÙ | Energy |
|---|
| 10 W LED bulb for 1 hour | 10 Wh = 36 kJ |
| Heating 1 L of water from 20 °C to 100 °C | ≈ 93 Wh |
| Belgian reference household, per year | ≈ 3,500 kWh |
| Data centres worldwide in 2024 (≈ 1.5% of world electricity) | ≈ 415 TWh |
| Data centres worldwide in 2030 (IEA projection) | ≈ 945 TWh |
Source for data centres : IEA, Energy and AI, 2025.
As we will see, a single query costs a fraction of a Wh. The question is rather the number of queries and the energy needed to train the models.
°persist_js_stateÂÙ$f1afaf8c-d9ad-446a-9826-9c4cda19993fЧrunning§runtimeÎ ÑŒ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f1afaf8c-d9ad-446a-9826-9c4cda19993f¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°œäÂU·has_pluto_hook_features¤bodyÙ*new_token (generic function with 1 method)°persist_js_stateÂÙ$f7ca3ff7-b5cf-452b-b955-7219e7397324Чrunning§runtimeÍS�¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f7ca3ff7-b5cf-452b-b955-7219e7397324¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeÙ!application/vnd.pluto.tree+object¬rootassignee¦iter_1²last_run_timestampËAÚ¼°�¡·has_pluto_hook_features¤body…¦prefix£BPE¨objectid¯428a722b92ec227¤type¦struct¬prefix_short£BPE¨elements’’¤text’«"ZabdZabac"ªtext/plain’¥pairs’…¦prefix½Dict{Tuple{Char, Char}, Char}¨objectid°76d6a7ae87f095b3¤type¤Dict¬prefix_short¤Dict¨elements‘’’ƒ¨objectid°fb8c82017ecb7739¤type¥Tuple¨elements’’’£'a'ªtext/plain’’£'a'ªtext/plainÙ!application/vnd.pluto.tree+object’£'Z'ªtext/plainÙ!application/vnd.pluto.tree+object°persist_js_stateÂÙ$5f05e717-a51a-4a99-bb11-cc493217f93fЧrunning§runtimeÎíá¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$5f05e717-a51a-4a99-bb11-cc493217f93f¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š›Ÿ•·has_pluto_hook_features¤bodyÚÙ*Norm of gradient increases exponentially with depth. Issue for deep neural net. Consider output
$$\begin{bmatrix}
y_{1,1} & \ldots & y_{1,d_\text{emb}}\\
\vdots & \ddots & \vdots\\
y_{d_\text{batch},1} & \ldots & y_{d_\text{batch},d_\text{emb}}
\end{bmatrix}$$
Normalization : $y_{i,j} \mapsto g(y_{i,j} - \mu_{i,j})/\sigma_{i,j}$ for gain $g$, mean $\mu$ and standard deviation $\sigma$.
Batch normalization : $\sigma_{i,j} = \sigma_{j}$ [IS15]
Layer normalization : $\sigma_{i,j} = \sigma_{i}$ [BKH16]
Batch norm depends on the batch hence is tricky to implement. Layer normalization is used in [VSP+17].
°persist_js_stateÂÙ$b56e9e56-e74a-401b-b4b5-f36bb33341d5Чrunning§runtimeÎ ¡*¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$b56e9e56-e74a-401b-b4b5-f36bb33341d5¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°š
Ü·has_pluto_hook_features¤bodyÚb
[HZRS15] K. He, X. Zhang, S. Ren and J. Sun. Deep Residual Learning for Image Recognition (Dec 2015), arXiv:1512.03385. Accessed on Nov 12, 2024.
°persist_js_stateÂÙ$57c2c944-0d91-489d-8ad7-f5520e71ef3eЧrunning§runtimeÎ V‡¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$57c2c944-0d91-489d-8ad7-f5520e71ef3e¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�—©½·has_pluto_hook_features¤bodyÙPByte Pair Encoding
°persist_js_stateÂÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fbЧrunning§runtimeÎ C=¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fb¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�žy…·has_pluto_hook_features¤bodyÙZInference vs web search
°persist_js_stateÂÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5Чrunning§runtimeÎ y¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�šàç·has_pluto_hook_features¤bodyÙ\
°persist_js_stateÂÙ$f8330700-e964-4e19-9c55-2b11df45789eЧrunning§runtimeÎ îF¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f8330700-e964-4e19-9c55-2b11df45789e¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�š$ÿ·has_pluto_hook_features¤bodyÙZEmbedding sizes in LLMs
°persist_js_stateÂÙ$b245ada9-1677-45bc-a883-bc213a2105b7Чrunning§runtimeÎ p˜¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$b245ada9-1677-45bc-a883-bc213a2105b7¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�¯0”·has_pluto_hook_features¤bodyÚ�Google search also needs an expensive precomputation : crawling the web, building the index and computing the PageRank.
1998 : "a PageRank for 26 million web pages can be computed in a few hours on a medium size workstation" Brin and Page, 1998.
Today : the index covers "hundreds of billions of webpages" and is "well over 100,000,000 gigabytes" (Google).
Back-of-the-envelope : $10^{11}$ pages × 50 links × 2 FLOPs × 50 power iterations $\approx 5 \cdot 10^{14}$ FLOP for the PageRank, that is $10^{10}$ times less than GPT-4. However, PageRank is only a small part of the cost : crawling, parsing and storing the index dominate and there is no public figure for its energy. Note that LLMs also need the web to be crawled to obtain their training data.
°persist_js_stateÂÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ecЧrunning§runtimeÎ$œÚ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ec¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°šEü·has_pluto_hook_features¤bodyÚñÝWhat is the time complexity of inference with respect to $d_\text{emb}$, $n_\text{voc}$, $n_\text{ctx}$, $d_\text{ff}$, $h$ and $N$ ?
| Input | Parameters | Time |
|---|
| $CX + P \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}$ | $W_j^V \in \mathbb{R}^{d_v \times d_\text{emb}}$ | $O(d_v d_\text{emb} n_\text{ctx})$ |
| $CX + P \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}$ | $W_j^K, W_j^Q \in \mathbb{R}^{d_k \times d_\text{emb}}$ | $O(d_k d_\text{emb} n_\text{ctx})$ |
| $K, Q \in \mathbb{R}^{d_k \times n_\text{ctx}}$ | | $O(d_k n_\text{ctx}^2)$ |
| $V \in \mathbb{R}^{d_v \times n_\text{ctx}}, ... \in \mathbb{R}^{n_\text{ctx} \times n_\text{ctx}}$ | | $O(d_v n_\text{ctx}^2)$ |
| $... \in \mathbb{R}^{d_v \times n_\text{ctx}}$ | $W^O \in \mathbb{R}^{d_\text{emb} \times d_v}$ | $O(d_\text{emb} d_v n_\text{ctx})$ |
| $... \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}$ | $W_1 \in \mathbb{R}^{d_\text{ff} \times d_\text{emb}}$ | $O(d_\text{emb} d_\text{ff} n_\text{ctx})$ |
| $... \in \mathbb{R}^{d_\text{ff} \times n_\text{ctx}}$ | $W_2 \in \mathbb{R}^{d_\text{emb} \times d_\text{ff}}$ | $O(d_\text{emb} d_\text{ff} n_\text{ctx})$ |
So for $N$ layers (ignoring the complexity of the embedding):
$$O(Nn_\text{ctx}(n_\text{ctx}(d_v + d_k) + d_\text{emb}(d_v+d_k+d_\text{ff})))$$
Assuming that $d_v, d_k, d_\text{ff}$ has the same scale as $d_\text{emb}$:
$$O(Nn_\text{ctx}^2d_\text{emb} + Nn_\text{ctx}d_\text{emb}^2)$$

°persist_js_stateÂÙ$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5ddЧrunning§runtimeÎoè\¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°šv›9·has_pluto_hook_features¤bodyÚ„ÞPre-activation for residual neural networks introduced in [HZRS16] and used in GPT-2 [RWC+19]. See figure on the right.
Layer normalization is applied before attention and feed-forward, leaving an identity residual path. A final layer normalization follows the $N$ blocks.
Rotary Positional Encoding [SLP+23] replaces $W^K(Cx_i + p_i)$ and $W^Q(Cx_i + p_i)$ by $R^i W^KCx_i$ and $R^i W^QCx_i$ where $R$ is a rotation matrix. Advantage : $\langle k_i, q_j \rangle$ contains $R^{i - j}$ → relative difference of position.
°persist_js_stateÂÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45Чrunning§runtimeÎ× Å¾¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45¹depends_on_disabled_cells¦queued¤logs’ˆ¥group¥utils¤lineÿ£msg’ÙXLoading bibliography from `/home/runner/work/LINMA2472/LINMA2472/Lectures/biblio.bib`...ªtext/plain¥level¤Info¢id¹Main_workspace#3_a1ed908b§cell_idÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45¦kwargs�¤fileÙ7/home/runner/work/LINMA2472/LINMA2472/Lectures/utils.jlˆ¥group¥utils¤lineÿ£msg’²Loading completed.ªtext/plain¥level¤Info¢id¹Main_workspace#3_d0c2d7aa§cell_idÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45¦kwargs�¤fileÙ7/home/runner/work/LINMA2472/LINMA2472/Lectures/utils.jl¦output†¤mimeÙ!application/vnd.pluto.tree+object¬rootassignee¦biblio²last_run_timestampËAÚ¼°˜ånµ·has_pluto_hook_features¤body…¦prefixÙ(DocumenterCitations.CitationBibliography¨objectid°bfac4fc0ec33fe94¤type¦struct¬prefix_short´CitationBibliography¨elementsœ’§bibfile’Ù;"/home/runner/work/LINMA2472/LINMA2472/Lectures/biblio.bib"ªtext/plain’¥style’¬AlphaStyle()ªtext/plain’ªinsert_css’¤trueªtext/plain’ªshow_hover’¤trueªtext/plain’®show_backlinks’¤trueªtext/plain’§entries’…¦prefixÙ9OrderedCollections.OrderedDict{String, BibInternal.Entry}¨objectid°67a36a1af9937fec¤type¤Dict¬prefix_short«OrderedDict¨elements›’’¸"arechiga2023Dragguided"ªtext/plain’…¦prefix¥Entry¨objectid°3634d57a6fc08510¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°f45a2585d7d92123¤type¦struct¬prefix_short¦Access¨elements“’£doi’»"10.48550/arXiv.2306.09935"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù!"http://arxiv.org/abs/2306.09935"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°195d2d4b74c6885b¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°5c9158b2610a0faf¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¦"June"ªtext/plain’¤year’¦"2023"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid¯cddae8bdbc7d8d0¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°2183054acef567a7¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’¬"2306.09935"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’¸"arechiga2023Dragguided"ªtext/plain’¢in’…¦prefix¢In¨objectid°bd4cf4363b79b8a5¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’²"arXiv:2306.09935"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°70d38d413577302d¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù;"Drag-Guided Diffusion Models for Vehicle Image Generation"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’"ba2016Layer"ªtext/plain’…¦prefix¥Entry¨objectid°9fc08ff2eec6ad49¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°8d6486162109689e¤type¦struct¬prefix_short¦Access¨elements“’£doi’»"10.48550/arXiv.1607.06450"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù!"http://arxiv.org/abs/1607.06450"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°e048d8ca82a632fe¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°c809205db2cebaa0¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¦"July"ªtext/plain’¤year’¦"2016"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°302eb1362271fc14¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°ac64071b138d2c2b¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’¬"1607.06450"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’"ba2016Layer"ªtext/plain’¢in’…¦prefix¢In¨objectid°2e8c7337a4c0d9a2¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’²"arXiv:1607.06450"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°eb037df029b96d79¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’¹"Layer {{Normalization}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’´"bahdanau2016Neural"ªtext/plain’…¦prefix¥Entry¨objectid°b466661afd43cf22¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°ab7a73d42104c5d7¤type¦struct¬prefix_short¦Access¨elements“’£doi’º"10.48550/arXiv.1409.0473"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù "http://arxiv.org/abs/1409.0473"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°4d1a1da75c2985c2¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°cb906d6a65d51f82¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¥"May"ªtext/plain’¤year’¦"2016"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°9fce55f781923091¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°2935460f0ee56ae6¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’«"1409.0473"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’´"bahdanau2016Neural"ªtext/plain’¢in’…¦prefix¢In¨objectid°fa93cc5ffa14e8fc¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’±"arXiv:1409.0473"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°6b65c29c2a7b9e7b¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’ÙW"Neural {{Machine Translation}} by {{Jointly Learning}} to {{Align}} and {{Translate}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’²"bengio2000Neural"ªtext/plain’…¦prefix¥Entry¨objectid°447cce44db90a3da¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°e2f24048c6c99cd6¤type¦struct¬prefix_short¦Access¨elements“’£doi’¢""ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ùn"https://proceedings.neurips.cc/pape"[93m[1m ⋯ 36 bytes ⋯ [22m[39m"bf572b5940d7d9a8fa4c-Abstract.html"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°428ae28647040a20¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’Ù7"Advances in {{Neural Information Processing Systems}}"ªtext/plain’¤date’…¦prefix¤Date¨objectid¯d76741a550985b8¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¢""ªtext/plain’¤year’¦"2000"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°beacc899bf47ebbd¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid¯d1f4caf2f8d6d2a¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’¢""ªtext/plain’¦eprint’¢""ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’²"bengio2000Neural"ªtext/plain’¢in’…¦prefix¢In¨objectid°679c3c49b9a9b110¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’¢""ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’«"MIT Press"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¤"13"ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°8646f160dfa8b390¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù+"A {{Neural Probabilistic Language Model}}"ªtext/plain’¤type’¯"inproceedings"ªtext/plainÙ!application/vnd.pluto.tree+object’’»"bradley2024ClassifierFree"ªtext/plain’…¦prefix¥Entry¨objectid°f9568e6c60790e8f¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°4dd0dc756140887c¤type¦struct¬prefix_short¦Access¨elements“’£doi’¢""ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù,"https://openreview.net/forum?id=8K36RkrI7N"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°a1fcae9a05f826fe¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’ÙM"The {{Thirteenth International Conference}} on {{Learning Representations}}"ªtext/plain’¤date’…¦prefix¤Date¨objectid°2cf7433e0909f5bf¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’©"October"ªtext/plain’¤year’¦"2024"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°579fb1d6f13e6799¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid¯d1f4caf2f8d6d2a¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’¢""ªtext/plain’¦eprint’¢""ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’»"bradley2024ClassifierFree"ªtext/plain’¢in’…¦prefix¢In¨objectid°65e642c9e0261cc0¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’¢""ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’¢""ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°cad578b1e8c34ea1¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù;"Classifier-{{Free Guidance}} Is a {{Predictor-Corrector}}"ªtext/plain’¤type’¯"inproceedings"ªtext/plainÙ!application/vnd.pluto.tree+object’’²"britz2017Massive"ªtext/plain’…¦prefix¥Entry¨objectid°56b99c526cd73f11¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°a30cc6bffe564c76¤type¦struct¬prefix_short¦Access¨elements“’£doi’»"10.48550/arXiv.1703.03906"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù!"http://arxiv.org/abs/1703.03906"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°608066e702fa6e3d¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°f1083a01d970c7d5¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’§"March"ªtext/plain’¤year’¦"2017"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°b1c855dfa588dcd1¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°53a834c7b4a1652e¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’¬"1703.03906"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’²"britz2017Massive"ªtext/plain’¢in’…¦prefix¢In¨objectid°60039b91202a02ea¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’²"arXiv:1703.03906"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°e5384f589df5d406¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’ÙI"Massive {{Exploration}} of {{Neural Machine Translation Architectures}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’³"brown2020Language"ªtext/plain’…¦prefix¥Entry¨objectid°bf00a5ee85ae6a2a¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°cd50b61de508b700¤type¦struct¬prefix_short¦Access¨elements“’£doi’»"10.48550/arXiv.2005.14165"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù!"http://arxiv.org/abs/2005.14165"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°c33f72b8c85a18cb¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°87e71ceee8081b78¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¦"July"ªtext/plain’¤year’¦"2020"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°1c360c20fa4688b2¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°fc3fb3939df590f7¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’¬"2005.14165"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’³"brown2020Language"ªtext/plain’¢in’…¦prefix¢In¨objectid°9f18c19f70d24eb4¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’²"arXiv:2005.14165"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°c28accd558977fd8¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù/"Language {{Models}} Are {{Few-Shot Learners}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’¶"burgert2023Diffusion"ªtext/plain’…¦prefix¥Entry¨objectid°90ebdc44be690e2f¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°7871e5abb012efe8¤type¦struct¬prefix_short¦Access¨elements“’£doi’»"10.48550/arXiv.2312.03817"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù!"http://arxiv.org/abs/2312.03817"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°70effbe923c79955¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°9187a69e042e0ae9¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’ª"December"ªtext/plain’¤year’¦"2023"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°c6aa40314f3b0ec0¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°1f2ed3014161845b¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’¬"2312.03817"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’¶"burgert2023Diffusion"ªtext/plain’¢in’…¦prefix¢In¨objectid°c2a191f4b5852dbf¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’²"arXiv:2312.03817"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°2e4824c19ff86d67¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù?"Diffusion {{Illusions}}: {{Hiding Images}} in {{Plain Sight}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’³"cho2014Properties"ªtext/plain’…¦prefix¥Entry¨objectid°61b8f35ce5daf37d¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°b2b4f985fef788bb¤type¦struct¬prefix_short¦Access¨elements“’£doi’º"10.48550/arXiv.1409.1259"ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù "http://arxiv.org/abs/1409.1259"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°7e3e144dbeea092e¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’¢""ªtext/plain’¤date’…¦prefix¤Date¨objectid°e0d003272dec03dc¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’©"October"ªtext/plain’¤year’¦"2014"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°45a6dadfcdbb89ef¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid°bca0d6882f556f51¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’§"arXiv"ªtext/plain’¦eprint’«"1409.1259"ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’³"cho2014Properties"ªtext/plain’¢in’…¦prefix¢In¨objectid°5cc2f78480ab813c¤type¦struct¬prefix_short¢In¨elementsž’§address’¢""ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’¢""ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’±"arXiv:1409.1259"ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¢""ªtext/plain’©publisher’§"arXiv"ªtext/plain’¦school’¢""ªtext/plain’¦series’¢""ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°4a99c908d97d4405¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’ÙY"On the {{Properties}} of {{Neural Machine Translation}}: {{Encoder-Decoder Approaches}}"ªtext/plain’¤type’¦"misc"ªtext/plainÙ!application/vnd.pluto.tree+object’’·"dhariwal2024Diffusion"ªtext/plain’…¦prefix¥Entry¨objectid°862b30307c872a85¤type¦struct¬prefix_short¥Entry¨elementsœ’¦access’…¦prefix¦Access¨objectid°28c594d5bb7e9683¤type¦struct¬prefix_short¦Access¨elements“’£doi’¢""ªtext/plain’¬howpublished’¢""ªtext/plain’£url’Ù0"https://dl.acm.org/doi/10.5555/3540261.3540933"ªtext/plainÙ!application/vnd.pluto.tree+object’§authors’…¦prefix°BibInternal.Name¨objectid°3def6b413e578b4a¤type¥Array¬prefix_short ¨elements‘¤moreÙ!application/vnd.pluto.tree+object’©booktitle’Ùo"Proceedings of the 35th {{Internati"[93m[1m ⋯ 27 bytes ⋯ [22m[39m"al Information Processing Systems}}"ªtext/plain’¤date’…¦prefix¤Date¨objectid°287e7dab246d0a90¤type¦struct¬prefix_short¤Date¨elements“’£day’¢""ªtext/plain’¥month’¦"June"ªtext/plain’¤year’¦"2024"ªtext/plainÙ!application/vnd.pluto.tree+object’§editors’…¦prefix°BibInternal.Name¨objectid°473cae59d0f24663¤type¥Array¬prefix_short ¨elements�Ù!application/vnd.pluto.tree+object’¦eprint’…¦prefix¦Eprint¨objectid¯d1f4caf2f8d6d2a¤type¦struct¬prefix_short¦Eprint¨elements“’®archive_prefix’¢""ªtext/plain’¦eprint’¢""ªtext/plain’primary_class’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¢id’·"dhariwal2024Diffusion"ªtext/plain’¢in’…¦prefix¢In¨objectid°c1002b636c4a579c¤type¦struct¬prefix_short¢In¨elementsž’§address’³"Red Hook, NY, USA"ªtext/plain’§chapter’¢""ªtext/plain’§edition’¢""ªtext/plain’«institution’¢""ªtext/plain’¤isbn’³"978-1-71384-539-3"ªtext/plain’¤issn’¢""ªtext/plain’§journal’¢""ªtext/plain’¦number’¢""ªtext/plain’¬organization’¢""ªtext/plain’¥pages’¬"8780--8794"ªtext/plain’©publisher’¸"Curran Associates Inc."ªtext/plain’¦school’¢""ªtext/plain’¦series’®"{{NIPS}} '21"ªtext/plain’¦volume’¢""ªtext/plainÙ!application/vnd.pluto.tree+object’¦fields’…¦prefix´Dict{String, String}¨objectid°c7c7d394d3d9f4a3¤type¤Dict¬prefix_short¤Dict¨elements‘¤moreÙ!application/vnd.pluto.tree+object’¤note’¢""ªtext/plain’¥title’Ù3"Diffusion Models Beat {{GANs}} on Image Synthesis"ªtext/plain’¤type’¯"inproceedings"ªtext/plainÙ!application/vnd.pluto.tree+object¤moreÙ!application/vnd.pluto.tree+object’©citations’…¦prefixÙ-OrderedCollections.OrderedDict{String, Int64}¨objectid°2057b39ceaab1da5¤type¤Dict¬prefix_short«OrderedDict¨elements�Ù!application/vnd.pluto.tree+object’®page_citations’…¦prefix¹Dict{String, Set{String}}¨objectid°b0ad94698b56dd2c¤type¤Dict¬prefix_short¤Dict¨elements�Ù!application/vnd.pluto.tree+object’ªanchor_map’…¦prefix´Documenter.AnchorMap¨objectid°a5ee99a77518f990¤type¦struct¬prefix_short©AnchorMap¨elements’’£map’…¦prefixÙ5Dict{String, Dict{String, Vector{Documenter.Anchor}}}¨objectid°10c588ca4b86e354¤type¤Dict¬prefix_short¤Dict¨elements�Ù!application/vnd.pluto.tree+object’¥count’¡0ªtext/plainÙ!application/vnd.pluto.tree+object’«anchor_keys’…¦prefixÙPBijections.Bijection{String, String, Dict{String, String}, Dict{String, String}}¨objectid°e2209e2b6c4f6f76¤type¤Dict¬prefix_short©Bijection¨elements�Ù!application/vnd.pluto.tree+object’hover_entries’…¦prefixÙ#Dict{String, Tuple{String, String}}¨objectid°cf45d8bf48877e17¤type¤Dict¬prefix_short¤Dict¨elements�Ù!application/vnd.pluto.tree+object’©backlinks’…¦prefixÙ6Dict{String, Vector{DocumenterCitations.CitationSite}}¨objectid°f365e9fe10c7a978¤type¤Dict¬prefix_short¤Dict¨elements�Ù!application/vnd.pluto.tree+object°persist_js_stateÂÙ$3176c675-ce53-4471-b13b-1d694f5efdd8Чrunning§runtimeÎ
ĸdepends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$3176c675-ce53-4471-b13b-1d694f5efdd8¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�ž^¢·has_pluto_hook_features¤bodyÙjEstimating the inference energy
°persist_js_stateÂÙ$006f44b0-c10d-44c3-8300-183e706b605dЧrunning§runtimeÎ r—¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$006f44b0-c10d-44c3-8300-183e706b605d¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�›l ·has_pluto_hook_features¤bodyÚ8Further readings
This dictionary viewpoint can also be view as a memory store that recalls by similarity, called associate memory. See this blog post for more details.
°persist_js_stateÂÙ$a14e505e-2e4a-4c73-8133-7560ba58916bЧrunning§runtimeÎ ù”¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$a14e505e-2e4a-4c73-8133-7560ba58916b¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�œÙ¶·has_pluto_hook_features¤bodyÙdKey-Value (KV) cache
°persist_js_stateÂÙ$2a7e5096-1e8d-4506-96d2-86de0a7d39aaЧrunning§runtimeÎ ]"¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$2a7e5096-1e8d-4506-96d2-86de0a7d39aa¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™+†·has_pluto_hook_features¤bodyÙ´Forcing $D = C^\top$ appears to work well in practice [PW17], this is what is used in [VSP+17].
°persist_js_stateÂÙ$d605d0a5-7511-4a32-9e1b-1c2befa43796Чrunning§runtimeÎ
zó¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d605d0a5-7511-4a32-9e1b-1c2befa43796¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�ŸR‡·has_pluto_hook_features¤bodyÚuNetwork : $\approx 0.06$ kWh/GB in 2015, halving every 2 years Aslan et al., 2018. Extrapolating to $\approx 0.002$ kWh/GB today, a 2.5 MB search result page (HTTP Archive, 2024) costs $\approx 0.005$ Wh and a text answer of a few kB much less. Moreover, the power of the network is mostly fixed : it does not grow with the data volume so the marginal cost of a query is close to zero Mytton et al., 2024.
User device : A 20 W laptop used for 30 seconds consumes $\approx 0.17$ Wh, the same order as the server side of the query! User devices account for more than half of the emissions of the ICT sector Malmodin et al., 2024.
°persist_js_stateÂÙ$6712c883-b407-47e1-a666-4de05f8f8d6eЧrunning§runtimeÎzRå¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$6712c883-b407-47e1-a666-4de05f8f8d6e¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°–¹ª·has_pluto_hook_features¤bodyÛ Ç$$\begin{multline}
\hat{p}(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}}) = \\
\text{softmax}(C^\top W_2 \tanh(W_1
\begin{bmatrix}
C x_{-1}\\
\vdots\\
C x_{-n_\text{ctx}}
\end{bmatrix}
))
\end{multline}$$
°persist_js_stateÂÙ$7be26ee1-de61-4b66-b221-5c5c728c2901Чrunning§runtimeÎ í¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$7be26ee1-de61-4b66-b221-5c5c728c2901¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°��~A·has_pluto_hook_features¤bodyÙROrders of magnitude
°persist_js_stateÂÙ$04e9b912-6712-4290-acc4-f24bb27a1469Чrunning§runtimeÎ M¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$04e9b912-6712-4290-acc4-f24bb27a1469¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°��.Ç·has_pluto_hook_features¤bodyÙRMachine translation
°persist_js_stateÂÙ$af8194a1-a358-4cf7-b446-6b377cb76687Чrunning§runtimeÎ Må¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$af8194a1-a358-4cf7-b446-6b377cb76687¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�œ:.·has_pluto_hook_features¤bodyÚoThe feed-forward network is implemented independently for the output of each query so each query can be processed independently through each layer. The next layer allows each queries to then look at the results of the previous layer for past (because of the mask) queries.
°persist_js_stateÂÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03Чrunning§runtimeζ|L¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™þ¤{·has_pluto_hook_features¤bodyÚæ
[SHB16] R. Sennrich, B. Haddow and A. Birch. Neural Machine Translation of Rare Words with Subword Units (Jun 2016), arXiv:1508.07909. Accessed on Oct 23, 2024.
[KR18] T. Kudo and J. Richardson. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing (Aug 2018), arXiv:1808.06226. Accessed on Oct 23, 2024.
°persist_js_stateÂÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30Чrunning§runtimeÎ ¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�¯É`·has_pluto_hook_features¤bodyÚy
Scope : the energy of the GPUs only, of the full servers or of the whole data centre differ by $\approx 2.4\times$ Elsworth et al., 2025.
Average vs tail : company figures are medians or averages. Long contexts and reasoning models cost 10 to 100× more.
Dates : the 0.3 Wh of a web search dates from 2009. Both web search and LLMs became much more efficient since then.
Fixed vs marginal : idle servers and networks consume energy even without queries. BLOOM consumed $\approx 0.28$ kWh every 10 minutes while idle Luccioni et al., 2022.
Rebound effect : the energy per query decreases quickly but the number of queries grows even faster. The demand of data centres is expected to more than double by 2030 IEA, 2025.
°persist_js_stateÂÙ$aa10118d-0991-4284-97f8-3a0728636a66Чrunning§runtimeÎ ©¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$aa10118d-0991-4284-97f8-3a0728636a66¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�®æë·has_pluto_hook_features¤bodyÙT
°persist_js_stateÂÙ$e2eca085-9f99-4e3a-9db4-e7f692aedd34Чrunning§runtimeÎ }û¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$e2eca085-9f99-4e3a-9db4-e7f692aedd34¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�™¦Û·has_pluto_hook_features¤bodyÙxText to vectors : step 2 → embedding
°persist_js_stateÂÙ$d014e6aa-92f6-4ca1-be47-516565d1bb20Чrunning§runtimeÎ FÜj¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$d014e6aa-92f6-4ca1-be47-516565d1bb20¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™<ð$·has_pluto_hook_features¤bodyÛ ¦8Heads focus on different aspects. Their outputs are combined with $W^O \in \mathbb{R}^{d_\text{emb} \times hd_v}$:
$$\begin{align}
\text{head}_j & = \text{Attention}(W_j^VV, W_j^KK, W_j^QQ)\\
\text{MultiHead}(V, K, Q)
& =
W^O\text{vcat}(\text{head}_1, \ldots, \text{head}_h)
\end{align}$$
See [VSP+17; Figure 2] on the right.
Similarly, in the masked case:
$$\begin{align}
\text{head}_j & = \text{Masked-Attention}(W_j^VV, W_j^KK, W_j^QQ)\\
\text{Masked-MultiHead}&(V, K, Q)
=
W^O\text{vcat}(\text{head}_1, \ldots, \text{head}_h)
\end{align}$$
°persist_js_stateÂÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057aeЧrunning§runtimeÎ O׸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057ae¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�¯f<·has_pluto_hook_features¤bodyÙ:Caveats
°persist_js_stateÂÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5eЧrunning§runtimeÎ
}9¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5e¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�œ¾©·has_pluto_hook_features¤bodyÚéLet $\hat{y}_{i,j}$ be the column of $\hat{Y}_i$ at the position of $x_j$, with $\hat{Y}_0 = CX + P$. In layer $i$, $\hat{y}_{i,j}$ depends on
$\hat{y}_{i-1,j}$ for the query, the residual connection and the feed-forward network,
$\hat{y}_{i-1,k}$ for $k < j$ only through their keys $W_\ell^K\hat{y}_{i-1,k}$ and values $W_\ell^V\hat{y}_{i-1,k}$ for each head $\ell$,
not on $\hat{y}_{i-1,k}$ for $k > j$ because of the mask.
Given the keys and values of the past positions, the computation of $\hat{y}_{i,j}$ is independent of the other positions.
°persist_js_stateÂÙ$a57f2149-c876-4421-a958-e9aed1f892a6Чrunning§runtimeÎ3>:u¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$a57f2149-c876-4421-a958-e9aed1f892a6¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°œjFзhas_pluto_hook_features¤bodyÚî
mathjax_figure
mathjax_figure(draw; width, height)
Luxor only draws the shapes, the labels are overlaid in HTML so that they are typeset by the MathJax of Pluto. draw(label) draws the shapes with the origin at the center and adds labels with label(text, x, y; color, fontsize, halign).
°persist_js_stateÂÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59Чrunning§runtimeÍQq¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeÙ!application/vnd.pluto.tree+object¬rootassigneeÀ²last_run_timestampËAÚ¼°–fÅÅ·has_pluto_hook_features¤body…¦prefix¥Int64¨objectid°7ace6821419dbfb5¤type¥Array¬prefix_short ¨elements’’’¡1ªtext/plain’’¡1ªtext/plain°persist_js_stateÂÙ$453544fc-0e3e-4e04-8c0c-192f3a038884Чrunning§runtimeÎ ñ
¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$453544fc-0e3e-4e04-8c0c-192f3a038884¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�›¿·has_pluto_hook_features¤bodyÙRPositional encoding
°persist_js_stateÂÙ$0583ee0c-3802-4e81-b179-a80a82493b43Чrunning§runtimeÎ
O¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$0583ee0c-3802-4e81-b179-a80a82493b43¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™+O‹·has_pluto_hook_features¤bodyÚ@Byte Pair Encoding algorithm [SHB16] greedily merges the most frequent pair of tokens over the dataset into a new token. Most used implementations are SentencePiece [KR18] and tiktoken (play with it here). For instance, on this example, the pair ('a', 'a') is the most frequent so we substitute it by a new token, say 'Z':
°persist_js_stateÂÙ$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2Чrunning§runtimeÎ }/¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°–ٜ˷has_pluto_hook_features¤bodyÚøIs the KV cache equivalent to recomputing everything for the new context window ?
No! Without cache, we would compute new columns $\hat{y}_{i,j}'$ that only depend on $x_{-n_\text{ctx}+1}, \ldots, x_j$. The cached $\hat{y}_{i,j}$ were computed while $x_{-n_\text{ctx}}$ was still in the context window so they also depend on $x_{-n_\text{ctx}}$ (dashed red edges). The cached columns have one more token in their context!
°persist_js_stateÂÙ$01372b00-ecb2-42bd-b408-13234717d969Чrunning§runtimeÎ Ú:¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$01372b00-ecb2-42bd-b408-13234717d969¹depends_on_disabled_cells¦queued¤logs�¦output†¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚ¼°œå–·has_pluto_hook_features¤bodyÙ+new_token (generic function with 2 methods)°persist_js_stateÂÙ$f4366cf6-2be0-42b8-96c4-120be3f5c25eЧrunning§runtimeÎ YÆh¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$f4366cf6-2be0-42b8-96c4-120be3f5c25e¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°™!Í-·has_pluto_hook_features¤bodyÚ°persist_js_stateÂÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962Чrunning§runtimeÎ œ£¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°�žú3·has_pluto_hook_features¤bodyÚ
Ë| Query | Wh / query | Scope | Source |
|---|
| Google search (2009) | 0.3 | Google servers, including building the index | Hölzle, 2009 |
| Text generation, models ≤ 11B | ≈ 0.05 | Measured on 8×A100 | Luccioni et al., 2024 |
| Llama-3.1-8B, short / long prompt | 0.05 / 0.44 | Modelled, data centre | Jegham et al., 2025 |
| Gemini Apps, median text prompt (May 2025) | 0.24 (0.10 accelerators only) | Measured, full stack | Elsworth et al., 2025 |
| GPT-4o, 500 output tokens | ≈ 0.3 | Estimate | You, Epoch AI, 2025 |
| ChatGPT, average query | 0.34 | No methodology given | Altman, 2025 |
| Llama-3.1-405B, short / long prompt | 2.2 / 25 | Modelled, data centre | Jegham et al., 2025 |
| o3 (reasoning), short / long prompt | 1.2 / 12 | Modelled, data centre | Jegham et al., 2025 |
| Image generation | ≈ 2.9 | Measured on 8×A100 | Luccioni et al., 2024 |
Small models and median prompts : same order as a (2009) web search.
Large models, long contexts and reasoning : 1 to 2 orders of magnitude more.
Google reports a 33× reduction of the energy per prompt in 12 months Elsworth et al., 2025. There is no recent public figure for a web search.
°persist_js_stateÂÙ$c8923675-e73e-4621-82b9-966d8b003b97Чrunning§runtimeÎ Lo¸depends_on_skipped_cellsµpublished_object_keys�§errored§cell_idÙ$c8923675-e73e-4621-82b9-966d8b003b97¹depends_on_disabled_cells¦queued¤logs�¦output†¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚ¼°��ì·has_pluto_hook_features¤bodyÙb
°persist_js_state±published_objects€julia_version§v1.13.3±cell_dependenciesÞ “Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3„´precedence_heuristic §cell_idÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬LaTeXStrings‘Ù$32621224-a782-4bf6-9570-562cf2bb7360¬Luxor.origin�¤HTML�¦@L_str�¦HAlign�¤html�¥@draw�¬Luxor.sethue�¤text�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2¥Point�°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¬Luxor.finish�¸LaTeXStrings.latexstring�¤Base�§@md_str�¨fontsize�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bb„´precedence_heuristic §cell_idÙ$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bb´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$6622f9f0-cecc-476e-9d49-7d651f433b9f„´precedence_heuristic §cell_idÙ$6622f9f0-cecc-476e-9d49-7d651f433b9f´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$941ad4d9-3406-463e-9af9-96687cd1ecea„´precedence_heuristic §cell_idÙ$941ad4d9-3406-463e-9af9-96687cd1ecea´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$70f395b2-f8c2-44d5-b0af-702659dd7fee„´precedence_heuristic §cell_idÙ$70f395b2-f8c2-44d5-b0af-702659dd7fee´downstream_cells_map€²upstream_cells_map�¤dict‘Ù$d558636d-c714-4033-ae73-5b92c3cdedf3Ù$771d39a5-74dc-494e-929e-1164bb08b983„´precedence_heuristic §cell_idÙ$771d39a5-74dc-494e-929e-1164bb08b983´downstream_cells_map€²upstream_cells_map‚¤llms‘Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc¥table‘Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf„´precedence_heuristic §cell_idÙ$85a10748-8d19-44a8-a1c5-0d13b093f1bf´downstream_cells_map�°draw_transformer˜Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1Ù$18c26901-85eb-45ac-89bf-b03bd255007aÙ$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ecÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3²upstream_cells_map‹¥scale�¦sethue�ªsetopacity�¤text�§readpng�¥Point�°Luxor.placeimage�£box�©translate�¨fontsize�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$9057ce05-ba9b-46c9-847b-514e219f93a1„´precedence_heuristic §cell_idÙ$9057ce05-ba9b-46c9-847b-514e219f93a1´downstream_cells_map�©kv_needed‘Ù$083fcca6-10e0-4483-a754-19d1f2720ec7²upstream_cells_map‹¨CheckBox�«PlutoRunner�¯Core.applicable�¤Core�¨Base.get�¨getindex�¥@bind�·PlutoRunner.create_bond�¤Base�¢qa�§@md_str�Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8e„´precedence_heuristic §cell_idÙ$4f1d5112-dbac-4eb6-8518-0dc4193c3f8e´downstream_cells_map�£bibŸÙ$61dc1905-338f-4bfd-a158-2f6bacff769eÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03Ù$97463c54-7cc7-4497-a8a6-6422f5f582bdÙ$eb18303f-3dfb-4b87-90f2-f6dc542d7221Ù$76e2f97b-1c06-40cd-b134-d5155aa5587dÙ$75ca478c-916f-464a-9435-8208ee726d50Ù$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2aÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012Ù$e41d13ca-1dc1-45ae-9fa6-a83c4101120dÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076Ù$b56e9e56-e74a-401b-b4b5-f36bb33341d5Ù$2a8433e3-9a3b-487b-abf3-09278ea42389Ù$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfcÙ$45efc71d-d5f8-474e-9b89-e72fac7110fdÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313²upstream_cells_map‚§bibrefs�¦biblio‘Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45Ù$93200f46-7c8f-4362-a445-43c57b50a2d2„´precedence_heuristic §cell_idÙ$93200f46-7c8f-4362-a445-43c57b50a2d2´downstream_cells_map€²upstream_cells_map‚¤llms‘Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc¥names�Ù$2a8433e3-9a3b-487b-abf3-09278ea42389„´precedence_heuristic §cell_idÙ$2a8433e3-9a3b-487b-abf3-09278ea42389´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$61dc1905-338f-4bfd-a158-2f6bacff769e„´precedence_heuristic §cell_idÙ$61dc1905-338f-4bfd-a158-2f6bacff769e´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$eb18303f-3dfb-4b87-90f2-f6dc542d7221„´precedence_heuristic §cell_idÙ$eb18303f-3dfb-4b87-90f2-f6dc542d7221´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$76e2f97b-1c06-40cd-b134-d5155aa5587d„´precedence_heuristic §cell_idÙ$76e2f97b-1c06-40cd-b134-d5155aa5587d´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$f6f7376e-9984-4289-b8ff-9d47e5358791„´precedence_heuristic §cell_idÙ$f6f7376e-9984-4289-b8ff-9d47e5358791´downstream_cells_mapƒ³DocumenterCitations�§Logging�£CSV‘Ù$579a203b-e6f7-4190-b874-18b00a5c3f77²upstream_cells_map€Ù$b1a924f4-e2f0-445c-830f-94287a0e52f7„´precedence_heuristic §cell_idÙ$b1a924f4-e2f0-445c-830f-94287a0e52f7´downstream_cells_map�°numerical_lookup‘Ù$95504a74-d5ef-4fb7-83a0-88914c7cbc59²upstream_cells_map…§findmax�§collect�¤keys�¦values�£dot�Ù$f2cba2aa-c541-4692-a441-e65741750a15„´precedence_heuristic §cell_idÙ$f2cba2aa-c541-4692-a441-e65741750a15´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$6fc13413-53de-4c75-9b9e-620e0b7f8a1f„´precedence_heuristic §cell_idÙ$6fc13413-53de-4c75-9b9e-620e0b7f8a1f´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$ccf2dc71-b883-497a-bc58-29ffaf9ea4ad„´precedence_heuristic §cell_idÙ$ccf2dc71-b883-497a-bc58-29ffaf9ea4ad´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2a„´precedence_heuristic §cell_idÙ$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2a´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$86101f07-67c5-4df2-911c-4013c44d6c5b„´precedence_heuristic §cell_idÙ$86101f07-67c5-4df2-911c-4013c44d6c5b´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$a3efd921-eb14-4901-9d6c-800cc812fe02„´precedence_heuristic §cell_idÙ$a3efd921-eb14-4901-9d6c-800cc812fe02´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$321b2919-62f7-42fd-9ae3-8ab9bc309525„´precedence_heuristic §cell_idÙ$321b2919-62f7-42fd-9ae3-8ab9bc309525´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$6800afbf-8ac6-4308-b4cc-b37da57e42c1„´precedence_heuristic §cell_idÙ$6800afbf-8ac6-4308-b4cc-b37da57e42c1´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$18c26901-85eb-45ac-89bf-b03bd255007a„´precedence_heuristic §cell_idÙ$18c26901-85eb-45ac-89bf-b03bd255007a´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87¬Luxor.finish�¤Base�£img�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$bcbb3db2-85b3-4cb0-9309-f5c032d14da5„´precedence_heuristic §cell_idÙ$bcbb3db2-85b3-4cb0-9309-f5c032d14da5´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$4e10271c-49f8-4f1d-869c-5fa11275d7f6„´precedence_heuristic §cell_idÙ$4e10271c-49f8-4f1d-869c-5fa11275d7f6´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$c1437dcc-22cb-424f-9b8e-326172f82d86„´precedence_heuristic §cell_idÙ$c1437dcc-22cb-424f-9b8e-326172f82d86´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$c09ec483-9fcf-48e7-b3c0-2508289e3cf3„´precedence_heuristic §cell_idÙ$c09ec483-9fcf-48e7-b3c0-2508289e3cf3´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060„´precedence_heuristic §cell_idÙ$37484e4a-87ad-4022-a8d9-e189fbbc8060´downstream_cells_map€²upstream_cells_map‰¡/�¥round�energy_tokens‘Ù$ba00b790-67ac-49b8-86ee-6df42107f245§@md_str�¨getindex�energy_params‘Ù$ba00b790-67ac-49b8-86ee-6df42107f245«energy_util‘Ù$ba00b790-67ac-49b8-86ee-6df42107f245¯energy_overhead‘Ù$ba00b790-67ac-49b8-86ee-6df42107f245¡*�Ù$1bbf2152-4fdf-4ed2-9bdf-95d699824d11„´precedence_heuristic §cell_idÙ$1bbf2152-4fdf-4ed2-9bdf-95d699824d11´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$a0d935f9-185a-4608-ab9f-954fa0e66f44„´precedence_heuristic §cell_idÙ$a0d935f9-185a-4608-ab9f-954fa0e66f44´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$728c16b7-50cd-43fe-a0d7-61d37952a6b3„´precedence_heuristic §cell_idÙ$728c16b7-50cd-43fe-a0d7-61d37952a6b3´downstream_cells_map€²upstream_cells_map�ªpair_stats‘Ù$89305cae-098f-4644-9109-d00f1e3bc04cÙ$1faa4ab2-6c93-47dc-b631-8be52780fe7d„´precedence_heuristic §cell_idÙ$1faa4ab2-6c93-47dc-b631-8be52780fe7d´downstream_cells_map€²upstream_cells_map‚¤dict‘Ù$d558636d-c714-4033-ae73-5b92c3cdedf3®softmax_lookup‘Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9Ù$c3a9a0ce-3450-4b17-8696-2ab8534b29f2„´precedence_heuristic §cell_idÙ$c3a9a0ce-3450-4b17-8696-2ab8534b29f2´downstream_cells_map�¦iter_3�²upstream_cells_map‚©new_token’Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969¦iter_2‘Ù$736920df-e4bb-4535-b982-e397aa0a782dÙ$e383bb72-49a1-4df1-84c3-b95a2ffe00f5„´precedence_heuristic §cell_idÙ$e383bb72-49a1-4df1-84c3-b95a2ffe00f5´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$6aa690e9-389f-4398-abae-b95060db4d90„´precedence_heuristic §cell_idÙ$6aa690e9-389f-4398-abae-b95060db4d90´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$4b61363d-87c9-4755-8286-44df34e9dd6a„´precedence_heuristic §cell_idÙ$4b61363d-87c9-4755-8286-44df34e9dd6a´downstream_cells_map€²upstream_cells_map†¤Base�Base.getindex�©@html_str�¢qa�§@md_str�®Base.Docs.HTML�Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87„´precedence_heuristic §cell_idÙ$94ae440d-0644-49db-9461-f1a1ff1d7f87´downstream_cells_map�¤citeÜ Ù$f4366cf6-2be0-42b8-96c4-120be3f5c25eÙ$0583ee0c-3802-4e81-b179-a80a82493b43Ù$2a7e5096-1e8d-4506-96d2-86de0a7d39aaÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904Ù$55a09acc-84da-491c-86ba-9a66f4ea52feÙ$225e58ba-b78d-4a0a-be4f-ad642c879b93Ù$a21fbc70-9137-4d0e-8c8c-cbdc5269778fÙ$86101f07-67c5-4df2-911c-4013c44d6c5bÙ$5150d8f3-6e85-43f2-801a-eae5cc3e3095Ù$d014e6aa-92f6-4ca1-be47-516565d1bb20Ù$18c26901-85eb-45ac-89bf-b03bd255007aÙ$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5ddÙ$25b79953-fd7c-46c1-b760-d57c09910981Ù$c1437dcc-22cb-424f-9b8e-326172f82d86²upstream_cells_map‚§bibcite�¦biblio‘Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45Ù$55435b26-7fc3-4c8b-8013-6fd4fb65a08e„´precedence_heuristic §cell_idÙ$55435b26-7fc3-4c8b-8013-6fd4fb65a08e´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$225e58ba-b78d-4a0a-be4f-ad642c879b93„´precedence_heuristic §cell_idÙ$225e58ba-b78d-4a0a-be4f-ad642c879b93´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$32621224-a782-4bf6-9570-562cf2bb7360„´precedence_heuristic§cell_idÙ$32621224-a782-4bf6-9570-562cf2bb7360´downstream_cells_mapŠLinearAlgebra�MathTeXEngine�°HypertextLiteral’Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418Ù$a57f2149-c876-4421-a958-e9aed1f892a6²PlutoTeachingTools‘Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418§PlutoUI‘Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418¬PrettyTables�¬LaTeXStrings‘Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3²ExperimentalLayout�ªDataFrames�¥LuxorœÙ$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1Ù$18c26901-85eb-45ac-89bf-b03bd255007aÙ$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5ddÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ecÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bfÙ$a57f2149-c876-4421-a958-e9aed1f892a6Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85²upstream_cells_map€Ù$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0„´precedence_heuristic §cell_idÙ$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$0c0c1163-0aec-4089-9acc-539b3a86d0b3„´precedence_heuristic §cell_idÙ$0c0c1163-0aec-4089-9acc-539b3a86d0b3´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85„´precedence_heuristic §cell_idÙ$293266c6-01e7-4da0-8caf-56b1ce0aae85´downstream_cells_map‚¨#436#val�±transformer_graph’Ù$083fcca6-10e0-4483-a754-19d1f2720ec7Ù$52c97de1-3d75-4b51-8397-86ea67af6c77²upstream_cells_mapÞ !¡/�¥first�¦isless�«Luxor.arrow�¡-�ªLuxor.line�°Luxor.setopacity�Luxor.setline�«Luxor.Point�¦length�¡+�¤@doc�¡>�¢<=�£sum�¬Luxor.sethue�¢!=�Luxor.setdash�¥Union�¡:�º#___this_pluto_module_name�¡<�®mathjax_figure‘Ù$a57f2149-c876-4421-a958-e9aed1f892a6¤last�¡!�¢==�¥Tuple�¢in�¬Luxor.circle�©Luxor.box�®Luxor.distance�¡*�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1„´precedence_heuristic §cell_idÙ$c5be3956-5102-4d88-bfdb-9813c0555fe1´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10„´precedence_heuristic §cell_idÙ$7e27c349-ee76-46bd-b1c2-a9ce54974e10´downstream_cells_map�¥table“Ù$771d39a5-74dc-494e-929e-1164bb08b983Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3²upstream_cells_mapŒ§nothing�¦String�¡:�¨Markdown�¡!�©ismissing�®Markdown.parse�¤sort�©isnothing�¬pretty_table�¦unique�¥names�Ù$2e8b1a77-1f04-4035-8d82-4061d81ecb7a„´precedence_heuristic §cell_idÙ$2e8b1a77-1f04-4035-8d82-4061d81ecb7a´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$29474a70-32eb-4281-8626-87819afa7267„´precedence_heuristic §cell_idÙ$29474a70-32eb-4281-8626-87819afa7267´downstream_cells_map�¨add_pair‘Ù$01372b00-ecb2-42bd-b408-13234717d969²upstream_cells_map†¤copy�¥push!�¤prod�§replace�£BPE‘Ù$d050a7ee-3aa7-4539-a236-5b6446599ded¢=>�Ù$6bff7bca-ea1d-44c6-b8c3-040250f90654„´precedence_heuristic §cell_idÙ$6bff7bca-ea1d-44c6-b8c3-040250f90654´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$0322b582-219b-4fbb-92e5-c82ff635615c„´precedence_heuristic §cell_idÙ$0322b582-219b-4fbb-92e5-c82ff635615c´downstream_cells_map�ºdraw_preactivation_decoder‘Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd²upstream_cells_mapÞ ¡/�¨fontface�¦sethue�¥arrow�¡-�¦length�¡+�§nothing�¤text�¤line�©enumerate�¥Point�¦circle�£box�¤poly�§setline�¨fontsize�¡*�Ù$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1b„´precedence_heuristic §cell_idÙ$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1b´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$6f72e8a5-819d-474c-a725-7f7318d964d7„´precedence_heuristic§cell_idÙ$6f72e8a5-819d-474c-a725-7f7318d964d7´downstream_cells_map€²upstream_cells_map�§include�Ù$b7583418-f4fb-4c63-b421-b5b9af269768„´precedence_heuristic §cell_idÙ$b7583418-f4fb-4c63-b421-b5b9af269768´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$a873f760-bfc1-489f-a58e-75e12afa54f2„´precedence_heuristic §cell_idÙ$a873f760-bfc1-489f-a58e-75e12afa54f2´downstream_cells_map�©highlight—Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1Ù$18c26901-85eb-45ac-89bf-b03bd255007aÙ$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3²upstream_cells_map…£box�ªpolysmooth�¦sethue�¥Point�ªsetopacity�Ù$76ba4e9b-8bb0-47c4-b607-2ca711f035e6„´precedence_heuristic §cell_idÙ$76ba4e9b-8bb0-47c4-b607-2ca711f035e6´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$b2982908-a2ec-436a-abae-4dcdf896a3af„´precedence_heuristic §cell_idÙ$b2982908-a2ec-436a-abae-4dcdf896a3af´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0d„´precedence_heuristic §cell_idÙ$91abc03b-fef7-4f93-96fc-13f1cf654f0d´downstream_cells_map€²upstream_cells_mapÞ ¤llms‘Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc°Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¥table‘Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$a5b20939-9afa-48c0-aa67-cbca6bc99804„´precedence_heuristic §cell_idÙ$a5b20939-9afa-48c0-aa67-cbca6bc99804´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475„´precedence_heuristic §cell_idÙ$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$beccf4e8-1b01-4cb2-b23c-bc5db604f21c„´precedence_heuristic §cell_idÙ$beccf4e8-1b01-4cb2-b23c-bc5db604f21c´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$e41d13ca-1dc1-45ae-9fa6-a83c4101120d„´precedence_heuristic §cell_idÙ$e41d13ca-1dc1-45ae-9fa6-a83c4101120d´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$be4a698a-848d-47bf-b825-24bc94621767„´precedence_heuristic §cell_idÙ$be4a698a-848d-47bf-b825-24bc94621767´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418„´precedence_heuristic §cell_idÙ$5058e4eb-c53d-4468-b7ff-4f04ded96418´downstream_cells_map€²upstream_cells_map‰°HypertextLiteral‘Ù$32621224-a782-4bf6-9570-562cf2bb7360¸HypertextLiteral.content�¤@htl�·HypertextLiteral.Result�·PlutoUI.TableOfContents�·HypertextLiteral.Bypass�§PlutoUI‘Ù$32621224-a782-4bf6-9570-562cf2bb7360²PlutoTeachingTools‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$PlutoTeachingTools.ChooseDisplayMode�Ù$5150d8f3-6e85-43f2-801a-eae5cc3e3095„´precedence_heuristic §cell_idÙ$5150d8f3-6e85-43f2-801a-eae5cc3e3095´downstream_cells_map€²upstream_cells_map…¦HAlign�£img�§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$8c27b182-0c3c-4c19-9619-df62b7dd6bf0„´precedence_heuristic §cell_idÙ$8c27b182-0c3c-4c19-9619-df62b7dd6bf0´downstream_cells_map€²upstream_cells_mapƒ¦HAlign�§@md_str�¨getindex�Ù$a3380780-ec5e-46c1-a0e0-af64b3e1da1c„´precedence_heuristic §cell_idÙ$a3380780-ec5e-46c1-a0e0-af64b3e1da1c´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$9a8eef1b-27c0-4d57-a389-53708ade9058„´precedence_heuristic §cell_idÙ$9a8eef1b-27c0-4d57-a389-53708ade9058´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc„´precedence_heuristic §cell_idÙ$f39305ea-f7f5-440e-ac55-c83e27f6e7fc´downstream_cells_map�¤llms”Ù$771d39a5-74dc-494e-929e-1164bb08b983Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$93200f46-7c8f-4362-a445-43c57b50a2d2²upstream_cells_map�©load_llms‘Ù$579a203b-e6f7-4190-b874-18b00a5c3f77Ù$f7ca738d-5215-4e91-a2f3-a5ff10911313„´precedence_heuristic §cell_idÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$c7f318b9-30e6-4b79-b7da-52f70904d246„´precedence_heuristic §cell_idÙ$c7f318b9-30e6-4b79-b7da-52f70904d246´downstream_cells_map�ªsubstitute�²upstream_cells_map‹¦isless�¡-�§replace�£min�¤prod�¡+�¦String�¤Char�¢=>�¥Tuple�§minimum�Ù$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfc„´precedence_heuristic §cell_idÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfc´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$8b78360a-21cb-4574-a84d-46ea4d0cedb1„´precedence_heuristic §cell_idÙ$8b78360a-21cb-4574-a84d-46ea4d0cedb1´downstream_cells_map€²upstream_cells_map�£img�Ù$95ec4140-9147-11ef-2af4-5528bad0e6f5„´precedence_heuristic §cell_idÙ$95ec4140-9147-11ef-2af4-5528bad0e6f5´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9„´precedence_heuristic §cell_idÙ$77f446ac-6030-48f2-9bea-93c427f9fcb9´downstream_cells_map�®softmax_lookup‘Ù$1faa4ab2-6c93-47dc-b631-8be52780fe7d²upstream_cells_map‹¦values�¤keys�£sum�¥@show�£zip�©Base.repr�¬Base.println�¤Base�£dot�¡*�§softmax‘Ù$570fa160-3adb-463e-99b8-b7dd05076908Ù$d05e6f0f-0081-4fb6-91e9-ac2f58beda4a„´precedence_heuristic §cell_idÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4a´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$d54b5390-0ec0-4ff8-ab18-51726482ca46„´precedence_heuristic §cell_idÙ$d54b5390-0ec0-4ff8-ab18-51726482ca46´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f572e113-b36b-4a6b-96c7-c26f100e1ad4„´precedence_heuristic §cell_idÙ$f572e113-b36b-4a6b-96c7-c26f100e1ad4´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89„´precedence_heuristic §cell_idÙ$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$a21fbc70-9137-4d0e-8c8c-cbdc5269778f„´precedence_heuristic §cell_idÙ$a21fbc70-9137-4d0e-8c8c-cbdc5269778f´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$ba00b790-67ac-49b8-86ee-6df42107f245„´precedence_heuristic §cell_idÙ$ba00b790-67ac-49b8-86ee-6df42107f245´downstream_cells_map„energy_tokens‘Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060energy_params‘Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060«energy_util‘Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060¯energy_overhead‘Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060²upstream_cells_map�«PlutoRunner�¯Core.applicable�¤Core�¨Base.get�¨getindex�¦Select�¥@bind�·PlutoRunner.create_bond�¡:�¢=>�¤Base�§@md_str�¦Slider�Ù$c3db7eb2-356a-428f-9777-6369662d8b06„´precedence_heuristic §cell_idÙ$c3db7eb2-356a-428f-9777-6369662d8b06´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$9e898325-e9e2-45bd-af74-3dd86f00f7b5„´precedence_heuristic §cell_idÙ$9e898325-e9e2-45bd-af74-3dd86f00f7b5´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$97463c54-7cc7-4497-a8a6-6422f5f582bd„´precedence_heuristic §cell_idÙ$97463c54-7cc7-4497-a8a6-6422f5f582bd´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012„´precedence_heuristic §cell_idÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$89305cae-098f-4644-9109-d00f1e3bc04c„´precedence_heuristic §cell_idÙ$89305cae-098f-4644-9109-d00f1e3bc04c´downstream_cells_map�ªpair_stats’Ù$728c16b7-50cd-43fe-a0d7-61d37952a6b3Ù$01372b00-ecb2-42bd-b408-13234717d969²upstream_cells_map�¦isless�§nextind�©eachindex�£get�©lastindex�¡+�¡>�¤Dict�¦String�¡<�¤Char�¥Tuple�£Int�Ù$bcf7667f-f99b-4d10-af84-5d3879f1db5d„´precedence_heuristic §cell_idÙ$bcf7667f-f99b-4d10-af84-5d3879f1db5d´downstream_cells_map€²upstream_cells_map†¤Base�Base.getindex�©@html_str�¢qa�§@md_str�®Base.Docs.HTML�Ù$25b79953-fd7c-46c1-b760-d57c09910981„´precedence_heuristic §cell_idÙ$25b79953-fd7c-46c1-b760-d57c09910981´downstream_cells_map€²upstream_cells_map„¢qa�§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$c032b3ff-c539-4e38-81d0-39b28b3a8076„´precedence_heuristic §cell_idÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$b9a962a6-4f4f-4676-95b6-0c2cfd536031„´precedence_heuristic §cell_idÙ$b9a962a6-4f4f-4676-95b6-0c2cfd536031´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$cb6b040d-c213-4ecb-a047-fb66407752a3„´precedence_heuristic §cell_idÙ$cb6b040d-c213-4ecb-a047-fb66407752a3´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$45efc71d-d5f8-474e-9b89-e72fac7110fd„´precedence_heuristic §cell_idÙ$45efc71d-d5f8-474e-9b89-e72fac7110fd´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$083fcca6-10e0-4483-a754-19d1f2720ec7„´precedence_heuristic §cell_idÙ$083fcca6-10e0-4483-a754-19d1f2720ec7´downstream_cells_map€²upstream_cells_map‚©kv_needed‘Ù$9057ce05-ba9b-46c9-847b-514e219f93a1±transformer_graph‘Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85Ù$d050a7ee-3aa7-4539-a236-5b6446599ded„´precedence_heuristic §cell_idÙ$d050a7ee-3aa7-4539-a236-5b6446599ded´downstream_cells_map�£BPE“Ù$29474a70-32eb-4281-8626-87819afa7267Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969²upstream_cells_map„¦String�¥Tuple�¤Char�¤Dict�Ù$579a203b-e6f7-4190-b874-18b00a5c3f77„´precedence_heuristic §cell_idÙ$579a203b-e6f7-4190-b874-18b00a5c3f77´downstream_cells_map�©load_llms‘Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc²upstream_cells_map…¨CSV.File�©DataFrame�£CSV‘Ù$f6f7376e-9984-4289-b8ff-9d47e5358791¢=>�§rename!�Ù$570fa160-3adb-463e-99b8-b7dd05076908„´precedence_heuristic §cell_idÙ$570fa160-3adb-463e-99b8-b7dd05076908´downstream_cells_map�§softmax‘Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9²upstream_cells_mapƒ¡/�£exp�£sum�Ù$9cb90e76-3bb5-41ff-bc79-c4949400d904„´precedence_heuristic §cell_idÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$75ca478c-916f-464a-9435-8208ee726d50„´precedence_heuristic §cell_idÙ$75ca478c-916f-464a-9435-8208ee726d50´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03„´precedence_heuristic §cell_idÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$1d02cfbb-9d6b-41fd-9621-0103a9110813„´precedence_heuristic §cell_idÙ$1d02cfbb-9d6b-41fd-9621-0103a9110813´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$9ff95a9a-192b-4a12-8e2e-7acd6659c066„´precedence_heuristic §cell_idÙ$9ff95a9a-192b-4a12-8e2e-7acd6659c066´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f95a6de6-5e02-4237-88ba-ec44ef3d38c3„´precedence_heuristic §cell_idÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3´downstream_cells_map€²upstream_cells_mapÞ ¤llms‘Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc°Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¥table‘Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$ed5b5702-4cca-4116-a70f-4a562178f490„´precedence_heuristic §cell_idÙ$ed5b5702-4cca-4116-a70f-4a562178f490´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$736920df-e4bb-4535-b982-e397aa0a782d„´precedence_heuristic §cell_idÙ$736920df-e4bb-4535-b982-e397aa0a782d´downstream_cells_map�¦iter_2‘Ù$c3a9a0ce-3450-4b17-8696-2ab8534b29f2²upstream_cells_map‚¦iter_1‘Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324©new_token’Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969Ù$55a09acc-84da-491c-86ba-9a66f4ea52fe„´precedence_heuristic §cell_idÙ$55a09acc-84da-491c-86ba-9a66f4ea52fe´downstream_cells_map€²upstream_cells_map…¦HAlign�£img�§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$52c97de1-3d75-4b51-8397-86ea67af6c77„´precedence_heuristic §cell_idÙ$52c97de1-3d75-4b51-8397-86ea67af6c77´downstream_cells_map€²upstream_cells_map�±transformer_graph‘Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85Ù$a6a6a352-7479-4f3f-9abb-900eb901466c„´precedence_heuristic §cell_idÙ$a6a6a352-7479-4f3f-9abb-900eb901466c´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$d558636d-c714-4033-ae73-5b92c3cdedf3„´precedence_heuristic §cell_idÙ$d558636d-c714-4033-ae73-5b92c3cdedf3´downstream_cells_map�¤dict“Ù$70f395b2-f8c2-44d5-b0af-702659dd7feeÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59Ù$1faa4ab2-6c93-47dc-b631-8be52780fe7d²upstream_cells_map‚¢=>�¤Dict�Ù$8d231f2c-4b0c-4c37-a746-16e98d4cafc8„´precedence_heuristic §cell_idÙ$8d231f2c-4b0c-4c37-a746-16e98d4cafc8´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbe„´precedence_heuristic §cell_idÙ$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbe´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993f„´precedence_heuristic §cell_idÙ$f1afaf8c-d9ad-446a-9826-9c4cda19993f´downstream_cells_map�©new_token“Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324Ù$736920df-e4bb-4535-b982-e397aa0a782dÙ$c3a9a0ce-3450-4b17-8696-2ab8534b29f2²upstream_cells_mapƒ¦String�£BPE‘Ù$d050a7ee-3aa7-4539-a236-5b6446599ded¤Dict�Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324„´precedence_heuristic §cell_idÙ$f7ca3ff7-b5cf-452b-b955-7219e7397324´downstream_cells_map�¦iter_1‘Ù$736920df-e4bb-4535-b982-e397aa0a782d²upstream_cells_map�©new_token’Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969Ù$5f05e717-a51a-4a99-bb11-cc493217f93f„´precedence_heuristic §cell_idÙ$5f05e717-a51a-4a99-bb11-cc493217f93f´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�©highlight‘Ù$a873f760-bfc1-489f-a58e-75e12afa54f2°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$b56e9e56-e74a-401b-b4b5-f36bb33341d5„´precedence_heuristic §cell_idÙ$b56e9e56-e74a-401b-b4b5-f36bb33341d5´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$57c2c944-0d91-489d-8ad7-f5520e71ef3e„´precedence_heuristic §cell_idÙ$57c2c944-0d91-489d-8ad7-f5520e71ef3e´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$32fc8313-f3e8-489f-b0a1-ba740bc744fb„´precedence_heuristic §cell_idÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fb´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$bf563783-9784-4c74-a7b1-6d7a3ed618c5„´precedence_heuristic §cell_idÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$f8330700-e964-4e19-9c55-2b11df45789e„´precedence_heuristic §cell_idÙ$f8330700-e964-4e19-9c55-2b11df45789e´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$b245ada9-1677-45bc-a883-bc213a2105b7„´precedence_heuristic §cell_idÙ$b245ada9-1677-45bc-a883-bc213a2105b7´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ec„´precedence_heuristic §cell_idÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ec´downstream_cells_map€²upstream_cells_mapÞ ¥scale�°Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�§readpng�Luxor.preview�Luxor.Drawing�Base.getindex�°Luxor.placeimage�°draw_transformer‘Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf¬Luxor.finish�¤Base�¢qa�©translate�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd„´precedence_heuristic §cell_idÙ$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd´downstream_cells_map€²upstream_cells_mapÞ °Luxor.background�¬Luxor.origin�¤HTML�¦HAlign�¤html�¥@draw�¬Luxor.sethue�Luxor.preview�Luxor.Drawing�Base.getindex�ºdraw_preactivation_decoder‘Ù$0322b582-219b-4fbb-92e5-c82ff635615c¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87¬Luxor.finish�¤Base�§@md_str�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45„´precedence_heuristic §cell_idÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45´downstream_cells_map�¦biblio’Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8e²upstream_cells_map�¬load_biblio!�Ù$3176c675-ce53-4471-b13b-1d694f5efdd8„´precedence_heuristic §cell_idÙ$3176c675-ce53-4471-b13b-1d694f5efdd8´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$006f44b0-c10d-44c3-8300-183e706b605d„´precedence_heuristic §cell_idÙ$006f44b0-c10d-44c3-8300-183e706b605d´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$a14e505e-2e4a-4c73-8133-7560ba58916b„´precedence_heuristic §cell_idÙ$a14e505e-2e4a-4c73-8133-7560ba58916b´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$2a7e5096-1e8d-4506-96d2-86de0a7d39aa„´precedence_heuristic §cell_idÙ$2a7e5096-1e8d-4506-96d2-86de0a7d39aa´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$d605d0a5-7511-4a32-9e1b-1c2befa43796„´precedence_heuristic §cell_idÙ$d605d0a5-7511-4a32-9e1b-1c2befa43796´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$6712c883-b407-47e1-a666-4de05f8f8d6e„´precedence_heuristic §cell_idÙ$6712c883-b407-47e1-a666-4de05f8f8d6e´downstream_cells_map€²upstream_cells_map…¦HAlign�£img�§@md_str�¨getindex�¢=>�Ù$7be26ee1-de61-4b66-b221-5c5c728c2901„´precedence_heuristic §cell_idÙ$7be26ee1-de61-4b66-b221-5c5c728c2901´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$04e9b912-6712-4290-acc4-f24bb27a1469„´precedence_heuristic §cell_idÙ$04e9b912-6712-4290-acc4-f24bb27a1469´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$af8194a1-a358-4cf7-b446-6b377cb76687„´precedence_heuristic §cell_idÙ$af8194a1-a358-4cf7-b446-6b377cb76687´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03„´precedence_heuristic §cell_idÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03´downstream_cells_map€²upstream_cells_map�£bib‘Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30„´precedence_heuristic §cell_idÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$aa10118d-0991-4284-97f8-3a0728636a66„´precedence_heuristic §cell_idÙ$aa10118d-0991-4284-97f8-3a0728636a66´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$e2eca085-9f99-4e3a-9db4-e7f692aedd34„´precedence_heuristic §cell_idÙ$e2eca085-9f99-4e3a-9db4-e7f692aedd34´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$d014e6aa-92f6-4ca1-be47-516565d1bb20„´precedence_heuristic §cell_idÙ$d014e6aa-92f6-4ca1-be47-516565d1bb20´downstream_cells_map€²upstream_cells_map†¦HAlign�£img�§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87¢=>�Ù$b4bfde70-510d-4ed5-acb2-7f0cdc3057ae„´precedence_heuristic §cell_idÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057ae´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$e6420fcc-3b76-4dd5-b172-c7381afb5e5e„´precedence_heuristic §cell_idÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5e´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$a57f2149-c876-4421-a958-e9aed1f892a6„´precedence_heuristic §cell_idÙ$a57f2149-c876-4421-a958-e9aed1f892a6´downstream_cells_map‚¨#440#val�®mathjax_figure‘Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85²upstream_cells_mapÞ °HypertextLiteral‘Ù$32621224-a782-4bf6-9570-562cf2bb7360¡/�¸HypertextLiteral.content�¤@htl�°Luxor.background�£Any�¬Luxor.origin�¤HTML�¯Luxor.svgstring�¡+�¤@doc�¥Union�Luxor.Drawing�¥push!�·HypertextLiteral.Result�º#___this_pluto_module_name�¬Luxor.finish�·HypertextLiteral.Bypass�¢==�¥Tuple�¥Luxor‘Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù HypertextLiteral.attribute_value�Ù$95504a74-d5ef-4fb7-83a0-88914c7cbc59„´precedence_heuristic §cell_idÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59´downstream_cells_map€²upstream_cells_map‚°numerical_lookup‘Ù$b1a924f4-e2f0-445c-830f-94287a0e52f7¤dict‘Ù$d558636d-c714-4033-ae73-5b92c3cdedf3Ù$453544fc-0e3e-4e04-8c0c-192f3a038884„´precedence_heuristic §cell_idÙ$453544fc-0e3e-4e04-8c0c-192f3a038884´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$0583ee0c-3802-4e81-b179-a80a82493b43„´precedence_heuristic §cell_idÙ$0583ee0c-3802-4e81-b179-a80a82493b43´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2„´precedence_heuristic §cell_idÙ$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2´downstream_cells_map€²upstream_cells_mapƒ¢qa�§@md_str�¨getindex�Ù$01372b00-ecb2-42bd-b408-13234717d969„´precedence_heuristic §cell_idÙ$01372b00-ecb2-42bd-b408-13234717d969´downstream_cells_map�©new_token“Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324Ù$736920df-e4bb-4535-b982-e397aa0a782dÙ$c3a9a0ce-3450-4b17-8696-2ab8534b29f2²upstream_cells_mapЦisless�§findmax�¡+�§minimum�£BPE‘Ù$d050a7ee-3aa7-4539-a236-5b6446599ded¨add_pair‘Ù$29474a70-32eb-4281-8626-87819afa7267ªpair_stats‘Ù$89305cae-098f-4644-9109-d00f1e3bc04c¢=>�¡-�£min�Ù$f4366cf6-2be0-42b8-96c4-120be3f5c25e„´precedence_heuristic §cell_idÙ$f4366cf6-2be0-42b8-96c4-120be3f5c25e´downstream_cells_map€²upstream_cells_mapƒ§@md_str�¨getindex�¤cite‘Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$22f68d24-1132-4e45-b8fb-7a1f362f0962„´precedence_heuristic §cell_idÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�Ù$c8923675-e73e-4621-82b9-966d8b003b97„´precedence_heuristic §cell_idÙ$c8923675-e73e-4621-82b9-966d8b003b97´downstream_cells_map€²upstream_cells_map‚§@md_str�¨getindex�®last_save_timeËAÚ¼°�‚µ«notebook_idÙ$1f76e9f8-dcd5-11f1-bf3f-a59c0631e4a9«in_temp_dir¨metadata€¥bonds€´cell_execution_orderÜ “Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$6f72e8a5-819d-474c-a725-7f7318d964d7Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418Ù$95ec4140-9147-11ef-2af4-5528bad0e6f5Ù$c09ec483-9fcf-48e7-b3c0-2508289e3cf3Ù$beccf4e8-1b01-4cb2-b23c-bc5db604f21cÙ$ccf2dc71-b883-497a-bc58-29ffaf9ea4adÙ$1bbf2152-4fdf-4ed2-9bdf-95d699824d11Ù$57c2c944-0d91-489d-8ad7-f5520e71ef3eÙ$c3db7eb2-356a-428f-9777-6369662d8b06Ù$2e8b1a77-1f04-4035-8d82-4061d81ecb7aÙ$ed5b5702-4cca-4116-a70f-4a562178f490Ù$e2eca085-9f99-4e3a-9db4-e7f692aedd34Ù$9e898325-e9e2-45bd-af74-3dd86f00f7b5Ù$bcf7667f-f99b-4d10-af84-5d3879f1db5dÙ$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0Ù$6622f9f0-cecc-476e-9d49-7d651f433b9fÙ$6aa690e9-389f-4398-abae-b95060db4d90Ù$6712c883-b407-47e1-a666-4de05f8f8d6eÙ$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1bÙ$f8330700-e964-4e19-9c55-2b11df45789eÙ$4e10271c-49f8-4f1d-869c-5fa11275d7f6Ù$d54b5390-0ec0-4ff8-ab18-51726482ca46Ù$6800afbf-8ac6-4308-b4cc-b37da57e42c1Ù$55435b26-7fc3-4c8b-8013-6fd4fb65a08eÙ$bcbb3db2-85b3-4cb0-9309-f5c032d14da5Ù$d558636d-c714-4033-ae73-5b92c3cdedf3Ù$70f395b2-f8c2-44d5-b0af-702659dd7feeÙ$b1a924f4-e2f0-445c-830f-94287a0e52f7Ù$95504a74-d5ef-4fb7-83a0-88914c7cbc59Ù$8d231f2c-4b0c-4c37-a746-16e98d4cafc8Ù$570fa160-3adb-463e-99b8-b7dd05076908Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9Ù$1faa4ab2-6c93-47dc-b631-8be52780fe7dÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5Ù$9ff95a9a-192b-4a12-8e2e-7acd6659c066Ù$76ba4e9b-8bb0-47c4-b607-2ca711f035e6Ù$8c27b182-0c3c-4c19-9619-df62b7dd6bf0Ù$0c0c1163-0aec-4089-9acc-539b3a86d0b3Ù$b7583418-f4fb-4c63-b421-b5b9af269768Ù$6fc13413-53de-4c75-9b9e-620e0b7f8a1fÙ$006f44b0-c10d-44c3-8300-183e706b605dÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4aÙ$a3efd921-eb14-4901-9d6c-800cc812fe02Ù$4b61363d-87c9-4755-8286-44df34e9dd6aÙ$453544fc-0e3e-4e04-8c0c-192f3a038884Ù$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bbÙ$f2cba2aa-c541-4692-a441-e65741750a15Ù$e383bb72-49a1-4df1-84c3-b95a2ffe00f5Ù$af8194a1-a358-4cf7-b446-6b377cb76687Ù$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475Ù$a5b20939-9afa-48c0-aa67-cbca6bc99804Ù$b2982908-a2ec-436a-abae-4dcdf896a3afÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5eÙ$9057ce05-ba9b-46c9-847b-514e219f93a1Ù$a14e505e-2e4a-4c73-8133-7560ba58916bÙ$9a8eef1b-27c0-4d57-a389-53708ade9058Ù$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2Ù$b9a962a6-4f4f-4676-95b6-0c2cfd536031Ù$a0d935f9-185a-4608-ab9f-954fa0e66f44Ù$941ad4d9-3406-463e-9af9-96687cd1eceaÙ$c8923675-e73e-4621-82b9-966d8b003b97Ù$04e9b912-6712-4290-acc4-f24bb27a1469Ù$8b78360a-21cb-4574-a84d-46ea4d0cedb1Ù$6bff7bca-ea1d-44c6-b8c3-040250f90654Ù$321b2919-62f7-42fd-9ae3-8ab9bc309525Ù$7be26ee1-de61-4b66-b221-5c5c728c2901Ù$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbeÙ$3176c675-ce53-4471-b13b-1d694f5efdd8Ù$ba00b790-67ac-49b8-86ee-6df42107f245Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060Ù$a3380780-ec5e-46c1-a0e0-af64b3e1da1cÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fbÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962Ù$be4a698a-848d-47bf-b825-24bc94621767Ù$d605d0a5-7511-4a32-9e1b-1c2befa43796Ù$cb6b040d-c213-4ecb-a047-fb66407752a3Ù$1d02cfbb-9d6b-41fd-9621-0103a9110813Ù$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89Ù$aa10118d-0991-4284-97f8-3a0728636a66Ù$b245ada9-1677-45bc-a883-bc213a2105b7Ù$a6a6a352-7479-4f3f-9abb-900eb901466cÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057aeÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30Ù$f572e113-b36b-4a6b-96c7-c26f100e1ad4Ù$f6f7376e-9984-4289-b8ff-9d47e5358791Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$f4366cf6-2be0-42b8-96c4-120be3f5c25eÙ$0583ee0c-3802-4e81-b179-a80a82493b43Ù$2a7e5096-1e8d-4506-96d2-86de0a7d39aaÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904Ù$55a09acc-84da-491c-86ba-9a66f4ea52feÙ$225e58ba-b78d-4a0a-be4f-ad642c879b93Ù$a21fbc70-9137-4d0e-8c8c-cbdc5269778fÙ$86101f07-67c5-4df2-911c-4013c44d6c5bÙ$5150d8f3-6e85-43f2-801a-eae5cc3e3095Ù$d014e6aa-92f6-4ca1-be47-516565d1bb20Ù$25b79953-fd7c-46c1-b760-d57c09910981Ù$c1437dcc-22cb-424f-9b8e-326172f82d86Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$61dc1905-338f-4bfd-a158-2f6bacff769eÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03Ù$97463c54-7cc7-4497-a8a6-6422f5f582bdÙ$eb18303f-3dfb-4b87-90f2-f6dc542d7221Ù$76e2f97b-1c06-40cd-b134-d5155aa5587dÙ$75ca478c-916f-464a-9435-8208ee726d50Ù$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2aÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012Ù$e41d13ca-1dc1-45ae-9fa6-a83c4101120dÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076Ù$b56e9e56-e74a-401b-b4b5-f36bb33341d5Ù$2a8433e3-9a3b-487b-abf3-09278ea42389Ù$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfcÙ$45efc71d-d5f8-474e-9b89-e72fac7110fdÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bfÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ecÙ$0322b582-219b-4fbb-92e5-c82ff635615cÙ$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5ddÙ$a873f760-bfc1-489f-a58e-75e12afa54f2Ù$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1Ù$18c26901-85eb-45ac-89bf-b03bd255007aÙ$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3Ù$a57f2149-c876-4421-a958-e9aed1f892a6Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85Ù$083fcca6-10e0-4483-a754-19d1f2720ec7Ù$52c97de1-3d75-4b51-8397-86ea67af6c77Ù$d050a7ee-3aa7-4539-a236-5b6446599dedÙ$29474a70-32eb-4281-8626-87819afa7267Ù$89305cae-098f-4644-9109-d00f1e3bc04cÙ$728c16b7-50cd-43fe-a0d7-61d37952a6b3Ù$c7f318b9-30e6-4b79-b7da-52f70904d246Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324Ù$736920df-e4bb-4535-b982-e397aa0a782dÙ$c3a9a0ce-3450-4b17-8696-2ab8534b29f2Ù$579a203b-e6f7-4190-b874-18b00a5c3f77Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fcÙ$93200f46-7c8f-4362-a445-43c57b50a2d2Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10Ù$771d39a5-74dc-494e-929e-1164bb08b983Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3®process_status¥readyªcell_orderÜ “Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418Ù$95ec4140-9147-11ef-2af4-5528bad0e6f5Ù$f4366cf6-2be0-42b8-96c4-120be3f5c25eÙ$61dc1905-338f-4bfd-a158-2f6bacff769eÙ$c09ec483-9fcf-48e7-b3c0-2508289e3cf3Ù$beccf4e8-1b01-4cb2-b23c-bc5db604f21cÙ$ccf2dc71-b883-497a-bc58-29ffaf9ea4adÙ$1bbf2152-4fdf-4ed2-9bdf-95d699824d11Ù$57c2c944-0d91-489d-8ad7-f5520e71ef3eÙ$0583ee0c-3802-4e81-b179-a80a82493b43Ù$728c16b7-50cd-43fe-a0d7-61d37952a6b3Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324Ù$736920df-e4bb-4535-b982-e397aa0a782dÙ$c3db7eb2-356a-428f-9777-6369662d8b06Ù$c3a9a0ce-3450-4b17-8696-2ab8534b29f2Ù$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03Ù$2e8b1a77-1f04-4035-8d82-4061d81ecb7aÙ$ed5b5702-4cca-4116-a70f-4a562178f490Ù$771d39a5-74dc-494e-929e-1164bb08b983Ù$97463c54-7cc7-4497-a8a6-6422f5f582bdÙ$e2eca085-9f99-4e3a-9db4-e7f692aedd34Ù$9e898325-e9e2-45bd-af74-3dd86f00f7b5Ù$bcf7667f-f99b-4d10-af84-5d3879f1db5dÙ$2a7e5096-1e8d-4506-96d2-86de0a7d39aaÙ$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0Ù$eb18303f-3dfb-4b87-90f2-f6dc542d7221Ù$6622f9f0-cecc-476e-9d49-7d651f433b9fÙ$6aa690e9-389f-4398-abae-b95060db4d90Ù$9cb90e76-3bb5-41ff-bc79-c4949400d904Ù$6712c883-b407-47e1-a666-4de05f8f8d6eÙ$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1bÙ$76e2f97b-1c06-40cd-b134-d5155aa5587dÙ$f8330700-e964-4e19-9c55-2b11df45789eÙ$91abc03b-fef7-4f93-96fc-13f1cf654f0dÙ$75ca478c-916f-464a-9435-8208ee726d50Ù$4e10271c-49f8-4f1d-869c-5fa11275d7f6Ù$55a09acc-84da-491c-86ba-9a66f4ea52feÙ$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2aÙ$d54b5390-0ec0-4ff8-ab18-51726482ca46Ù$225e58ba-b78d-4a0a-be4f-ad642c879b93Ù$a21fbc70-9137-4d0e-8c8c-cbdc5269778fÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012Ù$6800afbf-8ac6-4308-b4cc-b37da57e42c1Ù$55435b26-7fc3-4c8b-8013-6fd4fb65a08eÙ$bcbb3db2-85b3-4cb0-9309-f5c032d14da5Ù$d558636d-c714-4033-ae73-5b92c3cdedf3Ù$70f395b2-f8c2-44d5-b0af-702659dd7feeÙ$b1a924f4-e2f0-445c-830f-94287a0e52f7Ù$95504a74-d5ef-4fb7-83a0-88914c7cbc59Ù$8d231f2c-4b0c-4c37-a746-16e98d4cafc8Ù$86101f07-67c5-4df2-911c-4013c44d6c5bÙ$570fa160-3adb-463e-99b8-b7dd05076908Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9Ù$1faa4ab2-6c93-47dc-b631-8be52780fe7dÙ$e41d13ca-1dc1-45ae-9fa6-a83c4101120dÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5Ù$9ff95a9a-192b-4a12-8e2e-7acd6659c066Ù$5150d8f3-6e85-43f2-801a-eae5cc3e3095Ù$c032b3ff-c539-4e38-81d0-39b28b3a8076Ù$76ba4e9b-8bb0-47c4-b607-2ca711f035e6Ù$8c27b182-0c3c-4c19-9619-df62b7dd6bf0Ù$0c0c1163-0aec-4089-9acc-539b3a86d0b3Ù$b7583418-f4fb-4c63-b421-b5b9af269768Ù$d014e6aa-92f6-4ca1-be47-516565d1bb20Ù$6fc13413-53de-4c75-9b9e-620e0b7f8a1fÙ$006f44b0-c10d-44c3-8300-183e706b605dÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4aÙ$a3efd921-eb14-4901-9d6c-800cc812fe02Ù$b9caae1a-38aa-4d01-9cda-3d6782fb0e03Ù$4b61363d-87c9-4755-8286-44df34e9dd6aÙ$453544fc-0e3e-4e04-8c0c-192f3a038884Ù$c5be3956-5102-4d88-bfdb-9813c0555fe1Ù$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bbÙ$18c26901-85eb-45ac-89bf-b03bd255007aÙ$b56e9e56-e74a-401b-b4b5-f36bb33341d5Ù$f2cba2aa-c541-4692-a441-e65741750a15Ù$5f05e717-a51a-4a99-bb11-cc493217f93fÙ$2a8433e3-9a3b-487b-abf3-09278ea42389Ù$e383bb72-49a1-4df1-84c3-b95a2ffe00f5Ù$f95a6de6-5e02-4237-88ba-ec44ef3d38c3Ù$af8194a1-a358-4cf7-b446-6b377cb76687Ù$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5ddÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfcÙ$a5b20939-9afa-48c0-aa67-cbca6bc99804Ù$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ecÙ$25b79953-fd7c-46c1-b760-d57c09910981Ù$45efc71d-d5f8-474e-9b89-e72fac7110fdÙ$b2982908-a2ec-436a-abae-4dcdf896a3afÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5eÙ$083fcca6-10e0-4483-a754-19d1f2720ec7Ù$9057ce05-ba9b-46c9-847b-514e219f93a1Ù$a14e505e-2e4a-4c73-8133-7560ba58916bÙ$9a8eef1b-27c0-4d57-a389-53708ade9058Ù$52c97de1-3d75-4b51-8397-86ea67af6c77Ù$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2Ù$b9a962a6-4f4f-4676-95b6-0c2cfd536031Ù$a0d935f9-185a-4608-ab9f-954fa0e66f44Ù$941ad4d9-3406-463e-9af9-96687cd1eceaÙ$c8923675-e73e-4621-82b9-966d8b003b97Ù$04e9b912-6712-4290-acc4-f24bb27a1469Ù$c1437dcc-22cb-424f-9b8e-326172f82d86Ù$8b78360a-21cb-4574-a84d-46ea4d0cedb1Ù$f7ca738d-5215-4e91-a2f3-a5ff10911313Ù$6bff7bca-ea1d-44c6-b8c3-040250f90654Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3Ù$321b2919-62f7-42fd-9ae3-8ab9bc309525Ù$7be26ee1-de61-4b66-b221-5c5c728c2901Ù$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbeÙ$3176c675-ce53-4471-b13b-1d694f5efdd8Ù$ba00b790-67ac-49b8-86ee-6df42107f245Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060Ù$a3380780-ec5e-46c1-a0e0-af64b3e1da1cÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fbÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962Ù$be4a698a-848d-47bf-b825-24bc94621767Ù$d605d0a5-7511-4a32-9e1b-1c2befa43796Ù$cb6b040d-c213-4ecb-a047-fb66407752a3Ù$1d02cfbb-9d6b-41fd-9621-0103a9110813Ù$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89Ù$aa10118d-0991-4284-97f8-3a0728636a66Ù$b245ada9-1677-45bc-a883-bc213a2105b7Ù$a6a6a352-7479-4f3f-9abb-900eb901466cÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057aeÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30Ù$f572e113-b36b-4a6b-96c7-c26f100e1ad4Ù$32621224-a782-4bf6-9570-562cf2bb7360Ù$f6f7376e-9984-4289-b8ff-9d47e5358791Ù$6f72e8a5-819d-474c-a725-7f7318d964d7Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8eÙ$85a10748-8d19-44a8-a1c5-0d13b093f1bfÙ$0322b582-219b-4fbb-92e5-c82ff635615cÙ$a873f760-bfc1-489f-a58e-75e12afa54f2Ù$a57f2149-c876-4421-a958-e9aed1f892a6Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85Ù$d050a7ee-3aa7-4539-a236-5b6446599dedÙ$29474a70-32eb-4281-8626-87819afa7267Ù$89305cae-098f-4644-9109-d00f1e3bc04cÙ$c7f318b9-30e6-4b79-b7da-52f70904d246Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993fÙ$01372b00-ecb2-42bd-b408-13234717d969Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fcÙ$579a203b-e6f7-4190-b874-18b00a5c3f77Ù$93200f46-7c8f-4362-a445-43c57b50a2d2Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10«cell_inputsÞ “Ù$3d8add97-59e1-444a-838b-85c2a2ac60b3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$3d8add97-59e1-444a-838b-85c2a2ac60b3«code_foldedäcodeÚ8HAlign(
md"""
*Cross-Attention* between
* values and keys ``E(CX + P)`` where ``E`` is the encoder, and ``X`` is the matrix of input tokens
* query ``Q`` depending on past output ``Y`` and number of layers already applied
```math
\text{MultiHead}(E(CX + P), E(CX + P), Q)
```
The embedding vectors ``CX`` take then different projections
for value, key, query and also for different heads!
```math
\begin{multline}
\text{head}_j = \text{Attention}(W_j^VV, W_j^KK, W_j^QQ)\\
\text{where } V = K = E(CX + P)
\end{multline}
```
""",
HTML(html(@draw begin
draw_transformer(false)
highlight(31, -97, 205, 5)
#sethue("red")
#setopacity(1)
fontsize(32)
text("CX + P", Point(-60, 252), halign = :center)
text("CY + P", Point(65, 252), halign = :center)
text(L"E(CX + P)", Point(-70, -160), halign = :center)
end 300 400)),
)Ù$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bb„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$92e01e21-ca77-43fc-9bf8-0c5a7aaed1bb«code_foldedäcodeºmd"## Residual connection"Ù$6622f9f0-cecc-476e-9d49-7d651f433b9f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6622f9f0-cecc-476e-9d49-7d651f433b9f«code_foldedäcodeÙ!md"# Pre-transformers approaches"Ù$941ad4d9-3406-463e-9af9-96687cd1ecea„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$941ad4d9-3406-463e-9af9-96687cd1ecea«code_foldedäcodeÚ|qa(md"What is the analogy with reverse-mode AD ? What is the cost ?",
md"""
The forward pass of reverse-mode AD stores the intermediate values so that the backward pass does not recompute them.
Similarly, the KV cache stores the keys and values computed by the forward pass of the previous token so that the forward pass of the next token does not recompute them.
We trade memory for time:
* **Memory** : ``O(Nhn_\text{ctx}(d_k + d_v))`` for the cached keys and values.
* **Time** : each new token costs ``O(Nn_\text{ctx}d_\text{emb} + Nd_\text{emb}^2)`` instead of ``O(Nn_\text{ctx}^2d_\text{emb} + Nn_\text{ctx}d_\text{emb}^2)``.
""")Ù$70f395b2-f8c2-44d5-b0af-702659dd7fee„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$70f395b2-f8c2-44d5-b0af-702659dd7fee«code_folded¤code¬dict[[1, 0]]Ù$771d39a5-74dc-494e-929e-1164bb08b983„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$771d39a5-74dc-494e-929e-1164bb08b983«code_foldedäcodeÙ™table(llms, mandatory_columns = ["``n_\\text{ctx}``"], included_columns = [
"Name",
"Ref",
"``n_\\text{voc}``",
"``n_\\text{ctx}``",
"Tokenizer",
])Ù$85a10748-8d19-44a8-a1c5-0d13b093f1bf„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$85a10748-8d19-44a8-a1c5-0d13b093f1bf«code_folded¤codeÚ‰function draw_transformer(decoder_only = true)
scale(0.4, 0.4)
Luxor.placeimage(readpng("images/transformer.png"), centered = true)
if decoder_only
sethue("red")
setopacity(0.4)
box(Point(-350, -160), Point(320, 20), :fill)
box(Point(-350, 20), Point(0, 460), :fill)
translate(Point(-170, -190))
setopacity(1)
fontsize(32)
text("Not used for now", halign = :center)
end
endÙ$9057ce05-ba9b-46c9-847b-514e219f93a1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9057ce05-ba9b-46c9-847b-514e219f93a1«code_foldedäcodeÚ`qa(md"What do we need to compute to predict ``x_0`` ?", md"""
Only the columns with a path to ``\hat{y}_{N,-1}`` in the computational graph.
These are exactly the nodes reached by the backward pass of reverse-mode AD from ``\hat{y}_{N,-1}``, the other ones do not contribute to the prediction of ``x_0``.
* In the last layer, computing ``\hat{y}_{N,j}`` for ``j < -1`` is **useless** to predict ``x_0``!
* In the other layers, every position is needed but the positions ``k < -1`` are only needed through their keys and values.
Highlight what is needed to predict ``x_0``: $(@bind kv_needed CheckBox())
""")Ù$4f1d5112-dbac-4eb6-8518-0dc4193c3f8e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4f1d5112-dbac-4eb6-8518-0dc4193c3f8e«code_folded¤codeÙ'bib(args...) = bibrefs(biblio, args...)Ù$93200f46-7c8f-4362-a445-43c57b50a2d2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$93200f46-7c8f-4362-a445-43c57b50a2d2«code_folded¤code«names(llms)Ù$2a8433e3-9a3b-487b-abf3-09278ea42389„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2a8433e3-9a3b-487b-abf3-09278ea42389«code_foldedäcodeÙ?bib(["ioffe2015Batch", "ba2016Layer", "vaswani2017Attentiona"])Ù$61dc1905-338f-4bfd-a158-2f6bacff769e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$61dc1905-338f-4bfd-a158-2f6bacff769e«code_foldedäcodeÙ4bib(["goodfellow2016Deep", "vaswani2017Attentiona"])Ù$eb18303f-3dfb-4b87-90f2-f6dc542d7221„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$eb18303f-3dfb-4b87-90f2-f6dc542d7221«code_foldedäcodeÙ0bib(["press2017Using", "vaswani2017Attentiona"])Ù$76e2f97b-1c06-40cd-b134-d5155aa5587d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$76e2f97b-1c06-40cd-b134-d5155aa5587d«code_foldedäcode¹bib(["bengio2000Neural"])Ù$f6f7376e-9984-4289-b8ff-9d47e5358791„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f6f7376e-9984-4289-b8ff-9d47e5358791«code_folded¤codeÙ(import DocumenterCitations, CSV, LoggingÙ$b1a924f4-e2f0-445c-830f-94287a0e52f7„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b1a924f4-e2f0-445c-830f-94287a0e52f7«code_folded¤codeÙ„function numerical_lookup(dict, query)
_, i = findmax([dot(query, key) for key in keys(dict)])
return collect(values(dict))[i]
endÙ$f2cba2aa-c541-4692-a441-e65741750a15„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f2cba2aa-c541-4692-a441-e65741750a15«code_foldedäcodeºmd"## Layer normalization"Ù$6fc13413-53de-4c75-9b9e-620e0b7f8a1f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6fc13413-53de-4c75-9b9e-620e0b7f8a1f«code_foldedäcodeÙoqa(md"Is ``W^O`` needed if ``h = 1`` ?", md"No, if ``h = 1``, we can merge ``W^OW_1^V`` into a new ``W_1^V``.")Ù$ccf2dc71-b883-497a-bc58-29ffaf9ea4ad„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$ccf2dc71-b883-497a-bc58-29ffaf9ea4ad«code_foldedäcodeÙ0md"## Text to vectors : step 1 → tokenization"Ù$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5b4a67a9-e33e-4dc6-b9f0-fd9a2cca6f2a«code_foldedäcodeÙ3bib(["mikolov2010Recurrent", "goodfellow2016Deep"])Ù$86101f07-67c5-4df2-911c-4013c44d6c5b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$86101f07-67c5-4df2-911c-4013c44d6c5b«code_foldedäcodeÚ0md"""
Attention head provides a differentiable numerical dictionary $(cite("bahdanau2016Neural"))
```math
\begin{align}
\alpha
& =
\text{softmax}(\langle q, k_1 \rangle, \ldots, \langle q, k_{n_\text{ctx}}\rangle)
&
\text{Attention}(q, k, v)
& =
\sum_{i=1}^{n_\text{ctx}} \alpha_i v_i
\end{align}
```
"""Ù$a3efd921-eb14-4901-9d6c-800cc812fe02„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a3efd921-eb14-4901-9d6c-800cc812fe02«code_foldedäcodeµmd"## Self-Attention"Ù$321b2919-62f7-42fd-9ae3-8ab9bc309525„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$321b2919-62f7-42fd-9ae3-8ab9bc309525«code_foldedäcodeÙ md"# Energy consumption of LLMs"Ù$6800afbf-8ac6-4308-b4cc-b37da57e42c1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6800afbf-8ac6-4308-b4cc-b37da57e42c1«code_foldedäcode¿md"# Attention is all you need"Ù$18c26901-85eb-45ac-89bf-b03bd255007a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$18c26901-85eb-45ac-89bf-b03bd255007a«code_foldedäcodeÚHAlign(
md"""
Residual connection $(cite("he2015Deep"))
$(img("resnet"))
""",
HTML(html(@draw begin
draw_transformer()
highlight(210, -85, 290, -45)
highlight(360, -75, 440, 50)
highlight(210, 210, 290, 250)
highlight(360, 220, 440, 420)
end 300 400))
)Ù$bcbb3db2-85b3-4cb0-9309-f5c032d14da5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$bcbb3db2-85b3-4cb0-9309-f5c032d14da5«code_foldedäcodeÙ°md"
What would a numerical dictionary look like ? Consider keys ``k_i \in \mathbb{R}^{d_k}`` and values ``v_i \in \mathbb{R}^{d_v}``. Given a query ``q \in \mathbb{R}^{d_k}``,"Ù$4e10271c-49f8-4f1d-869c-5fa11275d7f6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4e10271c-49f8-4f1d-869c-5fa11275d7f6«code_foldedäcodeÙ&md"## Recurrent neural networks (RNN)"Ù$c1437dcc-22cb-424f-9b8e-326172f82d86„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c1437dcc-22cb-424f-9b8e-326172f82d86«code_foldedäcodeÙ÷md"""
* LSTM **encoder** → **context** → LSTM **decoder** $(cite("sutskever2014Sequence")). See $(cite("sutskever2014Sequence", "Figure 1")) below.
* Issue with *encoder bottleneck*. All information has to be summarized in the **context**.
"""Ù$c09ec483-9fcf-48e7-b3c0-2508289e3cf3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c09ec483-9fcf-48e7-b3c0-2508289e3cf3«code_foldedäcode¼md"## Autoregressive Models"Ù$37484e4a-87ad-4022-a8d9-e189fbbc8060„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$37484e4a-87ad-4022-a8d9-e189fbbc8060«code_foldedäcodeÚ8let
flop = 2 * energy_params * 1e9 * energy_tokens
joules = flop / (1.4e12 * energy_util) * energy_overhead
md"""
Back-of-the-envelope estimate : $(round(flop, sigdigits = 2)) FLOP → $(round(joules / 3600, sigdigits = 2)) Wh per query → $(round(joules / 10, sigdigits = 2)) seconds of a 10 W LED bulb.
This ignores the processing of the prompt and the attention cost which grows with the context length.
The overhead factor 1.7 is the ratio between full stack and active TPU energy reported in [Elsworth et al., 2025](https://arxiv.org/abs/2508.15734).
"""
endÙ$1bbf2152-4fdf-4ed2-9bdf-95d699824d11„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1bbf2152-4fdf-4ed2-9bdf-95d699824d11«code_foldedäcodeÚjmd"""
#### Why not encode each letter ?
* **Idea** : Turn each letter into its one-hot encoding in ``\mathbb{R}^{26}``.
* **Issue** : The "past text" only has ``n_\text{ctx}`` characters so ``n_\text{ctx}`` must be **large** but transformers have a complexity **quadratic** in ``n_\text{ctx}``!
* **Practical details** : Text is encoded with [UTF-8](https://en.wikipedia.org/wiki/UTF-8) so each character is encoded into 1 to 4 bytes. We encode each byte to a vector in ``\mathbb{R}^{256}`` but care must be taken not to generate invalid UTF-8.
#### Why not encode each word ?
* **Idea** : Turn each word into its one-hot encoding in ``\mathbb{R}^n``. The value of ``n`` is the number of words. Depending on the language ([source](https://en.wikipedia.org/wiki/List_of_dictionaries_by_number_of_words)):
| Language | French | English | Dutch | German |
|----------|---------|---------|---------|---------|
| ``n`` | 408,078 | 350,000 | 350,000 | 200,000 |
* **Issue** : The value of ``n`` is **too large**. We cannot trust the words of languages to be a tokenization that optimally compresses text for our dataset.
"""Ù$a0d935f9-185a-4608-ab9f-954fa0e66f44„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a0d935f9-185a-4608-ab9f-954fa0e66f44«code_foldedäcodeÚ»qa(md"When would the KV cache not make sense ?",
md"""
When the transformer learns a logic that is not auto-regressive.
For instance, if the tokens describe the board of a game and we only want the next move given the state of the board, the context is fixed.
There is then no reason for a column computed with an extra token in its context to be better.
But in that case, we would not generate tokens in a rolling horizon fashion anyway.
""")Ù$728c16b7-50cd-43fe-a0d7-61d37952a6b3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$728c16b7-50cd-43fe-a0d7-61d37952a6b3«code_folded¤code¹pair_stats("aaabdaaabac")Ù$1faa4ab2-6c93-47dc-b631-8be52780fe7d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1faa4ab2-6c93-47dc-b631-8be52780fe7d«code_folded¤codeÙ softmax_lookup(dict, [0.8, 0.2])Ù$c3a9a0ce-3450-4b17-8696-2ab8534b29f2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c3a9a0ce-3450-4b17-8696-2ab8534b29f2«code_folded¤codeºiter_3 = new_token(iter_2)Ù$e383bb72-49a1-4df1-84c3-b95a2ffe00f5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e383bb72-49a1-4df1-84c3-b95a2ffe00f5«code_foldedäcode»md"## Feed-Forward network"Ù$6aa690e9-389f-4398-abae-b95060db4d90„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6aa690e9-389f-4398-abae-b95060db4d90«code_foldedäcode·md"## Shared embedding"Ù$4b61363d-87c9-4755-8286-44df34e9dd6a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4b61363d-87c9-4755-8286-44df34e9dd6a«code_foldedäcodeÙíqa(
html"Is the order between the tokens taken into account by the model ?",
md"""
No. Since the same matrices ``W_j^V``, ``W_j^K`` and ``W_j^Q`` multiply the different position. The **position** information is completely **lost**!
"""
)Ù$94ae440d-0644-49db-9461-f1a1ff1d7f87„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$94ae440d-0644-49db-9461-f1a1ff1d7f87«code_folded¤codeÙ(cite(args...) = bibcite(biblio, args...)Ù$55435b26-7fc3-4c8b-8013-6fd4fb65a08e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$55435b26-7fc3-4c8b-8013-6fd4fb65a08e«code_foldedäcode»md"## Numerical dictionary"Ù$225e58ba-b78d-4a0a-be4f-ad642c879b93„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$225e58ba-b78d-4a0a-be4f-ad642c879b93«code_foldedäcodeÚšmd"""
It's difficult to model long-term dependencies as their gradient either vanish or explodes exponentially (think of the power method) $(cite("goodfellow2016Deep", "Section 10.7"))
*Gated* extensions attempting to solve this issue $(cite("goodfellow2016Deep", "Section 10.10")):
* Long short-term memory (LSTM) $(cite("graves2014Generating"))
* Gated recurrent unit (GRU) $(cite("cho2014Properties"))
"""Ù$32621224-a782-4bf6-9570-562cf2bb7360„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$32621224-a782-4bf6-9570-562cf2bb7360«code_folded¤codeÙ¥using PlutoUI, DataFrames, PrettyTables, LinearAlgebra, Luxor, LaTeXStrings, MathTeXEngine, PlutoUI, PlutoUI.ExperimentalLayout, HypertextLiteral, PlutoTeachingToolsÙ$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e86a57ae-3945-4cbf-b2e4-a96f4b5295e0«code_foldedäcodeÚ?qa(md"Why don't we use ``D = C^{-1}`` ?",
md"""
The ``C`` matrix is not invertible as it is not square. Given a vector ``u``, the vector ``v = Du`` could be computed to be the mininum norm solution of ``u = Cv``, this is what is done by `v = D \ u`. Computing `D \ u` would be more computational work than `D' \ u` but that's not the only reason `D'` is used.
If ``v`` is chosen so that ``u = Cv``, it means that ``v`` is a linear combination allowing to reconstruct ``u`` using the columns of ``C``.
From that perspective, ``v_i`` could be nonzero even though the direction of the vector ``u`` us far from the direction of the column ``C_{:i}``.
That does not correspond to what we want to do here.
Here, we want a probability vector ``p`` with high probability ``p_i`` when the direction of the column ``C_{:i}`` is close to the direction of ``u``.
In the vector ``v = C^\top u``, ``v_i = \langle C_{:i}, u \rangle`` is the scalar product between the ``i``th column of ``C`` and ``u`` hence it is a good measure of how close the direction are to each other.
Of course this assumes that the column of ``C`` have the same norm, but the hope is for the model to figure in training that the columns of ``C`` should have unit norm.
We can then simply apply softmax to turn these scalar products into probabilities ``p = \text{softmax}(v)``.
""")Ù$0c0c1163-0aec-4089-9acc-539b3a86d0b3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0c0c1163-0aec-4089-9acc-539b3a86d0b3«code_foldedäcodeÙbmd"""
```math
\text{Masked-Attention}(V, K, Q)\
=
V\text{softmax}(M + K^\top Q/\sqrt{d_k})
```
"""Ù$293266c6-01e7-4da0-8caf-56b1ce0aae85„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$293266c6-01e7-4da0-8caf-56b1ce0aae85«code_folded¤codeÚN"""
transformer_graph(; kv_cache = false, needed = false, n_ctx = 4, N = 3)
Draw the dependencies between the columns of ``\\hat{Y}_0, \\ldots, \\hat{Y}_N``.
If `needed` is `true`, the columns that are not needed to predict ``x_0`` are greyed out.
If `kv_cache` is `true`, show the computation of the next token with the KV cache.
"""
function transformer_graph(; kv_cache = false, needed = false, n_ctx = 4, N = 3, width = 640, height = 420)
mathjax_figure(; width, height) do label
colors = (computed = "#546b8c", kv = "#9673a6", cached = "#ffb570", new = "#67ab9f", unneeded = "#d0d0d0", extra = "#e06666")
js = kv_cache ? (-n_ctx:0) : (-n_ctx:-1)
Δx = 90
Δy = 80
X(j) = (j - (first(js) + last(js)) / 2) * Δx + 50
Y(i) = (N / 2 - i) * Δy
radius = 9
# Blocks of a layer from bottom to top with the colors of the figure
# of the paper: attention, add & norm, feed-forward, add & norm
blocks = [(14, "#FFE2BB"), (6, "#F2F4C1"), (14, "#C2E8F7"), (6, "#F2F4C1")]
block_width = 24
gap = 3
stack_height = sum(first, blocks) + gap * (length(blocks) - 1)
stack_bottom(i) = (Y(i - 1) + Y(i) + stack_height) / 2
attention(i, j) = Luxor.Point(X(j) - block_width / 2, stack_bottom(i) - first(blocks[1]) / 2)
in_window(j) = !kv_cache || j > -n_ctx
function status(i, j)
if kv_cache
j == 0 && return :new
(i == N || !in_window(j)) && return :unneeded
return :cached
end
needed && i == N && j != -1 && return :unneeded
return :computed
end
# Edges from position `k` in layer `i - 1` to position `j` in layer `i`
for i in 1:N, j in js, k in js
k <= j || continue
s = status(i, j)
if s == :new
in_window(k) || continue
color = k == j ? colors.computed : colors.kv
elseif kv_cache && s == :cached && !in_window(k)
# The cached values still have `x_{-n_ctx}` in their context
color = colors.extra
elseif s == :unneeded || (kv_cache && s == :cached)
color = colors.unneeded
else
color = k == j ? colors.computed : colors.kv
end
p = Luxor.Point(X(k), Y(i - 1))
# The past positions only enter through the attention
q = k == j ? Luxor.Point(X(j), Y(i) + radius) : attention(i, j)
d = (q - p) / Luxor.distance(p, q)
Luxor.sethue(color)
Luxor.setdash(color == colors.extra ? "dash" : "solid")
Luxor.arrow(p + radius * d, q; linewidth = k == j ? 1.5 : 1, arrowheadlength = 6)
end
Luxor.setdash("solid")
for i in 1:N, j in js
s = status(i, j)
opacity = s in (:computed, :new) ? 1 : s == :cached ? 0.5 : 0.25
y = stack_bottom(i)
for (h, color) in blocks
center = Luxor.Point(X(j), y - h / 2)
# Opaque white background so that the arrow behind is hidden
# when the block is drawn with some transparency
Luxor.setopacity(1)
Luxor.sethue("white")
Luxor.box(center, block_width, h, 2, :fill)
Luxor.setopacity(opacity)
Luxor.sethue(color)
Luxor.box(center, block_width, h, 2, :fill)
Luxor.sethue("black")
Luxor.setline(0.8)
Luxor.box(center, block_width, h, 2, :stroke)
y -= h + gap
end
end
Luxor.setopacity(1)
Luxor.setline(1.5)
for i in 0:N, j in js
s = status(i, j)
Luxor.sethue(colors[s])
Luxor.circle(Luxor.Point(X(j), Y(i)), radius, :fill)
end
for i in 0:N
label(i == N ? "\\hat{Y}_N" : "\\hat{Y}_$i", X(first(js)) - 60, Y(i))
end
for j in js
label("x_{$j}", X(j), Y(0) + 25; color = in_window(j) ? "black" : colors.unneeded)
s = status(N, j)
if s in (:computed, :new)
label("x_{$(j + 1)}", X(j), Y(N) - 25; color = j == last(js) ? "black" : "gray")
end
end
if kv_cache
y = Y(0) + 45
Luxor.sethue("black")
Luxor.setline(1)
Luxor.line(Luxor.Point(X(-n_ctx + 1) - 15, y), Luxor.Point(X(0) + 15, y), :stroke)
label("\\text{context window}", (X(-n_ctx + 1) + X(0)) / 2, y + 12; fontsize = 12)
y = -height / 2 + 15
for (x, s, text) in [(-280, :cached, "\\text{cached}"), (-170, :new, "\\text{computed for } x_1"), (-20, :unneeded, "\\text{not needed}")]
Luxor.sethue(colors[s])
Luxor.circle(Luxor.Point(x, y), 6, :fill)
label(text, x + 12, y; fontsize = 12, halign = :left)
end
Luxor.sethue(colors.extra)
Luxor.setdash("dash")
Luxor.line(Luxor.Point(110, y), Luxor.Point(135, y), :stroke)
Luxor.setdash("solid")
label("x_{$(-n_ctx)} \\text{ in the context}", 143, y; fontsize = 12, halign = :left)
else
y = -height / 2 + 15
for (x, color, text) in [(-200, colors.computed, "same position"), (-40, colors.kv, "past positions through keys and values")]
Luxor.sethue(color)
Luxor.line(Luxor.Point(x, y), Luxor.Point(x + 25, y), :stroke)
label("\\text{$text}", x + 33, y; fontsize = 12, halign = :left)
end
end
end
endÙ$c5be3956-5102-4d88-bfdb-9813c0555fe1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c5be3956-5102-4d88-bfdb-9813c0555fe1«code_foldedäcodeÚ�HAlign(
md"""
Cannot sum ``Cx_i + e_i`` with one-hot encoding ``e_i \in \mathbb{R}^{n_\text{ctx}}`` as the dimension of ``Cx_i`` is ``\mathbb{R}^{d_\text{emb}}``.
So we also add a positional embedding ``P`` : ``Cx_i + Pe_i = Cx_i + p_i``.
With Self-Attention:
```math
\text{Self-MultiHead}(CX + P, CX + P, CX + P)
```
""",
HTML(html(@draw begin
draw_transformer()
highlight(310, 410, 530, 510)
end 300 400))
)Ù$7e27c349-ee76-46bd-b1c2-a9ce54974e10„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7e27c349-ee76-46bd-b1c2-a9ce54974e10«code_folded¤codeÚ«function table(df; mandatory_columns = String[], included_columns = nothing)
for col in mandatory_columns
df = df[(!ismissing).(df[!, col]), :]
end
if !isnothing(included_columns)
df = unique(df[!, included_columns])
end
Markdown.parse(pretty_table(
String,
sort(df),
backend = :markdown,
column_labels = names(df),
allow_markdown_in_cells = true,
formatters = [(v, _, _) -> ismissing(v) ? "" : v],
))
endÙ$2e8b1a77-1f04-4035-8d82-4061d81ecb7a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2e8b1a77-1f04-4035-8d82-4061d81ecb7a«code_foldedäcodeÙ)md"## Increasing length of \"past text\""Ù$29474a70-32eb-4281-8626-87819afa7267„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$29474a70-32eb-4281-8626-87819afa7267«code_folded¤codeÙ™function add_pair(bpe::BPE, subs)
pairs = copy(bpe.pairs)
push!(pairs, subs)
return BPE(replace(bpe.text, prod(subs.first) => subs.second), pairs)
endÙ$6bff7bca-ea1d-44c6-b8c3-040250f90654„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6bff7bca-ea1d-44c6-b8c3-040250f90654«code_foldedäcode¶md"## Cross-Attention"Ù$0322b582-219b-4fbb-92e5-c82ff635615c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0322b582-219b-4fbb-92e5-c82ff635615c«code_folded¤codeÚ
ºfunction draw_preactivation_decoder()
x = -20
function block(y, height, color, lines)
sethue(color)
box(Point(x, y), 164, height, 4, :fill)
sethue("black")
box(Point(x, y), 164, height, 4, :stroke)
for (i, line) in enumerate(lines)
text(
line,
Point(x, y + 15 * (i - (length(lines) + 1) / 2));
halign = :center,
valign = :middle,
)
end
end
function connection(y1, y2)
arrow(Point(x, y1), Point(x, y2); linewidth = 1.5, arrowheadlength = 6)
end
function addition(y)
sethue("#F2F4C1")
circle(Point(x, y), 9, :fill)
sethue("black")
circle(Point(x, y), 9, :stroke)
text("+", Point(x, y); halign = :center, valign = :middle)
end
function residual(y1, y2)
circle(Point(x, y1), 2.5, :fill)
poly([Point(x, y1), Point(88, y1), Point(88, y2)]; action = :stroke)
arrow(Point(88, y2), Point(x + 9, y2); linewidth = 1.5, arrowheadlength = 6)
end
fontface("sans")
fontsize(13)
setline(1.5)
sethue("#F4F4F4")
box(Point(-5, 30), 224, 296, 12, :fill)
sethue("black")
box(Point(-5, 30), 224, 296, 12, :stroke)
text("N×", Point(128, 30); halign = :center, valign = :middle)
# Each residual path bypasses both the normalization and its sub-block.
residual(163, 49)
residual(29, -86)
connection(189, 151)
block(139, 24, "#F2F4C1", ["Layer Norm"])
connection(127, 115)
block(94, 42, "#FFE2BB", ["Masked Multi-Head", "Attention"])
connection(73, 58)
addition(49)
connection(40, 16)
block(4, 24, "#F2F4C1", ["Layer Norm"])
connection(-8, -20)
block(-40, 40, "#C2E8F7", ["Feed-Forward"])
connection(-60, -77)
addition(-86)
# The final normalization is outside the repeated decoder block.
connection(-95, -132)
block(-144, 24, "#F2F4C1", ["Layer Norm"])
connection(-156, -171)
block(-183, 24, "#DADDEF", ["Linear"])
connection(-195, -211)
block(-223, 24, "#CEE8D1", ["Softmax"])
connection(-235, -246)
text("Output probabilities", Point(x, -260); halign = :center, valign = :middle)
addition(198)
connection(217, 207)
block(232, 30, "#F9DEDE", ["Input Embedding"])
connection(255, 247)
text("Inputs (shifted right)", Point(x, 267); halign = :center, valign = :middle)
line(Point(100, 211), Point(100, 198), :stroke)
arrow(Point(100, 198), Point(x + 9, 198); linewidth = 1.5, arrowheadlength = 6)
text("Positional", Point(100, 220); halign = :center, valign = :middle)
text("Encoding", Point(100, 235); halign = :center, valign = :middle)
return nothing
endÙ$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c4bebd0d-eacf-4db4-b5b3-4dca50ab9e1b«code_foldedäcodeÚ£qa(md"What are the number of columns of ``W_1`` and number of rows of ``W_2`` now ?",
md"""
The matrix ``W_1`` has ``n_\text{ctx}d_\text{emb}`` columns. Assuming ``d_\text{emb} \ll n_\text{voc}`` and ``n_\text{ctx} \gg 1``, this is much smaller than the number ``n_\text{ctx}n_\text{voc}`` that we would have without the embedding. The number of rows of ``W_2`` is ``n_\text{voc}``, unaffected by the embedding.
""")Ù$6f72e8a5-819d-474c-a725-7f7318d964d7„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6f72e8a5-819d-474c-a725-7f7318d964d7«code_folded¤code³include("utils.jl")Ù$b7583418-f4fb-4c63-b421-b5b9af269768„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b7583418-f4fb-4c63-b421-b5b9af269768«code_foldedäcode»md"## Multi-Head Attention"Ù$a873f760-bfc1-489f-a58e-75e12afa54f2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a873f760-bfc1-489f-a58e-75e12afa54f2«code_folded¤codeÚfunction highlight(a, b, c, d)
sethue("green")
setopacity(0.4)
#box(Point(a, b), Point(c, d), :fill)
polysmooth(box(Point(a, b), Point(c, d), vertices=true), 10, action = :fill)
setopacity(1)
polysmooth(box(Point(a, b), Point(c, d), vertices=true), 10, action = :stroke)
endÙ$76ba4e9b-8bb0-47c4-b607-2ca711f035e6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$76ba4e9b-8bb0-47c4-b607-2ca711f035e6«code_foldedäcode·md"## Masked Attention"Ù$b2982908-a2ec-436a-abae-4dcdf896a3af„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b2982908-a2ec-436a-abae-4dcdf896a3af«code_foldedäcodeÙ%md"## Dependencies between positions"Ù$91abc03b-fef7-4f93-96fc-13f1cf654f0d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$91abc03b-fef7-4f93-96fc-13f1cf654f0d«code_foldedäcodeÚtHAlign(
md"""
See the table below for the size of embeddings of large language models:
$(table(llms, mandatory_columns = ["``d_\\text{emb}``"], included_columns = [
"Name",
"Num params",
"Ref",
"``n_\\text{voc}``",
"``d_\\text{emb}``",
]))
""",
HTML(html(@draw begin
draw_transformer()
highlight(200, -160, 375, -110)
highlight(200, 490, 375, 565)
end 300 400));
)Ù$a5b20939-9afa-48c0-aa67-cbca6bc99804„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a5b20939-9afa-48c0-aa67-cbca6bc99804«code_foldedäcode³md"## Cost of LLMs"Ù$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$79e6c4a8-cc1e-40cc-bb09-e9a7a9a8e475«code_foldedäcode½md"## Transformer variations"Ù$beccf4e8-1b01-4cb2-b23c-bc5db604f21c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$beccf4e8-1b01-4cb2-b23c-bc5db604f21c«code_foldedäcodeÚmd"""
Given a sequence of ``n_\text{ctx}`` past vectors ``x_{-1}, \ldots, x_{-n_\text{ctx}} \in \mathbb{R}^{n}``, "predict" the next ones. Key idea : *receding horizon*:
```math
\begin{align}
& p(x_0, x_1 | x_{-1}, \ldots, x_{-n_\text{ctx}})\\
& = p(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}})p(x_1 | x_0, x_{-1}, \ldots, x_{-n_\text{ctx}+1}, \textcolor{red}{x_{-n_\text{ctx}}})\\
& \approx p(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}}) p(x_{1} | x_0, x_{-1}, \ldots, x_{-n_\text{ctx}+1})
\end{align}
```
* **Model** : Probability of next vector ``\hat{p}(x_0 | X)`` where ``X`` concatenates ``x_{-1}, \ldots, x_{-n_\text{ctx}}``.
* **Loss** : Cross-entropy : ``\mathcal{L}_{\hat{p}}(X) \triangleq H(p, \hat{p}) = -\textbf{E}_p[\log(\hat{p})] = -\sum_{x_0} p(x_0 | X) \log(\hat{p}(x_0 | X))``
* Particular case for ``p(x_0 | X) = \delta_y`` : ``\mathcal{L}_{\hat{p}}(X) = -\log(\hat{p}(y | X))``
#### What about Language Models ?
Given "past text", predict the "following text". How to turn text into vectors of ``\mathbb{R}^n`` ?
"""Ù$e41d13ca-1dc1-45ae-9fa6-a83c4101120d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e41d13ca-1dc1-45ae-9fa6-a83c4101120d«code_foldedäcode»bib(["bahdanau2016Neural"])Ù$be4a698a-848d-47bf-b825-24bc94621767„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$be4a698a-848d-47bf-b825-24bc94621767«code_foldedäcode¿md"## Network and local device"Ù$5058e4eb-c53d-4468-b7ff-4f04ded96418„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5058e4eb-c53d-4468-b7ff-4f04ded96418«code_foldedäcodeÙÞ@htl("""
Transformers in Large Language Models (LLMs)
Benoît Legat
$(PlutoTeachingTools.ChooseDisplayMode())
$(PlutoUI.TableOfContents(depth=1))
""")Ù$5150d8f3-6e85-43f2-801a-eae5cc3e3095„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5150d8f3-6e85-43f2-801a-eae5cc3e3095«code_foldedäcodeÚ§HAlign(
md"""
`softmax` is then applied to each **column**:
```math
\text{softmax}(K^\top Q/\sqrt{d_k})
```
Division by ``\sqrt{d_k}`` scales the input of softmax to
preferable regions $(cite("vaswani2017Attentiona", "Secton 3.2.1")).
Illustrated on the right from $(cite("bahdanau2016Neural", "Figure 3(a)")).
```math
\text{Attention}(V, K, Q) = V\text{softmax}(K^\top Q/\sqrt{d_k})
```
""",
img("attention_matrix"),
)Ù$8c27b182-0c3c-4c19-9619-df62b7dd6bf0„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8c27b182-0c3c-4c19-9619-df62b7dd6bf0«code_foldedäcodeÚMHAlign(
md"""
💡 **Key idea** In the model for ``\hat{p}(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}})``, incorporate sub-models
```math
\begin{align}
\bar{p}(&x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}})\\
\bar{p}(&x_{-1} | x_{-2}, \ldots, x_{-n_\text{ctx}})\\
& \quad\qquad\vdots\\
\bar{p}(&x_{-n_\text{ctx}+1} | x_{-n_\text{ctx}}).
\end{align}
```
""",
md"""
Mask prevents ``\hat{p}`` to look at inputs in the future:
```math
M
=
\begin{bmatrix}
0 & 0 & \cdots & 0\\
-\infty & 0 & \ddots & \vdots\\
\vdots & \ddots & \ddots & 0\\
-\infty & \cdots & -\infty & 0
\end{bmatrix}
```
""",
)Ù$a3380780-ec5e-46c1-a0e0-af64b3e1da1c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a3380780-ec5e-46c1-a0e0-af64b3e1da1c«code_foldedäcodeÚèqa(md"Why is the GPU utilization so low when generating tokens ?",
md"""
Generating one token requires a matrix-**vector** product with each weight matrix so each weight read from the GPU memory is only used for 2 FLOPs.
Decoding is therefore limited by the **memory bandwidth**, not by the FLOPs.
Data centres **batch** the queries of many users together so that each weight read from memory is used for a matrix-**matrix** product.
A local GPU serving a single user cannot do that.
""")Ù$9a8eef1b-27c0-4d57-a389-53708ade9058„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9a8eef1b-27c0-4d57-a389-53708ade9058«code_foldedäcodeÚ�md"""
Once ``x_0`` is generated, the context window moves to ``x_{-n_\text{ctx}+1}, \ldots, x_0``.
💡 For each layer ``i < N``, the keys and values of ``\hat{y}_{i,j}`` for ``j < 0`` were already computed for the previous token, let's **cache** them!
Since the positions are independent given these keys and values, we only need to compute the new column ``\hat{y}_{i,0}`` of each layer ``i``.
"""Ù$f39305ea-f7f5-440e-ac55-c83e27f6e7fc„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f39305ea-f7f5-440e-ac55-c83e27f6e7fc«code_folded¤code²llms = load_llms()Ù$f7ca738d-5215-4e91-a2f3-a5ff10911313„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f7ca738d-5215-4e91-a2f3-a5ff10911313«code_foldedäcode¼bib("sutskever2014Sequence")Ù$c7f318b9-30e6-4b79-b7da-52f70904d246„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c7f318b9-30e6-4b79-b7da-52f70904d246«code_folded¤codeÙ function substitute(text::String, pair::Tuple{Char,Char})
new_char = min('Z' + 1, minimum(text)) - 1
return replace(text, prod(pair.first) => pair.second)
endÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfc„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4dd7083a-e730-4f4b-bde8-fc1a5b08ebfc«code_foldedäcodeÙ@bib(["he2016Identity", "radford2019Language", "su2023RoFormer"])Ù$8b78360a-21cb-4574-a84d-46ea4d0cedb1„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8b78360a-21cb-4574-a84d-46ea4d0cedb1«code_foldedäcode¼img("sutskever2014Sequence")Ù$95ec4140-9147-11ef-2af4-5528bad0e6f5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$95ec4140-9147-11ef-2af4-5528bad0e6f5«code_foldedäcodeÙ"md"# Large Language Models (LLMs)"Ù$77f446ac-6030-48f2-9bea-93c427f9fcb9„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$77f446ac-6030-48f2-9bea-93c427f9fcb9«code_folded¤codeÙ»function softmax_lookup(dict, query)
ks = keys(dict)
α = softmax([dot(query, key) for key in keys(dict)])
@show α
return sum(α * value for (α, value) in zip(α, values(dict)))
endÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4a„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d05e6f0f-0081-4fb6-91e9-ac2f58beda4a«code_foldedäcode¾md"# Decoder-only transformer"Ù$d54b5390-0ec0-4ff8-ab18-51726482ca46„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d54b5390-0ec0-4ff8-ab18-51726482ca46«code_foldedäcode¹md"## Extensions of RNNs"Ù$f572e113-b36b-4a6b-96c7-c26f100e1ad4„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f572e113-b36b-4a6b-96c7-c26f100e1ad4«code_foldedäcode¬md"## Utils"Ù$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f544e9fc-fb9a-4c32-b215-fa3a0eb37b89«code_foldedäcodeÚ%md"""
| Model | Energy | Scope | Source |
|---|---|---|---|
| GPT-3 (175B) | 1.3 GWh | Estimate, V100, PUE 1.1 | [Patterson et al., 2021](https://arxiv.org/abs/2104.10350) |
| BLOOM (176B) | 0.43 GWh | Measured, A100, GPUs only | [Luccioni et al., 2022](https://arxiv.org/abs/2211.02001) |
| OLMo 2 (7B / 13B) | 0.16 / 0.23 GWh | Measured, GPUs only | [Morrison et al., 2025](https://arxiv.org/abs/2503.05804) |
| Llama 3.1 (405B) | ≤ 21.6 GWh | 30.84M H100 hours × 700 W | [Meta model card](https://huggingface.co/meta-llama/Llama-3.1-405B) |
| GPT-4 | ``\approx 2 \cdot 10^{25}`` FLOP | Estimate | [Epoch AI](https://epoch.ai/data-insights/models-over-1e25-flop) |
* 21.6 GWh ≈ 6,000 Belgian households for a year.
* Experiments and development add a significant amount on top of the final training run : ≈ 50% for OLMo 2 [Morrison et al., 2025](https://arxiv.org/abs/2503.05804).
* The COâ‚‚ depends a lot on the electricity grid : BLOOM was trained on the French grid with 57 gCOâ‚‚/kWh [Luccioni et al., 2022](https://arxiv.org/abs/2211.02001).
"""Ù$a21fbc70-9137-4d0e-8c8c-cbdc5269778f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a21fbc70-9137-4d0e-8c8c-cbdc5269778f«code_foldedäcodeÙÙmd"""
Recently, Mamba suggests a solution to the complexity issue $(cite("gu2024Mamba")). As it scales better with ``n_\text{ctx}``, it is even suggested to get rid of the tokenizer : $(cite("wang2024MambaByte")).
"""Ù$ba00b790-67ac-49b8-86ee-6df42107f245„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$ba00b790-67ac-49b8-86ee-6df42107f245«code_foldedäcodeÚfmd"""
A dense transformer with ``N`` parameters needs ``\approx 2N`` FLOPs per generated token (one multiplication and one addition per weight), thanks to the KV cache.
For a mixture of experts, ``N`` is the number of **active** parameters.
An NVIDIA H100 performs ``\approx 989 \cdot 10^{12}`` FLOP/s (dense BF16) at 700 W so ``\approx 1.4 \cdot 10^{12}`` FLOP/J at **peak** utilization.
* Parameters ``N`` : $(@bind energy_params Select([1 => "1B", 8 => "8B", 70 => "70B", 405 => "405B"], default = 8))
* Generated tokens : $(@bind energy_tokens Slider([100, 300, 1000, 3000, 10000], default = 300, show_value = true))
* GPU utilization : $(@bind energy_util Slider([0.01, 0.03, 0.1, 0.3, 1.0], default = 0.1, show_value = true))
* Overhead factor (host CPU, idle machines, cooling) : $(@bind energy_overhead Slider(1.0:0.1:2.0, default = 1.7, show_value = true))
"""Ù$c3db7eb2-356a-428f-9777-6369662d8b06„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c3db7eb2-356a-428f-9777-6369662d8b06«code_foldedäcodeÙOmd"""
Note that the new tokens can also be part of the most frequence pair!
"""Ù$9e898325-e9e2-45bd-af74-3dd86f00f7b5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9e898325-e9e2-45bd-af74-3dd86f00f7b5«code_foldedäcodeÚ md"""
Consider one-hot encoding with vocabulary size ``n_\text{voc}`` and a *bigram model*
```math
\hat{p}(x_0 | x_{-1}) = \text{softmax}(W_d \tanh(\cdots\tanh(W_1 x_{-1})\cdots)
```
The matrix ``W_d`` has ``n_\text{voc}`` rows and ``W_1`` has ``n_\text{voc}`` columns → issue if ``n_\text{voc}`` is large
**Embedding** : Use vectors ``c_1, \ldots, c_{n_\text{voc}} \in \mathbb{R}^{d_\text{emb}}`` with *embedding size* (aka *hidden size*) ``d_\text{emb} \ll n_\text{voc}``.
Equivalently, we still use one-hot encoding but we add an encoder
``C \in \mathbb{R}^{d_\text{emb} \times n_\text{voc}}`` and decoder ``D \in \mathbb{R}^{n_\text{voc} \times d_\text{emb}}``
```math
\hat{p}(x_0 | x_{-1}) = \text{softmax}(D W_d \tanh(\cdots\tanh(W_1 C x_{-1})\cdots)
```
"""Ù$97463c54-7cc7-4497-a8a6-6422f5f582bd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$97463c54-7cc7-4497-a8a6-6422f5f582bd«code_foldedäcodeÙµbib(["team2024Gemini", "team2024Geminia", "team2024Gemma", "team2024Gemmaa", "sennrich2016Neural", "radford2019Language", "brown2020Language", "touvron2023Llama", "yu2023MEGABYTE"])Ù$8eafcfed-9771-4d99-b0c5-bd75a6dab012„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8eafcfed-9771-4d99-b0c5-bd75a6dab012«code_foldedäcodeÙlbib(["cho2014Properties", "graves2014Generating", "goodfellow2016Deep", "gu2024Mamba", "wang2024MambaByte"])Ù$89305cae-098f-4644-9109-d00f1e3bc04c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$89305cae-098f-4644-9109-d00f1e3bc04c«code_folded¤codeÙÿfunction pair_stats(text::String)
stats = Dict{Tuple{Char,Char},Int}()
for i in eachindex(text)
j = nextind(text, i)
if j > lastindex(text)
break
end
a = text[i]
b = text[j]
stats[(a, b)] = get(stats, (a, b), 0) + 1
end
return stats
endÙ$bcf7667f-f99b-4d10-af84-5d3879f1db5d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$bcf7667f-f99b-4d10-af84-5d3879f1db5d«code_foldedäcodeÚýqa(
html"What difference do you expect with respect to the previous model ?",
md"""
The products ``W_1C`` and ``DW_d`` have the same dimension as the matrices ``W_1`` and ``W_d`` of the previous model. So the expressive power of the model was not improved while we increased the number of parameters and we potentially made the loss function "even more nonconvex".
If the hidden dimension (i.e., the number of rows of ``C`` / columns of ``W_1`` or the number of rows of ``W_d`` / columns of ``D``) is much smaller than ``n_\text{voc}``, then it's faster to compute ``W_1(Cx)``. Moreover, we are forcing the matrix ``W_1C`` to have a low rank compared the model without ``C``. This means less expressivness but it might also prevent overfitting so the case isn't so clear.
The case become clearer when the input embedding ``C`` is shared between more than one character, i.e., ``n_\text{ctx} > 1``.
Same for the output embedding, ``D`` is useful when it is not preceded by a linear with which it can just be merged.
""")Ù$25b79953-fd7c-46c1-b760-d57c09910981„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$25b79953-fd7c-46c1-b760-d57c09910981«code_foldedäcodeÚ+qa(md"""
How does the number of parameters of transformers compare with $(cite("bengio2000Neural")) or RNNs for large ``n_\text{ctx}`` ?
""",
md"""
* The number of parameters of the transformer does **not** depend on ``n_\text{ctx}``.
* The number of parameters of $(cite("bengio2000Neural")) depends linearly with ``n_\text{ctx}``. Assuming that the number of hidden neurons scales proportionally with ``n_\text{ctx}``, the number of parameters even scales quadratically with ``n_\text{ctx}``!
* For RNNs, if the dimension of the internal state scales proportionally with ``n_\text{ctx}``, the number of parameters is also proportional with ``n_\text{ctx}``! If the dimension of the internal state is kept too small, increasing the context won't be so helpful, due to *encoder bottleneck*, see next slide.
""")Ù$c032b3ff-c539-4e38-81d0-39b28b3a8076„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c032b3ff-c539-4e38-81d0-39b28b3a8076«code_foldedäcodeÙ4bib(["bahdanau2016Neural", "vaswani2017Attentiona"])Ù$b9a962a6-4f4f-4676-95b6-0c2cfd536031„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b9a962a6-4f4f-4676-95b6-0c2cfd536031«code_foldedäcodeÚ\qa(md"Why does it still make sense ?",
md"""
Thanks to the mask, the transformer incorporates the sub-models (see Masked Attention)
```math
\bar{p}(x_{j+1} | x_j, \ldots, x_{-n_\text{ctx}})
```
so it is trained to predict the next token from contexts of every length from ``1`` to ``n_\text{ctx}``.
We can reasonably assume that the trained weights follow an auto-regressive logic in which one more token of context can only improve the prediction.
The same holds for the intermediate layers if we think of the columns of ``\text{softmax}(C^\top \hat{Y}_i)`` as intermediate probabilities ``\hat{p}_i``.
So the KV cache is **not equivalent** but, from an auto-regressive point of view, it should be **at least as good**.
Note that this is an assumption: the transformer is trained on each window from scratch, it never sees cached values during training!
""")Ù$cb6b040d-c213-4ecb-a047-fb66407752a3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$cb6b040d-c213-4ecb-a047-fb66407752a3«code_foldedäcodeÚ¢qa(md"Is a small model running on a local GPU cheaper than a web search ?",
md"""
A local GPU drawing 300 W and generating 100 tokens/s answers with 500 tokens in 5 s : ``300 \times 5 = 1500`` J ``\approx 0.4`` Wh.
A laptop drawing 40 W at 30 tokens/s : ``40 \times 17 \approx 700`` J ``\approx 0.2`` Wh.
This is the same order of magnitude as a web search (0.3 Wh in 2009) plus the device used to browse it.
There is no network and no data centre but a single user cannot batch the queries so the GPU is used less efficiently than in a data centre.
In a sense, the expensive training **compressed** the web into the weights so that the answer can be computed offline.
""")Ù$45efc71d-d5f8-474e-9b89-e72fac7110fd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$45efc71d-d5f8-474e-9b89-e72fac7110fd«code_foldedäcode·bib("bengio2000Neural")Ù$083fcca6-10e0-4483-a754-19d1f2720ec7„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$083fcca6-10e0-4483-a754-19d1f2720ec7«code_foldedäcodeÙ%transformer_graph(needed = kv_needed)Ù$d050a7ee-3aa7-4539-a236-5b6446599ded„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d050a7ee-3aa7-4539-a236-5b6446599ded«code_folded¤codeÙ@struct BPE
text::String
pairs::Dict{Tuple{Char,Char},Char}
endÙ$579a203b-e6f7-4190-b874-18b00a5c3f77„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$579a203b-e6f7-4190-b874-18b00a5c3f77«code_folded¤codeÚAfunction load_llms()
llms = DataFrame(CSV.File("llms.csv"))
rename!(llms, "Embedding dimension" => "``d_\\text{emb}``")
rename!(llms, "Vocabulary size" => "``n_\\text{voc}``")
rename!(llms, "Context window" => "``n_\\text{ctx}``")
rename!(llms, "Feed-Forward hidden dimension" => "``d_\\text{ff}``")
return llms
endÙ$570fa160-3adb-463e-99b8-b7dd05076908„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$570fa160-3adb-463e-99b8-b7dd05076908«code_folded¤codeÙ7function softmax(x)
y = exp.(x)
return y / sum(y)
endÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9cb90e76-3bb5-41ff-bc79-c4949400d904«code_foldedäcodeÙâmd"""
With ``n_\text{ctx} > 1``, the encoder ``C`` is shared by all tokens:
See for instance the network below taken from $(cite("bengio2000Neural", "Figure 1")), the first popular application of neural nets for languages:
"""Ù$75ca478c-916f-464a-9435-8208ee726d50„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$75ca478c-916f-464a-9435-8208ee726d50«code_foldedäcodeÙSbib(["team2024Gemma", "team2024Gemmaa", "radford2019Language", "touvron2023Llama"])Ù$b9caae1a-38aa-4d01-9cda-3d6782fb0e03„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b9caae1a-38aa-4d01-9cda-3d6782fb0e03«code_foldedäcodeÚ†HAlign(md"""
*Self-Attention* with embedding ``C`` is:
```math
\text{Masked-MultiHead}(CX, CX, CX)
```
The embedding vectors ``CX`` take then different projections
for value, key, query and also for different heads!
```math
\text{head}_j = \text{Masked-Attention}(W_j^VCX, W_j^KCX, W_j^QCX)
```
""",
HTML(html(@draw begin
draw_transformer()
highlight(200, 250, 375, 350)
end 300 400)),
)Ù$1d02cfbb-9d6b-41fd-9621-0103a9110813„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1d02cfbb-9d6b-41fd-9621-0103a9110813«code_foldedäcodeÙ%md"## Training : compressing the web"Ù$9ff95a9a-192b-4a12-8e2e-7acd6659c066„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$9ff95a9a-192b-4a12-8e2e-7acd6659c066«code_foldedäcodeÚ»md"""
```math
\begin{align}
Q & = \begin{bmatrix}
q_1 & \cdots & q_{n_\text{ctx}}
\end{bmatrix} &
K & = \begin{bmatrix}
k_1 & \cdots & k_{n_\text{ctx}}
\end{bmatrix} &
K^\top Q & =
\begin{bmatrix}
\langle k_1, q_1 \rangle & \cdots & \langle k_1, q_{n_\text{ctx}} \rangle\\
\vdots & \ddots & \vdots\\
\langle k_{n_\text{ctx}}, q_1 \rangle & \cdots & \langle k_{n_\text{ctx}}, q_{n_\text{ctx}} \rangle
\end{bmatrix}
\end{align}
```
"""Ù$f95a6de6-5e02-4237-88ba-ec44ef3d38c3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f95a6de6-5e02-4237-88ba-ec44ef3d38c3«code_foldedäcodeÚKHAlign(
md"""
Different weights
``W_1 \in \mathbb{R}^{d_\text{ff} \times d_\text{emb}}``, ``W_2 \in \mathbb{R}^{d_\text{emb} \times d_\text{ff}}`` for each layer:
```math
x \mapsto W_2\max(0, W_1x + b_1) + b_2
```
Expansion factor ``d_\text{ff} / d_\text{emb}`` is typically 4× like suggested in $(cite("vaswani2017Attentiona")) (but not for Gemma)
$(table(llms, mandatory_columns = ["``d_\\text{ff}``"], included_columns = [
"Name",
"Ref",
"``d_\\text{emb}``",
"``d_\\text{ff}``",
]))
""",
HTML(html(@draw begin
draw_transformer()
highlight(200, -45, 375, 25)
end 300 400)),
)Ù$ed5b5702-4cca-4116-a70f-4a562178f490„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$ed5b5702-4cca-4116-a70f-4a562178f490«code_foldedäcodeÙþmd"""
> **Challenging tradeoff**: Encode text to **increase** length of "past text" while keeping ``n_\text{ctx}`` and ``n`` **small** enough.
Length of "past text" increases with vocabulary size ``n_\text{voc}`` and context window ``n_\text{ctx}``.
"""Ù$736920df-e4bb-4535-b982-e397aa0a782d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$736920df-e4bb-4535-b982-e397aa0a782d«code_folded¤codeºiter_2 = new_token(iter_1)Ù$55a09acc-84da-491c-86ba-9a66f4ea52fe„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$55a09acc-84da-491c-86ba-9a66f4ea52fe«code_foldedäcodeÚÁHAlign(
md"""
```math
\begin{align}
h^{(t+1)} & = \tanh(Wh^{(t)} + Ux^{(t+1)} + b)\\
o^{(t)} &= Vh^{(t)} + c\\
\hat{y}^{(t)} &= \text{softmax}(o^{(t)})
\end{align}
```
Illustrated on the right $(cite("goodfellow2016Deep", "Figure 10.3")).
RNNs as language model showcased in $(cite("mikolov2010Recurrent")).
**Issue**: Training time and space complexity is proportional to ``n_\text{ctx}`` and **cannot parallelize** to speed up.
""",
img("RNN")
)Ù$52c97de1-3d75-4b51-8397-86ea67af6c77„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$52c97de1-3d75-4b51-8397-86ea67af6c77«code_foldedäcodeÙ"transformer_graph(kv_cache = true)Ù$a6a6a352-7479-4f3f-9abb-900eb901466c„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a6a6a352-7479-4f3f-9abb-900eb901466c«code_foldedäcodeÚžqa(md"After how many queries does the inference energy exceed the training energy ?",
md"""
With 21.6 GWh for training and 0.3 Wh per query : ``21.6 \cdot 10^9 / 0.3 \approx 7 \cdot 10^{10}`` queries.
With ``10^9`` queries per day, this takes about 72 days.
For a widely used model, the energy of the **inference** dominates over the lifetime of the model.
For a rarely used model, the **training** dominates.
""")Ù$d558636d-c714-4033-ae73-5b92c3cdedf3„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d558636d-c714-4033-ae73-5b92c3cdedf3«code_folded¤codeÙ0dict = Dict([1, 0] => [1, 1], [0, 1] => [-1, 1])Ù$8d231f2c-4b0c-4c37-a746-16e98d4cafc8„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8d231f2c-4b0c-4c37-a746-16e98d4cafc8«code_foldedäcodeµmd"## Attention head"Ù$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbe„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d069517e-c9c9-4b1e-b55f-1ff5d1ccabbe«code_foldedäcodeÚbmd"""
| | Energy |
|---|---|
| 10 W LED bulb for 1 hour | 10 Wh = 36 kJ |
| Heating 1 L of water from 20 °C to 100 °C | ≈ 93 Wh |
| Belgian reference household, per year | ≈ 3,500 kWh |
| Data centres worldwide in 2024 (≈ 1.5% of world electricity) | ≈ 415 TWh |
| Data centres worldwide in 2030 (IEA projection) | ≈ 945 TWh |
Source for data centres : [IEA, *Energy and AI*, 2025](https://www.iea.org/reports/energy-and-ai/executive-summary).
As we will see, a **single** query costs a fraction of a Wh. The question is rather the number of queries and the energy needed to train the models.
"""Ù$f1afaf8c-d9ad-446a-9826-9c4cda19993f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f1afaf8c-d9ad-446a-9826-9c4cda19993f«code_folded¤codeÙ6new_token(text::String) = new_token(BPE(text, Dict()))Ù$f7ca3ff7-b5cf-452b-b955-7219e7397324„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f7ca3ff7-b5cf-452b-b955-7219e7397324«code_folded¤codeÙ!iter_1 = new_token("aaabdaaabac")Ù$5f05e717-a51a-4a99-bb11-cc493217f93f„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$5f05e717-a51a-4a99-bb11-cc493217f93f«code_foldedäcodeÚrHAlign(
md"""
Norm of gradient increases exponentially with depth.
Issue for deep neural net.
Consider output
```math
\begin{bmatrix}
y_{1,1} & \ldots & y_{1,d_\text{emb}}\\
\vdots & \ddots & \vdots\\
y_{d_\text{batch},1} & \ldots & y_{d_\text{batch},d_\text{emb}}
\end{bmatrix}
```
Normalization : ``y_{i,j} \mapsto g(y_{i,j} - \mu_{i,j})/\sigma_{i,j}`` for gain ``g``, mean ``\mu`` and standard deviation ``\sigma``.
* Batch normalization : ``\sigma_{i,j} = \sigma_{j}`` $(cite("ioffe2015Batch"))
* Layer normalization : ``\sigma_{i,j} = \sigma_{i}`` $(cite("ba2016Layer"))
Batch norm depends on the batch hence [is tricky to implement](https://www.youtube.com/watch?v=P6sfmUTpUmc). Layer normalization is used in $(cite("vaswani2017Attentiona")).
""",
HTML(html(@draw begin
draw_transformer()
highlight(290, -85, 360, -45)
highlight(290, 210, 360, 250)
end 300 400))
)Ù$b56e9e56-e74a-401b-b4b5-f36bb33341d5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b56e9e56-e74a-401b-b4b5-f36bb33341d5«code_foldedäcode³bib(["he2015Deep"])Ù$57c2c944-0d91-489d-8ad7-f5520e71ef3e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$57c2c944-0d91-489d-8ad7-f5520e71ef3e«code_foldedäcode¹md"## Byte Pair Encoding"Ù$32fc8313-f3e8-489f-b0a1-ba740bc744fb„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$32fc8313-f3e8-489f-b0a1-ba740bc744fb«code_foldedäcode¾md"## Inference vs web search"Ù$bf563783-9784-4c74-a7b1-6d7a3ed618c5„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$bf563783-9784-4c74-a7b1-6d7a3ed618c5«code_foldedäcode¿md"## Matrix form of attention"Ù$f8330700-e964-4e19-9c55-2b11df45789e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f8330700-e964-4e19-9c55-2b11df45789e«code_foldedäcode¾md"## Embedding sizes in LLMs"Ù$b245ada9-1677-45bc-a883-bc213a2105b7„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b245ada9-1677-45bc-a883-bc213a2105b7«code_foldedäcodeÚœmd"""
Google search also needs an expensive precomputation : crawling the web, building the index and computing the PageRank.
* 1998 : "a PageRank for 26 million web pages can be computed in a few hours on a medium size workstation" [Brin and Page, 1998](http://infolab.stanford.edu/~backrub/google.html).
* Today : the index covers "hundreds of billions of webpages" and is "well over 100,000,000 gigabytes" ([Google](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/)).
Back-of-the-envelope : ``10^{11}`` pages × 50 links × 2 FLOPs × 50 power iterations ``\approx 5 \cdot 10^{14}`` FLOP for the PageRank, that is ``10^{10}`` times less than GPT-4.
However, PageRank is only a small part of the cost : crawling, parsing and storing the index dominate and there is **no public figure** for its energy.
Note that LLMs also need the web to be crawled to obtain their training data.
"""Ù$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ec„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$8d6ec2b3-997e-4df5-a3b2-c1dffa53d0ec«code_foldedäcodeÚ½qa(
md"What is the time complexity of inference with respect to ``d_\text{emb}``, ``n_\text{voc}``, ``n_\text{ctx}``, ``d_\text{ff}``, ``h`` and ``N`` ?",
HAlign(
md"""
| Input | Parameters | Time |
|-------|------------|------|
| ``CX + P \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}`` | ``W_j^V \in \mathbb{R}^{d_v \times d_\text{emb}}`` | ``O(d_v d_\text{emb} n_\text{ctx})`` |
| ``CX + P \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}`` | ``W_j^K, W_j^Q \in \mathbb{R}^{d_k \times d_\text{emb}}`` | ``O(d_k d_\text{emb} n_\text{ctx})`` |
| ``K, Q \in \mathbb{R}^{d_k \times n_\text{ctx}}`` | | ``O(d_k n_\text{ctx}^2)`` |
| ``V \in \mathbb{R}^{d_v \times n_\text{ctx}}, ... \in \mathbb{R}^{n_\text{ctx} \times n_\text{ctx}}`` | | ``O(d_v n_\text{ctx}^2)`` |
| ``... \in \mathbb{R}^{d_v \times n_\text{ctx}}`` | ``W^O \in \mathbb{R}^{d_\text{emb} \times d_v}`` | ``O(d_\text{emb} d_v n_\text{ctx})`` |
| ``... \in \mathbb{R}^{d_\text{emb} \times n_\text{ctx}}`` | ``W_1 \in \mathbb{R}^{d_\text{ff} \times d_\text{emb}}`` | ``O(d_\text{emb} d_\text{ff} n_\text{ctx})`` |
| ``... \in \mathbb{R}^{d_\text{ff} \times n_\text{ctx}}`` | ``W_2 \in \mathbb{R}^{d_\text{emb} \times d_\text{ff}}`` | ``O(d_\text{emb} d_\text{ff} n_\text{ctx})`` |
So for ``N`` layers (ignoring the complexity of the embedding):
```math
O(Nn_\text{ctx}(n_\text{ctx}(d_v + d_k) + d_\text{emb}(d_v+d_k+d_\text{ff})))
```
Assuming that ``d_v, d_k, d_\text{ff}`` has the same scale as ``d_\text{emb}``:
```math
O(Nn_\text{ctx}^2d_\text{emb} + Nn_\text{ctx}d_\text{emb}^2)
```
""",
HTML(html(@draw begin
draw_transformer()
translate(-10, 150)
scale(0.6)
Luxor.placeimage(readpng("images/multi-head.png"), centered = true)
end 300 400))
)
)Ù$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d1ba8da3-add8-4dbe-9ebf-9a32fa5cd5dd«code_foldedäcodeÚêHAlign(
md"""
*Pre-activation* for residual neural networks introduced in $(cite("he2016Identity")) and used in GPT-2 $(cite("radford2019Language")). See figure on the right.
Layer normalization is applied **before** attention and feed-forward, leaving an **identity** residual path. A final layer normalization follows the ``N`` blocks.
*Rotary Positional Encoding* $(cite("su2023RoFormer")) replaces
``W^K(Cx_i + p_i)`` and ``W^Q(Cx_i + p_i)``
by ``R^i W^KCx_i`` and ``R^i W^QCx_i`` where ``R`` is a rotation matrix.
Advantage : ``\langle k_i, q_j \rangle`` contains ``R^{i - j}`` → **relative** difference of position.
""",
HTML(html(@draw begin
draw_preactivation_decoder()
end 300 560)),
)Ù$1d5b1b7c-828c-4a16-b446-cff21b015d45„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$1d5b1b7c-828c-4a16-b446-cff21b015d45«code_folded¤code·biblio = load_biblio!()Ù$3176c675-ce53-4471-b13b-1d694f5efdd8„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$3176c675-ce53-4471-b13b-1d694f5efdd8«code_foldedäcodeÙ&md"## Estimating the inference energy"Ù$006f44b0-c10d-44c3-8300-183e706b605d„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$006f44b0-c10d-44c3-8300-183e706b605d«code_foldedäcodeÙìmd"""
#### Further readings
This dictionary viewpoint can also be view as a *memory store* that recalls by similarity, called *associate memory*. See [this blog post](https://alexwang.ai/posts/associative-memory/) for more details.
"""Ù$a14e505e-2e4a-4c73-8133-7560ba58916b„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a14e505e-2e4a-4c73-8133-7560ba58916b«code_foldedäcode»md"## Key-Value (KV) cache"Ù$2a7e5096-1e8d-4506-96d2-86de0a7d39aa„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2a7e5096-1e8d-4506-96d2-86de0a7d39aa«code_foldedäcodeÙ–md"""
Forcing ``D = C^\top`` appears to work well in practice $(cite("press2017Using")), this is what is used in $(cite("vaswani2017Attentiona")).
"""Ù$d605d0a5-7511-4a32-9e1b-1c2befa43796„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d605d0a5-7511-4a32-9e1b-1c2befa43796«code_foldedäcodeÚ€md"""
**Network** : ``\approx 0.06`` kWh/GB in 2015, halving every 2 years [Aslan et al., 2018](https://doi.org/10.1111/jiec.12630).
Extrapolating to ``\approx 0.002`` kWh/GB today, a 2.5 MB search result page ([HTTP Archive, 2024](https://almanac.httparchive.org/en/2024/page-weight)) costs ``\approx 0.005`` Wh and a text answer of a few kB much less.
Moreover, the power of the network is mostly **fixed** : it does not grow with the data volume so the **marginal** cost of a query is close to zero [Mytton et al., 2024](https://ideas.repec.org/a/bla/inecol/v28y2024i4p966-980.html).
**User device** : A 20 W laptop used for 30 seconds consumes ``\approx 0.17`` Wh, the **same order** as the server side of the query!
User devices account for more than half of the emissions of the ICT sector [Malmodin et al., 2024](https://ideas.repec.org/a/eee/telpol/v48y2024i3s0308596123002124.html).
"""Ù$6712c883-b407-47e1-a666-4de05f8f8d6e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$6712c883-b407-47e1-a666-4de05f8f8d6e«code_foldedäcodeÚHAlign(
md"""
```math
\begin{multline}
\hat{p}(x_0 | x_{-1}, \ldots, x_{-n_\text{ctx}}) = \\
\text{softmax}(C^\top W_2 \tanh(W_1
\begin{bmatrix}
C x_{-1}\\
\vdots\\
C x_{-n_\text{ctx}}
\end{bmatrix}
))
\end{multline}
```
""",
img("bengio2000Neural", :width => 250),
)Ù$7be26ee1-de61-4b66-b221-5c5c728c2901„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$7be26ee1-de61-4b66-b221-5c5c728c2901«code_foldedäcodeºmd"## Orders of magnitude"Ù$04e9b912-6712-4290-acc4-f24bb27a1469„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$04e9b912-6712-4290-acc4-f24bb27a1469«code_foldedäcodeºmd"## Machine translation"Ù$af8194a1-a358-4cf7-b446-6b377cb76687„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$af8194a1-a358-4cf7-b446-6b377cb76687«code_foldedäcodeÚ md"The feed-forward network is implemented **independently** for the output of each query so each query can be processed independently through each **layer**. The next layer allows each queries to then look at the results of the previous layer for **past** (because of the mask) queries."Ù$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$4df0a18d-cb14-41b1-ba40-fd6bfcbb0b03«code_foldedäcodeÙ4bib(["sennrich2016Neural", "kudo2018SentencePiece"])Ù$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$2b8fa47a-9761-4d16-9daa-bbb6a0c36f30«code_foldedäcodeÚ€md"""
* **Scope** : the energy of the GPUs only, of the full servers or of the whole data centre differ by ``\approx 2.4\times`` [Elsworth et al., 2025](https://arxiv.org/abs/2508.15734).
* **Average vs tail** : company figures are medians or averages. Long contexts and reasoning models cost 10 to 100× more.
* **Dates** : the 0.3 Wh of a web search dates from 2009. Both web search and LLMs became much more efficient since then.
* **Fixed vs marginal** : idle servers and networks consume energy even without queries. BLOOM consumed ``\approx 0.28`` kWh every 10 minutes while idle [Luccioni et al., 2022](https://arxiv.org/abs/2211.02001).
* **Rebound effect** : the energy per query decreases quickly but the number of queries grows even faster. The demand of data centres is expected to more than double by 2030 [IEA, 2025](https://www.iea.org/reports/energy-and-ai/executive-summary).
"""Ù$aa10118d-0991-4284-97f8-3a0728636a66„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$aa10118d-0991-4284-97f8-3a0728636a66«code_foldedäcode»md"## Training vs PageRank"Ù$e2eca085-9f99-4e3a-9db4-e7f692aedd34„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e2eca085-9f99-4e3a-9db4-e7f692aedd34«code_foldedäcodeÙ-md"## Text to vectors : step 2 → embedding"Ù$d014e6aa-92f6-4ca1-be47-516565d1bb20„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$d014e6aa-92f6-4ca1-be47-516565d1bb20«code_foldedäcodeÚžHAlign((
md"""
Heads focus on different aspects. Their outputs are **combined** with ``W^O \in \mathbb{R}^{d_\text{emb} \times hd_v}``:
```math
\begin{align}
\text{head}_j & = \text{Attention}(W_j^VV, W_j^KK, W_j^QQ)\\
\text{MultiHead}(V, K, Q)
& =
W^O\text{vcat}(\text{head}_1, \ldots, \text{head}_h)
\end{align}
```
See $(cite("vaswani2017Attentiona", "Figure 2")) on the right.
Similarly, in the masked case:
```math
\begin{align}
\text{head}_j & = \text{Masked-Attention}(W_j^VV, W_j^KK, W_j^QQ)\\
\text{Masked-MultiHead}&(V, K, Q)
=
W^O\text{vcat}(\text{head}_1, \ldots, \text{head}_h)
\end{align}
```
""",
img("multi-head", :width => 250)),
[70, 30],
)Ù$b4bfde70-510d-4ed5-acb2-7f0cdc3057ae„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$b4bfde70-510d-4ed5-acb2-7f0cdc3057ae«code_foldedäcode®md"## Caveats"Ù$e6420fcc-3b76-4dd5-b172-c7381afb5e5e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$e6420fcc-3b76-4dd5-b172-c7381afb5e5e«code_foldedäcodeÚ\md"""
Let ``\hat{y}_{i,j}`` be the column of ``\hat{Y}_i`` at the position of ``x_j``, with ``\hat{Y}_0 = CX + P``.
In layer ``i``, ``\hat{y}_{i,j}`` depends on
* ``\hat{y}_{i-1,j}`` for the query, the residual connection and the feed-forward network,
* ``\hat{y}_{i-1,k}`` for ``k < j`` **only** through their keys ``W_\ell^K\hat{y}_{i-1,k}`` and values ``W_\ell^V\hat{y}_{i-1,k}`` for each head ``\ell``,
* not on ``\hat{y}_{i-1,k}`` for ``k > j`` because of the mask.
Given the keys and values of the past positions, the computation of ``\hat{y}_{i,j}`` is **independent** of the other positions.
"""Ù$a57f2149-c876-4421-a958-e9aed1f892a6„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$a57f2149-c876-4421-a958-e9aed1f892a6«code_folded¤codeÚ$"""
mathjax_figure(draw; width, height)
Luxor only draws the shapes, the labels are overlaid in HTML so that
they are typeset by the MathJax of Pluto.
`draw(label)` draws the shapes with the origin at the center and adds labels
with `label(text, x, y; color, fontsize, halign)`.
"""
function mathjax_figure(draw; width, height)
labels = []
function label(text, x, y; color = "black", fontsize = 14, halign = :center)
dx = halign == :left ? "0" : "-50%"
push!(labels, @htl("""\$$text\$"""))
end
Luxor.Drawing(width, height, :svg)
Luxor.origin()
Luxor.background("white")
draw(label)
Luxor.finish()
svg = Luxor.svgstring()
@htl("""""")
endÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$95504a74-d5ef-4fb7-83a0-88914c7cbc59«code_folded¤codeÙ"numerical_lookup(dict, [0.8, 0.2])Ù$453544fc-0e3e-4e04-8c0c-192f3a038884„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$453544fc-0e3e-4e04-8c0c-192f3a038884«code_foldedäcodeºmd"## Positional encoding"Ù$0583ee0c-3802-4e81-b179-a80a82493b43„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$0583ee0c-3802-4e81-b179-a80a82493b43«code_foldedäcodeÚÜmd"""
Byte Pair Encoding algorithm $(cite("sennrich2016Neural")) greedily merges the most frequent pair of tokens over the dataset into a new token.
Most used implementations are `SentencePiece` $(cite("kudo2018SentencePiece")) and `tiktoken` (play with it [here](https://tiktokenizer.vercel.app/)). For instance, on [this example](https://en.wikipedia.org/wiki/Byte_pair_encoding), the pair `('a', 'a')` is the most frequent so we substitute it by a new token, say `'Z'`:
"""Ù$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$728f5fdf-77a5-46c7-b3ee-01064ef1b7e2«code_foldedäcodeÚÆqa(md"Is the KV cache equivalent to recomputing everything for the new context window ?",
md"""
No! Without cache, we would compute new columns ``\hat{y}_{i,j}'`` that only depend on ``x_{-n_\text{ctx}+1}, \ldots, x_j``.
The cached ``\hat{y}_{i,j}`` were computed while ``x_{-n_\text{ctx}}`` was still in the context window so they also depend on ``x_{-n_\text{ctx}}`` (dashed red edges).
The cached columns have **one more token** in their context!
""")Ù$01372b00-ecb2-42bd-b408-13234717d969„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$01372b00-ecb2-42bd-b408-13234717d969«code_folded¤codeÙ°function new_token(bpe::BPE)
stats = pair_stats(bpe.text)
pair = findmax(stats)[2]
new_char = min('Z' + 1, minimum(bpe.text)) - 1
return add_pair(bpe, pair => new_char)
endÙ$f4366cf6-2be0-42b8-96c4-120be3f5c25e„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$f4366cf6-2be0-42b8-96c4-120be3f5c25e«code_foldedäcodeÚµmd"""
### References
* Recurrent neural networks : $(cite("goodfellow2016Deep", "Chapter 10")) and Section 4.7 of [The Elements of Differentiable Programming book](https://diffprog.github.io/)
* Transformers : $(cite("vaswani2017Attentiona")) and Section 4.8 of [The Elements of Differentiable Programming book](https://diffprog.github.io/)
* [Neural Networks: Zero to Hero](https://karpathy.ai/zero-to-hero.html) by Andrej Karpathy
"""Ù$22f68d24-1132-4e45-b8fb-7a1f362f0962„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$22f68d24-1132-4e45-b8fb-7a1f362f0962«code_foldedäcodeÚmmd"""
| Query | Wh / query | Scope | Source |
|---|---|---|---|
| Google search (2009) | 0.3 | Google servers, including building the index | [Hölzle, 2009](https://googleblog.blogspot.com/2009/01/powering-google-search.html) |
| Text generation, models ≤ 11B | ≈ 0.05 | Measured on 8×A100 | [Luccioni et al., 2024](https://arxiv.org/abs/2311.16863) |
| Llama-3.1-8B, short / long prompt | 0.05 / 0.44 | Modelled, data centre | [Jegham et al., 2025](https://arxiv.org/abs/2505.09598) |
| Gemini Apps, median text prompt (May 2025) | 0.24 (0.10 accelerators only) | Measured, full stack | [Elsworth et al., 2025](https://arxiv.org/abs/2508.15734) |
| GPT-4o, 500 output tokens | ≈ 0.3 | Estimate | [You, Epoch AI, 2025](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use) |
| ChatGPT, average query | 0.34 | No methodology given | [Altman, 2025](https://blog.samaltman.com/the-gentle-singularity) |
| Llama-3.1-405B, short / long prompt | 2.2 / 25 | Modelled, data centre | [Jegham et al., 2025](https://arxiv.org/abs/2505.09598) |
| o3 (reasoning), short / long prompt | 1.2 / 12 | Modelled, data centre | [Jegham et al., 2025](https://arxiv.org/abs/2505.09598) |
| Image generation | ≈ 2.9 | Measured on 8×A100 | [Luccioni et al., 2024](https://arxiv.org/abs/2311.16863) |
* Small models and median prompts : **same order** as a (2009) web search.
* Large models, long contexts and reasoning : **1 to 2 orders of magnitude** more.
* Google reports a 33× reduction of the energy per prompt in 12 months [Elsworth et al., 2025](https://arxiv.org/abs/2508.15734). There is no recent public figure for a web search.
"""Ù$c8923675-e73e-4621-82b9-966d8b003b97„¨metadataƒ¨disabled©show_logsîskip_as_script§cell_idÙ$c8923675-e73e-4621-82b9-966d8b003b97«code_foldedäcodeÙ!md"# Encoder-decoder transformer"´last_hot_reload_timeË ©shortpath¯transformers.jl¥nbpkgŠbusy_packages�Ù,waiting_for_permission_but_probably_disabled§enabledðterminal_outputs�°HypertextLiteralÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...MathTeXEngineÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...¥LuxorÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...¬PrettyTablesÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...¬LaTeXStringsÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...³DocumenterCitationsÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...ªnbpkg_syncÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...£CSVÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...§LoggingÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...§PlutoUIÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...LinearAlgebraÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...ªDataFramesÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...²PlutoTeachingToolsÚ“
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Project.toml`
[32m[1m Updating[22m[39m `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hkxipumrtv/Manifest.toml`
[90m[14a3606d] [39m[93m↑ MozillaCACerts_jll v2026.8.13 ⇒ v2026.10.20[39m
[90m[efcefdf7] [39m[93m↑ PCRE2_jll v10.46.0+0 ⇒ v10.47.0+0[39m
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...¶waiting_for_permission·restart_recommended_msgÀ´restart_required_msgÀ²installed_versionsްHypertextLiteral¥1.0.0MathTeXEngine¥0.6.9ªDataFrames¥1.8.2¬PrettyTables¥3.4.8¬LaTeXStrings¥1.4.1Ù!__internal_julia_manifest_version¦1.13.3³DocumenterCitations¥1.5.0£CSV¥1.0.0§Logging¦stdlib§PlutoUI¦0.7.83LinearAlgebra¦stdlib¸__internal_julia_version¦1.13.3¥Luxor¥4.5.0²PlutoTeachingTools¥0.4.7¯install_time_nsθ¹Äð¬instantiatedÃpluto_version¦v1.1.0