Merge branch 'presentation-slides' of github.com:NeilNjae/cipher-training into presen...

[cipher-training.git] / slides / caesar-break.html

diff --git a/slides/caesar-break.html b/slides/caesar-break.html
index 187719da5f81e687adcac89c30c90646663df232..15607e7665005c39396e342951187c35000bbb75 100644 (file)
--- a/slides/caesar-break.html
+++ b/slides/caesar-break.html
@@ -126,13 +126,15 @@ But before then how do we count the letters?
 * Read a file into a string
 ```python
 open()
-read()
+.read()
 ```
 * Count them
 ```python
 import collections
 ```
 
+Create the `language_models.py` file for this.
+
 ---
 
 # Canonical forms
@@ -150,16 +152,18 @@ Counting letters in _War and Peace_ gives all manner of junk.
 # Accents
 
 ```python
->>> caesar_encipher_letter('é', 1)
+>>> 'é' in string.ascii_letters
+>>> 'e' in string.ascii_letters
 ```
-What does it produce?
-
-What should it produce?
 
 ## Unicode, combining codepoints, and normal forms
 
 Text encodings will bite you when you least expect it.
 
+- **é** : LATIN SMALL LETTER E WITH ACUTE (U+00E9)
+
+- **e** + ** ́** : LATIN SMALL LETTER E (U+0065) + COMBINING ACUTE ACCENT (U+0301)
+
 * urlencoding is the other pain point.
 
 ---
@@ -190,6 +194,15 @@ def unaccent(text):
 
 ---
 
+# Find the frequencies of letters in English
+
+1. Read from `shakespeare.txt`, `sherlock-holmes.txt`, and `war-and-peace.txt`.
+2. Find the frequencies (`.update()`)
+3. Sort by count 
+4. Write counts to `count_1l.txt` (`'text{}\n'.format()`)
+
+---
+
 # Vector distances
 
 .float-right[![right-aligned Vector subtraction](vector-subtraction.svg)]
@@ -217,7 +230,7 @@ The higher the power used, the more weight is given to the largest differences i
 * L<sub>&infin;</sub> norm: 
 `\(\|\mathbf{a} - \mathbf{b}\| = \max_i{(\mathbf{a}_i - \mathbf{b}_i)} \)`
 
-neither of which will be that useful.)
+neither of which will be that useful here, but they keep cropping up.)
 ---
 
 # Normalisation of vectors
@@ -285,6 +298,54 @@ And the probability measure!
 
 ## Computing is an empircal science
 
+Let's do some experiments to find the best solution!
+
+---
+
+## Step 1: get **some** codebreaking working
+
+Let's start with the letter probability norm, because it's easy.
+
+## Step 2: build some other scoring functions
+
+We also need a way of passing the different functions to the keyfinding function.
+
+## Step 3: find the best scoring function
+
+Try them all on random ciphertexts, see which one works best.
+
+---
+
+# Reading letter probabilities
+
+1. Load the file `count_1l.txt` into a dict, with letters as keys.
+
+2. Normalise the counts (components of vector sum to 1): `$$ \hat{\mathbf{x}} = \frac{\mathbf{x}}{\| \mathbf{x} \|} = \frac{\mathbf{x}}{ \mathbf{x}_1 + \mathbf{x}_2 + \mathbf{x}_3 + \dots }$$`
+    * Return a new dict
+    * Remember the doctest!
+
+3. Create a dict `Pl` that gives the log probability of a letter
+
+4. Create a function `Pletters` that gives the probability of an iterable of letters
+    * What preconditions should this function have?
+    * Remember the doctest!
+
+---
+
+# Breaking caesar ciphers (at last!)
+
+## Remember the basic idea
+
+```
+for each key:
+    decipher with this key
+    how close is it to English?
+    remember the best key
+```
+
+Try it on the text in `2013/1a.ciphertext`. Does it work?
+
+---
 
 
     </textarea>