Instead of "scrambling" letters, modern encryption works by taking data and combining it with seemingly-random looking data in a clever way. It's similar to rot13 in two important ways: encryption and decryption are actually the same operation, and everything is in-place.
One approach is to have a big chunk of secret data lying around and use that to XOR. As long everyone who needs to send or read the original plaintext has the same secret data ... this should work. Couple of problems with this:
-
-
Problem 1: The secret data needs to be seemingly-random. You can't use the text from book or something. Any patterns in the secret data will show up in the encrypted version, and that's literally part how the allies beat the axis powers in WWII.
Show this thread -
Problem 2: You can't ever re-use the secret data. Again, patterns will show up. So you have to somehow securely get big wedges of secret random data (One time pads) around to everyone who needs them. Too hard.
Show this thread -
So in modern encryption, we *generate* the secret data we need from a small key, and those keys are much easier to get around and protect. This is what symmetric encryption algorithms really are: schemes for deterministically generating random data from a key.
Show this thread -
That "deterministic" part really matters: two people with the same key have to generate the *exact* same data, or else they won't be able to understand one another.
Show this thread -
You've probably heard of lots of these algorithms: AES, 3DES, DES, RC4, ChaCha20. All of these algorithms do this. It turns out that the math problem of taking a key and generating a random stream of data, one that has no patterns and is not predictable in any way, is hard.
Show this thread -
From that list, only AES and ChaCha20 are considered secure today, they others have been broken in interesting ways ... people figured out how to predict them. AES itself has a bit of patchy reputation, because ...
Show this thread -
Cryptographers: AES is the premier and most-analyzed encryption algorithm. Absolute gold standard!
Also Cryptographers: AES implementations in software (not hardware) are either insecure, or slow, or both. It wasn't designed with caching side-channels in mind.
Show this thread -
Don't worry too much if you didn't follow that. Just take this away: AES is mathematically awesome but hard to code, it's a good thing we almost always have hardware support to do it instead.
Show this thread -
... anyway ... moving on ... how do these encryption algorithms actually work? How do we take a key and securely generate a random stream of data? I'm going to simplify things a little here and talk about blocks.
Show this thread -
These algorithms take three inputs, and produce encrypted text. The inputs are the key, the plaintext ... and - surprise - something strange called an initialization vector. So for example: AES(key, IV, plaintext) -> encrypted_data
Show this thread -
The key and IV are combined to create a set of "starting conditions" for the algorithm; it's like an initial permutation, or scrambling, of some scrabble tiles. The same key and the IV will always produce these same starting conditions? Why do we have an IV, you ask? ...
Show this thread -
We have an IV so that we can encrypt multiple messages using the same key. Without an IV, each generated stream would always be the same, and that's no good. One of rules is that we can't re-use ever. So we need the IV to mix things up. But unlike the key, the IV can be public.
Show this thread -
So when you encrypt a message and send it to someone, you can send add "hey, here's the IV I used". Now it's still critical that we don't re-use the combination of key and IV, because those would generate the same random data.
Show this thread -
Two common ways to do this: 1) the IV is some kind of counter: we just increment it for every message. 2) the IV is generated randomly, and it's a big enough value that we don't worry too much about collisions. Anyway, I said I'd talk about blocks.
Show this thread -
So the key and IV are "mixed" or combined in some way to create starting conditions ... these conditions are really an initial "block" of random data. For AES128, the block is 128 bits, for AES256 it's ... 256, for ChaCha20 it's 512-bits.
Show this thread -
Now the magic and personality of the encryption algorithm really comes in. It's really all about how we generate one block after another, and each block is seemingly-random with predictable relationship to what came before or after, to someone who doesn't have the key.
Show this thread -
I'm not going to go too deep into how these work, but if you want to get a sense yourself: start by looking up linear congruential generators, which is a really simple function that can make a block of data "cycle" in a random non-repeating way.
Show this thread -
Then look up Feistel networks, which are sort of the next level of that. Look up S-Boxes then if you're more curious, and finally, take a look at how Salsa20 does the rotations that ChaCha20 uses. It's all more approachable than you might think!
Show this thread -
O.k. so now we know how random streams of data can be combined with plaintext to do encryption and decryption, and we sort of know how random streams of data are produced. Isn't that it?
Show this thread -
For full disk encryption, it almost is really. We basically just encrypt every block or sector of storage, under the same key, and using an IV that's derived from the "position" on disk. So we can always just decrypt any block anywhere on the disk, as long as we have the key.
Show this thread -
But there's a problem with this ... someone can mess with the encrypted data. If I change the value of any byte, even if I don't have the key, it will mess with the decrypted output. There's no real security against that kind of meddling.
Show this thread -
For sending messages and data over the network, that's not going to got it. We don't want people tampering with our information. So we need to add an integrity check! There's a few schemes for this.
Show this thread -
HMAC, GCM and Poly1305 are the common ones in use right now. In each case, these algorithms basically take the data as input, along with another key (an integrity key) and produce a MAC or a tag, which is just another piece of data that acts as a signature.
Show this thread -
So to encrypt, and protect, our string, one scheme might be: AES(key, IV, "Ovaltine") -> encrypted_output HMAC(key, encrypted_output) -> MAC and then on the wire, we send: IV | encrypted_output | MAC
Show this thread -
To decrypt, we check the MAC first by generating it again and making sure they are the same, and then we decrypt the output. Internally there are differences between how HMAC, GCM and Poly1305 generate these signatures, but you don't need to worry about that.
Show this thread -
Today, this combination of operations is wrapped up in function we call "AEAD" which means Authenticated Encryption with Additional Data, and it does all of this is a mostly-foolproof way for you. Basically: AEAD(key, IV, plaintext, additional_data) -> IV_encrypted_data_MAC
Show this thread -
The "additional data" is just any other data you might want to "prove" the sender has, but not send; like say some meta-data that establishes a permission. It's often left empty.
Show this thread -
Now you can still screw up with AEAD. If you re-use the same IV, that's bad!! There are attempts to make this better, my colleague Shay has been working on a cool scheme called SIV, and it adds a measure of protection against that too.
Show this thread -
If you do use unique IVs, modern encryption is really robust. In general, you could publish some encrypted text in the New York Times, and no-one will be able to crack it. This Is true even if /some/ of the text is known. For example ...
Show this thread -
In internet protocols a lot of the text is known, a HTTP server always responds the same way and the first few bytes are known and totally guessable. This doesn't matter at all - doesn't help an attacker figure anything else out even one bit. We've come a long way from WWII.
Show this thread - 10 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.