Select text to annotate, Click play in YouTube to begin
hello NDC Oslo I'm Dylan I'm going to talk to you about email because like email is kind of important right you ever get the thing where you can't remember your password so you click forgotten password they send you
an email you get locked out of your account you click the button they send you an email you want to go to sign into Microsoft azira and they say oh no hang on we just need to check it's really you how do they check it's really you they send you an email you sign up on a new
service so like we want to make sure that you're not a Spam bot or some kind of you know evil nefarious internet criminal overload I'm going to send you an email and you can use the email to verify it so email is a really really important
component in the fact that any of the stuff that we do online ever actually works at all so we're going to do a little experiment we're going to go here
now what I'd like all of you to do you're going to go to that web address and you are going to follow the instructions now there is a button that's going to about to appear on your phone screen that says don't press this button yet do you know what that means
there is a saying that if they wrote on a button do not press this button or it will destroy the universe the paint would not be dry before a human being had slapped the button do not press the button what I want you
to do is to follow the instructions on the screen there we go let's bring up our live update someone has matched the button that little yellow at sign there that's somebody who can't read the documentation and it's written in large friendly
letters how evil can it be so we're going to just let that rack up a little bit oh look at that look at that are someone who's already caused an error message well done congratulations there we go look at that
someone's already gone all the way through the process most of you it's good to see every little white dot on that that someone who is following the instructions no wait wait wait no no no no no no no
no stand up the instructions say you stand up so you stand up stand up stand up stand up stand up stand up everyone's standing up standing up standing up
that looks like a statistically valid sample right so what we're going to do I'm going to count down from five I'm going to get to zero you're allowed to press the button five four
three two one press the button and we're going to send you an email and when we do you are gonna go and check your inbox and when you get the
email follow the instructions in the email all right I see a couple going green I see a couple going pink some of the stuff's getting through some of the stuff is a gonna junk mail some of it is still cute some of it's still sent
uh the little black ones there are probably because I've pissed off every email provider on the planet doing the research for this talk and some of them have decided that they are just not going to talk to me anymore so we're getting a fair few of those
going through now it's around about now that you've been waiting long enough that if this email was like your two-factor authentication to get into Azure to fix a problem in production or the password reset so you
can get into your Gmail to get your damn plane ticket because you need to go to the airport you'd be getting a little bit why doesn't email work this should have arrived by now well friends I have some very distressing news for you
everything is working absolutely fine those of you who are still standing up bear with me one more second this is the SMTP specification latest version October 2008 about how we should try to
deliver email and what it says is if it doesn't work first time you should wait at least 30 minutes and then you should keep trying for four or five days before you finally decide that the email didn't work now anyone
who didn't get the email you are welcome to come down the front at the end and you can get one of these the sticker that should have been an email that I've I've got some of these printed out here saying take all of those these are the special NDC Oslo editions
of this so they won't look like this ever again now I'm going to talk to you about email and I'm going to talk to you about why we have ended up with this weird situation where the spec says that an email can take five days but we built
the internet on the fact it takes about 15 seconds how did we get here now my kind of credibility I guess my expertise in the field of email it's something I've been working with my entire career in uh
2002 when the internet had a connection wizard one of my first like big professional projects I ever worked on was a project for this company Spotlight is a a British directory of Show
Business job opportunities for actors and actresses all those kinds of things and uh I built an email-based job information service so if you were making a movie and you needed actors to be in your movie you could go on Spotlight and you could send out an
email now this was built in classic asp and I had a for Loop and what it would do is it would send an email and then it would responds.write a DOT so you'd get a screen that went dot dot dot dot dot and then at the end we'd respond stop
flashing we'd redirect it's in about 200 emails a day it was Conquest up but the system before that was built on fax machines so this was a pretty neat idea it was also a spectacularly successful
product it went from 200 emails a day to 500 to a thousand to two thousand to five thousand by the time I left Spotlight in 2018 this system was sending a quarter of a million emails
out every day and that's not a quarter of a million copies of the same newsletter it's a quarter of a million unique individual personalized emails now that was an interesting phenomenon that we noticed we catered mainly to the
English-speaking Show Business Market in London about two o'clock every afternoon show business would get back from its nice boozy lunch in the pub sit down and go I should probably send out some emails what do you think Brian and
they go on Spotlight they'd bang in a couple of things they'd send us out this happened at the same time as New York and Broadway had just finished their latte and were getting ready to get set into a solid day's work which was the
same time that Hollywood and Los Angeles have finished their power breakfast and started yelling at their assistance so when I say a quarter of a million emails a day it most of them went out in a three-hour window every afternoon that
is a sustained rate of about 10 to 12 emails per second for three hours every weekday for more than a decade now you probably have a picture in your head of what a professional actor looks
like most of them don't look like the ones you see on TV the ones you see on TV are the ones who have made it because they have a job right now the average professional actor looks like this
because they're at home waiting for their next audition and they are sitting there and they are refreshing their email now some of you might have had complaints like companies uh you worked at where you've sent email and people have complained that you've sent them anymore have you ever had someone to
phone you to complain that you hadn't sent them email because we used to get people ringing up going I haven't had any emails for a couple of hours is everything all right and we'd be like well we think so maybe there's just no work for you today but
we had to answer those calls and we had to be able to explain what was going on so my team and I got incredibly good at understanding everything there was to know about email delivery how does it work why does it work why might it fail
now the whole time this is going on because I'm an idiot I'm running an open source mail server for me and all my friends so Dylan beatty.net all the email ran on an old Hewlett Packard workstation that was locked in a server cupboard somewhere for about 10 or 15
years now I eventually stopped doing that and you'll find out why in the course of the talk but then at the start of this year who was NDC London I was yeah um some people who very nearly weren't
were people who work for a company I won't name whose Corporate email provider switched to a new spam filtering service the week before the conference and that spam filtering service mimecast decided that send grid
was sending junk and so all of their NDC conference tickets went to junk and we had to kind of jump in and figure out how did these people not get the tickets they'd paid money for that were sent directly to them and I thought that might be an interesting talk in this
because lots of us build systems that send email and lots of us kind of don't really understand all the weird quirks of how it works so email is older than networks the first
computer that kind of really got her any kind of traction was this thing the compatible time sharing systems about 1961 and you could send mail on this thing but send as a misnomer because when you sent mail the mail didn't go
anywhere it was the equivalent of leaving Post-it notes on a really expensive refrigerator because this computer was shared between 60 or 70 different researchers so when your colleague came to work the next morning
and logged into that terminal you'd left them a note that was the first use of electronic mail this kind of went on for a couple years and we're going to skip forward to 1969. now if the United States of
America was a Netflix show 1969 would be a season finale big budget they pulled out all the stops they did lots of really interesting stuff 69 was the year that people walked on the Moon it was the year the Boeing 747 first flew was
the year the Lockheed SR-71 first flew it was the year of Woodstock and the summer of love all that kind of stuff and it was the year that the upper net was invented
now there were networks before the arpanet the opponent was the advanced research projects agency Network and it looked like this this is a hand-drawn sketch of the entire network now before this there was a network but it was a
kind of ad hoc network of computers that would dial each other up once or twice a day and they'd swap messages and things but when the arpanet came along for the first time there was an idea of being connected to the net this was the
network and if you were on it other people could send messages to you now modern email is a combination of probably hundreds of people working on solving smart problems over decades on
all kinds of platforms but one person whose Innovation is still a significant part of the way we work with it was this guy it's Ray Tomlinson and he was working on an opponent Mail
system in 1971 and Rey invented at Rey is the person who went well hang on if we know the user's name and we know the arpanet host where they host their email we could put an at in the middle
because it's Alice at the machine now you notice that the hostname there is just MIT multix it's not a DOT anything it's not MIT multix.com it's just MIT multix because the network was so small that computers just had a name and if you
bought a new computer you wanted to get on the net you'd give it a name and then you talk to Jake this is Elizabeth fainler who everyone called Jake for for some obscure reason she ran the network information center
at uh Stanford from 1970 to 1988 and she maintained the host's file the definitive hosts file of the entire internet and every year or two she and
her team would print it because you could buy a print out of the hosts file and this listed all of the machines connected to the arpanet along with the person responsible for maintaining them telephone numbers street addresses and
email addresses now there are two kinds of people in the world there are the people who see technology and go hey this is really useful and there are the people who go hey those people think this is useful maybe I can use it to get rich
let's meet Gary Turk now uh Gary took I punched his name into Google to find a photo the guy's still alive he's still working this is his LinkedIn page like you could connect with him after this talk if you really
wanted to um but in 1978 Gary Turk was working for the digital Equipment Corporation he was a sales rep and his job was to sell these the deck system 20. now this thing
had built-in arpanet protocol support it was like you don't have to do anything special you could plug it into a network and it would just work and rightly or wrongly Gary thought well I reckon people who are on the upper net might be interested in knowing about this
computer and digital didn't have a whole lot of sales going on on the US West Coast they had a big office on the East Coast but West Coast you know California Portland those kind of places they didn't really have much of a presence so he got his assistant
to go through the arpanet directory and type in the email addresses of everybody on the American West Coast who had an email address 393 of them and put them now at this point they overflowed the header field so all the people who got this email got
an email which started with about 250 other people's email addresses and then right down at the end of it it says hey we invite you to come and see the deck system 2020 and we'll be giving
presentations now has any of you ever gotten trouble at work like maybe your boss said please don't do that again or maybe your boss shouted at you or maybe the police shouted at you or maybe the police shouted at your boss you ever
screw up so bad that the United States military yells at your boss because that's what happened major Raymond chehor who was the uh the U.S Air Force Major in charge of the
upper net because this was run by the United States Air Force you know U.S military project basically called Gary's boss and said you do not do this ever again we're very sorry now there's maybe a little window right about then when we
could have fixed the problem but everyone just went yeah don't send spam or the American Air Force Will shout at you we clear whatever I went yeah we understand we'll be good from now on we'll be nice we'll behave
we're gonna fast forward a little bit now now you know that uh demographic studies or populations kind of a group people into Generations there are the Baby Boomers and then there's there's Generation X and then there are the Millennials and then there's Generation
Z and then there's gen alpha or whatever they're called in the new one and there's this little Gap in the middle the micro generation we're the people who got email at the same time as everyone else in the world got email for
the first time now if you are Generation X it means you were born before Star Wars and if you are Millennial it means you were born after Return of the Jedi and the people in the middle are the
millennial Falcons represent how many of my fellow Falcons we got out there today awesome now the millennial it was kind of this weird micro generation we're the ones
who like grew up in a world where we had broadcast TV and our first job we had email so we kind of had the analog childhood and then our adulthood has been completely digital but it turns out that the internet is a millennial Falcon
because when Star Wars came out in 1977 there was the opera net it used NCP the network communication protocol everything was managed by the host's file which you had to go and get a copy
of from Stanford University and junk mail didn't exist no one had done that yet when Return of the Jedi came out in May 1983 the opponent had become the internet these NCP was gone TCP was
everywhere we're all running on this DNS had been invented we had a hierarchical domain name addressing system and junk email was very much a thing now uh one of the things invented in the
early 80s there was SMTP it's created by this guy John postel SMTP was published in 1982 and has been incredibly successful now the problem with successful systems is you need to
maintain an obscene level of backwards compatibility if you bring out an email system at any point in the last 40 years that doesn't talk to all the other people who are already using email no one's going to use it because it's
useless because you can't communicate it is the equivalent of going out and buying a 2022 MacBook M1 or M2 Pro and finding that it has a cassette player so that you can still load your
PowerPoint slides from 1982. this is how backwards compatible SMTP systems and relays have to be and one of the things defined in that request for comments RFC 821 was the format of an email address
now uh quit your hands how have you who's written code to validate email addresses keep your hands up if you use the regular expression yeah so I mean validating them in code is kind of but we know an email address
when we see one right like uh wait we're looking at our friends in The Avengers we got Iron Man at avengers.com that looks valid right um and Spider-Man well yeah that's probably going to work uh t'challa the
avengers.com that's gonna root obviously nothing wrong with that rocket plus Groot I'm not sure now what about Bruce the Hulk Banner at avengers.com what about no vision doesn't give a
about DNS Vision it's going to use IPv6 hard-coded literals in square brackets boom how many of these would your email validation routine have accepted they are all valid
these addresses are all completely valid email addresses according to the RFC specification for internet email addresses so maybe validity is not really the
question we should be asking ourselves we're going to pull some email addresses apart and see how they actually work now an email address has a domain part and this controls where it's going and it has a local part that controls what
happens when it gets there so what we're going to do we're going to walk through the process of actually speaking the raw protocol that we use to send email addresses so I'm going to use a tool called nslookup here cool kids use dig
because dig is what runs on Linux and everything but a dig doesn't come with windows so this is nslookup which is available on most platforms I'm going to do an NS lookup name server lookup and I'm going to say hey I want to know MX
set type equals MX mail exchanger records I want to know which computers on the internet are the mail exchanges for this domain fun with DOT email and it's going to say oh here you go look
smtp1 smtp2 there's a number in here called an Amex preference now some people call this priority that is a stupid name for it because a loan number is a higher priority preference number 10 means try this one
first if you can't get through the 10 go to preference number 20 the bigger number means wait try that one later so that's the computer we need to talk to we're going to talk to smtp1 so I'm going to telnet into that on Port 25 and
it's going to say hey nice to see you and I'm going to say hello I'm Dylan beatty.net and it's going to say 250 I'm smtp.fun with email and I'm going to say I have mail from Dylan dylanbeatty.net
and it is going to say 250 okay and I am going to say recipient 2 hello at fun now this is where it gets interesting because that recipient 2 is the point where we say to that system this is who
we want to talk to now I wanted to find out what actually works like never mind what the instruction manual says I don't care what's in the spec what actually works out here so I went on Google and I googled the best email providers for
doing stupid experiments and then I went no no let's go with something we're going to go with uh best business email hosting and I troll through a bunch of search results now one that I just want to highlight because it is the most wonderful piece
of hallucinatory Journalism I've ever seen this article from PC Magazine in 2020 the best hosted email providers are Salesforce
Microsoft Office 360 users should get their email from GoDaddy web hosting Zoho loyalists should use Zoho Mail and the best email for Microsoft organizations is teams
um in case you're not familiar with business email hosting this is like you go into cars magazine you say what are the best economy family cars in 2020 and it says well there's the Hyundai Tucson there's bicycles there's a submarine and
there's penguins I don't like yeah right um but I also found quite a lot of good solid articles that were recommendations for which companies people use in 2023
to host email and so I picked a bunch of those now disclaimer here this is not a review of business email Services I do not want anyone here going away and choosing an email platform
because of something that I said in this talk what I'm doing here is the equivalent of testing cars by filling them with gravel setting them on fire driving them off a bridge and seeing if they explode before they sink
it's interesting every provider I'm going to show you is a rock solid platform they all do email really really well until some idiot like me comes along and tries to poke them with a stick
so I identified from looking at all these articles the big players right now Office 365 and Google are the ones that really kind of dominate the space Zoho have a lot of support from people who don't want to use the big platforms protonmail is very popular because they
have really strong commitment to security and FastMail is uh so disclaimer fast mounted company I've used to host my mail for years and years personally I love them I have no problems with them um but I'm just a regular pay-in
customer of all these Services there's no I haven't talked to them there's no endorsements so I set up email domains for every single one of those set up all the DNS records all the anti-spam records everything and then I went through to see how many email addresses
I could register so let's try to challenge with the apostrophe in it Google yep no problem Office 365 yeah no problem go on to uh Zoho Mail it let me register
it but then when I click on it apparently they didn't tell the people who built their front end that there might be apostrophes in the Json and the email addresses so uh we're going to give that one a frowny face protonmail just says no you can't have that not
allowed and I go on a fast mail and I said please check this so no can't do that not allowed let's try rocket plus Groot now I go on a Gmail and I try and create a mailbox and it says you can't contain special characters so it looks
like you can't do that but Google has a weird feature and it's quite a neat feature if you have a bunch of email addresses so we've got Iron Man at gmail we got Iron Man plus Jarvis we got ion dot man we got ion dotman plus Jarvis we
got I don't know these all go to the same place because Google decided they don't want some situation where I'm Dylan Beatty and you're Dylan dot BT and we're different people they went that is too
complicated for humans to understand and the plus thing the nice thing about that is it's preserved so when you sign up for like a mailing list you could be like dylan.bt plus NDC gmail.com and then if that shows up in a data breach I
know which company the breach came from so this is plus tag tracking this is a Google feature it's not part of an email standard but it is worth knowing about if you're working with mail deliverability so actually rocket plus grew on Gmail is kind of a smiley face
um Microsoft 365 says no you can't have that so it's a cross for them Zoho Mail says invalid username all right fair enough protonmail nope you can't have that and FastMail says this name's already taken
which it isn't because I just registered that domain it's not taken but yeah we're going to give that one a a sad face now Bruce Banner in quotes the email spec says that you can have a quoted local part containing spaces
there is not a single provider on the planet that I found which will let you create a mailbox but then I thought all right well enough fun with the sensible addresses what about single apostrophe
Google yeah Microsoft yeah Zoho Mail no protonmail no FastMail no what about single hyphen now this is where Google says email cannot be a single hyphen I'm like what
about a double hyphen and Google goes oh yeah you can have that one that's fine Office 365 minus at is fine that's legitimate uh Zoho says no proton says
no that name is already taken again it's not I don't know what the validation thinks is going on here FastMail says no so this is the set of aliases that I managed to create there this is the Google admin for gg.fun with DOT email
to Charles in single apostrophes in minus minuses in underscore is in uh last one I wanted to try was just underscore at Google says yes Outlook says yes Zoho says well Zoho says I put
it in I click add and it says ass 101. and I'm like I wonder where that came from so I open up the Chrome Network inspector I take the request apart and it says status code 200 s101 now where I
come from 200 means okay right so I don't know why it's red but we're gonna go with we're not quite sure what happened there and finally underscore at FM FastMail just says great choice
so there we go so next I thought all right never mind creating mailboxes I'm going to create a catch-all address I'm going to create a dress that says anything at this domain should go to that person over there anything that's deliverable because I
wanted to see if I could push the limits of what spec email addresses would actually do that's where I hit a little bit of a problem because I went on a Gmail and I tried to send an email to quote Bruce space Banner quote and Gmail says no Gmail says you can't send mail
to that person it's not recognized and that's just it doesn't exist um protonmail their interface said the same thing email address format is not valid I'm thinking Hmm now what we can conclude from this is an email address
valid does it contain at least one at sign because if it doesn't no but if it does then it depends
can an email address be case sensitive the answer is yes if you're evil when SMTP was introduced in the early 1980s the way that it got rolled out was most people who already had a system
that had mail added an SMTP connector to it and most of those systems ran on Unix and Unix has a case-sensitive file system and it means that your mailbox has the same name as your mailbox file
which has the same name as your user which means that Alice with a little a and Alice with a big a could be different people on the same Unix machine and so when you connect an SMTP relay the spec says that once it gets to where
it's going if the local administrator of that system has made it case sensitive that should be delivered intact you should never ever modify the case of somebody else's email address now this
is where the airline industry turns up and goes we have Cobalt mainframes your email address will be uppercase whether you like it or not so when Alex Goes to Alice here goes to buy an email ticket and get it sent to her unix.box.edu
account she's not going to get anything because big Alice and medium Alice they've got mailboxes but Capital Alice doesn't exist so can email be case sensitive it can if you really want it to but it probably shouldn't be
so we're going to go back to here we're going to do our 1980s flavored we got a hello we got mail from we've got recipient to so it's okay we're going to say data and it's going to say go ahead end your data with a new line dot new line and we're just going to say Hello
dot boom 250 okay queued quit bye bye and there it is we sent an email using 1982 flavored raw SMTP typing into a Windows terminal the only challenge with this is finding a network that will
allow you to do it and we'll find out later why that's interesting now this is where I start thinking well hang on Gmail and proton won't let me send emails with spaces in them but I've just proved that we can hack into the Matrix
we can send code directly so I thought what if I go to protonmail look up their mail exchanger tell net into it hello mail from hallowed fun with email recipient to that it says Ah protocol error okay that's not going to work
let's try it again Zoho what have they got boom tell it into that one hello mail from recipient to now I'm not relaying because that's their domain I'm giving them money to
host that but apparently they've decided that that's relaying and they are not going to let me relay sad face let's try Google telnet boom there we go we're in hello mail from yep okay recipient two that's
not a valid RFC 5321 address also that's a lie it is a valid RFC 5321 address Google is lying to your face but they Google you know what are you gonna do so that doesn't work now I tried to do this
with the Microsoft account and uh I could not find any Network anywhere that Microsoft would let me inject mail from because every single place I tried I just got access denied band sending so
the NDC conference Wi-Fi didn't work Hotel Wi-Fi didn't work tell that on my phone didn't work onboard Wi-Fi on Norwegian air didn't work Deutsche Bahn Railway onboard Wi-Fi didn't work Microsoft are really good at working out
who might be trying to send spam from a moving train in Germany and making sure that that doesn't happen yeah Azure blocks Port 25. you can't send mail from Azure or AWS or Google
Cloud we'll talk about that in a bit yeah probably and everyone at NDC and everyone in the hotel I'm sorry it's research you know start with let's try it with FastMail so I got that mail exchanger tell it
hello mail from recipient two whoa okay data there we go this was sent to an address with spaces in it and it worked let's
see if it worked quit Okay bye there it is it worked delivered to an email address containing spaces because this is what the specification says email should do
now what we've just done there every one of those Services we have done the digital equivalent of like let's imagine that we want to email we want to send a letter a physical letter to this you
know who this is right this is Lazarus he's the president of Malawi and he lives here and so we want to send a letter to him so we're going to write a letter put it in envelope and we're going to go to Malawi get on a plane fly land a little long way we're going to go
out of town go up to the the presidential Palace at the top of the hill and we're going to go hello hello we have a letter for his Excellency now one that is not an efficient way to deliver mail and two he might be out in
which case we have a deliverability problem so what can we do instead well we do this we put a stamp on it and we put it in a mailbox and we assume that we can find
any mailbox anywhere on the planet we can put a letter into it and if we put the correct postage on it'll go where it needs to go now mailboxes work because there is a physical and financial cost
to sending printed mail you've got to buy a stamp you've got to buy printing have you seen how much printer ink costs like if you want to sell bad penis enlargement procedures to a hundred thousand people you're not going to do it using a Hewlett Packard inkjet
printer the business model just doesn't work so this kind of mail it's kind of self-regulating up to a point but let's uh let's have a look at what we've got in my inbox here now I've got
a message here from Lisa Williams she's looking for a special friendship web design prices on Google cross-border Payments International an investment opportunity for you your subscription
has been confirmed important message FedEx International [Applause] I have a new voicemail message let's give it up for the NDC Viking spam choir
into it you ever wondered why junk mail is called spam so spam is spiced ham it's this it's meat in a tin which has been around for years and Monty Python did a sketch in
the 70s where someone's in a cafe and the things on the menu are egg and spam and egg and bacon and spam and egg and spam and spam and sausage and spam and spam and spam and spam and spam and for reasons I will never understand uh in the cafe where the clip said there is a
chorus of Vikings who sing the spam song and I put the Monty Python clip in the video and then I thought I'm going to be annoyed with a bunch of loud hairy men we don't need a video clip we can do this one live
and yeah so spam email as a reference to that sketch where the noise of the spam spam spam just drowns out everything else that's going because that's what happens with junk mail we have these unsolicited emails and the reason we're getting them is because email was
invented by hippies and hippies suck at security now things just kind of rattled on a little while you know 1980s sort of came and went and for a long time in the 80s every mail server on the internet was on
open relay because peace and love man and Good Vibes and yeah we'll help you relay your mail to anywhere then the 90s comes along and what happens in the 90s is a lot of people who've never had email or the internet before they get the internet at
home and the thing you've got to realize about what was happening in the 90s is that stuff that was impossible is now happening daily someone gets their first email account and a couple of days in they get an email from their
granddaughter in Australia with a photo of the new baby and that was impossible like a year previously you could not do that it took three weeks to send photographs from Australia to Europe suddenly those things happen they're
like ah the internet's absolutely magic it's brilliant it's amazing the next thing in their inbox is Bill Gates saying he's going to give them a million dollars and everyone gets a free trip to Disneyland and they're like well the last thing was too good to be true but
then it happened maybe this one's too good to be true but maybe this one will happen as well we had no frame of reference as a population for what email couldn't could not do and what was possible on the internet and so junk
mail scams spams bad medication became an absolutely Rife problem most of it sent through open email relays now there were four approaches that we took to try and stop this one of them
was they made it against the law the controlling the assault of non-solicited pornography and marketing can spam Act of 2003 now as you can tell this was a resounding success because
after 2003 there was no more junk mail right yeah now it turns out that quite a lot of people aren't in America and quite a lot of people who are in America don't give a crap and even the ones who do it's very very hard to
prosecute them because it's ridiculously easy to comply with the specifics of the can spam act so law enforcement did not deliver a solution the second approach we tried was client-side filtering you have some
software that sorts out your email now most of us use that I use that you know I have Gmail and it puts stuff in my inbox and updates on social and that kind of stuff I'm not going to talk about that today because as a developer trying to send you an email what you're
doing with it after I've delivered it that is your problem and I don't care how you've got it set up you can filter anything you want there's very little consistency that's on you to get set up the way you want to do it there's maybe some interesting stuff here about trying
to training machine learning systems but like that email that I sent at the beginning of this session that all of you said yes I want to receive this email quite a lot of it went to junk how would you train an algorithm to know that you're in Oslo right now doing a live demo and this is important so when
I'm going to talk about client-side filtering what about verification one of the problems with unsolicited email is that you are not accountable there's no way of tracing the junk mail back to the person who sent it and if there was it
would be easier to find those people and tell them to stop what we need because we need peace and love and Authentication and the problem is they didn't include that because the hippies didn't think we were going to need it so when SMTP was
created it has no support no usernames no passwords no security no encryption absolutely nothing and so we looked at the SMTP and we went oh simple mail now we didn't want to make the complicated mail transfer protocol that might have
put people off so instead we made the extended simple mail transfer protocol now this was actually invented at Jim Henson's Creature Shop because uh the difference between SMTP and asmtp is the
SMTP you say hello and esmtpc you hey hello which is close enough so we're going to do an extended smte SMTP transcript we're going to tell that into my fun with email server and we're
going to say hello Dylan beatty.net and it's going to go oh cool you speak the like cool kids language look this is the stuff that I can do for you I support authentication using the plane system or the login system or the crime md5 system
I support start TLS which means we can switch this whole thing to a secure Channel and I do enhanced status codes and we're like all right well before we send our username password we probably want to switch to the secure TLS thing so we're going to start TLS by the way
lots of people still call this SSL because SSL was invented at Netscape in the 90s and everyone went yay secure sockets makes the internet secure SSL stopped being a good idea in 1999 but by then we'd all got used to talking about
SSL certificates these days it's actually TLS transport layer security so we're going to send start TLS and it's going to go yeah okay go ahead now we're secure start again on a secure Channel hello dylanbt.net there's our SMTP
handshake and now we can say this is our username and password meshed together base64 encoded so it's technically not encrypted it's just encoded but that's going to say authentication successful and now we're in so we've said this is
us look we're a paying customer please relay email for us but the fourth approach is let's stop the bad people being able to get onto the internet in the first place now there's a guy called John Gilmore John
is a very interesting person he's one of those people who I agree with everything he does right up to the point where I think he turns into a bit of a dick and then he kind of stops just past that point John was employee number five at
Sun Microsystems he was one of the founders of the Electronic Frontier Foundation he is a uh I've seen him described as an extreme libertarian cipherpunk activist and uh the most famous quote I've seen from John is this
one the net interprets censorship as damage and Roots around it if you start blocking ports because you don't like what people are doing the internet is designed to find another way rounded and you know he has taken this philosophy to
an extreme he runs an open mail relay if you go to hop.toad.com it will accept email from anyone on the planet on Port 25 and it will deliver it doesn't care who you are doesn't care where you came from which is kind of the libertarian
ethos in a nutshell like you shouldn't have to show your ID to be able to take advantage of Public Services which I kind of agree with except where the internet is concerned because it becomes too easy to abuse it now I really wanted
to use toad.com in the demos I put together and in the live demo we did at the beginning I couldn't because none of the networks that I'm able to connect to would let me relay mail through hop.to.com they just
said you can't do it I can't connect if you put a system in Azure on an Azure VM or you know an app service and say hey connect to that system on Port 25 says no you can't connect to anything on Port 25 don't send spam Google Cloud same
thing Amazon web services same thing most uh you know dial up most connected Wi-Fi most public Wi-Fi can't connect to anything on Port 25. so we can't use this open relay approach to send mail
the only way we've got a sending mail is to find someone we have a relationship with and use our username password to connect to their server so that's fine when we want to use our internet provider or our email service to send
mail to the rest of the world but what about receiving it because the beautiful thing about email is that anyone on the planet can send me a message and I should be able to get it and this is what sets it apart from things like WhatsApp and Twitter and
telegram there is no multinational corporation whose algorithm is showing me things they think I want to see I should be able to guarantee that anyone can send me an email and I will receive that email be able to look at it which
means it needs to be wide open if I'm hosting my mail on Gmail I can't expect everyone talking to me to set up a Google account to be able to do that likewise Microsoft so when a mail arrives When someone knocks on the door
of fun with email and says hey I got an email here for Dylan what can we do to determine whether that's legitimate well first we can look at the envelope we can look where did it come from where was this mail sent what was the network address
it came from does that appear in any of the public databases of known spamming Network addresses now this system kind of works up to a point it stops a lot of spam I'm not going to pretend it doesn't
it's also a massive pain in the ass because anyone who wants to give you a bad day can just find your email address and they can report it for spamming and these places most of them don't verify anything they're just like okay well we'll add you to the list and if it's a
problem the system administrator can come on and they can request to be taken off the list again now one people will do this just for for a lot two people will block entire blocks of IP addresses if your ISP has given the
next address along to someone who's sending junk mail you could find you've been blocked through no fault of your own you can only deal this from these things most of them say once every 48 hours there's about 10 or 12 of these services this is why I stopped running
my own mail server it's because I got fed up of going onto these forms and saying no you've blocked the entire network block again and I'm not sending spam I'm doing everything right stop cutting me off so this system doesn't
work it's too easy to abuse and it kind of relies on your innocent until proven guilty but proving you guilty just means someone went it was them and then we're gonna block you for the rest of your life unless you beg and grovel and try and get reinstated
let's talk about a better system now there's a thing called SPF the sender protect send a policy framework and this is beautiful because it's built on top of the things we had at right at the start of the 80s it's built on SMTP
and it's built on DNS if you get a the Google admin toolbox and you look up fun with email on their dig toolbox you can click through the records there's a thing here called txt records and in there you'll see this record here it
says txt time to live is short-lived and value equals what value on this thing is this V equals SPF one version is the sender policy framework version one then I got a rule here that says okay this
email that is allowed to send email for that domain fun with DOT email spf.smtp.com that's allowed SPF dot messaging engine.com that is allowed and anything else I got a little squiggly
thing says this might be suspicious so what that qualifies them in a plus means a pass this rule matches you can deliver the email question mark means don't care now that's not much use in production but it's really good for testing stuff because you can create your SP PF
records and see what matched without it changing any Behavior a tildeer is a soft fail flag it but deliver it and a minus if I put a minus in there now the plus is the default so normally you don't see those the records actually
look like this but that minus there I can put a rule that says look if anyone anywhere on the internet gets an email that says it comes from fun with DOT email and it doesn't match that rule you can just ditch it like that's fine with me I'm telling you that's junk I
guarantee these are the only people sending email on my behalf now there's a couple of other systems we're not going to go into today which do public key cryptography there's a thing called a decim domain key identified mail and there's a thing called dmarc which is
the same but it then sends you reports of anyone who's trying to pretend they're you and spam stuff but this idea of building records on top of DNS to manage email this is getting some traction this is working pretty well now
when we sent that a SMTP transcript earlier we basically sent a message just said hello like once we've thrown the envelope away we don't know where it came from we don't know who sent it we don't know who they think they sent it to we just know that it said hello we
want to send messages which are slightly more useful than that and so when we connect to our mail server and we get to the point where it says Hey data send us your message what we're going to do is we're going to send a block of headers this is who it's from this is who it's
to this is the subject this is the email body it's going to say okay 250 cubed we're going to say quit it's going to say buy and that is going to work brilliantly we get nice formatted email and this works great as long as
everything that you have to say can be expressed in seven bit ASCII because SMTP is built on ASCII now seven bit ASCII was a brilliant idea in the United States of America in 1960s so was
Richard Nixon and leaded gasoline like you know we can do a little better but backwards compatibility you can't ship a mail system which can't send mail to the people who are still old system so here
in 2023 we are still stuck with seven bit ASCII as the only way of sending anything over an email platform and so we have to use mime the multi-purpose internet mail extensions
what mime is is a way of taking just about any kind of content and turning it into seven bit ASCII sending it over the wire and rehydrating it at the other end now mine was actually pretty clever it's a very very elegant system so we got our
headers from to subject data's this now mine version 1.0 mine's supposed to have other versions it was supposed to have 1.1 1.5 2.0 3.0 but they never figured out how that was
going to work so basically we are stuck with 1.0 being a floating Point Boolean that if it's 1.0 yes and anything else means this is probably not valid mine now we're going to throw in a content type here we say this is a multi-part
mixed message and the boundary string is this now this can be anything you like as long as it doesn't appear in your message anywhere we're going to put in a message now this this is a multi-part message in mime format this is for old
old email readers that don't know about mime there aren't very many of them around anymore now we get to the good stuff so that's the header on our message I'm going to put in a boundary it says we're starting a block here and this
block is multi-part alternative that says we're sending you the same message but in two or three different ways so we've now got a different boundary we're going to say right this is the text plane version of this email hello we're
having fun with email and then we are going to have the HTML version hello we're having fun with email this bit is an HTML format and then we're going to close out that boundary block and then we're going to close out the other one so now we've put another container so
mime is actually a tree structure you can have containers inside containers inside containers and in this one we're going to say hey we're going to do an attachment here this is fun with email.png it's base64 encoded because that's 7-bit ASCII safe that's the file
name it should be an attachment so this should show up as a little paper clip in your email client and then we're going to have this massive great big long block of base64 encoded stuff now this is all well and good these are standards you can go out you can study
them you can figure out how they work you can write your own implementations if you are actually building an application that works with any of this stuff you have a whole bunch of challenges ahead of you not least
because email is an open standard and when you create code to send somebody a message you have absolutely no idea what device they are going to be using to look at that they might be reading it on
Yahoo they might be reading it on Outlook or an Android phone they might be using a screen reader there is a whole raft of different things out there and these days most email is HTML because HTML looks nice and because our
marketing teams want us to send HTML emails so we can have buttons and colors and links and those things in it HTML email is horrible it is one of humankind's terrible terrible mistakes
good news is that there are Services out there that mean you don't kind of have to touch the icky bits quite as much as you used to one of these is a thing called the mailjet markup language explain so mailjet is an email marketing
and relay company and mjml is their way of writing HTML emails there's another one which is called Foundation foundation for email then I have a logo but they do have a picture of a squid or a cuttlefish so I thought all right
we'll put that in here now fundamentally what these things do is they give you a language which looks a bit like HTML so I'm going to write an email and this is mailjet markup language so I got a male jet head with a preview I've got a
body with a section and a column now this is it looks like HTML it's an XML kind of compliant dialect it's not HTML what we're going to do is we're going to run this through the mailjet compiler because the HTML that clients actually
understand looks like this it is a sack of Horrors if you read this closely what we've got there is we've got a table inside a table inside a table and if you're on a
Microsoft Outlook or Internet Explorer platform you get two extra tables for free because the blue comments there are the Microsoft if this is Microsoft Outlook then include this extra chunk of markup now I know people who used to
write this stuff by hand I used to write this stuff by hand platforms like mailjet have made the whole thing so much easier to manage one of the things that they don't really do is general purpose templating they are transpilers
they have no idea of behavior they don't work like a view engine like Razer or something so what you can do we're not going to go into the specifics in this talk but you can have projects where you take mailjet and when you compile it you turn your mail jet into Razer and then
at runtime you put a model into your razer and you get populated HTML with individual order items and stuff at the other end if you want to see how that works I got some code up on GitHub you can go and poke around with it so you get to the end of it
and it works on your machine which is kind of pointless because why would you send yourself email you know how do you test this stuff if you're developing A system that is going to send emails there's a couple of rules that I absolutely live by so the first
one is you want an end point on your website somewhere where you can see what the email is supposed to look like um the NDC tickets platform that we use in London and Copenhagen and Porto
um every single order in that there's a link at the bottom where you can say hey show me what that email's supposed to look like now one this is useful just from being able to check details on something also when you're designing this stuff you can just keep refreshing
that page because it's way quicker to reload the page or to use like a hot reload or you know web plugin then I have to send yourself an email wait for it to come through go to your inbox open it up ah still doesn't look quite right round we go again so this gets you to
the point where it looks right next tool that I use a thing called paper cut now Papercut is a Windows application that just sits in the corner of your screen and your system notification tray and it's an SMTP server that doesn't send email every email that you send it
intercepts it and it goes yeah we got it here it is and it lets you go through and inspect all the properties so you can see your message in a browser view there that's pretty good you can see your message headers what exactly got
filled in by the code that you were using to generate your mail messages you can see the body you can see the individual sections how the mind format breaks down you can see the raw message content for the whole thing so if something doesn't look right this is a
great tool for diving in and figuring out how that works so you've got works on my machine in a browser you've got works on my machine with an SMTP relay let's see if it actually works on the internet
mail trap is a fantastic bit of kit that I've used on dozens of projects over the years mail trap again it pretends to be a mail server so you send your mail via mail trap and instead of sending it it intercepts it gives you some of the same
features that Papercut does so you can see different representations but then it's got tools like this so that's your HTML that's your HTML Source that's your raw message body this is your spam analysis this is all the reasons why
that email might get picked up as junk now if you look at this one real closely I got a 0.8 on there and I got a whole string of zeros but the 0.8 is because the domain name that I sent it from is less than 28 days old
and this is the kind of thing they pick up on if you're warming up a project for a big client who's going to send a lot of email you need to start sending email early because it can take a couple of months for their outgoing relay their domain name and everything to be trusted
and established by Services all over the internet I'm going to do the HTML check on here now this is going to say look this is the stuff that you ended up using in that email and this is where it's not going to work like you've got a thing in there so Gmail's mobile Webmail
is flag two issues that won't render properly on that all the way down to AOL and iOS and the FastMail desktop Webmail all these kind of platforms now let's assume we have done our due diligence we've tested all these platforms we've
designed the email the client signed it off and we've tested it on Gmail iOS Android Outlook Zoho proton FastMail Yahoo Thunderbird Eudora Vivaldi mail and Pine and then we get that WhatsApp from the client go and the email doesn't
look right and you're like what like send me a screenshot and they send you this and you go I absolutely believe that we all have the right to consume digital media in a
format of our choice using our chosen fonts color schemes devices layout screen reader accessibility whatever I also think that dark mode email is an Eldritch Horror that should never have
been allowed to live I wanted to find a friend of yours who's a graphic designer I want you to ask them to design you just ask them to design you a book cover or something and then mention in the brief oh yeah by the way uh we don't know yet if it's going
to be on black paper or white paper is that a problem because they're probably gonna slap you because the first thing you think about when you design things like marketing collateral is what are we going for here is this like dark with
light or is this light with dark giving the client control over that on something like email just opens you now there are CSS rules and selectors that are supposed to let us do multiple versions of everything they don't
actually work terribly reliably they're getting better but the situation we have now is if you've got transparent pngs in your email you just need to come up with something that works on both and often if it works on both it doesn't look great on either of
them you end up having to put like little outlines around your logo so they're still visible and they don't disappear into the background and this is you know I find it immensely frustrating as a developer whose job is
to maintain and support these kinds of things but I also think it's incredibly healthy and the reason why so many of these things exist validating email and is this email address valid
email is a moving Target it's not fixed nobody controls it it isn't like someone says email version 2 will be active as of today it is a distributed Federated system where
different people are free to implement parts of that protocol however works for them they do what works for their systems an email address is valid it worked yesterday that doesn't mean it still works today email that rendered
properly last week may not render properly this week something that didn't work last week someone might upgrade that client it works suddenly and as challenging and frustrating as this is as a developer I love it
because no one can mess with email Zuckerberg and his cronies can't come along and kick you off email the way they can kick you off WhatsApp or kick you off Instagram or Facebook or get you banned from Twitter or you're not
allowed to use signal anymore I can set up a server and you can set up a server and we can talk to each other and we can send each other email using Open Standards and open protocols and it is one of the last systems left which
allows us to do that without a multinational corporation wanting to harvest our data verify our accounts make sure we are who we say we are use algorithms to sell crap to us and throttle the whole process if it decides
it doesn't like us now that is where we are going to end the version of this talk that is going on to YouTube
End of transcript