CSV vs SPF vs DomainKeys (by alaric)
There are a number of technologies vying to protect us from spam, by providing channels for legitimate senders of email to prove that any given message comes from them, thus allowing them to build a reputation (and get whitelisted) while spammers' domains and/or mailserver IPs get bad reputations, and they can't spoof messages to steal the good reputations of others.
This is entirely separate to tools like SpamAssassin, which try to analyse emails once they've arrived, and allow a recipient to judge their spaminess regardless of who they've come from.
It's also entirely separate from things like HashCash, which work by letting senders attach a mathematical proof to their message that shows they've invested a second or so of computer time, thus distinguishing them from people who send thousands of messages per second.
CSV, SPF (aka Sender ID) and DomainKeys, work in three different ways.
CSV
CSV is all about checking that the sending mail server has a good reputation. The idea is that when a mail server connects to another mail server to send an email, the receiving mail server considers the name the mail server claims to be (sent in the SMTP HELO or EHLO). It firstly does a DNS lookup on the name and checks that the IP address of the sending mail server is one of those returned from the DNS. This checks that the mail server really is the one it claims to be. This is a standard check.
It then consults the DNS again, this time looking for special records placed there by the administrators of the domain containing the mail server. These "CSA records", if present, list which servers in the domain are allowed to be mail servers.
This allows an ISP, for example, to state that only their certified mail servers are allowed to send email - and that their customers are NOT allowed to send email themselves. Bear in mind that this check is done on name not IP address. So an ISP customer, if they had their own domain name, could set up a record called mail.example.com pointing to their IP at the ISP, and use it to send email, using their own example.com reputation. However, if they try and configure their mail server to claim to be customer123.isp.com, using their ISP's reputation, they will fail.
It also allows a virtual mail domain - such as my very own snell-pym.org.uk - to contain records stating that there are no mail servers in that domain. My email is sent by warhead.org.uk mail servers, but spammers (especially ones sending spam to me) might try and claim to be from snell-pym.org.uk mail servers.
The plan is that this check will be combined with a lookup in a reputation database to see if the domain has a good reputation (eg, it's in a whitelist) or has a bad reputation due to known spamming behaviour.
It leaves it up to the ISPs to block spoofing - CSV will happily let an email in claiming to be from president@whitehouse.gov as long as the mail server sending it has a good reputation. Therefore, ISP mail servers will need to be told to reject mail not claiming to be from their customers. This may be a pain for people who, working from home, send an email from bob@example.com via their ISP's mail servers, who will either need to ask their ISP to add bob@example.com to their list of allowed from addresses, or set up MSA to be able to send emails via their work mail server, from home.
SPF
SPF takes a slightly different tack. Instead of attaching reputations to domains of mail servers, it instead allows reputations to be attached to the domain used in the email From address.
To illustrate the example, when somebody sends an email, the from address might be bob@example.com, and the email might go through their ISP's mail server, mail.isp.com. CSV would check that the mail server actually is mail.isp.com, that the managers of isp.com are willing to take responsibility for email from mail.isp.com, and that isp.com has a good reputation. SPF, on the other hand, checks that the managers of example.com list mail.isp.com as being allowed to send email claiming to be from example.com addresses, then considers whether or not example.com has a good reputation.
The way it does this is, again, to put special records in the DNS. This time, there is a record in the example.com DNS listing mail servers that are allowed to send email for that domain.
If bob@example.com wants to send emails from home, then he talks to the example.com administrators to have his ISP's mail servers listed in the SPF records. With CSV, he would instead have to ask his ISP to make their mail server accept email from his address. Or, with either SPF or CSV, he could use MSA to send emails via his work mail server.
However, SPF is somewhat more complex to set up and use than CSV, for various reasons.
DomainKeys
DomainKeys takes another tack. With DomainKeys, the act of sending a message is changed somewhat; the sending mail server has to compute a digital signature of the message content and some key headers, which it then places in a special header in the message. Creating this digital signature requires knowledge of a secret key.
When the message is received, the signature can be checked by looking in the DNS of the domain name of the from address for a special record, which contains a public key. The public key can be used to check that the signature was made with the corresponding secret key, yet without revealing what the secret key is, thanks to some clever maths.
Similarly to SPF, this validates that the message indeed came from a mail server authorised to send emails from addresses in the domain. Unlike SPF, it does not require an exhaustive list of valid name servers to be maintained by the example.com managers, but if bob@example.com wants to send work emails via his home ISP, he really has to use MSA - it would not be sensible for the example.com managers to give a copy of their private key to Bob's ISP just so their mail servers can sign email from bob@example.com. Perhaps they can give a copy of the private key to Bob himself, and get him to set up special software on his home machine so that he can sign the messages before passing them to the ISP's mail server, but that would probably not be wise, since Bob's computer might be stolen, and setting up a new public and private key pair is more painful than just changing Bob's MSA password.
However, the big downside of DomainKeys is that it's complex. It involves actually processing outgoing messages to attach the digital signature, and it's a fairly complex process. Likewise, the process of checking the signature is quite complex. And a pair of keys have to be generated and distributed. It's a lot of work compared to the other two approaches.
Comparison
DomainKeys and SPF really provide the same kind of guarantee - that the message claiming to be from bob@example.com is indeed being sent by somebody authorised to do so - in different ways. CSV is different in that it doesn't care about the messages sent, it just guarantees that an incoming connection to a mail server is from an authorised mail server in the domain that it claims to be in.
All three then rely on some lists being published; for CSV, lists of domains that have mail servers in that are known to send or not send spam, and for DomainKeys/SPF, lists of email from address domains that are know to be or not be spammers. You get positive listings by gaining a good reputation, or by signing up with a whitelisting service like Bonded Sender or Habeas. You get negative listings by being reported spamming.
So which do I think is best?
Well, I'm not sure which will win in the long run. I think that SPF/DomainKeys complement CSV somewhat; both offer different kinds of protection, so it's best to have both. I've put both CSV and SPF records in my domains.
DomainKeys is a lot of work to do. I'd like to extend my MSA daemon to support the addition of DomainKeys signatures to outgoing email, but I'm waiting until somebody writes an implementation of the signing algorithm in Python (or that I can easily access from Python), rather than trying to do it myself.
The thing is, different recipients of email will be using different tests, so as a sender of email, ideally I should make sure my mail server domain is certified with CSV, my email address domain certifies those mail servers with SPF, and that my email is DomainKey signed with a public key appearing in my email address domain, to make sure I don't fail any of the tests.
And, right now, I don't actually check any of these things on my incoming mail; I need to upgrade to a newer version of Postfix so I can put in SMTP client validation plugins for SPF and CSV, then put a DomainKeys checker in the pipeline further down. All of which is a further pain.
And still, sadly, the vast bulk of email today is still sent without any such validation. Some of the big email providers - gmail, hotmail, aol, etc - are certifying their outgoing email with one or more of the above techniques, since spam costs them so much, but smaller companies and ISPs can't justify the time and effort. It's just perfectionist geeks like me that are doing it for the principle. But while lots and lots of valid email is not validated, recipients of email still have to pass incoming messages that are not validated through tools like SpamAssassin to try and fish out the spam.
Wed 13th Sep 2006 12:37 pm 
Subscribe
By andyjpb, Wed 13th Sep 2006 @ 1:08 pm
Hi,
SPF works on the Return-Path: header, not the From: header.
On Saturday I managed to end up in the return path of some spam. Since then I've been recieving between one and 4 hundred bounces a day. Now, all of the return path addresses are just a random string of characters @mydomain.com so I could just switch my catch all off and it would mostly go away. However, I've given out some addresses such as vendor@mydomain.com and now I can't remember them all and I don't want to loose the mail from them: I'm going to have to write a script to parse my inbox and workout which addreses I actually use.
Anyway, I spent some time on Saturday evening writing SPF records for my DNS. It hasn't eliminated the flow and I'm not sure if it's reduced it: not enough ISPs check for these things. However, apparently spammers are less likely to choose domains with SPF, etc records for use in their return paths so hopefully it will protect me in the future...
By alaric, Wed 13th Sep 2006 @ 1:15 pm
Yeah, I know about the return-path, I was just trying to avoid getting into too much detail. I didn't make any distinctions between RFC2821 MAIL FROM, RCPT TO, and the actual RFC2822 headers From and To, either, for the same reason. People actually wanting to implement SPF, CSV, and so on are recommended to read the specs rather than just do it from my summary 😉
Regarding SPF not stemming your flow: it may be that it's too late, the spams are all sent, and the bounces are just making their gradual way back to you. SPF records now won't help, in that case. But yes, I think a vanishingly small number of ISPs actually check any of these things. It's a lot easier to put SPF and CSV records in your DNS than it is to patch your MX mail servers to check that senders are sending mail properly, so hopefully it's worth people putting the records in, even though the return is minimal for now - because when a lot of domains (say, those behind 10% of legitimate emails sent) have such records there'll be a significant incentive to check for them on receipt. By then, the current releases of MTAs will probably have such support built in, too, not requiring any horrid patching.
By alaric, Fri 4th Mar 2016 @ 9:43 pm
Here's a more friendly introduction to DNS than the Wikipedia page I linked above, for the curious:
https://webhostinggeeks.com/guides/dns/