PCHelper.com - Technology Resources

PCHelper.com · Joined: 02 Mar 2007 Posts: 36

Explain how SPF works in 1 minute.

Domains use public records (DNS) to direct requests for different services (web, email, etc.) to the machines that perform those services.All domains already publish email (MX) records to tell the world what machines receive mail for the domain.
SPF works by domains publishing "reverse MX" records to tell the world what machines send mail from the domain. When receiving a message from a domain, the recipient can check those records to make sure mail is coming from where it should be coming from.

With SPF, those "reverse MX" records are easy to publish: one line in DNS is all it takes.

Do I have to publish spf for each of my smtp servers?

No. You should publish spf records for each and every domain you wish to protect from being used by spammers/virusses. If, for example, your domain is somedomain.tld and you furthermore have a subdomain www.somedomain.tld registered, you would publish for both somedomain.tld and www.subdomain.tld (the latter probably being set to "v=spf1 -all"). Note that you will have to publish for each and every A record, including any wildcard (*) or @ entries in your dns.

Can I bypass spf checks for certain servers?

Can I whitelist hosts on my dmz without making their adresses publicly available?

Often, you will want to allow certain servers to send mail through your smtp server. For example, if you have machines on your dmz that must be able to send status messages or you have some machines on your LAN that need to send out mail from your domain. In this case you will generally not want to publish these services in your public spf record (eg. "v=spf1 ip4:10.0.0.0/8 a -all") since this information is potentially valuable for hackers.

This can be solved in two ways. First of all, many spf implementations provide you with an option to put these adresses in a whitelist. This is just a list of hosts, which only needs to be available to your smtp server. A second option is to implement a 'local policy'. For details, consult the documentation that came with your specific spf implementation or search the list archives for 'whitelist' and 'local policy'.

My dns entries end in .local, how do I publish txt records for my domain?

You can not use your local dns server to publish txt records for your internet domains. You will have to use the dns server that serves your domain to the internet. Contact your dns provider and/or your hosting company, or look in the webpanel your dns/domain hoster provides.

What does SPF actually DO?

Suppose a spammer forges a hotmail.com address and tries to spam you.

He connects from somewhere other than hotmail.

When his message is sent, you see MAIL FROM: <forged_address@hotmail.com>, but you don't have to take his word for it. You can ask Hotmail if the IP address comes from their network.

(In this example) Hotmail publishes an SPF record. That record tells you (your computer) how to find out if the sending machine is allowed to send mail from Hotmail.

If Hotmail says they recognize the sending machine, it passes, and you can assume the sender is who they say they are. If the message fails SPF tests, it's a forgery. That's how you can tell it's probably a spammer.

Does it protect the "From:" header field?

SPF was designed to protect the envelope sender. That means the return-path that shows up in "MAIL FROM", and to a lesser extent the HELO argument that is supposed to be an FQDN.

The vast majority of SPF implementations today use the return-path as the subject of authentication and do not get involved with the header "From:".

Protecting authorship information is an important goal. However, the technical issues associated with protecting the "From:" header are much more numerous and challenging. The best way to protect the header "From:" is by using a cryptographic signature such as S/MIME, PGP, or (when it is released) Yahoo DomainKeys.

If you want to use the "From:" header as the subject of authentication with SPF, you need to be familiar with the following:

mailing lists
/etc/aliases-style forwarding
MUA "resend this message to"
web-generated email
the Sender header
the Resent-Sender and Resent-From headers
Does SPF break email forwarding?

Stuart Gathman's opinion is recorded at http://archives.listbox.com/spf-discuss@v2.listbox.com/200410/0488.html.

Yes, it does. You'll have to switch from forwarding, where the envelope sender is preserved, to remailing, where the envelope sender is changed. But don't worry, we're working on providing SRS patches for the four major opensource MTAs, so that when you upgrade to an SPF-aware version, this problem will be solved also.

If your forwarding runs through a commercial service like pobox.com, you shouldn't have to do anything. They have to change with the times, and perform the above rewriting automatically for you. SRS is a structured standard that helps them adapt.

What if I don't control my DNS records?

More and more DNS service providers are supporting TXT records. You can find a list of them at http://www.telefonica.net/web/news/spf/. If yours is not on the list, write to them and ask for TXT support!

SPF doesn't really STOP spam, does it?

We've heard the complaints -- Spammers can always get throwaway domains, etc.
At a high level, the answer is that we're moving from one paradigm to another: from "assumed innocent until proven guilty" to "assumed guilty unless proven innocent". The Aspen Framework brings two important tools to bear: reputation and accreditation. (A cartoon guide is available.)

We agree that throwaway domains will be the next step in the arms race. We can counter with:

fast automated blacklisting using spamtraps and attack detectors
simple reputation systems based on factors such as
age of domain according to whois
email profile of domain, eg. "too many unknown recipients"
call-back tests to see if the sender domain is able to receive mail.
The reputation system can advise a receiving MTA to defer or reject.
legal methods following the paper trail of who paid for the domain.
Here's an example of automated blacklisting in action:

A spammer spams.
The spam comes from an SPF-conformant domain.
That domain is on a widely published sender-domain blacklist.
The MTA rejects the message.
That domain is a throwaway, just-registered domain, and does not yet appear on blacklists.
The spam gets accepted by unsophisticated MTAs which do not use other traffic-analysis methods to impose a crude reputation system on unrecognized senders.
The spam also gets accepted by automated spamtraps.
The spamtraps add the domain to the blacklist.
(advanced) Some time later, the user checks email. Immediately before the display phase, the MUA re-tests the message against the blacklists, and discards it.
Thanks to the greater level of sender accountability, lawsuits may begin against the spammers, and registrars may be subpoenaed for domain owner information. SPF strengthens administrative and legal methods.
The spam comes from a non-SPF-conformant domain.
Initially,
Most legitimate mail will fall into this category.
Normal content filters get to do their job.
The usual false-positive/false-negative results apply.
Later,
Most legitimate mail will be SPF-conformant.
Some legitimate mail will not be SPF-conformant.
SPF-conformant receivers SHOULD receive non-conformant mail but MAY choose to perform additional filtering on it.
Eventually, as SMTP improves its immunity to spam, we hope spammers will get discouraged.
If the volume of spam decreases, legal and administrative approaches become more effective; right now they are simply swamped. If there are only 10 spammers in the world, law enforcement can focus on catching each one. If there are 10,000 spammers, law enforcement throws up its hands, calls it a societal problem, and says it doesn't have enough resources to tackle it.

The spam domain was registered with a domain registrar.
If the registrar is cooperative, we can find out from the registrar who the spammer was; and the registrar can stop accepting their registrations.
If the registrar is uncooperative, or if a spammer buys and runs a registrar, we can default-blacklist all their domains, in a political move similar to SPEWS's approach.
Alternatively, since spam is becoming increasingly illegal, we can subpoena the registrar to find out who registered the domain, and sue the spammer directly.
If the spammer registered the domain using false information, we can still go back to the credit card.
If the credit card was stolen, that's a crime which can be addressed using traditional means.
(20040702) Scott Kitterman has posted a suggested refinement to the above plan.

Can I run an SMTP server on my laptop.

If you run a personal domain, you can either not publish SPF records at all, or set up "v=spf1 ?all" for your domain, and you'll be able to send mail from your laptop no matter where you are.

If you are the customer of an ISP that publishes SPF records, your ISP should provide you with an SMTP server that you can authenticate to, using either POP-before-SMTP or SASL AUTH. Or you can ask them to exclude you from SPF using the a user-specific "exists" mechanism.

What does the logo mean?

I'm looking at an envelope. The return address claims to be from Blue Blob. Let's compare that to the postmark. Yup, the postmark says the message comes from Blue Blob. That means the return address is probably not made up. The message is good.

In email, the envelope sender, also known as the return path, is the return address. The SMTP client is the post office that applied the postmark. The blue blob is the domain name. If they don't match, maybe the message is a forgery. Maybe a bad guy made up the return address to fool you into opening the envelope. Maybe a spammer forged a Paypal return address to try to get your credit card number.

You can display the SPF logo on your website. Choose one that's right for you.

Why should SPF succeed when similar proposals have failed in the past?

The spam problem was never as bad in the past as it is now.

People are willing to put up with a lot more change and pain.

SPF is backed by pobox.com, which is willing to fund the development of SPF-enabling patches to MTAs, and to host the default fallback lookup domain for the purposes of guerilla adoption.

People who have shown interest in supporting SPF include Qualcomm (makers of Eudora), Tim O'Reilly (publisher of geek books), SpamAssassin, ActiveState (makers of PureMessage), MailArmory, Declude JunkMail, and others.

What is the "best guess" function?

Mail::SPF::Query provides a best_guess method, which pretends the domain had declared "a/24 mx/24 ptr". This is remarkably good at detecting unforged messages from domains which have not yet implemented SPF. It helps reduce false positives.

I've set up records, how do I test?

Once you've set up records, try them out.

You can also send mail to spfenabled@pobox.com and see what happens. If you send mail from an unlisted server it will be rejected. Please don't make up bogus addresses if that would cause random third parties to get mysterious bounce messages.

Is there a mailing list?

There are four actually. The big ones are for discussion and announcements. The little ones are for getting help with setting up SPF, and for developers who are implementing SPF related software like client libraries and SRS rewriters.

The lists receive between 5 and 50 messages a day.

There is a dedicated page all about these mailing lists and how to subscribe.

How do I search the mailing list archives?

Use the allmighty google: if you are looking for 'foo bar' in the spf-discuss list, then google for 'site:archives.listbox.com spf-discuss foo bar'.

The Demon Question: What about subdomains?

If I get mail from pielovers.demon.co.uk, and there's no SPF data for pielovers, should I go back one level and test SPF for demon.co.uk?
No. Each subdomain at Demon is a different customer, and each customer might have their own policy. It wouldn't make sense for Demon's policy to apply to all its customers by default; if Demon wants to do that, it can set up SPF records for each subdomain.

So the advice to SPF publishers is this: you should add an SPF record for each subdomain or hostname that has an A or MX record.

Sites with wildcard A or MX records should also have a wildcard SPF record, of the form: * IN TXT "v=spf1 -all"

(Thanks to Stuart Cheshire.)

What about Source IP Spoofing?

Spammers may spoof entire TCP sequences to get their mail out.
Knowledgeable people consider this unlikely. If you think this is a concern, you are invited to demonstrate an attack.

Whoa, that's an underscore in _spf.

I thought you could only have letters, numbers, and dashes in domain names, no underscores.
Apparently underscores are now okay. See RFC2181 section 11.

Underscores create a new secret dimension: that's the only way to guarantee you don't collide with an existing subdomain. SRV records use underscores for exactly this reason.

What about the simple case (implicit mx rule)?

When a domain has no MX records, we assume that an A record will suffice.
SPF can be similarly intuited. Mail::SPF::Query provides a "best_guess" method, which pretends the domain has "a/24 mx/24 ptr" defined. Even in the absence of SPF data, we can suggest that a transaction is legitimate, (though we can't suggest that it is not legitimate, only that we don't know). And finding legitimate transactions helps other antispam approaches reduce false positives.
What about going one step farther and listing all valid sender usernames as well as domains?

That's what the Exists mechanism is for.
Shouldn't you use SRV or EDNS0 records?

SRV records were, roughly speaking, meant for letting people add this sort of thing to DNS without having to add new record types. See also RFC2761, Extension Mechanisms for DNS.
Yes. But SRV records are hard for people to understand, and TXT records are easy. Fast widespread adoption is our goal. The Right Thing To Do is to get our own RRtype, and we will apply for it. We just don't expect to get it anytime soon.

(for SRV records, see http://dqd.com/~mayoff/tools/djbdns/make-record.adp)

My existing spam filters work well enough, why do I need spf?

All this seems like a lot of work. The spam filter that runs in my MUA or MDA catchs 99% of spam. Why should I care?
You may not care as an end-user, but your ISP does. Your ISP has to pay for the bandwidth of delivering all those messages to you. Your ISP spends time and money getting them down the pipe to wherever you delete them. And your ISP passes these costs on to you. If your ISP can block forgeries without having to download them at all, ultimately, you benefit.

Another reason to participate is that you can protect your own domain from being used by virusses, spammers and the lot.

What about the cracked, open-proxy DSL machines that are spam sources today?

Spammers are increasingly using compromised machines on fast connections to send spam.
Broadband providers will have to either force port 25 to go through their servers where they can keep an eye on traffic, or suffer the reputation costs of having their "forged" domain name in the envelope sender line. I know people who block Hotmail, Yahoo, and AOL simply because those are the most commonly forged domains; that hurts their legitimate users and ultimately it hurts them.

Given a large enough spam run, SPF becomes a DDOS against the forged domain!

Each SMTPd in a spam run may send a DNS query to the forged host's nameservers. At a million SMTPds, that's 100 megabytes of traffic!
DNS queries are still smaller than bounce messages. And most SPF lookups can be cached; only the relatively uncommon exists mechanism doesn't benefit from caching.

What about mailer-daemon messages whose sender address is <>?

When the sender address is <>, you don't have any domain to look up!
There are two solutions, one strong, one weak.

The weak solution uses the domain name presented in the HELO/EHLO command instead. This may work to catch careless spammers.

Careful spammers will find a way to get around this: consider a DynDNS-registered broadband host that has been compromised. The strong solution accepts the message, takes it apart, determines if it is in fact a bounce, and if it is a bounce, it extracts the Message-ID of the original message, and if the the Message-ID did not in fact originate from the system, it discards the message. This takes a bit of work, but in a world where automated spamtraps do most of the work and would add the hostname to known-spammer lists anyway, a spammer might as well use a non-null sender address anyway.

In either case an MTA should reject messages from null senders that have more than one recipient.

What policies besides SPF should I put in my MTA?

The SPF draft says that SPF checks are only meaningful in certain situations.
MTAs can block a lot of spam even before SPF checks occur.

Here are some suggestions that will block a lot of spam. Only messages that get past all these rules need to be SPF tested.

The envelope sender domain must have either an A or MX record.
The A or MX record of that sender domain must not be in:
0.0.0.0/8
10.0.0.0/8
127.0.0.0/8
172.16.0.0/12
192.168.0.0/16
The connecting client IP address must have a PTR record.
The HELO hostname must be a well formed FQDN that has an A record.
Note that rules 3 and 4 are often violated by legitimate but clueless domains who don't pay attention to these kinds of detail.

You can configure these settings in Postfix at http://www.postfix.org/uce.html

I'm an ISP. What should I keep in mind?

Vanity domain holders may point to you using an include: directive.
You should support SASL AUTH on ports 25 and 587. When your users need to phone home from an Internet cafe, port 25 may be blocked, but port 587 often isn't.
You may want to set a default of "unknown" by saying "?all" instead of "-all" for the first few months, to accommodate legitimate users who have been sending mail through somebody else's servers.
You can tell who these users are by adding an "exists:%{l}.%{i}._spf.ISP.COM" record, and grepping in your DNS server logs to see who's mailing through what machines. Then you can contact all nonconformant users and tell them to use your SMTP server.
You should never publish -all for customers domains without prior notice. Keep your customers informed about your spf roll-out, this will prevent them from being unpleasantly surprised by mail that suddenly is not delivered due to spf FAIL results.
How do I test/validate/check my record?

A variety of SPF checkers have been developed. Try:
the standard reference why.html page
Infinite Penguins
DNS Stuff (home of Declude JunkMail)
Does SPF stand for Sender Permitted From?

It used to, but that name turned out to be too slippery for the human mind to grasp, so we in early February 2004 we changed it to the more accurate "Sender Policy Framework".

Trivia: for marketing reasons we picked the acronym before we picked the expansion. First we went to Google to see if SPF already stood for anything outside the sunblock world. It is the name of a networking concept --- Shortest Path First --- but that's okay; every good acronym deserves at least two expansions. So we chose that name for the brand and today we're #1 on Google for "SPF".