Aw hell. How many things do I have to set up just so that I can send e-mails from my own domain?
The effect of all this seems to be less "making e-mail secure" and more "making it so that only Google, Apple, and Microsoft can send e-mail successfully"
OneDeuxTriSeiGo 4 days ago [-]
DKIM2 and DMARCbis are actually the opposite of this. They are long awaited fixes of brittle and often broken systems that are designed to now make providing secure email easier rather than harder.
They both have fairly clean migration paths and resolve a lot of the annoying edge cases that currently exist with authenticating and verifying email.
binkHN 3 days ago [-]
Thanks for this, it made me actually read the posted content, but there's a lot of content to digest and a lot MTAs will have to implement.
doubled112 4 days ago [-]
And sometimes if you do everything right, it still doesn’t work.
Recently I checked the IP against blacklists, waited a few months, did all of the other things, and then found out Microsoft bounces my entire VPS’s IP range. Appealing did not help.
They intermittently block Cloudflare email routing IPs too. All of these security measures and still it comes down to the IP address of your sender.
1over137 4 days ago [-]
Is it an el cheapo VPS?
doubled112 4 days ago [-]
Is Cloudflare a cheapo VPS?
It is a cheap VPS, but it would still be nice if there was a way to know (not assume) beforehand.
> 550 5.7.1 Unfortunately, messages from [IP ADDRESS] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3150).
> Your IP(s) qualify for conditional mitigation.
Still blocked. The system is working as expected.
inigyou 3 days ago [-]
Cloudflare is one of the shadiest hosts around, almost as bad as the big three clouds.
TZubiri 3 days ago [-]
You can't have the cake and eat it too.
Either you use cheap infrastructure that attackers can buy by the bulk for sybil attacks.
Or you pay a reasonable price for a slice of an IP block that doesn't share its reputation with elcheapos.
doubled112 3 days ago [-]
No disagreement here. I had never planned to use it for email, but I was bored and I already had it when I got an idea.
jeffbee 4 days ago [-]
I would simply like to point out that 550 at SMTP time is not a bounce.
doubled112 4 days ago [-]
You get an email bounce back with an error code in it. Is there some other definition I'm unaware of?
Wait, that is probably from the local mailer, and otherwise the sender may never know about the rejected messages.
b112 4 days ago [-]
5xx are bounces, a perm failure. 4xx error codes are temp (retry) errors.
jeffbee 3 days ago [-]
Incorrect. A bounce is a delivery status notification generated by a mailer after it has already accepted a message for delivery. A 5xx permanent error is a refusal to accept the message in the first place.
TZubiri 3 days ago [-]
Is this standarized terminology in some RFC like SMTP? Or is it presumably some well established folk lingo?
b112 3 days ago [-]
They're referring to the bounced notification message, not the fact that the mail bounced. Verb and noun.
A 5xx is a bounce, and results in a bounced email message. Variances always abound, but I'll stick with my 30 year old terminology, and it is correct.
jeffbee 3 days ago [-]
That's fair with regards to your statement, but "doubled112" assigned the agency to Microsoft, saying that "Microsoft bounces" their traffic, which is demonstrably not what is happening.
b112 2 days ago [-]
It is what's happening, as I've described. A 5xx is a bounce.
You don't agree with my terminology, but we're not disagreeing on what's happening. In this case, as per my dictionary, 'rejected' and 'bounced' are synonyms. If this was me, and my MTA hitting this Microsoft server, my MTA/server would see the bounce(5xx), then my MTA/server would generate a bounce message(email) which I'd receive in my inbox.
The bounce message is a result of the bounce. The action/cause is the bounce/reject 5xx.
How can you have a bounce message, without a bounce happening?
Again, you don't agree with this terminology, and that's fine. But I'm not pulling this out of a random hat, it's 30+ year old terminology that I've used with endless people locally and online. That doesn't mean your view is wrong, we may just be running into local variances in lingo.
There are all sorts of edge cases in our compute discourse. I ran into a company where they didn't use initialisms to discuss daemons/protocols, but treated them as acronyms. Of course, they didn't really discuss such things often. So when discussing smtp, they'd pronounce it 'sim-tee'. Of course, sntpd, smtpd, snmpd, and others are all pronunced 'sim-tee', which makes for a fun meeting. Think of it as 'I am groot' taken too far. For prudence, I kept enforcing initialisms, which of course resolved this.
Anyhow! Point is, well.. not really sure except info.
turpentine 3 days ago [-]
jeffbee isn't offering any explanation, maybe because it's obvious to them they don't think they need to, or something like that... so I'll volunteer one.
If you send an email, your client would talk to your local MTA (i.e. the SMTP server you own or are authorised to relay mail through, e.g. ISP). The local MTA usally just accepts the email to insert it into a queue for attempting delivery. When your MTA processes the queue, and talks to another and gets 5xx or 4xx response, your MTA will generate a "bounce" (non-delivery report) email that lands in your inbox with the details of the response it received.
So jeffbee is correct that when the local MTA gets a 5xx or 4xx response code in the SMTP session with target MTA, that /the response code is not a bounce/. Microsoft responding with 5xx or 4xx in the SMTP session, they are not bouncing the email. They are refusing to accept delivery.
For Microsoft to "bounce every email" from the original parent commenter, it would have accept each email first, and then use the return path address of each email accepted to send a bounce email asynchronously, i.e. the bounce is not part of the original session.
If a MTA talks to another MTA who accepts a message for delivery, they can then bounce the email at any later point via the address specified in the return path header. Why? Maybe incoming email is queued and scanned, because it would take too long to determine if it passes secondary rules when it's initially being accepted.
Given how this works, you could take an email inbox you received a year ago, or five... and send an email with whatever content to the return path address, and you have "bounced" the original email.
brightball 4 days ago [-]
DMARC isn't for sending email successfully, it's for preventing other people from impersonating your domain. Without it, there's nothing stopping anybody from sending an email saying it is from you@qurren.com. SPF tried. DKIM tried. Both of them had gaps.
When you use them together and have a DMARC policy that requires one of them or the other for successful delivery, it's the best current solution.
TZubiri 3 days ago [-]
Right, and when you don't configure DMARC successfully and the recipient requires DMARC, then you cannot send email successfully.
will4274 3 days ago [-]
Yeah. And when you don't configure your TLS certificate correctly, people can't connect to your webpage.
The anti-email authenticity standards gang has always smelled like the anti-TLS gang to me.
qurren 4 days ago [-]
Except I think I've had 1:1 personal e-mails from my domain go into a legitimate recipient's spam filter just because I didn't have DMARC set up and their mail server was flagging that "DMARC not set up == spammy domain"
drdexebtjl 4 days ago [-]
That is perfectly reasonable. Set it up correctly.
bombcar 4 days ago [-]
It is so much easier to set these things up with a frontier AI to walk you through the Byzantine steps.
denkmoon 3 days ago [-]
It takes an afternoon to set up DKIM and DMARC from scratch on a debian VPS. Yeah it's a little bit byzantine but it's not rocket science.
qurren 3 days ago [-]
Yeah I did that. Now it seems I have to set up DKIM2 and DMARC2 and DCRAP3 and DSHIT4 for another afternoon instead of just going to work and getting shit done.
denkmoon 3 days ago [-]
Good heavens an entire other afternoon after a decadal standards update. You’ll have to spend an afternoon upgrading off XP at some point too.
bombcar 4 days ago [-]
Too many admins just had it set to “no valid DMARC? Spam” instead of the more proper “failed DMARC? spam”.
Which is subtly different.
hinkley 4 days ago [-]
This sort of Regulatory Capture is quite old in the software field. People were already noticing it in the 90's.
Making a spec that contains a venn diagram of most of the features each of the signatories to the specification have implemented themselves ends up pulling the ladder up behind them. Each non-academic committee member discovers they're already more than 75% of the way to having completed the spec and any junior members or amateurs have years of work to do in order to catch up to Now. If any upstarts threaten to get within striking distance of an implementation you can always convene the committee again and discuss version 2 of the spec.
Mobile devices tamped this down just a little bit but mostly they lowered the slope of the line a hair and changed where the focus was a bit.
AceJohnny2 4 days ago [-]
> Aw hell. How many things do I have to set up just so that I can send e-mails from my own domain?
... said every spammer.
I'm sorry for your pain, and I'm in the same boat.
But it's important to understand that any sufficiently large, distributed-agent system (like federated email), will see the rise of parasites that will pump resources and diminish the value of the system.
What we're seeing here is an "immune" response to those parasites. We all pay for it.
I think this is an important lesson for anyone designing a distributed-agent system [1]. How do you design it so as to keep the bad actors out, or at least so their impact is negligeable?
[1] imma make my own email system! With blackjack, and hookers! oh wait...
assimpleaspossi 4 days ago [-]
>>parasites that will pump resources and diminish the value of the system.
Countries' legal systems really need to do something about them.
zbentley 3 days ago [-]
I don’t think they can. Spam, like speeding on highways and drug sales, is such an asymmetric enforcement area that I have very limited confidence that legal enforcement would make a significant dent in the volume. It’s far too technically easy to anonymously, repeatedly break anti-spam laws. This is an area where consortium enforcement (like the big inbox providers pushing solutions like DKIM2) is probably the most effective.
Don’t get me wrong, there are tons of areas where countries’ legal systems have no excuse for not enforcing the law more stringently (e.g. flagrant corruption in multiple regulatory bodies’ failure to enforce investment/wire fraud). But spam is part of the other category—technically difficult enough to crack down on that legal action is a waste of time. There are ways to change that, but they’re all either more centralization-prone, worse for privacy/liberty, or extremely expensive.
inigyou 3 days ago [-]
They keep finding that huge spam campaigns were run by one guy from his bedroom. I can't remember which specific spam campaign was recently caught, it might've been the phone spam about car insurance. It was one guy with a huge botnet.
In total they are a finite set, and even catching 5% of them will scare the rest.
zbentley 3 days ago [-]
I interpret that the opposite way: if a huge spam campaign can be run by a guy in his bedroom, there’s no way that larger spam operators can be effectively killed by legal action. They’ll just employ different guys in different bedrooms. The same thing is true with phone phishing scams—they’re not individually hard to eradicate, but the combination of lucrativeness and rapid-rebootability means that legal crackdowns in the past have not been effective at scaring the rest out of business.
inigyou 3 days ago [-]
Did you know you can stay out of jail if you tell the police who's paying you?
xtajv 3 days ago [-]
I think about software engineering as virtual construction.
In the physical world, we can have police that go after criminals who break down doors. But builders also have a responsibility to install locks.
And negligently failing to build a lock is actually not a great look if you want police to give you the time of day.
braiamp 4 days ago [-]
Eh, I read the article, and at most you only have to wait for your MTA to update to add the required headers and update your DNS records and you are golden. It still uses the same key you generated as far I'm aware.
braiamp 4 days ago [-]
Despite what everyone said, I'm excited specifically for DKIM2. As someone that had managed a mailing list, that one is probably the hardest thing to juggle around and DKIM2 layering seems to fix that issue neatly. I hope postfix has a guide proto.
peanut-walrus 4 days ago [-]
Missed opportunity to get rid of SPF. What I want to my DMARC policy to say: if someone is sending you an email that claims to be from my domain and it's not signed by one of the keys I have published under my domain, you should reject it, regardless where it came from.
And on the receiving side, the policy is similarly simple: if I receive any unsigned or unaligned email, I will reject it.
Edit: to clarify, I want there to be an option where I specify my DMARC policy to explicitly tell well-configured receiving servers "ignore whatever I have configured as my SPF record, only look at the signatures". There will no doubt be a long tail of mail servers where I will still need an SPF record for them to accept my mail.
Edit2: Another feature that I feel is lacking is ability to give dkim selectors a scope - e.g. this key is only valid for these particular From addresses.
ericpauley 3 days ago [-]
Best way to go about this is just blackhole SPF so it never passes, then set your DMARC alignment to strict. This approach prevents SPF from satisfying DMARC on the vast majority of providers [1].
In theory, yes, in practice this will currently result in deliverability issues, both for servers which don't speak dmarc and for spam signals. I have tried this :)
If this mode of operation was an explicit choice, this would give me the option to have a fallback SPF record for legacy mail systems, but most up-to-date servers will use the more secure and (for my use case) operationally simpler verification.
inigyou 3 days ago [-]
And I want mine to say: if it comes from my MX server it's legit, don't worry about the keys.
Because that's easy and keys and signing are hard.
justusthane 4 days ago [-]
Isn’t that already what DMARC does though? For DMARC to pass you need DKIM _or_ SPF alignment, not both. It’s designed that way because there are scenarios where SPF _can’t_ pass (email forwarding, mailing lists). So a well-configured mail server should accept your email regardless of SPF if DKIM is properly configured.
Re: specific keys for specific usernames: I can appreciate that you wish DKIM allowed for this, and I could imagine it being handy, but that was never the problem DKIM set out to solve — DKIM and SPF are all about be domain.
I’m also not sure it’s a great idea — the sender identity should be under the control of the sender. If you control the domain @foo.com, you could use that ability to assert that an email came from Bob, even if Bob never sent it. Contrast that with Bob signing the email using his own private key.
toast0 3 days ago [-]
> Isn’t that already what DMARC does though? For DMARC to pass you need DKIM _or_ SPF alignment, not both.
GP wants require DKIM, so SPF alone is insufficient to send mail from their domains.
If everyone did dmarc, you could set SPF to -all. But there are servers that check SPF but not DMARC. You need to pass SPF for those, so you need a passable SPF... but then DMARC will pass with only SPF.
sealedmailuk 3 days ago [-]
[flagged]
reader9274 3 days ago [-]
> Isn’t that already what DMARC does though? For DMARC to pass you need DKIM _or_ SPF alignment, not both.
But now if SPF passes and is aligned, DMARC passes (it doesn't matter what DKIM's status is). OP here wants to complete skip the SPF check entirely.
TZubiri 3 days ago [-]
>Missed opportunity to get rid of SPF.
> you should reject it, regardless where it came from.
Just don't use it yourself then?
"v=spf1 ?all"
AceJohnny2 4 days ago [-]
I suspect SPF is used because it's cheaper than performing cryptographic checks for each email. A (cached) DNS lookup and IP check on a connection is comparatively cheaper.
Animats 3 days ago [-]
Questions:
- How much infrastructure has to be fixed before this works, and in what order?
- Can you send mail from something that doesn't have a DNS entry? How does this affect the first hop from a desktop or mobile SMTP client?
- If an spam email came via SendGrid, Constant Spammer, or MailChump, are you going to be able to tell from the header signatures?
- If your headers are correct, are you guaranteed mail bounces for un-deliverable emails?
edelbitter 3 days ago [-]
> Can you send mail from something that doesn't have a DNS entry?
You never really could. Participating in public email exchange requires that the sender can resolve then "fully qualified" domain in your return address. Except after prior agreement or authentication, messages simply that fail this are not generally accepted.
> If your headers are correct, are you guaranteed mail bounces for un-deliverable emails?
Even better: You are more likely to see an immediate refusal instead of a delayed bounce, if the recipient exchange can during transmission already determine that they do not want message claiming to be originally transmitted from X to Y yet breaking their ability to check the signature added by X.
toast0 3 days ago [-]
> You never really could. Participating in public email exchange requires that the sender can resolve then "fully qualified" domain in your return address.
I thought you could send from <> for things that shouldn't bounce.
edelbitter 3 days ago [-]
If the return path is null that just clarifies unattended notifications should not be returned. The mail is still considered to be sent by the transmitting system, and when no mailbox is specified, the implied envelope sender simply defers to the "postmaster@your-ehlo-fqdn.example" address.
(Accepting mail at the "postmaster" mailbox is a mandatory part of SMTP, as is mentioning your fully qualified domain name in the "Hello" message when initiating the session. Public mail exchanges are free to, and often do, reject clients that submit anything other than resolvable domains there. Same with clients that use <> for applications other than those very limited "notification about specific quoted/referenced message" scenarios where the standards mandate <>.)
TZubiri 3 days ago [-]
>You never really could.
You can, open a TCP socket on port 25 to $IP, and just start sending email headers.
You can also use local domains, no DNS.
You can also leave a file in the /var/mail directory with the filename of the user.
zbentley 3 days ago [-]
> Can you send mail from something that doesn't have a DNS entry?
I hope not. Just like SSL, I think requiring a registrar+DNS server to vouch for a durable identity is an important barrier to abuse (and an important intervention point for violation reports).
bawolff 3 days ago [-]
> If an spam email came via SendGrid, Constant Spammer, or MailChump, are you going to be able to tell from the header signatures?
Couldn't you always tell this from headers? At the very least the recieved headers are going to be a give away.
bigbuppo 4 days ago [-]
Can someone distill this down to how it will be used by the big three email providers to make it impossible to use email except through them?
braiamp 4 days ago [-]
If anything, this moves it towards anyone having more access to everything. For example, reject isn't going to be treated anymore as a bounce. Now, provider policies still can and would be BS, but the standard doesn't tell them to do it certain way.
TacticalCoder 4 days ago [-]
> by the big three
Which big three?
Gmail has something like 1.8 billion users. iCloud mail around 1 billion.
Microsoft with 400 million users of its email is closer to Yahoo! Mail (225 million users) than to the big two.
AlotOfReading 4 days ago [-]
User numbers aren't the only factor. Microsoft has a much larger presence in commercial email than Apple. I suspect an outbound email from a personal provider is far more likely to be destined for an outlook inbox than one on iCloud.
comex 3 days ago [-]
I wonder how this post was composed. It's full of LLM-isms, but is also pretty informative and not too fluffy - basically, higher-quality than I'm used to seeing from LLM blog posts, especially at this length. Perhaps it was composed based on a detailed human outline? Or perhaps, could this be the power of Fable?
They spent a huge amount of complexity supporting mailing lists that claim mutated messages came FROM the original sender instead of FROM mailinglist@example.com. Is that use-case worth the additional effort?
edelbitter 3 days ago [-]
Yes! Forwarding is the reason mail flow authentication does not work very well today. Fix the forwarding use cases (and by extension: mailing lists & out of office arrangements) and we can finally just tell all senders:
"Your domains do not match, please use DMARC so we can automatically detect whether that is OK. No excuses, no exemptions, your use case can be dealt with using DMARC!"
bombcar 4 days ago [-]
Yes. Mailing lists are incredibly useful and very hard to change.
jeroenhd 4 days ago [-]
First time I'm reading about this, I'm glad there's progress in this area.
The JSON DSL for rewriting emails feels like a spammer/exploit vector waiting to happen. Some product is going to spam filter before applying reconstruction rules, or get tricked into applying reconstruction rules when it shouldn't, and spammers and scammer are going to abuse it.
Until either Google or Microsoft will adopt these standards, they'll remain effectively meaningless most likely. But even so, it's good to know people haven't given up on fixing email's spam problem entirely.
4 days ago [-]
dataflow 4 days ago [-]
I really don't understand what the original DKIM was not sufficient. Can someone ELI5? If you can verify that a message (including headers, which DKIK can sign) was signed by the outgoing server, then why isn't that the end of the story? Who cares how or why it got forwarded, or whatever else?
winstonwinston 3 days ago [-]
They want to allow forwarders (such as mailing lists) to modify signed messages all while keeping the original signed author email address. It is meant to replace ARC which is now deprecated.
You can now verify who changed what and when but it is still based on will and trust to accept what has been altered and therefore a security theatre.
It ‘solved’ a problem for a mailing list that insist on altering signed messages, even though they do not have to modify forwarded messages in my opinion and many lists do not.
brightball 4 days ago [-]
There are a few gaps with DKIM.
1. You have to set it up on every sending server. It's easier today but it wasn't always
2. You have to periodically rotate each of the keys that you setup because they can be cracked/stolen. Soon as somebody steals your key, they can impersonate anyone sending email from your domain.
3. Receiving email servers have no way of knowing if a message they received without a DKIM signature is supposed to include a DKIM signature, so simply not including one creates a scenario where receiving mail servers have to guess if the message was really from you.
crote 4 days ago [-]
2b. You have to publish the retired private keys, or else a recipient will retain undeniable proof of message authenticity.
Depending on your perspective, this can be either a feature or a bug.
bombcar 4 days ago [-]
The fallout from this has barely begun to be felt. It’s more important than the hypothetical quantum crypto stuff imo.
yasaheblasa 4 days ago [-]
1&2 sound worse after this update as described..
I'm not really sure why we are still bothering with this when DNSSEC progress means DANE like setups could solve the original E2E S/MIME issues of payment and domain indicating expectation of what its email senders are required to have for S/MIME.
There are some aspects of (possibly positive) deniability by an individual that probably still remain with DKIM but they kind of remain anyway with domain anchored S/MIME.
tptacek 3 days ago [-]
What DNSSEC progress?
yasaheblasa 3 days ago [-]
I don't consider 100% the goal since I'm happy if the average squatter/spammer/landing-page maker finds no value in steps to reaching more discerning clients given correlating filters. It seems likely to me that the percentage is near the tipping point where any serious organization is probably doing DNSSEC or having discussions about why they have IT problems and should be as serious now as they are for the email standards. For example, the validating DNSSEC looks higher than domains with functioning DKIM usage:
Sure, biggest Brand/Monopoly (and largely US) Tech is of course an interesting situation with a few different directions for interpretation. Yet, I think that many of them simply have to add DNSSEC, IPv6, etc as soon as the numbers finally look too much like 50%+, I.e. MS is about half the sites and already has selectively added it, usually where the consumer brand consequence is theirs via a SaaS product.
tptacek 3 days ago [-]
As you can see, after 30+ years of effort, we are nowhere close to 50%, or even 25%, of deployment on real sites.
yasaheblasa 3 days ago [-]
Secure shell had to wait quite a while for gradual adoption by anyone new and caskets of the older to pile up. SSL wasn't exactly shiny new in 2014 when LetsEncrypt was brought in to make it prevalent together with Google pressure. Similar pressure has the same groups picking DNSSEC experience up just in the last year.
If we put aside the advertising for a moment, the US had the same problems with every technology that is newer than landlines. Big investments in companies that will be prevented from failing will try to keep the US behind the trend but a few US companies will see that they better get ahead of where the rest of the world is going with or without US tech.
tptacek 2 days ago [-]
No, it didn't. SSH adoption was nearly universal with a year or two of its release. TLS was nearly universal on commercial sites long, long before LetsEncrypt. SSH and TLS are not comparable in adoption to DNSSEC.
dataflow 4 days ago [-]
1. I can't say I buy this excuse, but okay.
2. Is this an actual problem that has arisen with a worrying frequency in the past, or just a hypothetical? And how is it different from someone stealing your SSH key or TLS certificate?
3. Isn't it obvious from previous emails you've received from the same server?
brightball 3 days ago [-]
1. There are a lot of domains out there and all of the people who own them aren't necessarily technical enough to setup DKIM on their mail server. Ideally those people are using some type of service. SPF is much simpler in this regard.
2. This is a rather famous story about it happening.
I have no idea how widespread the issue is today but I had to do some analysis on it when I worked for dmarcian ahead of the Anti-Phishing Working Group conference and we found that a significant percentage of email from known malicious IPs associated with reported phishing was passing DKIM. Key rotation removes the problem. Many services like ProtonMail and Sendgrid will set you up with 2 CNAME's for your DKIM keys so that they can rotate them for you automatically.
3. Domains send emails from multiple servers. Sometimes dedicated email servers, Google/Outlook, Sendgrid, email marketing tools, etc. A receiving system has no way to validate whether any of the tools sending email claiming to be from your domain are actually from your domain. The first time you look at a DMARC report for a domain that's been around for a while, you will typically see that 90% or more of the messages claiming to be from your domain weren't from you at all.
dataflow 3 days ago [-]
Re: #3, shouldn't this be per-domain anyway, rather than per server? If a domain has one server signing and another one not signing then something feels wrong. It seems pretty fine to just look at what the domain did in the past as the basis, no?
brightball 3 days ago [-]
Since multiple services can send on behalf of the domain, DKIM has to be configured for each of them. A service provider, like Google, will likely use the same private key across any of their servers that is sending your mail but Sendgrid won't have access to that private key so they have to setup their own. Same goes for any other services that send mail using your domain.
As a receiving mail server, they have no way of knowing how many different parties are legitimately sending email on behalf of your domain.
SPF is per domain. You setup a DNS record at the domain (or subdomain) level that lists all of the IPs (or a DNS reference to a list of those IPs) that are authorized to send email on your behalf...but, that breaks with mail forwarding and mailing lists.
SPF is much simpler, absolutely. DKIM requires a private key to sign outgoing messages from every mail server sending mail on your behalf.
dataflow 3 days ago [-]
> A service provider, like Google, will likely use the same private key across any of their servers that is sending your mail but Sendgrid won't have access to that private key so they have to setup their own. Same goes for any other services that send mail using your domain.
I'm sorry, I'm so lost and confused. Why would a service like SendGrid be impersonating a random domain without being able to get a private key from the domain owner? Like you're saying SendGrid offers the ability to send emails from a domain like @gmail.com without its private key?
brightball 2 days ago [-]
I'll try to explain.
Say I decide to open a pirate themed gym called Slimmer Ye Timbers and buy the domain slimmeryetimbers.com.
If I want to setup a website, I will go to a hosting company, set something up, go into the DNS and point a couple of records for the top level domain and the www subdomain to the hosting company.
If I want to receive email sent to argh@slimmeryetimbers.com I need to setup a mail server (Google, Outlook, web host may offer one, could setup an open source one, etc) and once it's setup I go to the DNS to point an MX record at the mail server.
So far, if people try to lookup my website or send an email TO me everything is pretty straightforward.
Now, if I want to send email that says it's FROM argh@slimmeryetimers.com is where things get weird. Because I don't have to do anything.
Any server, anywhere can just do it and every mail server that receives a message saying it's from that domain has to try to figure out if it really is or isn't. This is where antispam rules come in. Without DMARC, SPF & DKIM in place receiving mail servers will build up trust in different IP addresses, typically from vendors who go out of their way to prevent email abuse specifically to protect the reputation of those IP addresses. The receiving mail server my do a reverse DNS lookup to see if the hostname matched. They might see if the MX server used by domain is the same server sending the message along with a ton of other algorithmic hoops to make a good judgement. Even with DMARC, SPF and DKIM in place they may still use those rules.
DMARC, when strictly enforced with p=reject, let's you signal the receiving mail server that all mail claiming to be from your domain can be validated and if it can't be validated that message isn't from you and can be discarded. All that DMARC does is say that if you receive a message from this domain, it should pass either SPF OR DKIM for this domain as well.
So let's say, for example that I'm using both Google Workspace and Sendgrid for email for slimmeryetimbers.com. I'll setup Gmail for my main professional email for myself and my staff and I'll setup Sendgrid to send email from the website (responses to contact forms, etc).
For SPF it can be pretty simple, a single TXT record:
Both Google and Sendgrid publish DNS records with the IP address of their servers and keep them up to date, so adding that include is all that we have to do.
DKIM is more complicated. Google will provide you with a DKIM record to put under a subdomain that includes the public DKIM key. Once they have verified it's setup, they will sign every email sent from your account with the private key that only they know. When a receiving server gets the message they will see that the message has been signed by DKIM and it will point to the subdomain where the public key was stored. By following the instructions from the signature in the email and using the public key, they can verify that the message was actually signed by the private key and hasn't been tampered with.
If we then setup Sendgrid they will give you their own DNS records for DKIM that work for their own private key. In both of these situations, we never get our hands on the private key because we are using a 3rd party service that doesn't disclose it. Sendgrid issues 2 CNAME records that point to DKIM public key records on their DNS so that they can control them. They'll send out messages signed with one private key and then later switch to a different private key while the original key and it's public record is changed, transparently without you having to deal with it.
If we were to setup our own mail servers, we would have access to it though. It's also entirely possible for somebody with access at an email company to go steal the private keys of their customers and sell them, or just use them. It's possible for those services or our mail servers to be hacked/compromised and the keys be stolen. Built in rotation process helps with this. It's the same reason that Let's Encrypt issues secure certificates that expire after 90 days (sometimes less). Just key rotation.
That was long but I hope it helps?
bawolff 3 days ago [-]
> Who cares how or why it got forwarded, or whatever else?
Because it broke enough niche usecases that lots of people didn't feel comfortable fully turning on dmarc in strict mode. Fixing that will hopefully spike adoption
stefan_lec 4 days ago [-]
Sounds pretty cool! I wonder if it closes enough holes that we could finally stop using SPF at all?
lousken 4 days ago [-]
Anyone migrated from exchange to stalwart? Curious about results
flowerthoughts 3 days ago [-]
Getting rid of ARC would be so nice, but of course this just means I need to add DKIM2 and need ARC for backwards compatibility.
SXX 3 days ago [-]
What always bugged me about whole email its that we still dont have two best and most reasonable practice to fight about abuse:
1 - Ability to pay once to provider give your domain Good reputation score to new or old domain and IP and whatever. Like pay once, be a good citizen.
2 - Or just use Hashcash or any other PoW.
This would really solve a problem with 99.9% of spam and allow actually more decentrolized email system.
Fact that no matter what you do its impossible to setup own emaik server to just send few emails a year with 100% guaranteed delivery is just beyond me.
edelbitter 3 days ago [-]
In a way, the "let the money decide who is a good citizen" already works. Just not in our favor:
Most of the spam I receive is not from some rando using an IP I know nothing about. Its from providers that keep making good money with their current anti-abuse.. strategy. As much as I would like, I cannot refuse all mail from certain providers, because some of their customers are "important" and non-abusive. But I get the good and the bad all laundered together¹.
¹) Some do allow recipients to distinguish, e.g. Sendgrid add an X-Entity-ID header which is a stable 1:1 or <small-number>:1 map between short ascii identifiers and customers. So if you store that mapping, you can reject the usual "From: <admin@bigcorp.example> but not sent using the account associated with bigcorp.example". (The way they easily could, if they cared.)
bawolff 3 days ago [-]
> Or just use Hashcash or any other PoW.
People love the idea of this, but i don't think it really makes sense.
How high do you set the PoW to? I dont think there is any middle ground that would actually deter attacks but not deter legit users.
SXX 3 days ago [-]
When I have personal email server and I need just ocasionally send email or two it can be as high as possible.
I just want them delivered not into Spam folder.
bawolff 3 days ago [-]
Yeah, but you pay the cost for outgoing messages and set the cost for incoming messages. The incentives aren't super well aligned, and its difficult to convince people to pay a cost to contact you when they cant force it to be reciprocal.
The point i was trying to make though, is that we are past the era of brute forcing email addresses to send spam. Most spam is at least a little targeted.
If you are an attacker and need to send 100,000 emails a week, that means your budget you can spend on each PoW is six seconds.
This is easily parallizable and you can get budget vps's for about $5/month (i dont know if its more ecconomically efficient to get more powerful VPS or lots of cheap ones, but your mail server is probably in the range of a budget vps so i think its a fair comparison).
So some back of the napkin math, if the marketer wants to send 100,000 emails a week it costs them very very roughly $1.25 per 6 seconds of PoW.
If they have a marketing budget of $500 (which seems very small by most advertising budgets). Then you need a proof of work that takes 40 minutes per email to deter them. I dont think most email providers would find that acceptable, and i also made a bunch of simplifying assumptions here that i think lean in the direction of making PoW sound better than it is.
roshanroyj 5 days ago [-]
[flagged]
Rendered at 12:03:25 GMT+0000 (Coordinated Universal Time) with Vercel.
The effect of all this seems to be less "making e-mail secure" and more "making it so that only Google, Apple, and Microsoft can send e-mail successfully"
They both have fairly clean migration paths and resolve a lot of the annoying edge cases that currently exist with authenticating and verifying email.
Recently I checked the IP against blacklists, waited a few months, did all of the other things, and then found out Microsoft bounces my entire VPS’s IP range. Appealing did not help.
They intermittently block Cloudflare email routing IPs too. All of these security measures and still it comes down to the IP address of your sender.
It is a cheap VPS, but it would still be nice if there was a way to know (not assume) beforehand.
> 550 5.7.1 Unfortunately, messages from [IP ADDRESS] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3150).
> Your IP(s) qualify for conditional mitigation.
Still blocked. The system is working as expected.
Either you use cheap infrastructure that attackers can buy by the bulk for sybil attacks.
Or you pay a reasonable price for a slice of an IP block that doesn't share its reputation with elcheapos.
Wait, that is probably from the local mailer, and otherwise the sender may never know about the rejected messages.
A 5xx is a bounce, and results in a bounced email message. Variances always abound, but I'll stick with my 30 year old terminology, and it is correct.
You don't agree with my terminology, but we're not disagreeing on what's happening. In this case, as per my dictionary, 'rejected' and 'bounced' are synonyms. If this was me, and my MTA hitting this Microsoft server, my MTA/server would see the bounce(5xx), then my MTA/server would generate a bounce message(email) which I'd receive in my inbox.
The bounce message is a result of the bounce. The action/cause is the bounce/reject 5xx.
How can you have a bounce message, without a bounce happening?
Again, you don't agree with this terminology, and that's fine. But I'm not pulling this out of a random hat, it's 30+ year old terminology that I've used with endless people locally and online. That doesn't mean your view is wrong, we may just be running into local variances in lingo.
There are all sorts of edge cases in our compute discourse. I ran into a company where they didn't use initialisms to discuss daemons/protocols, but treated them as acronyms. Of course, they didn't really discuss such things often. So when discussing smtp, they'd pronounce it 'sim-tee'. Of course, sntpd, smtpd, snmpd, and others are all pronunced 'sim-tee', which makes for a fun meeting. Think of it as 'I am groot' taken too far. For prudence, I kept enforcing initialisms, which of course resolved this.
Anyhow! Point is, well.. not really sure except info.
If you send an email, your client would talk to your local MTA (i.e. the SMTP server you own or are authorised to relay mail through, e.g. ISP). The local MTA usally just accepts the email to insert it into a queue for attempting delivery. When your MTA processes the queue, and talks to another and gets 5xx or 4xx response, your MTA will generate a "bounce" (non-delivery report) email that lands in your inbox with the details of the response it received.
So jeffbee is correct that when the local MTA gets a 5xx or 4xx response code in the SMTP session with target MTA, that /the response code is not a bounce/. Microsoft responding with 5xx or 4xx in the SMTP session, they are not bouncing the email. They are refusing to accept delivery.
For Microsoft to "bounce every email" from the original parent commenter, it would have accept each email first, and then use the return path address of each email accepted to send a bounce email asynchronously, i.e. the bounce is not part of the original session.
If a MTA talks to another MTA who accepts a message for delivery, they can then bounce the email at any later point via the address specified in the return path header. Why? Maybe incoming email is queued and scanned, because it would take too long to determine if it passes secondary rules when it's initially being accepted.
Given how this works, you could take an email inbox you received a year ago, or five... and send an email with whatever content to the return path address, and you have "bounced" the original email.
When you use them together and have a DMARC policy that requires one of them or the other for successful delivery, it's the best current solution.
The anti-email authenticity standards gang has always smelled like the anti-TLS gang to me.
Which is subtly different.
Making a spec that contains a venn diagram of most of the features each of the signatories to the specification have implemented themselves ends up pulling the ladder up behind them. Each non-academic committee member discovers they're already more than 75% of the way to having completed the spec and any junior members or amateurs have years of work to do in order to catch up to Now. If any upstarts threaten to get within striking distance of an implementation you can always convene the committee again and discuss version 2 of the spec.
Mobile devices tamped this down just a little bit but mostly they lowered the slope of the line a hair and changed where the focus was a bit.
... said every spammer.
I'm sorry for your pain, and I'm in the same boat.
But it's important to understand that any sufficiently large, distributed-agent system (like federated email), will see the rise of parasites that will pump resources and diminish the value of the system.
What we're seeing here is an "immune" response to those parasites. We all pay for it.
I think this is an important lesson for anyone designing a distributed-agent system [1]. How do you design it so as to keep the bad actors out, or at least so their impact is negligeable?
[1] imma make my own email system! With blackjack, and hookers! oh wait...
Countries' legal systems really need to do something about them.
Don’t get me wrong, there are tons of areas where countries’ legal systems have no excuse for not enforcing the law more stringently (e.g. flagrant corruption in multiple regulatory bodies’ failure to enforce investment/wire fraud). But spam is part of the other category—technically difficult enough to crack down on that legal action is a waste of time. There are ways to change that, but they’re all either more centralization-prone, worse for privacy/liberty, or extremely expensive.
In total they are a finite set, and even catching 5% of them will scare the rest.
In the physical world, we can have police that go after criminals who break down doors. But builders also have a responsibility to install locks.
And negligently failing to build a lock is actually not a great look if you want police to give you the time of day.
And on the receiving side, the policy is similarly simple: if I receive any unsigned or unaligned email, I will reject it.
Edit: to clarify, I want there to be an option where I specify my DMARC policy to explicitly tell well-configured receiving servers "ignore whatever I have configured as my SPF record, only look at the signatures". There will no doubt be a long tail of mail servers where I will still need an SPF record for them to accept my mail.
Edit2: Another feature that I feel is lacking is ability to give dkim selectors a scope - e.g. this key is only valid for these particular From addresses.
[1] https://taejoong.github.io/files/publications/hamza-2026-dma...
If this mode of operation was an explicit choice, this would give me the option to have a fallback SPF record for legacy mail systems, but most up-to-date servers will use the more secure and (for my use case) operationally simpler verification.
Because that's easy and keys and signing are hard.
Re: specific keys for specific usernames: I can appreciate that you wish DKIM allowed for this, and I could imagine it being handy, but that was never the problem DKIM set out to solve — DKIM and SPF are all about be domain.
I’m also not sure it’s a great idea — the sender identity should be under the control of the sender. If you control the domain @foo.com, you could use that ability to assert that an email came from Bob, even if Bob never sent it. Contrast that with Bob signing the email using his own private key.
GP wants require DKIM, so SPF alone is insufficient to send mail from their domains.
If everyone did dmarc, you could set SPF to -all. But there are servers that check SPF but not DMARC. You need to pass SPF for those, so you need a passable SPF... but then DMARC will pass with only SPF.
But now if SPF passes and is aligned, DMARC passes (it doesn't matter what DKIM's status is). OP here wants to complete skip the SPF check entirely.
> you should reject it, regardless where it came from.
Just don't use it yourself then?
"v=spf1 ?all"
- How much infrastructure has to be fixed before this works, and in what order?
- Can you send mail from something that doesn't have a DNS entry? How does this affect the first hop from a desktop or mobile SMTP client?
- If an spam email came via SendGrid, Constant Spammer, or MailChump, are you going to be able to tell from the header signatures?
- If your headers are correct, are you guaranteed mail bounces for un-deliverable emails?
You never really could. Participating in public email exchange requires that the sender can resolve then "fully qualified" domain in your return address. Except after prior agreement or authentication, messages simply that fail this are not generally accepted.
> If your headers are correct, are you guaranteed mail bounces for un-deliverable emails?
Even better: You are more likely to see an immediate refusal instead of a delayed bounce, if the recipient exchange can during transmission already determine that they do not want message claiming to be originally transmitted from X to Y yet breaking their ability to check the signature added by X.
I thought you could send from <> for things that shouldn't bounce.
You can, open a TCP socket on port 25 to $IP, and just start sending email headers.
You can also use local domains, no DNS.
You can also leave a file in the /var/mail directory with the filename of the user.
I hope not. Just like SSL, I think requiring a registrar+DNS server to vouch for a durable identity is an important barrier to abuse (and an important intervention point for violation reports).
Couldn't you always tell this from headers? At the very least the recieved headers are going to be a give away.
Which big three?
Gmail has something like 1.8 billion users. iCloud mail around 1 billion.
Microsoft with 400 million users of its email is closer to Yahoo! Mail (225 million users) than to the big two.
you're among the first few who have done it:
https://github.com/mjl-/mox/issues/404#issuecomment-43627498...
The JSON DSL for rewriting emails feels like a spammer/exploit vector waiting to happen. Some product is going to spam filter before applying reconstruction rules, or get tricked into applying reconstruction rules when it shouldn't, and spammers and scammer are going to abuse it.
Until either Google or Microsoft will adopt these standards, they'll remain effectively meaningless most likely. But even so, it's good to know people haven't given up on fixing email's spam problem entirely.
You can now verify who changed what and when but it is still based on will and trust to accept what has been altered and therefore a security theatre.
It ‘solved’ a problem for a mailing list that insist on altering signed messages, even though they do not have to modify forwarded messages in my opinion and many lists do not.
1. You have to set it up on every sending server. It's easier today but it wasn't always
2. You have to periodically rotate each of the keys that you setup because they can be cracked/stolen. Soon as somebody steals your key, they can impersonate anyone sending email from your domain.
3. Receiving email servers have no way of knowing if a message they received without a DKIM signature is supposed to include a DKIM signature, so simply not including one creates a scenario where receiving mail servers have to guess if the message was really from you.
Depending on your perspective, this can be either a feature or a bug.
There are some aspects of (possibly positive) deniability by an individual that probably still remain with DKIM but they kind of remain anyway with domain anchored S/MIME.
https://stats.labs.apnic.net/dnssec?s=Validating&d=01%2F06%2... https://stats.labs.apnic.net/dnssec?s=Validating&d=01%2F06%2... https://stats.labs.apnic.net/dnssec?s=Validating&d=01%2F06%2... https://stats.labs.apnic.net/dnssec?s=Validating&d=01%2F06%2...
https://dnssecmenot.fly.dev/
If we put aside the advertising for a moment, the US had the same problems with every technology that is newer than landlines. Big investments in companies that will be prevented from failing will try to keep the US behind the trend but a few US companies will see that they better get ahead of where the rest of the world is going with or without US tech.
2. Is this an actual problem that has arisen with a worrying frequency in the past, or just a hypothetical? And how is it different from someone stealing your SSH key or TLS certificate?
3. Isn't it obvious from previous emails you've received from the same server?
2. This is a rather famous story about it happening.
https://www.wired.com/2012/10/dkim-vulnerability-widespread/
I have no idea how widespread the issue is today but I had to do some analysis on it when I worked for dmarcian ahead of the Anti-Phishing Working Group conference and we found that a significant percentage of email from known malicious IPs associated with reported phishing was passing DKIM. Key rotation removes the problem. Many services like ProtonMail and Sendgrid will set you up with 2 CNAME's for your DKIM keys so that they can rotate them for you automatically.
3. Domains send emails from multiple servers. Sometimes dedicated email servers, Google/Outlook, Sendgrid, email marketing tools, etc. A receiving system has no way to validate whether any of the tools sending email claiming to be from your domain are actually from your domain. The first time you look at a DMARC report for a domain that's been around for a while, you will typically see that 90% or more of the messages claiming to be from your domain weren't from you at all.
As a receiving mail server, they have no way of knowing how many different parties are legitimately sending email on behalf of your domain.
SPF is per domain. You setup a DNS record at the domain (or subdomain) level that lists all of the IPs (or a DNS reference to a list of those IPs) that are authorized to send email on your behalf...but, that breaks with mail forwarding and mailing lists.
SPF is much simpler, absolutely. DKIM requires a private key to sign outgoing messages from every mail server sending mail on your behalf.
I'm sorry, I'm so lost and confused. Why would a service like SendGrid be impersonating a random domain without being able to get a private key from the domain owner? Like you're saying SendGrid offers the ability to send emails from a domain like @gmail.com without its private key?
Say I decide to open a pirate themed gym called Slimmer Ye Timbers and buy the domain slimmeryetimbers.com.
If I want to setup a website, I will go to a hosting company, set something up, go into the DNS and point a couple of records for the top level domain and the www subdomain to the hosting company.
If I want to receive email sent to argh@slimmeryetimbers.com I need to setup a mail server (Google, Outlook, web host may offer one, could setup an open source one, etc) and once it's setup I go to the DNS to point an MX record at the mail server.
So far, if people try to lookup my website or send an email TO me everything is pretty straightforward.
Now, if I want to send email that says it's FROM argh@slimmeryetimers.com is where things get weird. Because I don't have to do anything.
Any server, anywhere can just do it and every mail server that receives a message saying it's from that domain has to try to figure out if it really is or isn't. This is where antispam rules come in. Without DMARC, SPF & DKIM in place receiving mail servers will build up trust in different IP addresses, typically from vendors who go out of their way to prevent email abuse specifically to protect the reputation of those IP addresses. The receiving mail server my do a reverse DNS lookup to see if the hostname matched. They might see if the MX server used by domain is the same server sending the message along with a ton of other algorithmic hoops to make a good judgement. Even with DMARC, SPF and DKIM in place they may still use those rules.
DMARC, when strictly enforced with p=reject, let's you signal the receiving mail server that all mail claiming to be from your domain can be validated and if it can't be validated that message isn't from you and can be discarded. All that DMARC does is say that if you receive a message from this domain, it should pass either SPF OR DKIM for this domain as well.
So let's say, for example that I'm using both Google Workspace and Sendgrid for email for slimmeryetimbers.com. I'll setup Gmail for my main professional email for myself and my staff and I'll setup Sendgrid to send email from the website (responses to contact forms, etc).
For SPF it can be pretty simple, a single TXT record:
"v=spf1 include:_spf.google.com include:sendgrid.net ~all"
Both Google and Sendgrid publish DNS records with the IP address of their servers and keep them up to date, so adding that include is all that we have to do.
DKIM is more complicated. Google will provide you with a DKIM record to put under a subdomain that includes the public DKIM key. Once they have verified it's setup, they will sign every email sent from your account with the private key that only they know. When a receiving server gets the message they will see that the message has been signed by DKIM and it will point to the subdomain where the public key was stored. By following the instructions from the signature in the email and using the public key, they can verify that the message was actually signed by the private key and hasn't been tampered with.
If we then setup Sendgrid they will give you their own DNS records for DKIM that work for their own private key. In both of these situations, we never get our hands on the private key because we are using a 3rd party service that doesn't disclose it. Sendgrid issues 2 CNAME records that point to DKIM public key records on their DNS so that they can control them. They'll send out messages signed with one private key and then later switch to a different private key while the original key and it's public record is changed, transparently without you having to deal with it.
If we were to setup our own mail servers, we would have access to it though. It's also entirely possible for somebody with access at an email company to go steal the private keys of their customers and sell them, or just use them. It's possible for those services or our mail servers to be hacked/compromised and the keys be stolen. Built in rotation process helps with this. It's the same reason that Let's Encrypt issues secure certificates that expire after 90 days (sometimes less). Just key rotation.
That was long but I hope it helps?
Because it broke enough niche usecases that lots of people didn't feel comfortable fully turning on dmarc in strict mode. Fixing that will hopefully spike adoption
1 - Ability to pay once to provider give your domain Good reputation score to new or old domain and IP and whatever. Like pay once, be a good citizen.
2 - Or just use Hashcash or any other PoW.
This would really solve a problem with 99.9% of spam and allow actually more decentrolized email system.
Fact that no matter what you do its impossible to setup own emaik server to just send few emails a year with 100% guaranteed delivery is just beyond me.
¹) Some do allow recipients to distinguish, e.g. Sendgrid add an X-Entity-ID header which is a stable 1:1 or <small-number>:1 map between short ascii identifiers and customers. So if you store that mapping, you can reject the usual "From: <admin@bigcorp.example> but not sent using the account associated with bigcorp.example". (The way they easily could, if they cared.)
People love the idea of this, but i don't think it really makes sense.
How high do you set the PoW to? I dont think there is any middle ground that would actually deter attacks but not deter legit users.
I just want them delivered not into Spam folder.
The point i was trying to make though, is that we are past the era of brute forcing email addresses to send spam. Most spam is at least a little targeted.
If you are an attacker and need to send 100,000 emails a week, that means your budget you can spend on each PoW is six seconds.
This is easily parallizable and you can get budget vps's for about $5/month (i dont know if its more ecconomically efficient to get more powerful VPS or lots of cheap ones, but your mail server is probably in the range of a budget vps so i think its a fair comparison).
So some back of the napkin math, if the marketer wants to send 100,000 emails a week it costs them very very roughly $1.25 per 6 seconds of PoW.
If they have a marketing budget of $500 (which seems very small by most advertising budgets). Then you need a proof of work that takes 40 minutes per email to deter them. I dont think most email providers would find that acceptable, and i also made a bunch of simplifying assumptions here that i think lean in the direction of making PoW sound better than it is.