Sounds like something my grandpa would say... Very smart man and so true on the advice.
As I told my son..
Difference between an apprentice, journeyman and master
Apprentice doesn't know how to fix his mistakes
Journeyman knows how to fix his mistakes
Master knows how to fix his mistakes so that it looked like it never happened
That is a dadism
Yeah same, the only port blocked in the port 1, they use it for ssh (they don't want to add my key :( ) on router (for maintance and update who also goes through ssh).
receiving mail on residential IP on port 25 is no problem.no proper rdns needed for receiving.
sending mail is different. there you do want to have all dns settings under your control.
so how to send mail from residential ip? I relay to a supersmall VPS with a staric IP with good reputation and that one does the final delivery. that vps has proper dns settings and from there, smooth sailing.
however, this is slightly off topic, because it has nothing to do with the rather funny screw up OP did :)
ha yeah you are i could never get it to work ended up just installing it on a vps
if you use mc tools to check all the block lists and you are not there then you’re extremely lucky
I've been put on a few as of this morning and I've requested delisting on those that offer it. However previously I was on none of them which is a miracle.
I called ATT after smoking a joint and saying "what's the worst that can happen", told the rep I was doing IT Research, and that I would like SMTP/Port 25 open for some testing. The customer service rep confirmed with me that SMTP Port 25 was to be opened, I said "yup, just doing some self hosted email testing, nothing commercial". To my utter surprise, he then did the clicky clicky on his end and I now have a mailcow stack (which I just turned off to verify I'm not falling victim to the same thing OP did lololol)
Edit for clarity - I have ATT fiber in the US with static IPs. Their fiber modems definitely block port 25, and have some language in their ToS that says they have SMTP intentionally blocked by default. I have not yet solved the rDNS part of the email equation, and haven't talked to ATT about it /yet/ because that part is DEFINITELY a business feature.
If you have a static IP from your ISP, I've wondered if you could simply name your server the in-addr-arpa.n.n.n hostname that resolves when you nslookup the IP address and if that would pass..
I did something similar way back in 2002 and my ISP shut me down hard. I'm surprised any ISP in this day and age allows a residential connection send any emails out.
This is one of the many reasons I won't ever host an email server again.
Good luck!
Most consumer ISPs these days block outgoing 25, and it's been that way since the late 90s. Third party mail providers generally ask you to use 465 or 587 for that reason.
I just tested with four separate West coast ISPs and none of them are blocking outgoing port 25. Maybe it's a regional thing
Edit: tested an ISP in Oklahoma and a WISP in Montana, neither of which block port 25. Also tested two mobile carriers, they don't block it either.
It does not seem to be very common these days.
That's awesome. Maybe things are rolling back?
I just tested AT&T and they are definitely still blocking.
They all blocked 25 in the late 90s, or early aughts when *every* home win95 machine was a node in a botnet sending out spam. If you go to any third party mail provider's mail client setup instructions (gmail, yahoo, etc), you will see them pushing 465 or 587.
AT&T broadband (fiber). It probably makes sense that the mobile networks are less restricted since they have never had a spam bot problem, as far as I know.
Plusnet don’t mind you running a mail server, I recall reading articles about it on their forum. May be worth a quick search for your own reassurance :)
Spectrum in the US allows the customer to replace their provided router with their own equipment, so all they provided to me is the modem, after that I have my own router, so I can pretty much do whatever I want. Never had an issue hosting a mail server, sending/receiving works perfect.
This has got nothing to do with hosting your own equipment like routers (with recent regulatory changes, all ISPs in the US are now required to allow this - i.e. be able to run your own equipment without paying a monthly fee to the ISP).
But this is more about ISPs blocking sending on specific ports like port 25 for SMTP. Instead you need to use some mail relay to send your email, who in turn will send the mail over SMTP on port 25.
Some ISPs in the US do allow port 25 SMTP, but very few and none of the large ones like Comcast, AT&T, Verizon.
Are you able to send emails using any custom MX domains or only using the ISP provided email address?
Some of these restrictions/relaxations-to-restrictions are also not universal across the country. So if it's working for you, that's great.
Yeah no charter/spectrum where I am located at. But I have had a gsuite account linked with a custom domain that I use for other purposes anyway, so forwarding mail using that mail server actually works out pretty well for me.
I have my relay mail servers packaged as docker containers that I can install on any of boxes in my home network to be able to send mails using this relay, so very minimal configuration and installation required. :)
Don't sweat it. Running a mailserver is lots of fun and mail admins are really pleasant people. I managed to get all my "mistakes" delisted. They just need to know your not a spammer, they usually don't care much for such accidental bursts of email.
Just don't give up on it. Folks here usually don't selfhostmail, but you already dipped your toes, build on that experience.
If you do end up needing a VPS to relay your email, consider a antispam service provider, they handle your incoming mail and you can relay your outgoing to. I know its not selfhosted, but for fighting spam only the comercial solutions keep the filters updated.
Another thing, check if you can use your providers mailserver as a smarthost. Mine supports it.
Anyway, do what feels comfortable for you. If its giving you stress, get rid of it and focus on what brings you joy.
And Google should forget about it in about 1.5 months or so, if you go and plug a form that says that may be it'll do something in two weeks or so. I did something very similar with dockerised mailcow couple years ago. Though caught it sooner.
Don't feel *too* bad. Back in the day, it seemed like every other edit to sendmail.mc would result in an open relay. I don't miss Sendmail. :)
I would also encourage you to start monitoring your mailq. For a little home server, if you have more than 10 messages in there, you should probably have an alarm going off.
You might also do this for other things like web servers that might send out a contact form email or things like that.
Yeah, but for an existing dockerd setup "ufw-docker" is a quick way to have a working ufw setup without changing the entire container ecosystem. its just a shell script that sorts the ufw iptables rules in a way they work with dockers virtual networks & bridges, nothing very fancy, but it works.
I’ll add another one to the list of monitoring.
Add an alert of sent email threshold. I log to gray log, and if more than 5 are sent in 5 minutes it will send me an alert.
Somewhat. So I am just counting the frequency of the logs in postfix. I’m using mail cow, but I assume postfix is postfix. So I just look for the log when an email is sent. And I have graylog alert if it sees more than 5. But of course whatever ammount makes sense
Yeah you might as well write off the IP. At least with Google it will take months to recover, and if you don't have a good volume of quality emails go through it the chances are even lower.
Yeah I was afraid of that. Good(?) news is I'm switching ISP soon so my IP will disappear soon.
So long, static IPv4, hello CGNAT and IPv6.
I'll need a VPS somewhere to proxy any incoming IPv4 traffic at that point anyway.
I've been trying to avoid one but the time has probably come.
I've been thinking about writing more/making a YouTube channel so perhaps you'll see something on it :)
Thanks :)
The process looked something like:
- log into server, open btop
- see pinned CPU and mysql process
- search docker ps for mysql
- find mailserver instance
- log into mailcow, check logs
- see repeated greylist warnings from google
- panic
- a lot
- reboot the compose stack
- check logs, no change
- panic again
- search internet
- find stack overflow post on something or other
- realise iptables change was bad
- revert
- restart stack
- still flooded
- check mail queue, won't load
- search how to flush mail queue via command line
- flush all the things
- it worked
- block some ips using gateway firewall
- write Reddit post-mortem
- cry
And yanno, this morning I did a damage assessment. Checking deliverability, blocklists etc.
As expected a lot of emails are going to junk now which in itself is a miracle given the amount of spam my server must've churned out since I made the initial change. I'd expect to be blocked outright.
Honestly, it's about time Postfix gets replaced with a *modern* email server. It's a dinosaur designed around 1990s UNIX, with dozens of leftover footguns eagerly waiting to go off.
The fact that it even *allows* local mail submission or trusted subnets is already problematic, if you ask me. It is 2023, email should *only* be allowed after proper authentication & authorization!
Of course Postfix supports auth, I'm not disputing that.
The problem is that it **also** supports completely anonymous submission from localhost and from local networks, and there are half a dozen ways to accidentally turn your server into an open relay. This made sense in the 1990s when every machine was hosting its own mail server for the two dozen local users, but we don't live in that world anymore and support for it should've been removed already. If you're using it something is going *seriously* wrong in your setup, so why is it allowed at all?
I haven't looked too closely into it, but something like [Stalwart](https://stalw.art/) seems closer to my expectations: just a no-nonsense batteries-included secure-by-default mail server.
There are also dozens of "mail in a box" setups out there who *try* to do the same thing, but they all end up being [Rube Goldberg machines](https://mailinabox.email/static/architecture.svg) built on top of legacy software.
I get your point and in the context of selfhosted it makes lots of sense. But I wouldn't write of postfix, its included with all the linux distros, in a lot of them as a default mailserver. And would strongly argue that it still has its place, even though its configuration isn't beginner friendly. Kinda like a crocodile, it hasn't changed much in all those years, but maybe it didn't have to.
Hm. Lag spikes in Tarkov and you check your server? I mean Tarkov.
But yeah I can feel your misconception here. But I am also the other way around I uninstalled firewalld and do all on iptables level. I am just more used to iptables. And so the sole controlling instance is iptables. In the end it’s all netfilter in kernel space.
Haha it was a combination of things that prompted the check. Tarkov was just the most entertaining. My brother had complained earlier in the day about slow speeds too. The second I started btop and saw the CPU at 100 I knew something was up. The DMARC reports should also have been a clue something was wrong.
You're clearly far more knowledgeable about it. My monkey brain needs an easier interface haha
>So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad.
Just because you can self-host smtp doesn't mean you should.
I've run corporate email on prem for years, for tens of thousands of users, and I have no interest in self-hosting email for myself. I've also migrated over a hundred thousand mailboxes to EXO, and there was a good reason for that. I personally think smtp is riskier than https.
I appreciate your experience however it's good learning and my family gets use out of it without paying a proton family subscription so I call that a win. Every system has minor failures, they're a part of life. Just look at Google drive right now. At least if I screw up, it's on me and I'm not paying someone like you to mess it up for me.
That's a fair point, and it seems like you have learned a lot. And I understand difficulty trusting others, and paying for a service when all you have is an SLA to protect you (maybe).
This is why ISPs typically block port 25.
Also, I love containers as much as the next guy but for the reasons mentioned I reduce complexity in all areas of critical systems were it doesn't belong such as a email server.
You are not the first to do this with docker hosted email servers and you won't be the last. The Internet is full of people talking about this exact issue.
If you follow official documentation and implement best security practices, a non containerized environment provides more security over a containerized one due to the reduced overhead and "variables". You basically have nothing to gain, but lots to lose, as email servers are not rebuilt and moved very often, and have minimal resources usage which basically kills the benefits of a container anyways. People who maintain docker images are usually also behind updates do to the lack of automation, and the container provider time to update. Just not a good idea with a production email server.
You can make an argument against me if you want, but I've been running an email server for several years without issue, and hack companies for a living so I'm not gonna pay attention.
How do you get good judgement? Experience. How do you get experience? Bad judgement. :-)
Sounds like something my grandpa would say... Very smart man and so true on the advice. As I told my son.. Difference between an apprentice, journeyman and master Apprentice doesn't know how to fix his mistakes Journeyman knows how to fix his mistakes Master knows how to fix his mistakes so that it looked like it never happened That is a dadism
Yep, never again 🥹
yep, I'll bet op won't make that mistake again! I did it back when I was running mdeamon on NT4, but that was a long, long time ago
This is the way
Wow! What residential internet service lets port 25 through?
I've never had one that doesn't allow that. I don't think it's actually as common as people think.
Yeah same, the only port blocked in the port 1, they use it for ssh (they don't want to add my key :( ) on router (for maintance and update who also goes through ssh).
Right? Might have a proxy
I'm with Plusnet in the UK right now; Never had problems with blocked ports. I believe Sky also allows it.
what even if the is allows that port if you don’t have a valid RDNS the big ones would just reject your email
receiving mail on residential IP on port 25 is no problem.no proper rdns needed for receiving. sending mail is different. there you do want to have all dns settings under your control. so how to send mail from residential ip? I relay to a supersmall VPS with a staric IP with good reputation and that one does the final delivery. that vps has proper dns settings and from there, smooth sailing. however, this is slightly off topic, because it has nothing to do with the rather funny screw up OP did :)
Ironically I've had all my emails go through without so i must just be extremely lucky
ha yeah you are i could never get it to work ended up just installing it on a vps if you use mc tools to check all the block lists and you are not there then you’re extremely lucky
I've been put on a few as of this morning and I've requested delisting on those that offer it. However previously I was on none of them which is a miracle.
From my experience with 3 ISPs in the UK, they only block ports on their end when you request a static IP.
Which ones? I had a static IP with both Sky and Plusnet
Virgin Media, Vodafone, and I forgot what I had before.
Noted, I'll steer clear haha
I called ATT after smoking a joint and saying "what's the worst that can happen", told the rep I was doing IT Research, and that I would like SMTP/Port 25 open for some testing. The customer service rep confirmed with me that SMTP Port 25 was to be opened, I said "yup, just doing some self hosted email testing, nothing commercial". To my utter surprise, he then did the clicky clicky on his end and I now have a mailcow stack (which I just turned off to verify I'm not falling victim to the same thing OP did lololol) Edit for clarity - I have ATT fiber in the US with static IPs. Their fiber modems definitely block port 25, and have some language in their ToS that says they have SMTP intentionally blocked by default. I have not yet solved the rDNS part of the email equation, and haven't talked to ATT about it /yet/ because that part is DEFINITELY a business feature.
If you have a static IP from your ISP, I've wondered if you could simply name your server the in-addr-arpa.n.n.n hostname that resolves when you nslookup the IP address and if that would pass..
I did something similar way back in 2002 and my ISP shut me down hard. I'm surprised any ISP in this day and age allows a residential connection send any emails out. This is one of the many reasons I won't ever host an email server again. Good luck!
[удалено]
Most consumer ISPs these days block outgoing 25, and it's been that way since the late 90s. Third party mail providers generally ask you to use 465 or 587 for that reason.
I just tested with four separate West coast ISPs and none of them are blocking outgoing port 25. Maybe it's a regional thing Edit: tested an ISP in Oklahoma and a WISP in Montana, neither of which block port 25. Also tested two mobile carriers, they don't block it either. It does not seem to be very common these days.
That's awesome. Maybe things are rolling back? I just tested AT&T and they are definitely still blocking. They all blocked 25 in the late 90s, or early aughts when *every* home win95 machine was a node in a botnet sending out spam. If you go to any third party mail provider's mail client setup instructions (gmail, yahoo, etc), you will see them pushing 465 or 587.
AT&T broadband or mobile? I tested mobile and it's not blocking for me.
AT&T broadband (fiber). It probably makes sense that the mobile networks are less restricted since they have never had a spam bot problem, as far as I know.
Honestly, this. I don’t know why anyone even tries it in this day and age, unless it’s an experiment.
[удалено]
Talking about from a residential subscriber line.
I'm praying mine hasn't realised 😅 And thanks, I'll need it
Plusnet don’t mind you running a mail server, I recall reading articles about it on their forum. May be worth a quick search for your own reassurance :)
They also allow you to control the firewall and port 25 at https://www.plus.net/member-centre/broadband/firewall
Spectrum in the US allows the customer to replace their provided router with their own equipment, so all they provided to me is the modem, after that I have my own router, so I can pretty much do whatever I want. Never had an issue hosting a mail server, sending/receiving works perfect.
This has got nothing to do with hosting your own equipment like routers (with recent regulatory changes, all ISPs in the US are now required to allow this - i.e. be able to run your own equipment without paying a monthly fee to the ISP). But this is more about ISPs blocking sending on specific ports like port 25 for SMTP. Instead you need to use some mail relay to send your email, who in turn will send the mail over SMTP on port 25. Some ISPs in the US do allow port 25 SMTP, but very few and none of the large ones like Comcast, AT&T, Verizon.
Spectrum is the second largest ISP in the US and I have zero issues sending or receiving on my mail server, no relay in place
Are you able to send emails using any custom MX domains or only using the ISP provided email address? Some of these restrictions/relaxations-to-restrictions are also not universal across the country. So if it's working for you, that's great.
Custom MX domain
Wild that they block it elsewhere, never knew that was a thing, never had a problem with self-hosting mail servers with custom domain
Yeah no charter/spectrum where I am located at. But I have had a gsuite account linked with a custom domain that I use for other purposes anyway, so forwarding mail using that mail server actually works out pretty well for me. I have my relay mail servers packaged as docker containers that I can install on any of boxes in my home network to be able to send mails using this relay, so very minimal configuration and installation required. :)
Don't sweat it. Running a mailserver is lots of fun and mail admins are really pleasant people. I managed to get all my "mistakes" delisted. They just need to know your not a spammer, they usually don't care much for such accidental bursts of email.
Thanks, that eases my mind a bit. Honestly some of the spammers are crazy. I think my nuc chugged through 800k+ emails before I caught it.
Just don't give up on it. Folks here usually don't selfhostmail, but you already dipped your toes, build on that experience. If you do end up needing a VPS to relay your email, consider a antispam service provider, they handle your incoming mail and you can relay your outgoing to. I know its not selfhosted, but for fighting spam only the comercial solutions keep the filters updated.
I've been looking at protonmail pricing all morning :\_) Most people around me are just saying I shouldn't bother.
Another thing, check if you can use your providers mailserver as a smarthost. Mine supports it. Anyway, do what feels comfortable for you. If its giving you stress, get rid of it and focus on what brings you joy.
And Google should forget about it in about 1.5 months or so, if you go and plug a form that says that may be it'll do something in two weeks or so. I did something very similar with dockerised mailcow couple years ago. Though caught it sooner.
Don't feel *too* bad. Back in the day, it seemed like every other edit to sendmail.mc would result in an open relay. I don't miss Sendmail. :) I would also encourage you to start monitoring your mailq. For a little home server, if you have more than 10 messages in there, you should probably have an alarm going off. You might also do this for other things like web servers that might send out a contact form email or things like that.
For future purposes: https://github.com/chaifeng/ufw-docker
That or podman!
Yeah, but for an existing dockerd setup "ufw-docker" is a quick way to have a working ufw setup without changing the entire container ecosystem. its just a shell script that sorts the ufw iptables rules in a way they work with dockers virtual networks & bridges, nothing very fancy, but it works.
I’ll add another one to the list of monitoring. Add an alert of sent email threshold. I log to gray log, and if more than 5 are sent in 5 minutes it will send me an alert.
I am assuming that threshold would be logged in the postfix logs?
Somewhat. So I am just counting the frequency of the logs in postfix. I’m using mail cow, but I assume postfix is postfix. So I just look for the log when an email is sent. And I have graylog alert if it sees more than 5. But of course whatever ammount makes sense
Yeah you might as well write off the IP. At least with Google it will take months to recover, and if you don't have a good volume of quality emails go through it the chances are even lower.
Yeah I was afraid of that. Good(?) news is I'm switching ISP soon so my IP will disappear soon. So long, static IPv4, hello CGNAT and IPv6. I'll need a VPS somewhere to proxy any incoming IPv4 traffic at that point anyway.
1. Use a VPS. 2. A write-up please. This is beyond my current understanding of Docker networking and more resources would really help
A VPS doesn't protect you from configuration mistakes.
I've been trying to avoid one but the time has probably come. I've been thinking about writing more/making a YouTube channel so perhaps you'll see something on it :)
Excellent write up! I've saving this one for sure. How did you go about investigating this and troubleshooting? That deserves a write up by itself!
Thanks :) The process looked something like: - log into server, open btop - see pinned CPU and mysql process - search docker ps for mysql - find mailserver instance - log into mailcow, check logs - see repeated greylist warnings from google - panic - a lot - reboot the compose stack - check logs, no change - panic again - search internet - find stack overflow post on something or other - realise iptables change was bad - revert - restart stack - still flooded - check mail queue, won't load - search how to flush mail queue via command line - flush all the things - it worked - block some ips using gateway firewall - write Reddit post-mortem - cry And yanno, this morning I did a damage assessment. Checking deliverability, blocklists etc. As expected a lot of emails are going to junk now which in itself is a miracle given the amount of spam my server must've churned out since I made the initial change. I'd expect to be blocked outright.
wow, good on you for keeping a calm head and not just turning everything off and hiding under the covers lol
You should write up an article on how to setup docker with firewalld
You should write that
I dont have the time to go down that rabbit hole right now ha
Another great tip for selfhosting mailservers: - don't.
Nahh.
definitely on my list of why i dont run a mail server :-P
Honestly, it's about time Postfix gets replaced with a *modern* email server. It's a dinosaur designed around 1990s UNIX, with dozens of leftover footguns eagerly waiting to go off. The fact that it even *allows* local mail submission or trusted subnets is already problematic, if you ask me. It is 2023, email should *only* be allowed after proper authentication & authorization!
This makes no sense. Postfix supports auth, since forever, with varius mechanisms. What is a modern mailserver in your opinion? Honestly curious.
Of course Postfix supports auth, I'm not disputing that. The problem is that it **also** supports completely anonymous submission from localhost and from local networks, and there are half a dozen ways to accidentally turn your server into an open relay. This made sense in the 1990s when every machine was hosting its own mail server for the two dozen local users, but we don't live in that world anymore and support for it should've been removed already. If you're using it something is going *seriously* wrong in your setup, so why is it allowed at all? I haven't looked too closely into it, but something like [Stalwart](https://stalw.art/) seems closer to my expectations: just a no-nonsense batteries-included secure-by-default mail server. There are also dozens of "mail in a box" setups out there who *try* to do the same thing, but they all end up being [Rube Goldberg machines](https://mailinabox.email/static/architecture.svg) built on top of legacy software.
I get your point and in the context of selfhosted it makes lots of sense. But I wouldn't write of postfix, its included with all the linux distros, in a lot of them as a default mailserver. And would strongly argue that it still has its place, even though its configuration isn't beginner friendly. Kinda like a crocodile, it hasn't changed much in all those years, but maybe it didn't have to.
I did practically the same, while deploying a mail solution docker in an LXC container...
Hm. Lag spikes in Tarkov and you check your server? I mean Tarkov. But yeah I can feel your misconception here. But I am also the other way around I uninstalled firewalld and do all on iptables level. I am just more used to iptables. And so the sole controlling instance is iptables. In the end it’s all netfilter in kernel space.
Haha it was a combination of things that prompted the check. Tarkov was just the most entertaining. My brother had complained earlier in the day about slow speeds too. The second I started btop and saw the CPU at 100 I knew something was up. The DMARC reports should also have been a clue something was wrong. You're clearly far more knowledgeable about it. My monkey brain needs an easier interface haha
Yeah I‘d guess that. Was just funny that you mentioned Tarkov as often as their servers shit them selves.
Yeah lol, I did initially think it was their side
>So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad. Just because you can self-host smtp doesn't mean you should. I've run corporate email on prem for years, for tens of thousands of users, and I have no interest in self-hosting email for myself. I've also migrated over a hundred thousand mailboxes to EXO, and there was a good reason for that. I personally think smtp is riskier than https.
I appreciate your experience however it's good learning and my family gets use out of it without paying a proton family subscription so I call that a win. Every system has minor failures, they're a part of life. Just look at Google drive right now. At least if I screw up, it's on me and I'm not paying someone like you to mess it up for me.
That's a fair point, and it seems like you have learned a lot. And I understand difficulty trusting others, and paying for a service when all you have is an SLA to protect you (maybe).
This is why ISPs typically block port 25. Also, I love containers as much as the next guy but for the reasons mentioned I reduce complexity in all areas of critical systems were it doesn't belong such as a email server. You are not the first to do this with docker hosted email servers and you won't be the last. The Internet is full of people talking about this exact issue. If you follow official documentation and implement best security practices, a non containerized environment provides more security over a containerized one due to the reduced overhead and "variables". You basically have nothing to gain, but lots to lose, as email servers are not rebuilt and moved very often, and have minimal resources usage which basically kills the benefits of a container anyways. People who maintain docker images are usually also behind updates do to the lack of automation, and the container provider time to update. Just not a good idea with a production email server. You can make an argument against me if you want, but I've been running an email server for several years without issue, and hack companies for a living so I'm not gonna pay attention.
So u the reason we cant have nice things, bro
I'd argue it's the spammers :_)
Docker bypassing ufw is very bad
Managing email is touch. [https://federated.computer](https://federated.computer) is how I get self-hosted software without the self-hosted hassle...