Confirmation of the legality of mailings. Bulletproof hosting You get your VDS for free use

What can be hosted on bulletproof hosting

What is the price

The reason for this was repeated spam complaints from our hosting and VPS clients. It is not always possible to say with certainty whether these were intentional actions of customers or whether they themselves did not suspect that they had become a victim of spam bots. Whatever it was, the problem had to be solved.

Spam is not liked. Spam leaves a "black spot" on the face of the provider when its IP addresses are blacklisted, which affects all customers. Removing IP from blacklists is a special conversation. But this is one side of the coin. If it is possible to restore the reputation of an IP address, then it is much more difficult to restore the company's reputation and trust.

We decided to find a solution and implement a complex to protect and prevent unwanted mailings in the Unihost structure. After some brainstorming and discussion, we started testing and comparing what the SPAM/AV community has to offer.

There are many options on the market. However, most of the high-quality solutions are paid with tariffication of 1 license for 1 server or even for the number of outgoing/incoming messages, which would lead to higher tariffs. Therefore, they chose only among opensource.

Popular open source anti-spam solutions

Rspamd

It is suitable for systems of various sizes. It can integrate into various MTAs (Exim, Postfix, Sendmail and Haraka are described in the documentation) or work in SMTP proxy mode.

The message evaluation system is the same as in SpamAssassin, in particular, based on various factors: regular expressions, DNS block lists, white, gray, black lists, SPF, DKIM, statistics, hashes (fuzzy hashes) and others - only used in work other algorithms.

Rspamd supports extension via plugins.

Apache SpamAssassin

SA gained fame through the use of Bayesian filtering technology. Each message, when passing tests, receives a certain score and, when the threshold is reached, is placed in spam.

Easily integrates with almost any email service. Popular technologies are available in SA that are connected as plugins: DNSBL, SPF, DKIM, URIBL, SURBL, PSBL, Razor, RelayCountry, automatic whitelisting (AWL) and others.

Installation is generally not difficult. Once installed, SpamAssassin requires fine tuning and learning from spam emails.

ASSP

A platform-specific SMTP proxy server that receives messages before the MTA and analyzes it for spam.

All popular technologies are supported: whitelisting, greylisting, Bayesian filtering, DNSBL, DNSWL, URIBL, SPF, DKIM, SRS, virus scanning (with ClamAV), blocking or replacing attachments, and much more. MIME-encoded spam and pictures are detected (using Tesseract). Possibilities are expanded with the help of modules.

The project documentation is not always intelligible, and the instructions are often already outdated, but with some experience you can figure it out.

mailscanner

MailScanner is an all-inclusive solution for fighting phishing emails and checking mail for viruses and spam. It analyzes the content of the email, blocking attacks against email clients and HTML tags, checks attachments (prohibited extensions, double extensions, encrypted archives, etc.), controls email address spoofing, and much more.

MailScanner easily integrates with any MTA, the delivery includes ready-made configuration files. In addition to his own developments, he can use third-party solutions. SpamAssassin can be used to check for spam.

EFA project

There is another Open Source project - "eFa-project" - Email Filter Appliance. EFA was originally designed as a virtual appliance to run on Vmware or HyperV. The program uses ready-made packages MailScanner, Postfix, SpamAssasin (the whole list below) to stop spam and viruses, and they are already installed and configured to work properly in vm. This means that crutches are not needed - everything works out of the box.

The EFA includes the following components:

Postfix acts as an MTA (mail transfer agent) - reliable, fast, proven over the years;
The core of the spam filter - MailScanner - shoulder to shoulder with the antivirus take the whole hit;
Spam filter - SpamAssassin - detects spam messages. The framework includes many scoring systems, MTAs, and sets of regular expressions;
ClamAV is an antivirus that works with MailScanner;
MailWatch - a convenient web interface for working with MailScanner and other applications;
The content filter - DCC - defines mass mailing by sending hash sums of the message body to a special server, which in turn provides a response in the form of the number of hashes received. If the number exceeds the score=6 threshold, the message is considered spam;
Pyzor and - help SpamAssassin recognize spam more accurately using spam detection networks;
For gray listing, the postfix policy service is used to reduce the amount of spam that can be received by recipients;
For image recognition, the ImageCeberus module is used - it determines porn images, etc.
We chose EFA because the project includes all the best features of the above. In addition, our administrators already had some experience with it, so we chose EFA. Let's proceed to the description of the installation.

Installation and subsequent configuration of EFA

We decided to install it on a VPS with clean CentOS 6.8 x64, which acts as a relay server. First of all, you need to update all system utilities and components to the latest versions that are available in the repositories. For this we use the command:

Then install the wget and screen utilities if they were not installed:

After that, download the script that will install EFA:

Give the script permission to execute:

Run screen:

And run the script:

Now we can minimize our screen using the combination Ctrl + A + D.

After installation, you need to re-login to the server via ssh using the credentials for the first login. This is needed to run the initialization script and initial setup of EFA.

After logging in, the system prompts you to answer a few questions in order to set up EFA.

The list of questions looks like this:

After such a questionnaire, the entire list of answers is displayed. If something needs to be changed, dial the question number and enter new data. When ready to move on, type OK and press Enter. The system will start the auto-tuning process.

Upon completion of the configuration, the system will reboot and will be in full combat readiness.

This is a list of the main EFA options that are not editable through the MailWatch web interface. Therefore, it is good to know where to find them.

Manual setting of EFA

We went the hard way, but more flexible. Setting up EFA for yourself was not done through an interactive menu, but the configuration files were corrected. We wanted to not only set everything up, but also to understand all the components and understand what works and how.

First of all, in the main.cf file of the postfix settings, we added mynetworks, from which connections were accepted via SMTP. Then we set restrictions on helo requests, senders, recipients, and indicated paths to maps with ACCEPT or REJECT policies, subject to certain conditions. Also, inet_protocols has been changed to ipv4 to exclude ipv6 connections.

Then we changed the Spam Actions policy to Store in the configuration file /etc/MailScanner/MailScanner.conf. This means that if an email is identified as spam, it will be quarantined. This helps to further train SpamAssassin.

After these settings, we ran into the first problem. We were bombarded with thousands of letters from recipients [email protected], [email protected], [email protected] etc. The recipients were the same. We also received letters sent by MAILER-DAEMON, that is, in fact, without a sender.

As a result, we got a clogged queue without the ability to find normal, non-spam letters among the “red canvas”. We decided to REJECT such letters using the standard functionality of Postfix cards: helo_access, recipient_access, sender_access. Now harmful recipients and the like have successfully REJECTed. And those letters that were sent by MAILER-DAEMON are filtered out by helo requests.

When the queue was cleared, and our nerves calmed down, we began to configure SpamAssassin.

SpamAssassin training

SpamAssassin training is done on emails that have already been spammed. You can do this in two ways.

Via web interface

The first way is through the MailWatch web interface. In each letter, you can see the headers, the body, as well as the Bayesian score and other indicators. It looks like this:

After opening the letter, you can check the "SA Learn" checkbox and select one of several actions:

• As Ham - mark email as clean (Bayes algorithm training);
• As Spam - mark the message as spam (Bayes algorithm training);
• Forget - skip letter;
• As Spam+Report - mark the message as spam and send information about it to the spam detection network (razor + pyzor);
• As Ham+Revoke - mark the message as clean and send information about it to the spam detection network (razor + pyzor).

Through the console

This is done simply. The command looks like this:

In this command, the email with ID: 0DC5B48D4.A739D, which is in the spam email archive for the specific date /20170224/spam/, is marked as clean (not spam) bash--ham .

There is an opinion that it is enough to train SpamAssassin only for effective mail filtering. We decided to train SpamAssassin by feeding him absolutely all messages, both clean and spam. In addition, we found a database of spam emails and handed SA over to be torn to pieces.

Such training helped to more accurately calibrate the Bayesian algorithm. As a result, filtering is much more efficient. We conduct such trainings when the mail traffic is not very high in order to have time to analyze and capture the maximum number of letters.

In order for SpamAssassin to start working at full capacity, at the start it needs to feed about 1000 different letters. So be patient and start exercising.

It is still too early to talk about a complete victory over spam. However, now the number of spam complaints from our servers is zero. We won’t talk in more detail about the learning process itself now - I don’t want to reveal all the chips. Although, if you dig deeper into the settings, it's not difficult to figure it out.

For those who are too lazy to watch a 45 minute video on setting up SMTP on VDS

Preparatory stage

1) We register a mailbox (to which we will register all services). We enter all the data in the file so as not to lose it.

2) Register on reg.domainik.ru (or any other domain registration service) and buy a domain. If you want to get a domain for free, then read the article
We prescribe DNS Servers ns1.firstvds.ru and ns2.firstvds.ru (well, or NS of your hosting)

3) On firstvds.ru (or your hosting) we take a VDS server (the simplest one is 150r for the test, although the pros buy the most expensive one). Promo code for a 25% discount: 648439800. Confirmation of the number in the account through the sms-reg.com service Select the server template "Debian amd64" without ISPmanager. After payment, you need to wait until the server status becomes "Active" and an IP is assigned

Server Tuning

2) After registration, go to the ISPSystem Licenses section and enter the IP of our VDS. Next, go to the hosting (VDS) in the "Virtual Servers" section - click on our server and on top the "To Panel" button. There we select our server - "Reinstall" and select Debian-amd64-ispmngr and specify the password that was sent after activating the VDS. Next, wait a bit. In the "Containers" section, click "ISPmgr" and you should be transferred to ISPmanager

3) In the ISPmanager of your VDS, go to "Users" - "Create" and create a user, in the steps indicate your domain that you bought.

4) Next, go to "Domains-Domain Names" - your domain should already be there, select it and click NSy - "Create" and indicate the data that came in the letter to the mail that you specified when registering VDS. The subject of the email is "Changing DNS access settings…." enter all the data from the letter, including ns1.firstvds.ru and ns2.firstvds.ru and check the box "Apply to existing"

6) Create 4-5 mailboxes in the "Mailboxes" section.

DKIM setup

1) In the ISP panel of YOUR VDS server in the section go to "Settings-Software" find "Mail server" (SMTP) it should be "exim", if not - click the "Delete" button, then "Install" and select "exim -daemon-heavy" click "Next" - "Next" - "Finish". Installation completed. Next, select "Opendkim - DKIM filter" click "Install" (if you need to update the software in the process, then update it). When everything is installed (SMTP and DKIM) the status will be “yellow light”

2) Go to "Domains-Mail Domains" select our domain, click 2 times and check the box "Enable DKIM"

3) Go to "Domains-Domain Names" and see "Entries" there should be a long DKIM entry

Hiding IP

1) In the ISP panel of YOUR VDS server “System-File Manager”, click “Back” - then the etc folder, then exim4 and select the exim.conftemplate file 2 times click on it and edit it.
Paste the code after the +smtp_protocol_error line and before the TLS/SSL line and click "Save"

2) On the VDS server, in the "Containers" section, click "Restart"

Sending with eMail Mailer

1) In ePochta Mailer, click Settings-SMTP, set "SMTP only", click + and add our server.
We insert IP port 587. "Authorization" - AUTH PLAIN (obsolete). Encryption - "No". Login - one of our mails (which was registered in the panel on the VDC) and its password and insert our mail into the sender's email. "Threads" 1. For the "Wait" test, set 1 second. "After" 1 letters. Click OK.

2) We write any letter and click "Test" and check the deliverability to our mailboxes (Yandex, Mailru)

3) By analogy, the remaining mailboxes are added

4) Recommendations when mailing in the first days to send no more than 3000 letters / day to Mailru. The “Wait” setting is 7 seconds. It is also recommended to use
"Text substitution" for letter randomization. If the domain is banned, then a new domain is added and configured according to the instructions

The cost of the whole thing is 250 rubles.
Savings on viewing the course 1.5 hours of time.

I wish you all good luck!

Internet projects actively use email newsletters to communicate with customers.

• Advertising and informational, based on verified addresses
• Automatic: registration, ordering, order status change.
• Bulk spam mailings- letters sent without the consent of the receiving party

Yes, some of our clients send spam. Mail services are struggling with it - they add the IP addresses of the sending servers to the block lists. Mail from such addresses never gets into mailboxes again, but is immediately deleted.

We make sure that our customers do not send spam. For this filters are in effect- they identify malicious emails. As soon as the filter works, we block port 25, and the server can no longer send emails.

Automatic filters also work on letters, not spam. To prevent this from happening, apply in advance to be added to the white list.

White list

Works only for KVM servers. To be added to the white list, write a support request and let us know:

• How do subscribers agree to receive newsletters? Show a screenshot of the newsletter subscription form. If a person agrees with mailings when registering in the service, then send a screenshot of the registration form and an excerpt about mailings from the rules for using the resource.
• An example of a mailing letter: the original (the source code of the letter) and a screenshot of the letter. The letter must contain a link with the ability to unsubscribe from the mailing list.
• Screenshot of the page that opens from the link above. Unsubscribing should be simple, without authorization on the site.

Installing your own mail server, as a rule, does not cause any particular difficulties. A large number of ready-made instructions are available on the Web. Literally one command, and the 25th port is ready to go. It's fun when sent emails start bouncing back and recipients complain that messages aren't getting through. Here you like it or not, but you have to look for reasons and delve into technologies.

Who sends letters

Today, many web services offer the ability to link your domain to a service. Posting mail on Gmail or Yandex is especially popular. All messages will go through the SMTP server provided by them, a verified service provider will itself generate all the necessary headers and signatures that will allow you to pass through any spam filter. But this option is not always possible. For example, an organization has a large number of users, needs special settings for mail that are not available in cloud services. Or you use your own server with a portal, CMS or an online store from which you need to send messages.

By default, all PHP applications use the mail() function to send mail, which in turn sends them through the local SMTP server defined in php.ini .

sendmail_path = /usr/sbin/sendmail -t -i

Or in virtual host:

php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f [email protected]"

And although sendmail is written there in 100% of cases, in fact it can be a symlink, and Postfix or Exim sends mail. To send mail from the application, you can choose one of three options:

• The engine itself sometimes allows you to specify an external SMTP server (in default settings or through a plugin, in WordPress it is WP Mail SMTP or Easy WP SMTP). Just enter your account information and all problems are solved.
• Using a shim program that emulates a local SMTP server and sends messages through a mail account on a third-party server. SSMTP is very popular here.
• Using your mail server. Of course, you will have to configure it, but there are more configuration options.

We are interested in the last option. Let's figure out how to break through anti-spam technologies and guarantee the delivery of a message to the recipient. We will not filter spam ourselves. This is the topic of another article. As an experimental SMTP server, we will choose Postfix and Exim, they are popular on hosting, simple and understandable in settings, although the main questions will concern all SMTP servers.

How not to get into spam

Fighting spam is a headache for all mail administrators. And recently, just the other side of the coin has been relevant: spam filters are literally atrocious. Therefore, there is practically no spam in incoming mail, but normal messages are constantly disappearing somewhere, customers and management are nervous, and you have to additionally make sure that the message has reached the addressee. And after installing the SMTP server, you will most likely have to tinker with it so that messages reach at least somewhere. In particular, to evaluate the settings, you should see if letters are delivered to mailboxes of the main mail systems Gmail, Yandex, Mail.Ru. Usually at this stage the first difficulties appear, and you have to solve all the problems personally.

Mail services use a multi-level spam filtering system, so serious and secret that even their own technical support does not know about the principles. And each service has its own priorities. Although usually some hint about the reason for non-delivery is contained in the response letter of the service. The mail-tester.com service also helps in the analysis of the causes, it is enough to send a letter to the address indicated there and then, after the analysis, receive the result and a list of problems. Some of them can be checked and solved without setting up an SMTP server yet.

The fight against spam has spawned many technologies. The oldest of them is the blacklist, which contains all IPs and domains involved in sending spam, open relays, proxies and Dialup addresses used for remote access can also get here (that is, they theoretically should not send mail). Such blacklists are organized in different ways. Popular DNSBL (DNS blacklist) - blacklists in DNS format, which are easy to query. There are many bases available today, not all of them are popular and used. The problem is that there is no list for a specific mail service, how many and which ones they poll - this is a mystery.

Domain names, like IP addresses, can be used today. There is a possibility that a messaging service used them before you or the host hosted on it was hacked and sent spam. Accordingly, they may well fall into one of the DNSBLs and be a problem. Mail.Ru rejected letters from one IP precisely because it was on one of these half-forgotten lists, having got there in 2010. Moreover, Mail.Ru did not even bother to check the correctness of SPF and DKIM. The matter moved only when the IP was removed from the black list.

You can check the IP or domain yourself by sending a DNS query to the selected DNSBL server using the dig utility:

$ host -tA site.ex.dnsbl..ex.dnsbl.org not found: 3(NXDOMAIN)

But it is more convenient to use online services that check in several databases at once. The IP can be checked in dnsbl.info (59 bases) or whatismyipaddress.com (72 bases), the domain can also be checked in mxtoolbox.com (107 bases), spamhaus.org or multirbl.valli.org. If suddenly a domain or IP is on the list, it is better to immediately write to support and remove your address.

Correct DNS

When a message is received, the remote SMTP server first parses its header. The mailer only sends From, To, Date, Subject and X-Mailer. They are generally understandable and simply indicate from whom and where to send. The rest of the header is generated by both the SMTP server and the application that sends it. This, by the way, also needs to be taken into account, because letters sent via Telnet can go away, but not with Roundcube, simply because they have a different header. Roundcube, for example, substitutes its HELO/EHLO based on the server_name or localhost variable if it is not defined. So sometimes you just need to set it explicitly:

$rcmail_config["smtp_helo_host"] = "example.org";

The same applies to self-written PHP scripts.

During transmission, the letter will go through at least two SMTP servers, each of which also adds something of its own to the header. First of all, each server adds its Received: from. It is better to read them from bottom to top. The bottommost message is the sender's server, the topmost is the recipient's server. Although in reality there may be more servers, this is especially true when working with large service providers who, after receiving the letter, transfer it further, or when using an SMTP proxy along the way. To analyze the message path, you can use a service from Google, which will show in an understandable form all SMTP servers, transit times and SPF, DKIM and DMARC tests (more on them later).

The Received headers are different, although there are general rules. A typical one looks like this:

Received: from server.example.org (helo=server.example.org) by st15.provider.com with esmtps (Exim 4.80.1) (envelope-from )

Here the message was received from a server called server.example.org, has an IP of 1.2.3.4, the same name was used in the hello helo, received by Exim 4.80.1 of the server st15.provider.com. Message sent from [email protected] Having received such a header, the SMTP server starts checking the data. Breaks the domain and IP on DNSBL bases. Checks if a domain has an MX record. MX is initially used to find mail servers serving a given domain, its presence confirms that the domain is sending mail.

Next, it performs a reverse name resolution over IP through a reverse DNS query using a PTR record. That is, it will find out what name the server should be at the address from which the message came. This behavior was specified in RFC 2505 of February 1999 Anti-Spam Recommendations for SMTP MTAs. And although it has long been recognized that reverse zones are not sufficient for unambiguous identification of the sender and often lead to errors and delays, they are still supported. Therefore, they must match, otherwise the message will at least get a minus in the rating, and in the worst case, it will be discarded.

In our example, server.example.org should be assigned to IP 1.2.3.4. The DNS entry looks like this:

1.2.3.4.in-addr.arpa. IN PTR server.example.org

For IPv6, ip6.arpa is used. In principle, it is not necessary to know about the features of PTR, since PTR, with rare exceptions, is configured only by the hosting provider. And if it does not suit you, then you just need to contact support. You can check PTR using the query:

$ dig -x 1.2.3.4

In fact, the PTR record after deploying the VDS may point to the technical domain provided by the provider, like srv01.provider.net , in the VDS template the hostname is entered as Ubuntu1604 (it changes in /etc/hostname), in HELO/EHLO the SMTP server generally writes localhost .localdomain , and the email is from the example.org domain. The probability of delivering a letter under such conditions will rapidly approach zero. Although some services mark such inconsistencies as an error and conduct a full check.

I would especially like to draw attention to the fact that VDS usually has two IPv4 and v6. Therefore, everything that has been said applies to both versions, since a letter to one server can go over IPv4 and be delivered, while the other prefers to use IPv6, and the letter may not reach the recipient. At the same time, a lot of providers, providing IPv6, absolutely do not bother setting up a PTR record, and checking it returns an error. But Google, for example, prefers IPv6 and immediately discards the email if the PTR doesn't match the server name. In the response message of the service, it looks like this:

Continued available to members only

Option 1. Join the "site" community to read all the materials on the site

Membership in the community during the specified period will give you access to ALL Hacker materials, increase your personal cumulative discount and allow you to accumulate a professional Xakep Score rating!