Blogs

ISPConfig3 with Mail Quarantine, Roundcube and Amacube plugin

Netuxo recently migrated its mail hosting from an aging server using postfixadmin to a new server which forms part of our wider ISPConfig3 based hosting infrastructure. The default ISPConfig3 installation uses amavisd-new, all administered via the ISPConfig3 interface, which in our case is on a different server (a master-slave setup with one master server and several slave servers). We also wanted to provide the option to quarantine spam and virus emails, as several clients had expressed their fear of losing that important email because it could just get bounced (and they wouldn't know where this important email would come from, so whitelisting wasn't an option). It took us a while to figure out a good way of doing this, and this blog post describes what we did.

Our setup

Our setup is as follows:

  • A ISPConfig3 master server on server A. This server also provides webmail services using Roundcube.
  • A mailserver (server B), with ISPConfig3 as slave to the master on server A.
  • The ISPConfig3 Roundcube plugins, which allow users to control many aspects of their email account (such as vacation reply, forwarders, spam policy) from Roundcube.
  • A badly hacked version of the Amacube Roundcube plugin to access the mail quarantine. We did not want users to change their amavis settings using this plugin, as this is dealt with via the ISPConfig3 plugins.
  • For us as system admins a version of Mailzu to access the site or server quarantine.

Installation and configuration

We started with the default ISPConfig3 installation on a Debian Stretch server, adding server B as a slave to server A during installation. Server B only provides mail services, and in theory does not even need a web server, but this does not matter here. After setting up server B as mail server, we migrated all mail domains, accounts, aliases, forwarders etc from postfixadmin (this might be the topic for another blog post), so that they exist in ISPConfig3's databases on both, the master and slave servers.

The next step was the installation of the ISPConfig3 Roundcube plugins. The installation is well documented on the github page, so we do not cover it here. The set of plugins make use of the ISPConfig3 API interface and access the master server A. In our case, and following the installation instructions, this worked well out of the box. ISPConfig3 takes then care of replicating the settings to the slave server(s).

Mail Quarantine

We mostly followed the manual by Mathieu at https://uname.pingveno.net to set up mail quarantine. In our case we opted for creating the additional tables for mail quarantine in the Amavis/ISPConfig3 database, mostly because Amacube later gave us a lot of trouble trying to use two different databases. There is no problem with having these additional tables in the ISPConfig3 database on only the slave server B, as ISPConfig3 will just ignore them. Important is that these tables need to be on the server actually running amavisd-new, NOT on the master server (unless the master server happens to be your mail server).

Amacube

Amacube is where it gets complicated, and where we had to start hacking. For a start, Amacube is designed to access the amavis database directly, and in its original form allows a user to manipulate values in the amavis database. This is not what we wanted here, for two reasons:

  • As in our case we have a master->slave setup via ISPConfig3, manipulating the database on one server (server B) would get ISPConfig3 confused. Amacube does not do it the ISPConfig3 way - it just is not designed to do so (and there is nothing wrong with that).
  • Most of that side of the functionality of Amacube is provided by the set of ISPConfig3 Roundcube plugins, so there was no need for Amacube to provide that functionality.

What we were after was Amacube quarantine management, nothing more.

In a first step, we commented out the functionality related to the mail spam filter settings in Amacube. It probably would have been cleaner to do so via an additional config variable which would allow to disable this functionality, but in a first step we went for just commenting it out.

Then we got to the point we were interested in: Managing the Mail Quarantine. Amacube needs access (can be readonly) to the amavis/ISPConfig3 database on server B, which is the mail server. We created a MySQL user which can access this database remotely (remember, we have Roundcube installed on server A). In a first step, we had to adapt the query in Amacube quite a bit to make it work - some field names were different, some fields were in different tables, and some tables have different names in ISPConfig3 compared to Amavis. These fixes got us as far as showing the quarantined emails for exactly the logged in user. So far, so good.

In a second step we added some more to the query to:

  • Get all quarantined email for any alias of that user's email address, or any straight forwarder (account1@domain1.com->account2@domain2.com).
  • Get all quarantined email for a catchall if the user's email is setup as recipient for catchall, but exclude mail to other existing mailboxes.

This added some extra queries (to get aliases, other mailboxes (to exclude), etc), and some more complexity to the query getting the relevant quarantined emails.

The resulting hacked version of Amacube can be downloaded from https://netuxo.co.uk/sites/default/files/amacube-ispconfig-2017-08-07.tar.gz. We would be happy for anyone else to use this version of Amacube and to improve on it.

Mailzu

Mailzu was initially the preferred option for managing the Mail Quarantine, but in the end we opted against it, and to just keep it for "us" to be able to check the Mail Quarantine of the entire server. Mailzu does not seem to be under active development for many years already, and it seems unlikely to get picked up again. The nice thing is that it allows "us" (superusers) to see all quarantined email, but that is not a functionality we would want to provide to anyone else. Mailzu does not have domain admins, which would limit access to all emails from one domain - which would be a nice functionality to have.

More importantly, once we came across Amacube it was clear that from a user perspective the best option was to provide all functionality in one place - Roundcube. This means the quarantined email can easily be accessed from this one place. Although Amacube does not provide the option to preview a quarantined email, we did not consider this so crucial as to justify asking a user to log into a separate interface to manage their quarantined email. So a hacked Amacube it is in the end.

Netuxo is retiring Squirrelmail

How to transfer your addressbook to Roundcube

At the end of November 2015 Netuxo will retire Squirrelmail, which we know some of our clients still use to access their email. We started to provide the much more user friendly and modern Roundcube client a long time ago, and now it is time to retire Squirrelmail as it is no longer being developed very actively.

As some of our clients also use the Squirrelmail addressbook, we'll explain in this blog post how you can export your addressbook from Squirrelmail, and import it into Roundcube. This is a straightforward process and you should not encounter any problems.

So, if you are a Squirrelmail user, please follow the steps explained below BEFORE the end of November 2015. If you do not do so, you will no longer be able to access your Squirrelmail addressbook after that date.

1. Export your addressbook from Squirrelmail

Login to your email using the Squirrelmail client (https://mx.netuxo.co.uk/squirrelmail), and click on the "Addresses" link at the top. This will get you to your addressbook. You will now find some import and export options underneath. You can ignore the import options – if you want to import addresses, please do so into Roundcube only. What you need are the export options - at the very bottom.

You do not need to change any of the defaults here. Just click the button which says “Export to CSV File”. Depending on your browser, you might get different options.

In Firefox, you will be given the options “Open with” or “Save File”. You should save the file on your computer. Again, depending on your browser settings, you may be asked where to save the file (remember where you save it to!), or automatically save it in the default download location on your computer.

You have now successfully exported your Squirrelmail addressbook. The next step is to import the addressbook into Roundcube.

2. Import your addressbook into Roundcube

Now login into Roundcube at https://mx.netuxo.co.uk/roundcube and click on the "Address Book" link at the top. You will probably see in your left sidebar two address books: Personal addresses and Global addresses. The global addresses is relevant for all email accounts in your domain (ie all mailboxes @yourdomain), so if you want the addressbook to be available to all email accounts, you can import it there.

First, select the addressbook you want to import into. This is likely to be empty if you did not use Roundcube before.

You will see the icon for import at the top – it is marked in the screenshot.

When you click on this icon, you will be shown the import screen:

Under “Browse...”, select the file you exported from Squirrelmail. You can again select which addressbook the contacts will be imported into.

Click on “Import”, and the import will start. If you have a large addressbook, this might take a while.

You should finally get a message displaying the number of successfully imported addresses.

And that is it. Simple.

Of course, in the unlikely event you experience any problems, please do not hesitate to contact us.

Additional address book functions in Roundcube

We already mentioned above that Roundcube provides a global addressbook for each domain. This enables you to share your addresses among several email accounts within the same domain.

Roundcube also enables you to add external addressbooks from other servers in carddav format. Netuxo provides a carddav server, which enables you to synchronize your addressbook between Roundcube and other email clients – on your desktop or mobile phone. Please get in touch if you would like to find out more and/or read additional information here: https://mx.netuxo.co.uk/carddav.html

Netuxo fighting spam

Netuxo email hosting clients will have noticed that we have been stepping up our fight against spam in recent months. We have improved both spam and virus filtering, and we hope that you noticed many fewer spam emails getting through.

But filtering spam is only one part of the game - and it only deals with spam emails received by our servers. Now we are implementing two important email standards in relation to fighting spam, both of which deal with allowing the recipient of the email we are sending to verify that they are coming from an authorised server. Since August, we have been implementing SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail) on our mailserver, and we urge all our email hosting clients to get in touch with us so that we can implement both standards also for their domain. Below we briefly outline how these two standards work.

Sender Policy Framework (SPF)

SPF (see https://en.wikipedia.org/wiki/Sender_Policy_Framework) is a simple way to expose email spoofing. It allows the receiving mailserver to check that the mail had been sent via an authorised server. It does so by publishing a DNS (Domain Name System) record for the domain, which includes:

  • what servers (by name and/or IP address) are authorised to send emails originating from that domain
  • what to do if mail from a server not included in the above list is received.

Most spam software (such as spamassassin – which is what we use) will check the SPF DNS record of a domain, and if it exists it will verify that the mail in question comes from an authorised server. If not, the spam software can penalise the mail, making it more likely that it will be considered spam. The mechanism is simple and, unfortunately, its effectiveness is limited. Nevertheless, it helps fighting spam and is a widely used standard.

DomainKeys Identified Mail (DKIM)

DKIM (see https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) goes further. As the Wikipedia article explains: DKIM “allow[s] receiving mail exchangers to check that incoming mail from a domain is authorized by that domain's administrators and that the email (including attachments) has not been modified during transport. A digital signature included with the message can be validated by the recipient using the signer's public key published in the DNS.”

DKIM consists of several parts. The outgoing mailserver (our smtp server at mx.netuxo.co.uk) signs outgoing email (the entire message including attachments) of a domain using a specific private key, which is unique for each domain. This dkim signature is then included as a special header in the email. The receiving mailserver will retrieve the public key of the domain from the published DKIM DNS record, which then it can use to verify the signature. Thus, dkim allows to verify that the email comes from an authorised server (as it is signed) and that it has not been tampered with en-route (as the signature can be verified).

As with SPF, what is required is a specific DNS record for the domain which includes the public domain key.

Over to you

For now, we are only signing our own emails with DKIM, and we published a DNS record for netuxo.co.uk and netuxo.com . However, we would very much like to also sign your emails - which you send via our server. To do so, we need your help:

  • if we also manage your domain, we can add the required DNS records. However, we need you to tell us if you also send emails via others servers and which ones they are, in order for us to be able to create the correct SPF policy and DNS record. For DKIM, we can create the DKIM key pair and publish the respective DNS record;
  • if you manage your domain yourself, but have set mx.netuxo.co.uk as your MX record (meaning we deal with your email), then you will need to add the DNS records for both, SPF and DKIM, yourself. We can give you the details of the DNS record for DKIM (which include the public key), and can assist you in creating your SPF record, and in adding the DNS records to your domain. However, as this depends very much on who your domain registrar is, and whether your registrar supports the required DNS record -this goes beyond this blog post.

According to Kaspersky Lab, in Q1 2015, the proportion of spam in email traffic was 59.2% (see https://securelist.com/analysis/quarterly-spam-reports/69932/spam-and-phishing-in-the-first-quarter-of-2015/). This is down from the high estimates of over 80% from five years ago, but it not only a nuisance but also puts stress onto email systems. Fighting spam requires our continued efforts. So please get in touch and help us to do so...

Wandering on and looking ahead

This summer we've been working on a few projects that we have felt very comfortable with. One of these is the development of a new site called "Lost in Samsara" (see their portfolio entry and visit directly here (offsite link)). The project brings together opportunities for grassroots fair trade, a local "shop" and the free exchange of goods and services ("The Wheel"). Founders Alessia and Marvi say that they "Needed to focus on what really matters to us and get the courage to change what we don't like. At the same time, we see this project as our way to break the samsara cycle of unsustainable production, a production that causes suffering for both people and the environment. We would like to start acting differently in order to get different results and 'Lost in Samsara is our contribution to a more sustainable and fair world".

The site was launched a few weeks ago and there is a live crowd-funder in operation for the next 40 days, so if you can support the women on their mission to develop the project further please visit their Indiegogo page - details below. Further information and registration for The Wheel (community exchange) can be found here.

Looking ahead

With Drupal 8 on the near horizon one of the other things we have been engaged with in recent months is early discussions with a few of our long-term Drupal 6 clients about the "end of life" for that version and the migration and/or rebuild options and opportunities that will become available with the long-anticipated release of D8. Over the coming weeks we will be writing to all our Drupal 6 (and earlier!) users outlining that end of life policy and the range of possibilities for the future.

If you are interested in the development of D8 and how close it is to a stable release candidate, keep your eyes on this page.

Pages