DMS to retrieve messages from remote IMAP server -- WORKING WELL for inbox. Want to include subfolders too.

[Doni49]

With a lot of help from @polarathene, I was able to get my DMS installation working to download messages from my remote IMAP server. I'm currently trying to figure out how to get it to retrieve messages from subfolders and place them in their respective folders.

Right now, the focus is on the Junk folder. My remote server provides spam filtering and moves suspected spam into INBOX/Junk. I want to download those messages into the local server as INBOX/Junk.

The fetchmail documentation shows how to do it using the fetchmail CLI tool but I don't see anything about entering info in the fetchmail.cf file.

[glime-ai[bot]]

Please note that this response was created using generative AI, leveraging previously available information from the repository. This information may be helpful. However, it is our strong recommendation that you carefully consider the information before blindly applying it.

---

To configure Fetchmail to retrieve messages from subfolders such as INBOX/Junk on your remote IMAP server and place them into the corresponding folders on your local Docker Mailserver (DMS), you will need to modify the fetchmail.cf configuration file accordingly.

Here’s a basic example of how you might set up the fetchmail.cf to fetch emails from the Junk folder:

poll imap.remote-server.com protocol IMAP
user 'your-remote-user' there with password 'your-password' is 'your-local-user' here
folder 'INBOX.Junk'
mda "/usr/lib/dovecot/deliver -d your-local-user@your-local-domain.com"

In this configuration:

• Replace imap.remote-server.com with the address of your remote IMAP server.
• Replace your-remote-user with your username on the remote server.
• Replace your-password with your password on the remote server.
• Replace your-local-user and your-local-domain.com with the appropriate local user and domain managed by your DMS.

The folder directive specifies which folder on the remote server Fetchmail should check for new messages. The mda (Mail Delivery Agent) line specifies how the emails should be delivered to your local server, using Dovecot’s deliver agent.

Make sure to place this configuration in the fetchmail.cf file located in your DMS configuration directory, typically under docker-data/dms/config/.

If you need to fetch from multiple folders, you can repeat the configuration block for each folder, adjusting the folder directive as needed.

This comment was generated by Glime.

[Doni49]

Just after posting my last reply, I tried one more search and found something that looks promising. I'm going to tinker with this and post back with my results.

https://doc.dovecot.org/configuration_manual/protocols/lda/#lda

-m <mailbox>: Destination mailbox (default is INBOX). If the mailbox doesn’t exist, it will not be created (unless the lda_mailbox_autocreate setting is set to yes). If message couldn’t be saved to the mailbox for any reason, it’s delivered to INBOX instead.

If Sieve plugin is used, this mailbox is used as the “keep” action’s mailbox. It’s also used if there is no Sieve script or if the script fails for some reason.

Deliveries to namespace prefix will result in saving the mail to INBOX instead. For example if you have “Mail/” namespace, this allows you to specify dovecot-lda -m Mail/$mailbox where mail is stored to Mail/$mailbox or to INBOX if $mailbox is empty.

The mailbox name is specified the same as it’s visible in IMAP client. For example if you’ve a Maildir with .box.sub/ directory and your namespace configuration is prefix=INBOX/, separator=/, the correct way to deliver mail there is to use -m INBOX/box/sub

[Doni49]

:(

I tried this and it didn't work -- the message remains in the Junk folder on the remote server.

poll 'mail.remoteserver.com' proto imap
  user 'user@remoteserver.com'
  pass 'Password'
  is 'don@localserver'
  no sslcertck

poll 'mail.remoteserver.com' proto imap
user 'user@remoteserver.com' there with password 'Password' is 'don@localserver' here
folder 'INBOX.Junk'
mda "/usr/lib/dovecot/deliver -m INBOX/Junk"

I also tried replacing the 3rd line with this and got the same results.
folder 'INBOX/Junk'

[polarathene]

I'm not too familiar with the configuration being suggested by the AI here for delivery via Dovecot. We don't use the LDA ourselves in DMS, but LMTP (which Postfix sends through, and fetchmail would typically deliver to Postfix).

You perhaps still need to indicate the destination as before with the -d argument, in addition to the -m? I would probably focus on getting that to work first before worrying about -m or folder option, to confirm it delivers the mail. Once it delivers mail that way, then you could look at the additional folder config for both sides.

Regarding INBOX/Junk DMS may actually be configured for INBOX.Junk or .Junk, I can't recall exactly (EDIT: Apparently it's INBOX.Junk based on docs reference later in this reply), but if you have junk mail on your DMS account, you could verify this with the earlier command I showed you for running in the DMS container:

docker compose exec -it name-of-dms-container-here doveadm mailbox status -u john.doe@example.com messages INBOX.Junk

Just try adjusting that last value until the message count returned matches the actual junk mail messages stored 👍

---

The advice for using sieve is a valid alternative to consider though! I have a WIP rewrite on our docs for this as the current ones aren't in the best state.

You could use the sieve sub-addresses feature, but this would also allow anyone else to send mail to the same subaddress to direct mail to any configured folders. I'm not quite sure if mail delivered via fetchmail is exempt from junk/spam filtering (when enabled on DMS) either,

Place this file in your DMS config volume (mapped to /tmp/docker-mailserver in the container), with the filename of john.doe@example.com.dovecot.sieve (adjust the email address portion to match your user):

# This file should be in the DMS container at:
# /tmp/docker-mailserver/john.doe@example.com.dovecot.sieve

require ["envelope", "fileinto", "mailbox", "subaddress", "variables"];

if envelope :detail :matches "to" "*" {
  set :lower :upperfirst "tag" "${1}";
  if mailboxexists "INBOX.${1}" {
    fileinto "INBOX.${1}";
  } else {
    fileinto :create "INBOX.${tag}";
  }
}

That is from the current docs example, but the docs don't explain it too well yet.

• require is an "import" syntax to tell sieve software what it needs to use to run the script.
• The if line will then look for a + (:detail :matches) in the email address local-part (envelope), and anything to the right of the first match (*) as the ${1} value referenced on the next lines.
• set is lowercasing the matched value, while also setting the first letter to upper-case, so john.doe+junk@example.com => Junk, it assigns this to a new variable called tag.
• Next is a check if the mailbox with this tag name already exists, otherwise it will create it. As we can see there is a . separator, so that's what Dovecot expects, not the / you were using before. Thus, in case not obvious, you should avoid folder names with . in them as that would presumably break this feature.
• Mail should be delivered to that mailbox successfully 👍

So to integrate that with fetchmail, just change the local address to the one with subaddress like this:

poll 'mail.remoteserver.com' proto imap
  user 'user@remoteserver.com' there with password 'Password' is 'don+junk@localserver' here
  folder 'INBOX.Junk'

The folder value will need to be correct for whatever the remote mailserver is using to identify that remote mailbox, so you might need to experiment a bit there.

[Doni49]

Does DMS have spamassassin enabled by default? I've ready through the docs and if I'm reading everything correctly, it's automatically activated. If so:

1. Does it work well at catching most spam without a lot of false positives?
2. Is it easy to train SA to recognize spam by giving it messages that it missed? As best I can tell, I'm supposed to move any spam message that actually arrive in my inbox to the Junk folder. But if that's where SA puts messages it thinks are spam, it doesn't seem like that folder should be used for training SA. If legit messages (AKA Ham messages) are in the Junk folder when it does the training, wouldn't that just reinforce SA's assumption that the message is spam?

These answers might make this request moot as I was planning to let the remote server handle spam filtering. But once the remote server thinks a message is spam, it would be stored in the Junk folder on the remote server. I was trying to find a way to download those as well so I could look through them if a message I'm expecting arrive in my inbox. The other issue with doing it that way is that I'd have trouble training the remote server to recognize the spam it missed.

[polarathene]

Does DMS have spamassassin enabled by default? I've ready through the docs and if I'm reading everything correctly, it's automatically activated

It's been enabled by default... kinda. While the feature has historically been default enabled, it still relies on Amavis to actually work IIRC which still requires opt-in via ENV, so you'd need to toggle that on. The quality I can't comment too much on, some users have good and not so good experiences, we try to address any feedback there when possible.

We do have another feature rspamd, that is still in-progress with getting configured well enough that we'd be more confident making it the default in the future. It should be a much better choice, but I think presently has some worthwhile changes/improvements in the upcoming v14 release of DMS and possibly some added setup/config friction for the user to learn currently. I personally haven't tried it and it's not my area of the project.

I only work on the project itself, I've not actually deployed DMS so my input here is limited 😅

---

Regarding SA training, I am only familiar with SPAM/HAM concept and I believe our docs touch on this. There is also some existing issues you can search where that has been discussed in the past which might provide some more insight. I know one of the maintainers uses SA with their deployment and they're quite happy with the results.

The other issue with doing it that way is that I'd have trouble training the remote server to recognize the spam it missed.

Not sure what you can do there. You could either manage that remote spam from that service and move the mail to a different folder if it's not spam, which DMS then retrieves via fetchmail (you can configure to retrieve mail that has already been read/seen too). That's the only way for the remote service to be taught about false-positives AFAIK? Otherwise as far as it's concerned you're deleting anything that it places in the junk folder, and manage false-positives in your local DMS junk folder instead.

But like I said, I think when fetchmail delivers locally through Postfix it bypasses any SA/Amavis check, so it's unlikely to be processed through that, mail would always appear where you intend it to be then. You can deliver this mail to any other location too, it doesn't have to be the standard junk mail folder. The solution above with sieve subaddresses is just an approach that manually stores it in the junk folder.