As the GSoC 2017 final evaluation period just ended, my final work product is finally submitted. This post is a summary of my final work product.
- Plugin for Mailman Core.
- Enables creating a PGP mailing list, which has a list key, can receive and serve messages encrypted, can sign and receive signed messages from subscribers.
- Creates the
keyemail command, which is used for per-address user key management.
- Subscription to a PGP enabled mailing list the subscribing address to send and confirm an address public key, which the moderator must verify.
- Somewhat confirms the user has possession of the appropriate private key to the one sent on subscription.
- Has per-list settings for encryption/signatures/what to do with non encrypted / non signed messages, etc..
- Optionally exposes a REST API for list configuration.
- Has local archivers which can store the messages encrypted by the list key.
- Stores list and address keys in configurable key directories.
- Requires (some not merged) MRs in Mailman Core:
- Additional MR (not required):
- Required branches are merged and maintained at J08nY/mailman/plugin.
- To install, do
pip install mailman-pgp, warning: it will pull in a development version of Mailman Core and PGPy.
- A Django app, uses django-mailman3 and mailmanclient, integrates well with Postorius and HyperKitty.
- Provides management of PGP enabled mailing lists to the list owner, and of PGP related subscription settings to the subscriber.
- Requires (currently not merged) MRs in mailmanclient, django-mailman3, Postorius and HyperKitty:
- Some screenshots:
- A plugin for Mailman Core that turned out to be unnecessary for the working of django-pgpmailman, but implemented a similar feature as this MR.
- This plugin sends the events (and some information about them) from Mailman Core to a list of configurable endpoints using JSON in HTTP POST requests.
- mailmanclient/split-sources, merged
- Many many PRs to PGPy, a Python only implementation of OpenPGP. 19 PRs and counting. As PGPy was not and still is not feature complete in regards to RFC4880 I found out many times that it’s missing features/bugs broke mailman-pgp CI. It would not make sense fixing them locally, both from a software design perspective and open source software one aswell.
I think I met almost all goals that the project idea required and my original proposal stated, with the noteworthy exception of remote archiving to HyperKitty which I just couldn’t find a way to integrate.
Successfully created the mail list views. Inspired heavily by Postorius, to get the same look, both in templates and views. There is a list index view, which lists only PGP enabled lists, and their key fingerprints. This also allows one to download the list key as it’s linked from the list key fingerprint. The list name link leads to a list settings/info view. The info tab is available to any logged in user, while the settings are list owner only. All the per-list PGP settings are configurable there.
django-mailman3 template chunks#
In order to make plugging the django-mailman3 based apps together and deduplicate some of their code, as well as to integrate the django-pgpmailman app into any Postorius + HyperKitty project I refactored the direct references of Postorius to HyperKitty and vice versa.
This is done in the template chunk MR. It introduces a new template tag in django-mailman3, which is intended to be used by all django-mailman3 based apps to let other installed apps add their entries to the navbar and user menu. Which I are two main ways Postorius and HyperKitty reference each other.
This post is about my current plans on how to implement the web ui part of PGP enabled Mailman. It strives to integrate into the Mailman Suite and use its features to the maximum possible degree.
General idea: Refactor general stuff to django-mailman3, to allow apps to hook up together in Mailman Suite easily, and then use that to hook up django-pgpmailman.
Show PGP enabled public lists, with their key fingerprints, with the option to download their public keys, also show some of their configuration (so that subscribers can see that for example if they send a cleartext message to a list that requires encrypted messages, it will be bounced).
Enable list owner to configure the PGP related per-list configuration options.
Enable list owner to set/see the list key (private part). This is quite questionable and will have a site-level option to be turned off (the REST API will then not serve the list private key).
The same level of user key management as the
key command allows, with similar steps during key change/revocation.
Another django app is installed in the same project as Postorius + HyperKitty, django-pgpmailman. This app provides a list of PGP enabled mailing lists and their PGP related management in a similar way Postorius does, also user key management.
There are few places where Postorius refers to HyperKitty and vice versa, for adding the appropriate links/icons to the navbar as well as for the user menu entries. These references will be refactored to some mechanism in django-mailman3, which will allow any installed django app to add it’s entry to the navbar or the user menu. This will allow django-pgpmailman to hook up rather easily, without any direct references to it from Postorius/HyperKitty/django-mailman3.
The archiving web UI is a tougher nut to crack. I either have to develop a custom PGP mail archive frontend and integrate it similar to the PGP list management app, or integrate with HyperKitty transparently, so that archives are received encrypted, stored encrypted, and yet served to subscribers in clear. Developing a custom app is quite unrealistic and it would lack most HyperKitty functions.
However hooking up an encrypted message store to HyperKitty is also non-trivial, as HyperKitty is strongly tied with storing messages in it’s database and using a django Model to represent a message.
I currently have no realistic ideas, one that comes to mind, is to create a custom django database backend, that somehow stores everything encrypted, but thats a very unwieldy solution that likely won’t work well.
Fixed many little issues with the PGP plugin and PGPy. Got it to work quite nicely, below you can see a message received by a subscriber, by a PGP enabled discussion list, encrypted to his key, as shown by Thunderbird with the EnigMail plugin:
Also finally merged the finished
key revoke command to mailman-pgp/master.
This week was tough but productive. Temperatures spiking to 34°C in my hometown have a really bad effect on my daily productivity.
Setup instance with PGP plugin#
Finally got a complete mailman instance setup and running with J08nY/mailman/plugin + J08nY/mailman-pgp/master and J08nY/Postorius/plugin + J08nY/mailmanclient/plugin + mailman/HyperKitty/master + mailman/django-mailman3/master. The
plugin branches merge MR branches that introduce the plugin infrastructure for that particular Mailman component. For Mailman Core, the
plugin branch merges the
pluggable-components one introduces the concept of a
plugin to Mailman Core and replaces the
(pre|post)_hooks and is essential to let site admins easily add plugins to Mailman Core by simply installing them to the same environment as Mailman Core and some simple configuration to enable.
pluggable-workflows splits the subscription/unsubscription monolithic workflows into composable workflows, that are also pluggable by a plugin and set per-list.
list-style-descriptions are exposed via the REST api and Postorius uses them for displaying list style selection.
I even successfully created a PGP enabled discussion list through Postorius. Subscribed to it by sending the subscription request, confirming it, replying to the
key set <token> challenge with key attached, replying to the
key confirm <token> with the challenge body signed by the key being set. This would of course be followed by the moderator verifying the supplied key in any real application of PGP enabled lists, which is also supported.
The instance runs on a Raspberry Pi with 512MB RAM along with my web-server, mail-server and several other services, so don’t expect lightning fast performance, or it being up anyway, reserving the right for any extended downtime ;).
Working on proper key revocation behavior from the PGP plugin took much of my week as getting this right is pretty hard and the OpenPGP revocation mechanism is quite complex. The usual workflow for just an ordinary key change was already presented in one of my previous posts. However if the user needs to revoke a key with a revocation signature, we cannot use the old key to perform the key change challenge. Also, the key revocation can be only partial, as in a subkey being revoked, and the key can still be used for encryption and signing, then it’s usable for the PGP plugin and nothing needs to be done. This also gets more complex as when we allow a user to change his key without moderator approval, only with the challenge (which makes the user sign a challenge/statement signifying they are changing their key to the new one, by the old key). Then if the user revokes his former key using a reason for revocation that invalidates all signatures by that key(even former ones), we cannot trust the users current key, as the old one could have been compromised and used to set the new one.
For now, giving a mechanism for users to provide a revocation certificate that is verified merged with the key is implemented. If the revocation certificate revoked the key or a subkey/uid that makes the key not usable by the PGP plugin (the key can no longer be encrypted to or can sign) then the users key is reset and he/she has to send and confirm a new one with moderator approval necessary. That is almost completely implemented as it’s almost the same as the subscription challenge.
More PGPy work#
Necessary to make it usable, as for example, not having support for partial length headers would break handling of most messages encrypted with GPG as it likes to create plenty of packets with partial lengths. However, now I think that my development branch of PGPy is feature complete enough to support an instance of Mailman with the PGP plugin running.
Setup a Trello board to better track the issues that came up and keep my head sane:
Web UI integration#
The original proposal proposed adding support for PGP enabled lists to Postorius and HyperKitty directly, now when mailman-pgp is dynamically enabled in Mailman Core a similar approach needs to be taken with the Postorius and HyperKitty integration.
Thinking of doing local archiving very similar to the prototype archiver, encrypted by the list local-archive key. The remote archiving capability is a much tougher nut to crack and depends a lot on how the HyperKitty integration ends up looking.
The second evaluation period came quite fast after the first one, nonetheless the project advanced much further so quick recap of its current state is in order.
Since first evaluation#
SMTPS + STARTTLS#
Finally with working tests after upstream fixes in aiosmtpd.
Pluggable components (plugins)#
Rebased after the Click CLI processing branch was merged and got it to work nicely in the end. There was an issue with config processing, as plugins can now supply their own CLI commands and plugins are configured/specified in the Mailman core config, which can be supplied by a CLI option. So that created a bit of an argument processing/lazy-loading issue but looking some more at how Click works I was able to quickly resolve it. Config option is now an eager one and initializes Mailman before commands need to be listed/used.
Also rebased and added tests to get diffcov to 100% and fix one of the migrations in that branch, as it was broken before.
Key management (plugin)#
Implemented email commands for managing per-address PGP keys in plugin. It uses pluggable workflows to plug into the subscription process of a PGP enabled mailing list and requests the user key, also does mailback confirmation of it. The key is then available to the list moderator during subscription moderation, and more generally to the plugin for verifying signatures and encrypting. After subscription key management is also implemented where a user can change his set key, provided he can sign a challenge provided by the plugin with the old key. Key revocation handling is also necessary, but not yet done.
Outgoing processing (plugin)#
Implemented custom Bulk and Individual delivery classes for PGP enabled lists in plugin. These deliveries optionally encrypt the mail to subscribers keys, and/or sign with the list key. The bulk one retains anonymity of subscribers as the keyids are zeroed out of the PKESK(Public Key Encrypted Symmetric Key) packets which OpenPGP implementations should handle as a wildcard keyid and try decrypting with all usable private keys. It is also almost as efficient as it can be, as it only encrypts the message with one session key per chunk and then encrypts said session key to recipients in the chunk. The signing is also configurable.
Signature hash tracking (plugin)#
Implemented store of signature hashes from successful postings to a PGP enabled list which are then (optionally) used to stop replay of the same signature by Mailman in a future posting.
PyPI package (plugin)#
Created the PyPI package for the PGP plugin.
I would like to be a bit further in the project at this point, however I am very optimistic about the next days and weeks. After resolving some issues and TODOs I have for the current plugin implementation and setting up the live Mailman instance with the PGP plugin, which should be up by the end of the week. I believe I can start work on the other side of the REST API, of somehow hooking PGP enabled archives/lists to Postorius and HyperKitty.
Signature hash tracking#
It would be relatively easy to replay a signed message to a mailing list by a user as no kind of challenge-response is done on posting.
While signature replay checking is usually done on the end users point with against his keyring and messages he has so far received and their context, I think it is kind of expected of PGP enabled Mailman to also do this as it relays the messages.
On a successful posting to a PGP enabled mailing list (
AcceptedEvent to be precise), the message is searched for PGP signatures and their digests and key fingerprints are added into the
The signature rule then checks the digests from a posting against those in the
sighash table. If it finds it has previously accepted a message with any of those hashes it takes the
duplicate_sig_action which is per-list configurable.
This of course means that if the
duplicate_sig_action is not set to
Action.defer, which means the message gets rejected or dropped, that a signature that was sent to a list cannot be sent again. Of course if a user wants to send a signed message again, he can just resign it and send it again, the hashes won’t match. However sometimes it might be useful to post the message as signed originally, for example to prove something. However, I think it is worth it to keep this as a configurable option. Maybe with another option disabling the collection of signature hashes, for performance reasons.
Since a the plugin needs to process the messages for outgoing encryption on a per-recipient or per-chunk basis, it couldn’t be implemented as a Handler, I thought about implementing it as a custom OutgoingRunner but that didn’t work out either. However the
IMailTransportAgentDelivery interface is great for what the plugin needs to do. So there are two custom PGP enabled delivery classes that get selected by a custom
[mta] outgoing callable, one for bulk delivery and other for individual delivery.
I got the
pgpmailman-proposal repository almost completely up to date on changes to the plugin and my current proposed Core/Client/Postorius/HyperKitty changes and MRs.
As I remembered to ignore coverage of tests it dropped to 88% and I think it is currently in order to add more comprehensive tests for features implemented that check the edge cases which currently remain.
Setup live development instance#
I believe I got pretty far into the development without having a proper live development instance of Mailman Core + plugins and Postorius + HyperKitty + Client, so I’m going to set that up now and test manually. With time I might set this up as a public site/mailing list server, to demonstrate the features of the PGP plugin.
Currently thinking about implementing two encrypted archivers, one local, one remote. The local one would be similar to the prototype archiver but store messages encrypted, TBD how. The remote one will send the messages encrypted to a receiving archiver, the django-pgpmailman instance running next to HyperKitty.
With most of the essential stuff in the core plugin done, the web ui part of development can begin.
This week is around the halfway of GSoC 2017 project timeline, hence the title.
As I set out to do, I have now implemented most of the after subscription key management. The
key change command. The pre subscription key management is done via a custom dynamically loaded subscription policy/workflow. This is possible with my
pluggable-workflows branch/MR which introduces dynamically loaded workflows, which subscription and unsubscription policies are a part of.
On subscription the PGP enabled mailing list needs a users public key. An example conversation establishing that follows (shows a list with moderation + pgp enabled subscription policy):
Mailman User ------- ---- email@example.com firstname.lastname@example.org <- subscribe 1. 2. key set "token1" -> + instructions to reply and attach user key (signed by list) <- key set "token1" 3. (key attached as per RFC3156(PGP/MIME) or inline and we do our best to find it.) 4 key confirm "token2" -> + statement containing "token2" and user key fingerprint in body, with instructions to sign it (signed by list, encrypted to user key) <- key confirm "token2" 5. + must contain signed statement from step 4. (required to be signed by user key) 6. ... the rest of the subscription steps (moderation)...
Steps 3 and 5 can be optionally encrypted to the list key.
This workflow gives “some” assurance that the user really has the key she is subscribing with, however the trust in the address/user/key has to come from the list moderator and both my proposal and implementation rely on that. This workflow only proves that Mailman is communicating with someone that has the user key (can decrypt
"token2" and sign the
statement), not that that user is
Works similar to the
key set command on subscription. Works via a custom workflow, but not a subscription workflow/policy, just a workflow.
Mailman User ------- ---- email@example.com firstname.lastname@example.org <- key change 1. (new! key attached as per RFC3156(PGP/MIME) or inline and we do our best to find it.) 2. key confirm "token1" -> + statement containing "token1" and new! user key fingerprint in body, with instructions to sign it (signed by list, encrypted to new! user key) <- key confirm "token1" 3. + must contain signed statement from step 2. (required to be signed by old! user key)
Steps 1 and 3 can again be optionally encrypted.
This ensures that only someone with possesion of the old user key can change it. It requires the user to decrypt with the new key and sign a statement with the new key fingerprint as well as the token in it with the old key.
mailman-pgp @ PyPi I have now created a PyPi package for the mailman-pgp plugin as it’s maturing, and while still unusable I wanted the name.
I got test coverage of mailman-pgp to 95% which now actually means something as large parts of the codebase are implemented and not just stubs. Since this plugin requires features not merged into Mailman Core (or PGPy) for pipelines to work I maintain forks of both with branches that merge all required branches into a
plugin branch for Mailman Core and
dev branch for PGPy.
I had a look at working in outgoing encryption of messages for pgp enabled mailing lists. Encryption cannot be done as a handler, as it requires to process the messages per-recipient or per-batch of recipients. The only configurable/dynamic option currently in Mailman Core is the
[mta].outgoing callable. Which is not ideal since it itself has more functionality than simply dispatching to a correct
Delivery instance based on the list configuration, that would have to be duplicated in mailman-pgp. The
IMailTransportAgentDelivery interface looks promising though, so I’m thinking about refactoring the outgoing callable a bit to allow setting a custom one without the duplicities.
Signature hash tracking#
To prevent replay attacks for things where always sending and requiring a signed confirmation token in response is not practically doable, the pgp plugin will keep signature hashes with timestamps and posssibly key fingerprints in a DB/some other quickly searchable structure, and will check signatures it receives against the hashes in that structure. It will then act accordingly, for example not allowing a user to post the same message (with the same signature) again, since it could have been replayed.
I wanted to get the pgpmailman-proposal repository updated with current state/MRs but didn’t get to it this week, so might find time for it over the weekend as well as for some code documentation.
Just a quick update about the state of the project before I disappear from civilization for this weekend.
I have now finally got the pluggable workflows branch to test succesfully! With workflows being loaded dynamically, with workflow steps being saved completely, full backward compatibility, also database migrations and REST backward compatibility.
That branch is still missing more tests for diffcov to pass.
^^python 3.4 CI envs are currently broken, since
aiosmtpd 1.1 is python 3.5 only, so don’t mind that.
Building up on the pluggable workflows I have a base implementation of how the subscribers key gets to the mailing list and its moderator and also of how it’s confirmed.
SMTPS + STARTTLS#
During this week I also got to add SMTP over SSL/TLS support to
aiosmtpd, a lib which Mailman uses for it’s SMTP tests, so with that the SMTPS+STARTTLS MR has tests which pass and I think is ready for merging. Thanks Barry for the quick responses and help debugging over at
For next week I plan on cleaning up the pluggable workflows branch, finishing the pre-subscription key management, implementing after-subscription key management as well as getting back on track with documenting the mailman-pgp architecture and changes at the pgpmailman-proposal repo, which has been lacking.
As I note in one of my previous GSoC project updates, the PGP-enabled mailing lists require two new features from Mailman Core, both related to workflows. The first is to be able to inject steps into a subscription workflow to request the users pubkey before moderation checks. The second it to be able to require confirmation on all commands (to prevent replay attacks), although this could be avoided by storing the hashes from all the user sent signed commands. In this post I propose a way of refactoring the workflows into pluggable components.
Both of these requirements require changes to the way [Un]SubscriptionWorkflows and workflows in general work. Currently the subscription policy of a mailing list is determined by its
subscription_policy attribute of the
SubscriptionPolicy enum. That policy is enforced in the
SubscriptionWorkflow state machine, which curently works with a queue of states, and looks like this:
It is quite complex, and the
UnsubscriptionWorkflow has much of the same code except the verification and moderation parts.
What I propose is to make the [Un]SubscriptionWorkflows, more generally workflows, pluggable components which would work on a stack state machine and be composed of mixins like so:
This way a mailing list will have the name of its
SubscriptionWorkflow component in the
Which would mean for example an
SubscriptionPolicy.open would be transformed into a class like so:
@public @implementer(ISubscriptionWorkflow) class OpenSubscriptionPolicy(SubscriptionBase, VerificationMixin): """""" name = 'policy-open' description = 'An open subscription policy, only requires verification.' initial_state = 'prepare' def _step_prepare(self): self.push('do_subscription') self.push('verification_checks') self.push('sanity_checks')
I believe this simplifies the policies and also allows chaining and composing of workflow steps, which would enable plugins to add custom subscription policies enforcing more steps/checks during the [un]subscription process. For example an organization-only mailing list could add a check that checks for the subscribing address in the organization database, therefore removing the need for manual moderation of subscriptions.
This should also allow for requiring confirmation of all commands, by attaching a workflow instance to them. For the membership commands, this would be the corresponding [Un]SubscriptionWorkflows, and for other commands it would be currently disabled/empty workflow. Then a plugin could set the confirm workflow on commands it needs to.
I am currently working on this as a MR, although I am not sure if this is the right track to go on with workflows. I think something to split the monolithic [Un]SubscriptionWorkflows would be a nice thing in Mailman Core, as they share a lot of the same code and are overly complex for what they are, a linear series of checks with some skipping possible.