GSoC 2017 - Final work submission

As the GSoC 2017 final evaluation period just ended, my final work product is finally submitted. This post is a summary of my final work product.

Mailman-pgp#

  • repository@gitlab
  • docs@rtd
  • Plugin for Mailman Core.
  • Enables creating a PGP mailing list, which has a list key, can receive and serve messages encrypted, can sign and receive signed messages from subscribers.
  • Creates the key email command, which is used for per-address user key management.
  • Subscription to a PGP enabled mailing list the subscribing address to send and confirm an address public key, which the moderator must verify.
  • Somewhat confirms the user has possession of the appropriate private key to the one sent on subscription.
  • Has per-list settings for encryption/signatures/what to do with non encrypted / non signed messages, etc..
  • Optionally exposes a REST API for list configuration.
  • Has local archivers which can store the messages encrypted by the list key.
  • Stores list and address keys in configurable key directories.
  • Requires (some not merged) MRs in Mailman Core:
  • Additional MR (not required):
  • Required branches are merged and maintained at J08nY/mailman/plugin.
  • To install, do pip install mailman-pgp, warning: it will pull in a development version of Mailman Core and PGPy.

django-pgpmailman#

mailman-rest-events#

  • repository@gitlab
  • A plugin for Mailman Core that turned out to be unnecessary for the working of django-pgpmailman, but implemented a similar feature as this MR.
  • This plugin sends the events (and some information about them) from Mailman Core to a list of configurable endpoints using JSON in HTTP POST requests.

Other contributions#

Overall#

I think I met almost all goals that the project idea required and my original proposal stated, with the noteworthy exception of remote archiving to HyperKitty which I just couldn’t find a way to integrate.


GSoC 2017 - Web UI progress

django-pgpmailman progress#

Successfully created the mail list views. Inspired heavily by Postorius, to get the same look, both in templates and views. There is a list index view, which lists only PGP enabled lists, and their key fingerprints. This also allows one to download the list key as it’s linked from the list key fingerprint. The list name link leads to a list settings/info view. The info tab is available to any logged in user, while the settings are list owner only. All the per-list PGP settings are configurable there.

django-mailman3 template chunks#

In order to make plugging the django-mailman3 based apps together and deduplicate some of their code, as well as to integrate the django-pgpmailman app into any Postorius + HyperKitty project I refactored the direct references of Postorius to HyperKitty and vice versa.

This is done in the template chunk MR. It introduces a new template tag in django-mailman3, which is intended to be used by all django-mailman3 based apps to let other installed apps add their entries to the navbar and user menu. Which I are two main ways Postorius and HyperKitty reference each other.


GSoC 2017 - WebUI integration

This post is about my current plans on how to implement the web ui part of PGP enabled Mailman. It strives to integrate into the Mailman Suite and use its features to the maximum possible degree.

General idea: Refactor general stuff to django-mailman3, to allow apps to hook up together in Mailman Suite easily, and then use that to hook up django-pgpmailman.

Features#

Show PGP enabled public lists, with their key fingerprints, with the option to download their public keys, also show some of their configuration (so that subscribers can see that for example if they send a cleartext message to a list that requires encrypted messages, it will be bounced).

Enable list owner to configure the PGP related per-list configuration options.

Enable list owner to set/see the list key (private part). This is quite questionable and will have a site-level option to be turned off (the REST API will then not serve the list private key).

The same level of user key management as the key command allows, with similar steps during key change/revocation.

Implementation#

Another django app is installed in the same project as Postorius + HyperKitty, django-pgpmailman. This app provides a list of PGP enabled mailing lists and their PGP related management in a similar way Postorius does, also user key management.

There are few places where Postorius refers to HyperKitty and vice versa, for adding the appropriate links/icons to the navbar as well as for the user menu entries. These references will be refactored to some mechanism in django-mailman3, which will allow any installed django app to add it’s entry to the navbar or the user menu. This will allow django-pgpmailman to hook up rather easily, without any direct references to it from Postorius/HyperKitty/django-mailman3.

Archiving#

The archiving web UI is a tougher nut to crack. I either have to develop a custom PGP mail archive frontend and integrate it similar to the PGP list management app, or integrate with HyperKitty transparently, so that archives are received encrypted, stored encrypted, and yet served to subscribers in clear. Developing a custom app is quite unrealistic and it would lack most HyperKitty functions.

However hooking up an encrypted message store to HyperKitty is also non-trivial, as HyperKitty is strongly tied with storing messages in it’s database and using a django Model to represent a message.

I currently have no realistic ideas, one that comes to mind, is to create a custom django database backend, that somehow stores everything encrypted, but thats a very unwieldy solution that likely won’t work well.

Other progress#

Fixed many little issues with the PGP plugin and PGPy. Got it to work quite nicely, below you can see a message received by a subscriber, by a PGP enabled discussion list, encrypted to his key, as shown by Thunderbird with the EnigMail plugin:

message_encrypted

Also finally merged the finished key revoke command to mailman-pgp/master.


GSoC 2017 - Progress

This week was tough but productive. Temperatures spiking to 34°C in my hometown have a really bad effect on my daily productivity.

Setup instance with PGP plugin#

Finally got a complete mailman instance setup and running with J08nY/mailman/plugin + J08nY/mailman-pgp/master and J08nY/Postorius/plugin + J08nY/mailmanclient/plugin + mailman/HyperKitty/master + mailman/django-mailman3/master. The plugin branches merge MR branches that introduce the plugin infrastructure for that particular Mailman component. For Mailman Core, the plugin branch merges the pluggable-components, pluggable-workflows and list-style-descriptions branches.

The pluggable-components one introduces the concept of a plugin to Mailman Core and replaces the (pre|post)_hooks and is essential to let site admins easily add plugins to Mailman Core by simply installing them to the same environment as Mailman Core and some simple configuration to enable. pluggable-workflows splits the subscription/unsubscription monolithic workflows into composable workflows, that are also pluggable by a plugin and set per-list. list-style-descriptions are exposed via the REST api and Postorius uses them for displaying list style selection.

I even successfully created a PGP enabled discussion list through Postorius. Subscribed to it by sending the subscription request, confirming it, replying to the key set <token> challenge with key attached, replying to the key confirm <token> with the challenge body signed by the key being set. This would of course be followed by the moderator verifying the supplied key in any real application of PGP enabled lists, which is also supported.

The instance runs on a Raspberry Pi with 512MB RAM along with my web-server, mail-server and several other services, so don’t expect lightning fast performance, or it being up anyway, reserving the right for any extended downtime ;).

Key revocation#

Working on proper key revocation behavior from the PGP plugin took much of my week as getting this right is pretty hard and the OpenPGP revocation mechanism is quite complex. The usual workflow for just an ordinary key change was already presented in one of my previous posts. However if the user needs to revoke a key with a revocation signature, we cannot use the old key to perform the key change challenge. Also, the key revocation can be only partial, as in a subkey being revoked, and the key can still be used for encryption and signing, then it’s usable for the PGP plugin and nothing needs to be done. This also gets more complex as when we allow a user to change his key without moderator approval, only with the challenge (which makes the user sign a challenge/statement signifying they are changing their key to the new one, by the old key). Then if the user revokes his former key using a reason for revocation that invalidates all signatures by that key(even former ones), we cannot trust the users current key, as the old one could have been compromised and used to set the new one.

For now, giving a mechanism for users to provide a revocation certificate that is verified merged with the key is implemented. If the revocation certificate revoked the key or a subkey/uid that makes the key not usable by the PGP plugin (the key can no longer be encrypted to or can sign) then the users key is reset and he/she has to send and confirm a new one with moderator approval necessary. That is almost completely implemented as it’s almost the same as the subscription challenge.

More PGPy work#

Necessary to make it usable, as for example, not having support for partial length headers would break handling of most messages encrypted with GPG as it likes to create plenty of packets with partial lengths. However, now I think that my development branch of PGPy is feature complete enough to support an instance of Mailman with the PGP plugin running.

Trello board#

Setup a Trello board to better track the issues that came up and keep my head sane: mailman-pgp trello board

Next up#

Web UI integration#

The original proposal proposed adding support for PGP enabled lists to Postorius and HyperKitty directly, now when mailman-pgp is dynamically enabled in Mailman Core a similar approach needs to be taken with the Postorius and HyperKitty integration.

Archiving#

Thinking of doing local archiving very similar to the prototype archiver, encrypted by the list local-archive key. The remote archiving capability is a much tougher nut to crack and depends a lot on how the HyperKitty integration ends up looking.


GSoC 2017 - Second evaluation

The second evaluation period came quite fast after the first one, nonetheless the project advanced much further so quick recap of its current state is in order.

Since first evaluation#

SMTPS + STARTTLS#

MR 286

Finally with working tests after upstream fixes in aiosmtpd.

Pluggable components (plugins)#

MR 288

Rebased after the Click CLI processing branch was merged and got it to work nicely in the end. There was an issue with config processing, as plugins can now supply their own CLI commands and plugins are configured/specified in the Mailman core config, which can be supplied by a CLI option. So that created a bit of an argument processing/lazy-loading issue but looking some more at how Click works I was able to quickly resolve it. Config option is now an eager one and initializes Mailman before commands need to be listed/used.

Pluggable workflows#

MR 299

Also rebased and added tests to get diffcov to 100% and fix one of the migrations in that branch, as it was broken before.

Key management (plugin)#

Implemented email commands for managing per-address PGP keys in plugin. It uses pluggable workflows to plug into the subscription process of a PGP enabled mailing list and requests the user key, also does mailback confirmation of it. The key is then available to the list moderator during subscription moderation, and more generally to the plugin for verifying signatures and encrypting. After subscription key management is also implemented where a user can change his set key, provided he can sign a challenge provided by the plugin with the old key. Key revocation handling is also necessary, but not yet done.

Outgoing processing (plugin)#

Implemented custom Bulk and Individual delivery classes for PGP enabled lists in plugin. These deliveries optionally encrypt the mail to subscribers keys, and/or sign with the list key. The bulk one retains anonymity of subscribers as the keyids are zeroed out of the PKESK(Public Key Encrypted Symmetric Key) packets which OpenPGP implementations should handle as a wildcard keyid and try decrypting with all usable private keys. It is also almost as efficient as it can be, as it only encrypts the message with one session key per chunk and then encrypts said session key to recipients in the chunk. The signing is also configurable.

Signature hash tracking (plugin)#

Implemented store of signature hashes from successful postings to a PGP enabled list which are then (optionally) used to stop replay of the same signature by Mailman in a future posting.

PyPI package (plugin)#

mailman-pgp @ PyPI

Created the PyPI package for the PGP plugin.

Overall#

I would like to be a bit further in the project at this point, however I am very optimistic about the next days and weeks. After resolving some issues and TODOs I have for the current plugin implementation and setting up the live Mailman instance with the PGP plugin, which should be up by the end of the week. I believe I can start work on the other side of the REST API, of somehow hooking PGP enabled archives/lists to Postorius and HyperKitty.


GSoC 2017 - Post title goes here

Signature hash tracking#

It would be relatively easy to replay a signed message to a mailing list by a user as no kind of challenge-response is done on posting.

While signature replay checking is usually done on the end users point with against his keyring and messages he has so far received and their context, I think it is kind of expected of PGP enabled Mailman to also do this as it relays the messages.

On a successful posting to a PGP enabled mailing list (AcceptedEvent to be precise), the message is searched for PGP signatures and their digests and key fingerprints are added into the sighash table.

The signature rule then checks the digests from a posting against those in the sighash table. If it finds it has previously accepted a message with any of those hashes it takes the duplicate_sig_action which is per-list configurable.

This of course means that if the duplicate_sig_action is not set to Action.defer, which means the message gets rejected or dropped, that a signature that was sent to a list cannot be sent again. Of course if a user wants to send a signed message again, he can just resign it and send it again, the hashes won’t match. However sometimes it might be useful to post the message as signed originally, for example to prove something. However, I think it is worth it to keep this as a configurable option. Maybe with another option disabling the collection of signature hashes, for performance reasons.

Outgoing processing#

Since a the plugin needs to process the messages for outgoing encryption on a per-recipient or per-chunk basis, it couldn’t be implemented as a Handler, I thought about implementing it as a custom OutgoingRunner but that didn’t work out either. However the IMailTransportAgentDelivery interface is great for what the plugin needs to do. So there are two custom PGP enabled delivery classes that get selected by a custom [mta] outgoing callable, one for bulk delivery and other for individual delivery.

Documentation#

I got the pgpmailman-proposal repository almost completely up to date on changes to the plugin and my current proposed Core/Client/Postorius/HyperKitty changes and MRs.

Next up#

More testing#

As I remembered to ignore coverage of tests it dropped to 88% and I think it is currently in order to add more comprehensive tests for features implemented that check the edge cases which currently remain.

Setup live development instance#

I believe I got pretty far into the development without having a proper live development instance of Mailman Core + plugins and Postorius + HyperKitty + Client, so I’m going to set that up now and test manually. With time I might set this up as a public site/mailing list server, to demonstrate the features of the PGP plugin.

Archivers#

Currently thinking about implementing two encrypted archivers, one local, one remote. The local one would be similar to the prototype archiver but store messages encrypted, TBD how. The remote one will send the messages encrypted to a receiving archiver, the django-pgpmailman instance running next to HyperKitty.

django-pgpmailman#

With most of the essential stuff in the core plugin done, the web ui part of development can begin.


GSoC 2017 - The point of no return

This week is around the halfway of GSoC 2017 project timeline, hence the title.

Key management#

As I set out to do, I have now implemented most of the after subscription key management. The key change command. The pre subscription key management is done via a custom dynamically loaded subscription policy/workflow. This is possible with my pluggable-workflows branch/MR which introduces dynamically loaded workflows, which subscription and unsubscription policies are a part of.

Subscription#

On subscription the PGP enabled mailing list needs a users public key. An example conversation establishing that follows (shows a list with moderation + pgp enabled subscription policy):

Mailman                                                  User
-------                                                  ----
list@example.com                                         anne@example.com
                    <- subscribe                         1.


2.                  key set "token1" ->
                    + instructions to reply and attach user key
                    (signed by list)


                    <- key set "token1"                  3.
                    (key attached as per RFC3156(PGP/MIME)
                    or inline and we do our best to find it.)


4                   key confirm "token2" ->
                    + statement containing "token2" and
                    user key fingerprint in body, with instructions
                    to sign it
                    (signed by list, encrypted to user key)


                    <- key confirm "token2"              5.
                    + must contain signed statement from step 4.
                    (required to be signed by user key)


6.                  ... the rest of the subscription steps (moderation)...

Steps 3 and 5 can be optionally encrypted to the list key.

This workflow gives “some” assurance that the user really has the key she is subscribing with, however the trust in the address/user/key has to come from the list moderator and both my proposal and implementation rely on that. This workflow only proves that Mailman is communicating with someone that has the user key (can decrypt "token2" and sign the statement), not that that user is anne@example.com.

Key change#

Works similar to the key set command on subscription. Works via a custom workflow, but not a subscription workflow/policy, just a workflow.

Mailman                                                  User
-------                                                  ----
list@example.com                                         anne@example.com
                    <- key change                        1.
                    (new! key attached as per RFC3156(PGP/MIME)
                    or inline and we do our best to find it.)


2.                  key confirm "token1" ->
                    + statement containing "token1" and
                    new! user key fingerprint in body, with instructions
                    to sign it
                    (signed by list, encrypted to new! user key)


                    <- key confirm "token1"              3.
                    + must contain signed statement from step 2.
                    (required to be signed by old! user key)

Steps 1 and 3 can again be optionally encrypted.

This ensures that only someone with possesion of the old user key can change it. It requires the user to decrypt with the new key and sign a statement with the new key fingerprint as well as the token in it with the old key.

PyPi package#

mailman-pgp @ PyPi I have now created a PyPi package for the mailman-pgp plugin as it’s maturing, and while still unusable I wanted the name.

Test coverage#

I got test coverage of mailman-pgp to 95% which now actually means something as large parts of the codebase are implemented and not just stubs. Since this plugin requires features not merged into Mailman Core (or PGPy) for pipelines to work I maintain forks of both with branches that merge all required branches into a plugin branch for Mailman Core and dev branch for PGPy.

Next up#

Outgoing#

I had a look at working in outgoing encryption of messages for pgp enabled mailing lists. Encryption cannot be done as a handler, as it requires to process the messages per-recipient or per-batch of recipients. The only configurable/dynamic option currently in Mailman Core is the [mta].outgoing callable. Which is not ideal since it itself has more functionality than simply dispatching to a correct Delivery instance based on the list configuration, that would have to be duplicated in mailman-pgp. The IMailTransportAgentDelivery interface looks promising though, so I’m thinking about refactoring the outgoing callable a bit to allow setting a custom one without the duplicities.

Signature hash tracking#

To prevent replay attacks for things where always sending and requiring a signed confirmation token in response is not practically doable, the pgp plugin will keep signature hashes with timestamps and posssibly key fingerprints in a DB/some other quickly searchable structure, and will check signatures it receives against the hashes in that structure. It will then act accordingly, for example not allowing a user to post the same message (with the same signature) again, since it could have been replayed.

Documentation#

I wanted to get the pgpmailman-proposal repository updated with current state/MRs but didn’t get to it this week, so might find time for it over the weekend as well as for some code documentation.


GSoC 2017 - Another quick update

Just a quick update about the state of the project before I disappear from civilization for this weekend.

Pluggable workflows#

MR @ gitlab

I have now finally got the pluggable workflows branch to test succesfully! With workflows being loaded dynamically, with workflow steps being saved completely, full backward compatibility, also database migrations and REST backward compatibility.

That branch is still missing more tests for diffcov to pass.

https://gitlab.com/J08nY/mailman/pipelines/9645615

^^python 3.4 CI envs are currently broken, since aiosmtpd 1.1 is python 3.5 only, so don’t mind that.

Key management#

branch @ gitlab

Building up on the pluggable workflows I have a base implementation of how the subscribers key gets to the mailing list and its moderator and also of how it’s confirmed.

SMTPS + STARTTLS#

MR @ gitlab

During this week I also got to add SMTP over SSL/TLS support to aiosmtpd, a lib which Mailman uses for it’s SMTP tests, so with that the SMTPS+STARTTLS MR has tests which pass and I think is ready for merging. Thanks Barry for the quick responses and help debugging over at aiosmtpd!

Next up#

For next week I plan on cleaning up the pluggable workflows branch, finishing the pre-subscription key management, implementing after-subscription key management as well as getting back on track with documenting the mailman-pgp architecture and changes at the pgpmailman-proposal repo, which has been lacking.


GSoC 2017 - Pluggable Workflows

As I note in one of my previous GSoC project updates, the PGP-enabled mailing lists require two new features from Mailman Core, both related to workflows. The first is to be able to inject steps into a subscription workflow to request the users pubkey before moderation checks. The second it to be able to require confirmation on all commands (to prevent replay attacks), although this could be avoided by storing the hashes from all the user sent signed commands. In this post I propose a way of refactoring the workflows into pluggable components.

Current workflows#

Both of these requirements require changes to the way [Un]SubscriptionWorkflows and workflows in general work. Currently the subscription policy of a mailing list is determined by its subscription_policy attribute of the SubscriptionPolicy enum. That policy is enforced in the SubscriptionWorkflow state machine, which curently works with a queue of states, and looks like this:

sub_flow_before

It is quite complex, and the UnsubscriptionWorkflow has much of the same code except the verification and moderation parts.

Proposal#

What I propose is to make the [Un]SubscriptionWorkflows, more generally workflows, pluggable components which would work on a stack state machine and be composed of mixins like so:

sub_flow_components

This way a mailing list will have the name of its SubscriptionWorkflow component in the subscription_policy attribute. Which would mean for example an SubscriptionPolicy.open would be transformed into a class like so:

@public
@implementer(ISubscriptionWorkflow)
class OpenSubscriptionPolicy(SubscriptionBase, VerificationMixin):
    """"""

    name = 'policy-open'
    description = 'An open subscription policy, only requires verification.'
    initial_state = 'prepare'

    def _step_prepare(self):
        self.push('do_subscription')
        self.push('verification_checks')
        self.push('sanity_checks')

I believe this simplifies the policies and also allows chaining and composing of workflow steps, which would enable plugins to add custom subscription policies enforcing more steps/checks during the [un]subscription process. For example an organization-only mailing list could add a check that checks for the subscribing address in the organization database, therefore removing the need for manual moderation of subscriptions.

This should also allow for requiring confirmation of all commands, by attaching a workflow instance to them. For the membership commands, this would be the corresponding [Un]SubscriptionWorkflows, and for other commands it would be currently disabled/empty workflow. Then a plugin could set the confirm workflow on commands it needs to.

Comments?#

I am currently working on this as a MR, although I am not sure if this is the right track to go on with workflows. I think something to split the monolithic [Un]SubscriptionWorkflows would be a nice thing in Mailman Core, as they share a lot of the same code and are overly complex for what they are, a linear series of checks with some skipping possible.


GSoC 2017 - First evaluation

The first evaluation period is here, I have already sent in my evaluation so I thought I would recap the state of the project.

During the first two weeks or so, I mostly focused on features in Mailman Core that would be necessary for making a plugin like this work, my MRs introduce an easy to use/configure plugin infrastructure. Where plugins can provide components and even custom REST routes in Mailman Core. This with just installing the plugin into the same environment as Mailman Core and some config additions. Also some minor additions of selecting the list style on creation in Postorius and appropriate changes in mailman-client.

During the last two weeks I focused on work on the mailman-pgp plugin itself, along with its helper plugin mailman-rest-events. The REST events plugin is finished, apart from some tests and bug-fixing and it does its job of sending Mailman Core events to RESTful application endpoints fine.

The mailman-pgp plugin currently has working list-key generation, decryption in incoming runner, signature checking rule and general inline and PGP/MIME message handling utilities. I also got its tests to work with Mailman Core test layers so it can be actually integration tested as it would work with Mailman Core. Along with that I setup Gitlab CI pipelines with coverage testing and currently tests pass!

As I worked in PGP handling in messages I needed a good python PGP library, which I found, but since some features mailman-pgp requires were not implemented/buggy I made quite a lot of PRs to this library as well.

Key management is the next thing I would like to work on after finishing tests of the current implementation. After that outgoing message encryption and signing. When these things work I would then move on to implement the appropriate things in the django-pgpmailman web app which hooks into Postorius and HyperKitty.

Overall I am quite confident in where the project is now. Seeing it run tests with Mailman Core running and pass (just some basic tests atm) feels good.