Articles tagged:   GSoC 2017

GSoC 2017 - PGP handling progress

This week things moved along with the mailman-pgp plugin. As I finally had a good PGP library to work with as well as some basic plugin API patches outstanding to Mailman Core.

So with that I implemented large parts of the mailman_pgp.pgp package, including tests, as well as a basic signature checking rule. Also, as the package now contains tests, I added a gitlab CI config and started running it.

repo @ neuromancer.sk repo @ gitlab

Wrappers#

Created two classes, which wrap an email.message.Message instance (or mailman.email.message.Message) and expose a common public PGP handling API on said instance. One understands PGP/MIME (as per RFC1847 and RFC3156) and the other somewhat understands inline PGP. They both give methods to check presence of a PGP signed/encrypted message or PGP keys, to verify signatures of a message, to decrypt it or collect the attached keys. There are currently almost complete tests for handling signatures and partial for encryption.

MIMEWrapper#

This one was rather easy to implement, as RFC3156 and RFC1847 are both very precise but also rather simple standards (well compared that to 90+ page RFC4880 [OpenPGP]).

InlineWrapper#

As there is really no “Inline PGP standard” this wrapper improvises with how it looks for signatures/messages/keys. In it’s current form it only accepts a non-multipart message and looks for ASCII-armored blobs in it.

Signature rule#

The concrete way of integrating the PGP plugin into Mailman Core has changed over time. The original version of my proposal planned on working in the Mailman Core tree and as that changed to a plugin, new requirements of integration popped up, us stuff couldn’t just be changed for this one plugin to work.

Because of the nature of encryption the PGP plugin needs to get it’s hands on the messages before the rest of Mailmans rules. This could be done with a rule in a custom chain, however as I understand rules, they are, well, rules, they shouldn’t change the message they are processing in any way. In my understanding of this architecture, if Mailman was written in a language with a const identifier, the message would have it in the chain links process method. It would work as a rule, just wouldn’t feel nice and fit in with the architecture.

That’s why I first wanted to implement all handling of incoming messages in a custom incoming runner. This runners class is then set as the runner for the in queue and the default runner is moved to another queue, perhaps in_default. The custom incoming runner does it’s thing before passing messages back to the default incoming runners queue. However only encryption has this issue of needing to change the message, as rules and the rest need it in cleartext. So I decided to split off signature handling to a custom rule, and only handle encryption in the incoming runner.

I would like to get rid of the not-so-nice redirection of the default incoming runner altogether but I see no other way to decrypt the incoming message to a PGP enabled list before other rules get it.

The signature rule processes messages using the wrappers described above according to a per-list signature handling configuration:

Option Default
unsigned_msg_action bounce
inline_pgp_action None (pass-through)
expired_sig_action bounce
revoked_sig_action bounce
malformed_sig_action bounce
duplicate_sig_action bounce
strip_original_sig False
sign_outgoing False

I think the options are quite self-explanatory. They are checked in order and taken if the message matches the option and the set action is not None. So with the default configuration of the PGP enabled mailing list, an unsigned message will be bounced, inline PGP is allowed, signatures by expired keys are bounced (expired at time of checking), signatures by revoked keys are bounced, signatures that don’t verify are bounced (maybe a better name for them than malformed?) and finally signatures that the Mailman PGP plugin already processed are bounced as well. This is to prevent replay attacks and is something I think people would expect Mailman to do, although I am not sure if the plugin should do it.

This signature rule is almost complete, with some edge-cases remaining and the checking of duplicate signatures also remaining.


GSoC 2017 - Finally a nice PGP library

Since my last post, I’ve been looking for a suitable PGP python library as I realized my original candidate python-pgp just wouldn’t work:

  • Its not actively maintained.
  • Its tests/docs etc.. are lacking.
  • Its security is questionable.

I’ve since found and tried using python-gnupg, gnupg, python-gpgmime and finally PGPy.

Considered#

python-gnupg#

repo@bitbucket pypi A library that uses the GNUPG binary and its status interface to provide functions similar to GNUPG command-line functionality.

gnupg#

repo@github pypi A fork of the original python-gnupg, that fixes security issues mainly considering unvalidated input passed to the GNUPG binary. However it diverges from the original library and doesn’t offer some functionality the original does.

python-gpgmime#

repo@github pypi This library provides PGP/MIME functionality on top of python-gnupg, although I got it to work with the gnupg fork after some changes. Only handles RFC 3156 partially (no combined method, sign + encrypt).

However, all of these libraries require the GPG binary, mostly work only with some specific versions of it and break terribly when it replies unexpectedly. They all suffer from having to work with the GPG binary and having to adjust to its API changes, where one additional response word (in a new version of GPG) causes the operation to fail completely. And since these work with the GPG binary they only provide what functionality it offers, so no direct PGP packet manipulation, no signing with keys from a different keyring and so on.

Selected#

PGPy#

repo@github pypi This library is an (almost complete) implementaton of the OpenPGP format (RFC 4880), that doesn’t use the GPG binary but implements packet parsing, encryption/decryption, signing and other PGP features using python and OpenSSL. Also it seems that the library is in active development, accepts PRs and has an up-to-date pypi distribution. It has an extensive test suite, when counting parametrized tests it sits at around 800+ tests.

Since it allows access to OpenPGP packets (and parses around 90% of them), it enables the Mailman PGP plugin to not strip the original senders signature (if configured to do so), or encrypt to recipients anonymously (zeroing keyids in PKESK) and so on.


GSoC 2017 - About the plugin

As last time I described outstanding PRs to Mailman Core and other Mailman’s components, this time I am going to describe the plugin I am building on top of those components to enable PGP encrypted mailing lists.

In fact there are two plugins to present, with the first one being a general and quick plugin example that showcases how the new plugin API might be used and also provides some nice services for the PGP plugin.

mailman-rest-events#

repo @ gitlab

This plugin subscribes itself to receive events in Mailman Core and sends them to urls configured. It is a variation of what mailman!264 is intending to do, based on the plugin API implemented in my MRs. It currently offers only basic functionality, a URL is specified in config per endpoint along with an API key (very similar to mailman-hyperkitty) and a regex of event class names that will be sent. The data sent currently is only the event class name in a JSON object. Config can have many endpoints specified with the form shown below.

[endpoints.django_mm3]
url: http://localhost/django/api
api_key: Something_secret_here
events: .*

Results in a call such as this one:

POST /django/api?key=Something_secret_here HTTP/1.1
...
Content-Type: application/json

{"event": "DomainCreatedEvent"}

The plugin is currently very bare bones and proper serialization of events is underway. I am thinking of recursively serializing the non-private event attributes until an easily serializable object is reached, such as an Address or a MailingList or anything that has a unique string associated with it and can be queried using the REST API and such a string. So for example:

{
  "name": "DomainCreatedEvent",
  "event": {
    "domain": "example.com"
  }
}

or

{
  "name": "AddressVerificationEvent",
  "event": {
    "address": "anne@example.org"
  }
}

This plugin will be used by the mailman-pgp plugin to send events to django-mailman3 which will distribute them to Postorius, HyperKitty and the django app created to enable PGP encrypted mailing lists.

mailman-pgp#

repo @ gitlab

First of all, I think I came up with a name that better fits with mailman-hyperkitty being an archiver “plugin” and in general with how plugins for projects are named in python (flask_ext_… and so on).

As the general functionality and layout of the plugin was already described I will mostly just summarise issues I am currently working on tackling.

low level PGP work#

I originally proposed using python-pgp (github pypi) to use to parse and create OpenPGP packets, as other libraries mostly offer only a high-level API or are not so feature rich. However this library seems quite un-maintained, so either a working fork should be established or a different library used. I am currently looking at both options. Will contact the original author as well as the current pypi package owner and see what’s possible.

public key on subscription#

Since a PGP enabled mailing list needs to know the public key of a subscribed address at the point when moderator approval is required, some mechanism to do so is necessary. This needs to happen before moderator approval, as the moderator / list owner should be able to see / download the provided key to verify it (or at least its fingerprint). Currently I see two options, an extension of the proposed key command or some changes to the current Subscription and Unsubscription workflows.

Extending the proposed key command would let the user use said command to send and confirm his public key, before a subscription to an encrypted list. This is similar to how schleuder does it. They essentially run somewhat of a mailing key-server where the users might submit keys via custom headers in message body.

However the current SubscriptionWorkflow and UnsubscriptionWorkflow are really close to being able to require a custom additional step during which the subscribers key is received and confirmed. Maybe making [Un]SubscriptionWorkflow pluggable components, that can be set on a per-list basis, similar to owner/posting chains or pipelines.

The second solution is definitely a more heavy-handed one, although might be a better one in the long run. As it allows plugins to provide [un]subscription steps which might be a nice use for plugins in Mailman Core. I can imagine for example subscription verification where the plugin verifies that the subscriber is a customer of some company by showing the user a confirmation token in the customer system etc..

require confirmation for all commands#

To address replay attacks where an attacker listens on a user performing a command by sending it to the list_id-confirm address and then sends the same message again in the future to replay the command, all commands will need to require confirmation. This stops the attacker since he cannot obtain the cleartext of the message containing the confirmation token, or even if he could, he cannot forge the signature on the confirmation message (leaving endpoint security out-of-scope). To reuse the confirm command, Workflows and confirmation would have to be refactored out from subscription management.


GSoC 2017 - About a week in

This week has been quite productive. I started laying the groundwork for the encrypted lists plugin by making changes from core_changes to different components of both Mailman Core, Mailman-Client and Postorius. I will describe the MRs and ideas behind them here.

Mailman Core#

Move pipelines to their own package and instantiate dynamically#

mailman!287

This MR makes Pipelines in Mailman Core work in the same dynamic way as Handlers,Rules,Chains and other similar Mailman’s components work.

Add plugin infrastructure#

mailman!288

This MR is the biggest one and certainly the most important one for the pgpmailman plugin and other potential Mailman plugins. It adds a notion of a plugin to Mailman Core. To explain what I mean by a plugin, it’s best to look at its example configuration:

[plugin.example]
path: example_plugin
class: example_plugin.plugin.ExamplePlugin
enable: yes
configuration: python:example_plugin.config.example

This configuration defines a plugin called example (section name). Which resides in the example_plugin package with the example_plugin.plugin.ExamplePlugin class implementing the IPlugin interface, which I will describe in a moment. This plugin is enabled and can use the path example_plugin.config.example.cfg to load it’s configuration.

Alright now what does this actually do / change / provide in Mailman Core?

  • The plugin’s path package is searched for pluggable components if the plugin is enabled and has a non-empty path value. A new function similar to find_components is introduced and used, find_pluggable_components. The components are searched for in subpackages named after what is being searched, common to Mailman Core:
    • example_plugin.rules -> IRule
    • example_plugin.chains -> IChain
    • example_plugin.commands -> IEmailCommand
    • example_plugin.handlers -> IHandler
    • example_plugin.styles -> IStyle
    • example_plugin.pipelines -> IPipeline (after mailman!287 lands)
  • The plugin’s class is instantiated if the plugin is enabled and has a non-empty class value (and implements the IPlugin interface). The IPlugin interface has several methods:
    • pre_hook() -> Is called after instantiation but before Mailman’s db initialization, really just moved the existing pre_hook configurable callable into a plugin infrastructure. Return value unused.
    • post_hook() -> Is called after pre_hook and after initialization of Mailman’s db and other dynamic components such as commands, rules, handlers… Return value unused.
    • rest_object() -> Is called whenever a REST client calls the /plugins/<plugin.name>/... route, to obtain an object which will respond to REST queries for the plugin rooted at aforementioned route. This method can return None to signify the plugin doesn’t want to expose any REST routes, in which case any calls by clients return 404. Otherwise the object should use the routing system used by Mailman’s REST api (@child(), on_get and so on).
    • Mailman also sets the name attribute on the plugin object after instantiation, before pre_hook to the plugin’s name in configuration.
  • The plugin can use the path in it’s configuration option to load it’s configuration if it needs a config file. The same path resoultion rules apply as for the archivers configuration.

Add description attribute to styles#

mailman!289

Mailing list styles currently have an attribute for name, which for the default styles is:

 legacy-announce
 legacy-default

Which is not particularly human-readable, well it is, but doesn’t look like something to be shown in a web ui such as Postorius. Mapping these style names to their descriptions in Postorius is possible, however since the styles are dynamically found and instantiated, it’s not something practically doable.

So this MR adds a description attribute to IStyle and adds the appropriate descriptions to the default styles. Also exposes said description to REST clients.

Mailman-Client#

Add bindings for specifying list style#

mailmanclient!28

This MR adds missing access to the lists/styles REST api endpoint and another argument style_name to the Mailman-Client’s create_list call, which is already recognized in Core and will create a new list with selected style. This will then be used by Postorius.

Add plugin bindings#

mailmanclient!34

This MR depends on mailman!288 as it really just exposes the new REST api added with plugins to Mailman-Client. Since this new api gives plugins control over all the requests to paths after their root, the api in Mailman-Client can not be as high-level and abstracted as it is for lists and other core features. It provides access to plugin’s configuration and a Plugin.call(path, data, method) similar to Connection.call.

Postorius#

Add list style selection to list creation view#

postorius!214

This MR depends on mailmanclient!28 and mailman!289. It adds another field to the list creation form, prompting the user to select a list style, with the default being pre-selected and list style descriptions being shown.


GSoC 2017 - Integrating with Postorius and Hyperkitty

Since a plugin-like out-of-tree approach is required for implementing encrypted lists into Mailman, a straight forward integration into Postorius and HyperKitty (as first proposed) by making them “aware” of the encrypted lists plugin is not possible.

Thus a new approach for providing their functionality and conforming to the project requirements is necessary. I see three possible pathways forward and a middle-ground between them.

Standalone django app#

A new django app will be created, using django-mailman3 as a base, that will implement all the web based functionality for encrypted mailing lists, such as:

  • Displaying the List key for all public encrypted mailing lists.
  • List key management for list admins
  • User key management
  • Encrypted archives, that are server unencrypted (effectively replaces HyperKitty for encrypted lists)

This app will then be run besides Postorius.

A fork/patchset approach#

This approach will create a fork of Postorius and HyperKitty that will integrate changes necessary for the encrypted lists plugin seamlessly. Thus users wanting to use encrypted mailing lists will have to setup Postorius and HyperKitty from this fork.

Wrenching it in#

This approach tries to integrate all of the functionality using configurable options of Postorius and HyperKitty. For example storing messages encrypted could be done via a custom django.db.backend. Receiving messages encrypted could be done via a small custom django app that will receive them, decrypt and pass to HyperKitty decrypted.

A middle-ground#

Somewhat of a middle ground seems to be most sensible. A standalone app will be necessary to provide functionality that is simply not possible to be integrated into Postorius and HyperKitty sensibly. This app will mostly provide key management (user and list), receive the messages encrypted and so on. However Postorius and HyperKitty will work with the least amount of “wrenching it in” as possible.


GSoC 2017 - PGPMailman plugin

Structure#

  • pgpmailman - A Core plugin.

    • styles - Both styles generate a list keypair based on plugin settings on list creation as well as set other attributes for an encrypted mailing list. Such as the custom encrypted chain.

      • EncryptedDefaultStyle
      • EncryptedAnnounceStyle
    • pgp

    • rules

      • EncryptionRule - Decrypts message and enforces per-list encryption requirements.
      • SignatureRule - Checks message signature and enforces per-list signature requirements. Strips signature to msgdata.
    • chains

      • EncryptedOwnerChain - Passes message through EncryptionRule before letting it continue default-owner-chain.
      • EncryptedPostingChain - Passes message through EncryptionRule and SignatureRulebefore letting it continue default-posting-chain.
    • commands

      • KeyEmailCommand - Handles user key management through the key command.
      • KeyCLICommand
    • runners

      • EncryptedOutgoingRunner - Encrypts and optionally signs for configured lists. This is a runner and not a Pipeline since we need to encrypt all outgoing messages, so digests, virgin messages, posts…
    • archivers

      • EncryptedHyperKittyArchiver - Fetches list archive public keys from pgphyperkitty, uses them to send messages to archive encrypted, for encrypted lists.
  • pgphyperkitty- Django app, receives messages encrypted with list archive keys, decrypts them and passes them to HyperKitty. Also generates, holds and provides(public part) list archive keypairs.

Issues#

Rules shouldn’t change the message as they process it, since, well they are “rules” not “handlers”. However other Mailman’s rules need the message plaintext after decrypting for processing, so they cannot receive the message as it arrived encrypted. One solution would be to use a custom Incoming runner to sort-of unwrap the PGP/MIME message from encryption and signature (storing it in msgdata) and then pass it to the default Incoming runner to run chains. This would also make sense from a PGP/MIME standpoint as the “real” message is really the encrypted content so for that’s what other Mailman rules should get in the msg object.

Storing of user and list related metadata is unsolved. This means storing the user key fingerprint for each user / address, as well as storing per-list configuration. The list key fingerprint might not need to be stored as the key can be looked up by gnupg.user_id == list_fqdn. The plugin can have it’s own db although this means data duplication. A solution I see would be to allow plugins to store arbitrary metadata in the core db attached to certain models. So that the mailinglist model would have another attribute plugin_data of PickleType which would really be a dict with the plugin name as keys, with additional api allowing a plugin to access it’s metadata stored in this object.

Extending the Postorious and Hyperkitty interface is unsolved. Since plugins cannot add routes to the REST api. A solution I see that would be general and also require little changes to core would be to let plugins add custom routes to /plugins/<plugin_name>/... which could be then accessed either from a Postorious plugin or a custom app. Maybe with falcon.API.add_sink()?

Even when extending the REST api at core side is possible as just described, that still leaves integrating into HyperKitty and Postorius somehow, out-of-tree, as a django app. Looking at how HyperKitty and Postorius plug into each other currently, we see direct references at each other in the templates:

    {% if 'hyperkitty' in INSTALLED_APPS %}
    {# insert hyperkitty link #}
    {% endif %}

While that might be okay for integrating HyperKitty and Postorius together, and it’s even used at very few places and seems to not be the best solution, it’s not practically usable for integrating this plugin. Ideal integration will encrypt messages when sending them to HyperKitty while showing them decrypted to users. I am thinking of having a custom encrypted HyperKitty archiver that will handle encrypting with proper archive key at the core side and a custom django app at the HyperKitty side that will receive the message, decrypt it and pass it to HyperKitty decrypted. That however leaves the messages decrypted in HyperKitty’s DB so doesn’t really fulfill project requirements. In addition a custom DB engine can be used in HyperKitty’s settings that wraps whatever DB engine and encrypts/decrypts objects on the fly, although I am not sure of the doability of this.

Installation#

Just installing the plugin into the same virtualenv as the Mailman instance, and setting up the proper options in mailman.cfg should work.


GSoC 2017 - Mailman encrypted lists update

Plugin API enhancements in Core#

To cleanly implement encrypted mailing lists as a plugin to Mailman Core I propose several general changes to the plugin api, to allow for cleaner integration of plugins, more flexibility and easier plugin deployment. First I present the current state of pluggability in Mailman core and then the proposed changes.

Current state#

Relevant mailman-developers thread from GSoC 2015

Example plugin

  • A plugin creator has many ways of “injecting” his code to run at certain phases of Mailman’s operation, since Mailman looks for its classes and components dynamically, it doesn’t care whether they are from a plugin or originally from core.
    • Implementing IHandler, IChain, IRule, IEmailCommand, ICLISubCommand or IStyle and placing modules containing the classes in the appropriate directories where Mailman finds them and instantiates them.
    • Implementing IRunner and adding the custom runner to mailman.cfg.
    • Implementing IArchiver and adding the custom archiver to mailman.cfg.
    • Implementing IMailTransportAgentLifecycle and setting the custom MTA in mailman.cfg.
    • Setting a custom callable in pre_hook or post_hook (only one callable per hook).
  • Core config variable ext_dir is unused.
  • There is not much documentation / examples of plugin creation, however core is very well documented so it’s just a matter of figuring out what’s pluggable or not.

Proposed changes#

  • Add configuration option similar to config.styles.paths to Handlers, Rules, Chains etc.. Which Mailman will use to look for components, so that plugins can just be accessible on the python path and not necessarily inside the Mailman package. Or use just one path per plugin and a standardized plugin structure:
    • plugin package
      • handlers
      • rules
      • chains
      • commands
      • pipelines
      • styles
  • Let plugins add Pipelines the same way they can add Handlers, Rules etc…
    • This means refactoring BasePipeline, OwnerPipeline, PostingPipeline, VirginPipeline from mailman.core.pipelines.py into a package mailman.pipelines
    • Use find_components
  • Let plugins subscribe to receive events.
  • Let List creator specify List Style when creating it through Postorius.
  • Allow multiple callables in pre_hook and post_hook run in order specified.
  • Drop ext_dir.

1   2
Next