As last time I described outstanding PRs to Mailman Core and other Mailman’s components, this time I am going to describe the plugin I am building on top of those components to enable PGP encrypted mailing lists.

In fact there are two plugins to present, with the first one being a general and quick plugin example that showcases how the new plugin API might be used and also provides some nice services for the PGP plugin.

mailman-rest-events#

repo @ gitlab

This plugin subscribes itself to receive events in Mailman Core and sends them to urls configured. It is a variation of what mailman!264 is intending to do, based on the plugin API implemented in my MRs. It currently offers only basic functionality, a URL is specified in config per endpoint along with an API key (very similar to mailman-hyperkitty) and a regex of event class names that will be sent. The data sent currently is only the event class name in a JSON object. Config can have many endpoints specified with the form shown below.

[endpoints.django_mm3]
url: http://localhost/django/api
api_key: Something_secret_here
events: .*

Results in a call such as this one:

POST /django/api?key=Something_secret_here HTTP/1.1
...
Content-Type: application/json

{"event": "DomainCreatedEvent"}

The plugin is currently very bare bones and proper serialization of events is underway. I am thinking of recursively serializing the non-private event attributes until an easily serializable object is reached, such as an Address or a MailingList or anything that has a unique string associated with it and can be queried using the REST API and such a string. So for example:

{
  "name": "DomainCreatedEvent",
  "event": {
    "domain": "example.com"
  }
}

or

{
  "name": "AddressVerificationEvent",
  "event": {
    "address": "anne@example.org"
  }
}

This plugin will be used by the mailman-pgp plugin to send events to django-mailman3 which will distribute them to Postorius, HyperKitty and the django app created to enable PGP encrypted mailing lists.

mailman-pgp#

repo @ gitlab

First of all, I think I came up with a name that better fits with mailman-hyperkitty being an archiver “plugin” and in general with how plugins for projects are named in python (flask_ext_… and so on).

As the general functionality and layout of the plugin was already described I will mostly just summarise issues I am currently working on tackling.

low level PGP work#

I originally proposed using python-pgp (github pypi) to use to parse and create OpenPGP packets, as other libraries mostly offer only a high-level API or are not so feature rich. However this library seems quite un-maintained, so either a working fork should be established or a different library used. I am currently looking at both options. Will contact the original author as well as the current pypi package owner and see what’s possible.

public key on subscription#

Since a PGP enabled mailing list needs to know the public key of a subscribed address at the point when moderator approval is required, some mechanism to do so is necessary. This needs to happen before moderator approval, as the moderator / list owner should be able to see / download the provided key to verify it (or at least its fingerprint). Currently I see two options, an extension of the proposed key command or some changes to the current Subscription and Unsubscription workflows.

Extending the proposed key command would let the user use said command to send and confirm his public key, before a subscription to an encrypted list. This is similar to how schleuder does it. They essentially run somewhat of a mailing key-server where the users might submit keys via custom headers in message body.

However the current SubscriptionWorkflow and UnsubscriptionWorkflow are really close to being able to require a custom additional step during which the subscribers key is received and confirmed. Maybe making [Un]SubscriptionWorkflow pluggable components, that can be set on a per-list basis, similar to owner/posting chains or pipelines.

The second solution is definitely a more heavy-handed one, although might be a better one in the long run. As it allows plugins to provide [un]subscription steps which might be a nice use for plugins in Mailman Core. I can imagine for example subscription verification where the plugin verifies that the subscriber is a customer of some company by showing the user a confirmation token in the customer system etc..

require confirmation for all commands#

To address replay attacks where an attacker listens on a user performing a command by sending it to the list_id-confirm address and then sends the same message again in the future to replay the command, all commands will need to require confirmation. This stops the attacker since he cannot obtain the cleartext of the message containing the confirmation token, or even if he could, he cannot forge the signature on the confirmation message (leaving endpoint security out-of-scope). To reuse the confirm command, Workflows and confirmation would have to be refactored out from subscription management.

This week has been quite productive. I started laying the groundwork for the encrypted lists plugin by making changes from core_changes to different components of both Mailman Core, Mailman-Client and Postorius. I will describe the MRs and ideas behind them here.

Mailman Core#

Move pipelines to their own package and instantiate dynamically#

mailman!287

This MR makes Pipelines in Mailman Core work in the same dynamic way as Handlers,Rules,Chains and other similar Mailman’s components work.

Add plugin infrastructure#

mailman!288

This MR is the biggest one and certainly the most important one for the pgpmailman plugin and other potential Mailman plugins. It adds a notion of a plugin to Mailman Core. To explain what I mean by a plugin, it’s best to look at its example configuration:

[plugin.example]
path: example_plugin
class: example_plugin.plugin.ExamplePlugin
enable: yes
configuration: python:example_plugin.config.example

This configuration defines a plugin called example (section name). Which resides in the example_plugin package with the example_plugin.plugin.ExamplePlugin class implementing the IPlugin interface, which I will describe in a moment. This plugin is enabled and can use the path example_plugin.config.example.cfg to load it’s configuration.

Alright now what does this actually do / change / provide in Mailman Core?

• The plugin’s path package is searched for pluggable components if the plugin is enabled and has a non-empty path value. A new function similar to find_components is introduced and used, find_pluggable_components. The components are searched for in subpackages named after what is being searched, common to Mailman Core:
  • example_plugin.rules -> IRule
  • example_plugin.chains -> IChain
  • example_plugin.commands -> IEmailCommand
  • example_plugin.handlers -> IHandler
  • example_plugin.styles -> IStyle
  • example_plugin.pipelines -> IPipeline (after mailman!287 lands)
• The plugin’s class is instantiated if the plugin is enabled and has a non-empty class value (and implements the IPlugin interface). The IPlugin interface has several methods:
  • pre_hook() -> Is called after instantiation but before Mailman’s db initialization, really just moved the existing pre_hook configurable callable into a plugin infrastructure. Return value unused.
  • post_hook() -> Is called after pre_hook and after initialization of Mailman’s db and other dynamic components such as commands, rules, handlers… Return value unused.
  • rest_object() -> Is called whenever a REST client calls the /plugins/<plugin.name>/... route, to obtain an object which will respond to REST queries for the plugin rooted at aforementioned route. This method can return None to signify the plugin doesn’t want to expose any REST routes, in which case any calls by clients return 404. Otherwise the object should use the routing system used by Mailman’s REST api (@child(), on_get and so on).
  • Mailman also sets the name attribute on the plugin object after instantiation, before pre_hook to the plugin’s name in configuration.
• The plugin can use the path in it’s configuration option to load it’s configuration if it needs a config file. The same path resoultion rules apply as for the archivers configuration.

Add description attribute to styles#

mailman!289

Mailing list styles currently have an attribute for name, which for the default styles is:

 legacy-announce
 legacy-default

Which is not particularly human-readable, well it is, but doesn’t look like something to be shown in a web ui such as Postorius. Mapping these style names to their descriptions in Postorius is possible, however since the styles are dynamically found and instantiated, it’s not something practically doable.

So this MR adds a description attribute to IStyle and adds the appropriate descriptions to the default styles. Also exposes said description to REST clients.

Mailman-Client#

Add bindings for specifying list style#

mailmanclient!28

This MR adds missing access to the lists/styles REST api endpoint and another argument style_name to the Mailman-Client’s create_list call, which is already recognized in Core and will create a new list with selected style. This will then be used by Postorius.

Add plugin bindings#

mailmanclient!34

This MR depends on mailman!288 as it really just exposes the new REST api added with plugins to Mailman-Client. Since this new api gives plugins control over all the requests to paths after their root, the api in Mailman-Client can not be as high-level and abstracted as it is for lists and other core features. It provides access to plugin’s configuration and a Plugin.call(path, data, method) similar to Connection.call.

Postorius#

Add list style selection to list creation view#

postorius!214

This MR depends on mailmanclient!28 and mailman!289. It adds another field to the list creation form, prompting the user to select a list style, with the default being pre-selected and list style descriptions being shown.

Since a plugin-like out-of-tree approach is required for implementing encrypted lists into Mailman, a straight forward integration into Postorius and HyperKitty (as first proposed) by making them “aware” of the encrypted lists plugin is not possible.

Thus a new approach for providing their functionality and conforming to the project requirements is necessary. I see three possible pathways forward and a middle-ground between them.

Standalone django app#

A new django app will be created, using django-mailman3 as a base, that will implement all the web based functionality for encrypted mailing lists, such as:

• Displaying the List key for all public encrypted mailing lists.
• List key management for list admins
• User key management
• Encrypted archives, that are server unencrypted (effectively replaces HyperKitty for encrypted lists)

This app will then be run besides Postorius.

A fork/patchset approach#

This approach will create a fork of Postorius and HyperKitty that will integrate changes necessary for the encrypted lists plugin seamlessly. Thus users wanting to use encrypted mailing lists will have to setup Postorius and HyperKitty from this fork.

Wrenching it in#

This approach tries to integrate all of the functionality using configurable options of Postorius and HyperKitty. For example storing messages encrypted could be done via a custom django.db.backend. Receiving messages encrypted could be done via a small custom django app that will receive them, decrypt and pass to HyperKitty decrypted.

A middle-ground#

Somewhat of a middle ground seems to be most sensible. A standalone app will be necessary to provide functionality that is simply not possible to be integrated into Postorius and HyperKitty sensibly. This app will mostly provide key management (user and list), receive the messages encrypted and so on. However Postorius and HyperKitty will work with the least amount of “wrenching it in” as possible.

Structure#

• pgpmailman - A Core plugin.

  • styles - Both styles generate a list keypair based on plugin settings on list creation as well as set other attributes for an encrypted mailing list. Such as the custom encrypted chain.

    • EncryptedDefaultStyle
    • EncryptedAnnounceStyle

  • pgp

  • rules

    • EncryptionRule - Decrypts message and enforces per-list encryption requirements.
    • SignatureRule - Checks message signature and enforces per-list signature requirements. Strips signature to msgdata.

  • chains

    • EncryptedOwnerChain - Passes message through EncryptionRule before letting it continue default-owner-chain.
    • EncryptedPostingChain - Passes message through EncryptionRule and SignatureRulebefore letting it continue default-posting-chain.

  • commands

    • KeyEmailCommand - Handles user key management through the key command.
    • KeyCLICommand

  • runners

    • EncryptedOutgoingRunner - Encrypts and optionally signs for configured lists. This is a runner and not a Pipeline since we need to encrypt all outgoing messages, so digests, virgin messages, posts…

  • archivers

    • EncryptedHyperKittyArchiver - Fetches list archive public keys from pgphyperkitty, uses them to send messages to archive encrypted, for encrypted lists.

• pgphyperkitty- Django app, receives messages encrypted with list archive keys, decrypts them and passes them to HyperKitty. Also generates, holds and provides(public part) list archive keypairs.

Issues#

Rules shouldn’t change the message as they process it, since, well they are “rules” not “handlers”. However other Mailman’s rules need the message plaintext after decrypting for processing, so they cannot receive the message as it arrived encrypted. One solution would be to use a custom Incoming runner to sort-of unwrap the PGP/MIME message from encryption and signature (storing it in msgdata) and then pass it to the default Incoming runner to run chains. This would also make sense from a PGP/MIME standpoint as the “real” message is really the encrypted content so for that’s what other Mailman rules should get in the msg object.

Storing of user and list related metadata is unsolved. This means storing the user key fingerprint for each user / address, as well as storing per-list configuration. The list key fingerprint might not need to be stored as the key can be looked up by gnupg.user_id == list_fqdn. The plugin can have it’s own db although this means data duplication. A solution I see would be to allow plugins to store arbitrary metadata in the core db attached to certain models. So that the mailinglist model would have another attribute plugin_data of PickleType which would really be a dict with the plugin name as keys, with additional api allowing a plugin to access it’s metadata stored in this object.

Extending the Postorious and Hyperkitty interface is unsolved. Since plugins cannot add routes to the REST api. A solution I see that would be general and also require little changes to core would be to let plugins add custom routes to /plugins/<plugin_name>/... which could be then accessed either from a Postorious plugin or a custom app. Maybe with falcon.API.add_sink()?

Even when extending the REST api at core side is possible as just described, that still leaves integrating into HyperKitty and Postorius somehow, out-of-tree, as a django app. Looking at how HyperKitty and Postorius plug into each other currently, we see direct references at each other in the templates:

    {% if 'hyperkitty' in INSTALLED_APPS %}
    {# insert hyperkitty link #}
    {% endif %}

While that might be okay for integrating HyperKitty and Postorius together, and it’s even used at very few places and seems to not be the best solution, it’s not practically usable for integrating this plugin. Ideal integration will encrypt messages when sending them to HyperKitty while showing them decrypted to users. I am thinking of having a custom encrypted HyperKitty archiver that will handle encrypting with proper archive key at the core side and a custom django app at the HyperKitty side that will receive the message, decrypt it and pass it to HyperKitty decrypted. That however leaves the messages decrypted in HyperKitty’s DB so doesn’t really fulfill project requirements. In addition a custom DB engine can be used in HyperKitty’s settings that wraps whatever DB engine and encrypts/decrypts objects on the fly, although I am not sure of the doability of this.

Installation#

Just installing the plugin into the same virtualenv as the Mailman instance, and setting up the proper options in mailman.cfg should work.

Plugin API enhancements in Core#

To cleanly implement encrypted mailing lists as a plugin to Mailman Core I propose several general changes to the plugin api, to allow for cleaner integration of plugins, more flexibility and easier plugin deployment. First I present the current state of pluggability in Mailman core and then the proposed changes.

Current state#

Relevant mailman-developers thread from GSoC 2015

Example plugin

• A plugin creator has many ways of “injecting” his code to run at certain phases of Mailman’s operation, since Mailman looks for its classes and components dynamically, it doesn’t care whether they are from a plugin or originally from core.
  • Implementing IHandler, IChain, IRule, IEmailCommand, ICLISubCommand or IStyle and placing modules containing the classes in the appropriate directories where Mailman finds them and instantiates them.
  • Implementing IRunner and adding the custom runner to mailman.cfg.
  • Implementing IArchiver and adding the custom archiver to mailman.cfg.
  • Implementing IMailTransportAgentLifecycle and setting the custom MTA in mailman.cfg.
  • Setting a custom callable in pre_hook or post_hook (only one callable per hook).
• Core config variable ext_dir is unused.
• There is not much documentation / examples of plugin creation, however core is very well documented so it’s just a matter of figuring out what’s pluggable or not.

Proposed changes#

• Add configuration option similar to config.styles.paths to Handlers, Rules, Chains etc.. Which Mailman will use to look for components, so that plugins can just be accessible on the python path and not necessarily inside the Mailman package. Or use just one path per plugin and a standardized plugin structure:
  • plugin package
    • handlers
    • rules
    • chains
    • commands
    • pipelines
    • styles
• Let plugins add Pipelines the same way they can add Handlers, Rules etc…
  • This means refactoring BasePipeline, OwnerPipeline, PostingPipeline, VirginPipeline from mailman.core.pipelines.py into a package mailman.pipelines
  • Use find_components
• Let plugins subscribe to receive events.
• Let List creator specify List Style when creating it through Postorius.
• Allow multiple callables in pre_hook and post_hook run in order specified.
• Drop ext_dir.

ecgen is a tool for generating Elliptic curve domain parameters. While working on generating some interesting EC domain parameters for ECTester I found out there aren’t really any good tools for doing so. Cryptographic libraries don’t offer generating custom curves, they just offer preset ones or let you set custom ones. The only tools I found were ECB, LiDIA/GEC, MIRACL and cm. They all have their drawbacks however, ECB is limited in what parameters are modifiable and is closed source, LiDIA is unmaintained, and cm also doesn’t support the flexibility in parameters I needed. To add to that I lost the link to MIRACL and could not remember it’s name. So ecgen is what I started working on.

Since Elliptic Curve domain parameters are a rather complex object, generating them with various constraints and parameters is also complex and there are various algorithms for doing so. Generally two methods are used in practice. One is randomized and works as you might imagine from the name, by generating random domain parameters within some constraints, computing the rest of the parameters and hoping they satisfy the rest of the constraints. A variation of this is the ANSI X9.62 verifiably random algorithm. The other method is based on the theory of Complex Multiplication and is able to directly generate Elliptic Curve domain parameters with required constraints.

Generally the biggest problem when generating Elliptic Curve domain parameters is calculating the order of the curve specified by the curve equation. Since when this order is known the rest of the parameters (generators, group structure…) are found rather easily.

Computing this order given the equation is hard, there are several rather complex algorithms, some of which are fast for curves over $\mathbb{F}_{2^m}$ (AGM), some for curves over $\mathbb{F}_p$ (Schoof’s, SEA).

Usage#

ecgen --fp/--f2m BITS

Field specification#

• --f2mBinary field.
• --fpPrime field.

Generation options#

• -c / --count=COUNTGenerate multiple curves.
• -i / --invalidGenerate a set of invalid curves, for a given curve (using Invalid curve algorithm).
• -k / --cofactor=BOUNDGenerate a curve with cofactor up to BOUND TODO - NOT FINISHED
• --anomalous Generate an anomalous curve (of trace one, with field order equal to curve order).
• -K / --koblitzGenerate a Koblitz curve (a = 0).
• -n / --order=ORDERGenerate a curve with given ORDER (using Complex Multiplication). TODO - NOT IMPLEMENTED
• -p / --primeGenerate a curve with prime order.
• --points=TYPEGenerate points of given TYPE (random/prime/none).
• -r / --randomGenerate a random curve (using Random approach).
• -s / --seed[=SEED]Generate a curve from SEED (ANSI X9.62 verifiable procedure). TODO - NOT IMPLEMENTED
• -u / --uniqueGenerate a curve with only one generator.

IO options#

• -t / --format=FORMATFormat to output in. One of [csv,json], default is json.
• -f / --input=FILEInput from FILE.
• -o / --output=FILEOutput into FILE. Overwrites any existing file!
• -a / --appendAppend to output file (don’t overwrite).
• -v / --verbose[=FILE]Verbose logging (to stdout or FILE).

Other#

• -d / --data-dir=DIRSet PARI/GP data directory (containing seadata package).
• -m / --memory=SIZEUse PARI stack of SIZE (can have suffix k/m/g).
• --threads=NUMUse NUM threads.
• --thread-stack=SIZEUse PARI stack of SIZE (per thread, can have suffix k/m/g).

Examples#

Generate a prime field, prime order, uniquely generated 192-bit curve, don’t ask for input try random values:

> ecgen --fp -r -p -u 192

Generate 5 random, binary field, 163-bit koblitz curves:

> ecgen --f2m -r -k -c5 163

Generate invalid curves to a file, for a given prime field 192-bit curve:

> ecgen --fp -i --output=invalid.json 192
p: <input prime>
a: <input a param>
b: <input b param>

Generation methods#

Three different EC curve parameters generation methods are implemented.

Efficient Algorithms for Generating Elliptic Curves over Finite Fields Suitable for Use in Cryptography - [Baier]

Generation Methods of Elliptic Curves - [Baier, Buchmann]

Random approach#

• Generates field and equation parameters:
  • randomly
  • using ANSI X9.62 verifiably random method(from seed), until a curve with requested properties appears.
  • given input
• Can generate curves repeatedly until one satisfies requested properties:
  • -p / --prime generates curves until a prime order curve is found.
  • -K / --koblitz generates a curve with fixed A = 0 parameter.
  • -u / --unique generates a uniquely generated curve (with one generator/cyclic group).
  • etc..

Invalid curve generation#

• Generates invalid curves for a given curve.
• These curves have the same field, and A parameter in the short Weierstrass equation.
• Multiplication using some(most?) scalar multiplication algorithm proceeds the same way multiplication on the input curve would.
• Used with the -i / --invalid option
• Validation of Elliptic Curve Public Keys - [Antipa, Brown, Menezes, Struik, Vanstone]
• Differential Fault Attacks on Elliptic Curve Cryptosystems - [Biehl, Mayer, Muller]
• Practical Invalid Curve Attacks on TLS-ECDH - [Jager, Schwenk, Somorovksy]

Complex multiplication#

• Capable of generating a curve of a given (prime) order.
• Generates a subset of all Elliptic Curves over a given field.
• Used with the -n / --order option
• Constructing elliptic curves of prime order - [Broker, Stevenhagen]
• Generating Elliptic Curves of Prime Order - [Savas, Schmidt, Koc]
• Currently not implemented.

Build#

git clone https://github.com/J08nY/ecgen.git
cd ecgen
git submodule update --init
make

Requirements#

• PARI/GP

ecgen uses the PARI/GP library for elliptic curve arithmetic and it’s SEA point counting algorithm implementation. It also requires the additional seadata package (seadata and seadata-big recommended for large curves).

Libraries#

• lib/parson ©MIT
• lib/sha1 ©MPL / GPLv2 or later

parson is used to input and output JSON and is included in the lib/ directory.

A SHA-1 implementation by Paul Kocher, based on the SHA 180-1 Reference Implementation (for ANSI X9.62 algorithm) is used and also included in the lib/ directory.

License#

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

© Eastern Seaboard Phishing Authority