Fixpoint

2022-10-26

Classifying the Dovecot module menagerie by load point

Filed under: Email, Gales Linux, JWRD, MySQL, Software — Jacob Welsh @ 23:55

Upon receiving a resounding "meh" from the people code monkeys formerly considered "upstream" for the Dovecot IMAP server, we gained the clarity that we were indeed stuck with cleaning up yet another mess, to the degree it was to be cleaned up at all ; and we'd have to choose this degree carefully because holy hell there's still so much else to be done.

To bring some order to the discussion, let's start with an attempt at some previously missing definitions:

In Dovecot terminology, a module is one or more compilation units (.c files) compiled into a single shared object (.so file; contrary to appearances the suffix is not configurable), to be imported into process address space at startup, by the dynamic linker via dlopen.

The term plugin also comes up frequently, and seems to be more of a governance term than a technical one, referring to any collection of code built into any collection of modules - or perhaps even new executables - covering a particular class of optional features, and having some degree of independence in origin or maintenance from the main Dovecot codebase. That said, a number of plugins are currently bundled in the tree and covered in the documentation. Plugin support requires module support, but isn't the only thing that does.

Thus, our problem is that some number of possibly important features, which contributed to bringing Dovecot to the forefront of our search in the first place, currently require module support which currently assumes dynamic linking.

At first I'd been considering only the first layer of the problem, namely the build system, charitably proceeding on the hunch that the rot was contained to that layer. That's how it usually goes with onions after all, right? Perhaps it simply wasn't set up to link the modules in static library form (.a files) into the required executables so all I'd have to do was track down what those were and do some Makefile fiddling and we'd be set. Hey, since modules worked fine in static configuration for the Linux kernel, Python, PHP etc,(i) with only that weirdo SBCL falling flat on its face thus far, how bad could it be?

Pretty bad, it turns out, as you've no doubt guessed by now. Seeing as a module compiles to a library, and a library is a collection of named procedures and variables without a main program entry point, if there's no code to refer to any of those procedures and variables (symbols) then they can't possibly have any effect.(ii) And just as the build system was written without a hint of support for statically linking modules, so the code was written without a hint of support for statically initializing them.

Since I'd achieved a working build without module support, we at least had the option of leaving it out and doing without plugins. But without documentation especially of what all the internal, non-plugin modules were, we'd be proceeding blind to what missing pieces might come to bite us down the line.

As I'd learned from fixing the static build failure, most parts of Dovecot that act on modules - loading, unloading, initializing, deinitializing,(iii) and looking up symbols - don't call dlopen/dlsym directly but go through a "modules API", found at src/lib/module-dir.c. Thus my next step was to scan the codebase for all uses of those functions and draw up a complete map of what calls them directly and what those get linked into at the file level; that is, identifying all executables that could possibly call them, without fully tracing the call chains. While this was certainly helpful, the list of executables turned out to be pretty much everything so it wasn't enough.

I sketched out a next guiding question, roughly, to narrow the search to just the parts of the code that load modules so as to look at what they're doing and what modules they might load.(iv) What soon came to light was that there are several different species of modules, corresponding loosely to the different places in the code that load them. Some of these can be seen just by looking at the build system because they're installed to a particular subdirectory of the top-level module installation path ($(prefix)/lib/dovecot aka moduledir aka pkglibdir), while others are more implicit.

With the looking done, what became sorely needed was to organize the findings into some consistent structure for easy reference, so as to finally make some decisions and keep track of progress without getting lost in the forest. Let us proceed to enumerate each of those load points, thereby labeling each species or type of module, and in some cases even listing each possible instance.

1.

  • File: lib-fs/fs-api.c
  • Function: fs_class_try_load_plugin
  • Search path: moduledir
  • Name list or pattern: libfs_<driver>.so
  • Name parameter source: caller, perhaps config file
  • Module type: Filesystem driver
  • Plugin modules: libfs_compress, libfs_crypt, libfs_mail_crypt
  • Non-plugin modules:
  • Calling executables: possibly many
  • Notes: Most drivers are builtin (fs_classes_init); modules are only used for plugin-supplied ones. Unclear if there's any difference between "class" and "driver". Uses module_get_symbol to load fs_class_<driver> so as to fs_class_register() it.
  • Proposed action: Remove load point, fs-compress and mail-crypt plugins.

2.

  • File: auth/main.c
  • Function: main_preinit
  • Search path: moduledir/auth
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Authentication
  • Plugin modules: lib20_auth_var_expand_crypt
  • Non-plugin modules: libauthdb_imap, libmech_gssapi, libauthdb_ldap, libauthdb_lua
  • Calling executables: libexec/auth
  • Notes: The var-expand-crypt plugin also builds its same source file into a distinct but identical non-"auth" module: lib20_var_expand_crypt. gssapi, ldap and lua modules may be initialized specially via auth/passdb.c (passdbs_init). There's another load point at auth_module_load, called by passdb_preinit, userdb_preinit and mech_register_init, which seems entirely redundant with this one, loading specific modules that would be already covered by the blanket loading here.
  • Proposed action: Remove load point, var-expand-crypt plugin, and non-plugin modules.

3.

  • File: config/config-parser.c
  • Function: config_parse_load_modules
  • Search path: moduledir/settings
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Settings
  • Plugin modules: pigeonhole (external)
  • Non-plugin modules:
  • Calling executables: bin/doveconf, libexec/config
  • Notes: Uses module_get_symbol_quiet to load: <modname>_set_roots, <modname>_service_settings_array, <modname>_service_settings.
  • Proposed action: Remove load point, leaving a stub where the relevant pigeonhole module(s) would be statically initialized if we integrate it.

4.

  • File: dict/main.c
  • Function: main_init
  • Search path: moduledir/dict
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Dictionary
  • Plugin modules:
  • Non-plugin modules: libdict_ldap (if LDAP_PLUGIN)
  • Calling executables: libexec/dict
  • Notes: libdict_ldap already has the option of being builtin.
  • Proposed action: Remove load point; in the build system and ifdefs for libdict_ldap, remove the plugin option so it's either builtin or absent.

5.

  • File: lib-ssl-iostream/iostream-ssl.c
  • Function: ssl_module_load
  • Search path: moduledir
  • Name list or pattern: libssl_iostream_openssl.so
  • Name parameter source: n/a
  • Module type: singular
  • Plugin modules:
  • Non-plugin modules: libssl_iostream_openssl (if BUILD_OPENSSL and HAVE_SSL)
  • Calling executables: possibly many
  • Notes:
  • Proposed action: Remove load point and statically call ssl_iostream_openssl_init (conditional on HAVE_SSL as it already is).

6.

  • File: login-common/main.c
  • Function: login_load_modules
  • Search path: configurable via global_login_settings->login_plugin_dir, default moduledir/login
  • Name list or pattern: configurable list via global_login_settings->login_plugins, default none
  • Name parameter source: n/a
  • Module type: Login process
  • Plugin modules:
  • Non-plugin modules:
  • Calling executables: libexec/imap-login, libexec/imap-urlauth-login, libexec/pop3-login, libexec/submission-login
  • Notes:
  • Proposed action: Remove load point.

7.

  • File: old-stats/main.c
  • Function: main_preinit
  • Search path: moduledir/old-stats
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Old statistics process
  • Plugin modules: libold_stats_mail
  • Non-plugin modules: libstats_auth
  • Calling executables: libexec/old-stats
  • Notes: Not to be confused with imap-old-stats which appears to be an ordinary Mail plugin.
  • Proposed action: Look at the modules and decide whether to always include or fully remove.

8.

  • File: doveadm/doveadm-mail.c
  • Function: doveadm_mail_init_finish
  • Search path: configurable via doveadm_settings->mail_plugin_dir, default moduledir
  • Name list or pattern: configurable via doveadm_settings->mail_plugins, default none
  • Name parameter source: n/a
  • Module type: Mail or perhaps a Mail Administration subtype?
  • Plugin modules: unknown, could be many
  • Non-plugin modules: unknown
  • Calling executables: bin/doveadm
  • Notes:
  • Proposed action: Replace with a static list of <modname>_init calls based on what plugins are enabled at compile time. Alternatively, retain the module loader code, adding an internally wired table of the <modname>_init functions so that all plugins get compiled and linked but only the user-specified ones initialized.

9.

  • File: doveadm/doveadm-pw.c
  • Function: cmd_pw
  • Search path: moduledir/auth
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Authentication
  • Plugin modules: lib20_auth_var_expand_crypt
  • Non-plugin modules: libauthdb_imap, libmech_gssapi, libauthdb_ldap, libauthdb_lua
  • Calling executables: bin/doveadm
  • Notes: It would seem that auth modules can add functionality both to the "auth" daemon process (handling authentication requests) and to doveadm (perhaps configuring users & passwords).
  • Proposed action: Remove load point, var-expand-crypt plugin, and non-plugin modules.

10.

  • File: doveadm/doveadm-util.c
  • Function: doveadm_load_modules
  • Search path: moduledir/doveadm
  • Name list or pattern: *.so
  • Name parameter source: n/a
  • Module type: Administration
  • Plugin modules: lib10_doveadm_acl_plugin, lib10_doveadm_quota_plugin, lib20_doveadm_fts_lucene_plugin (if BUILD_LUCENE), lib20_doveadm_fts_plugin, libdoveadm_mail_crypt_plugin
  • Non-plugin modules:
  • Calling executables: bin/doveadm, libexec/doveadm-server
  • Notes:
  • Proposed action: Remove load point and statically call <modname>_init for all such modules, each conditional on being enabled at compile time. Remove fts-lucene plugin.

11.

  • File: lib-storage/mail-user.c
  • Function: mail_user_try_load_class_plugin
  • Search path: user->set->mail_plugin_dir, default moduledir
  • Name list or pattern: <name>.so
  • Name parameter source: storage driver, perhaps parsed from configured mailbox path
  • Module type: Storage driver
  • Plugin modules: none?
  • Non-plugin modules: none?
  • Calling executables: libexec/imap etc.
  • Notes: Seems to be a vestige of modular mailbox formats which are now all builtin.
  • Proposed action: Remove load point.

12.

  • File: lmtp/lmtp-client.c
  • Function: client_load_modules
  • Search path: client->lmtp_set->mail_plugin_dir, default moduledir
  • Name list or pattern: client->lmtp_set->mail_plugins, default none
  • Name parameter source: n/a
  • Module type: Mail or perhaps LMTP?
  • Plugin modules: unknown, could be many
  • Non-plugin modules: unknown
  • Calling executables: libexec/lmtp
  • Notes: Can one module extend both mail (imap etc) and lmtp processes, or are these properly regarded as distinct module types?
  • Proposed action: If we do any sort of customized delivery, we'd be using the LDA, not LMTP; I'm pretty sure qmail doesn't speak the latter, and why should it. So... remove the whole LMTP tree??

13.

  • File: lib-dcrypt/dcrypt.c
  • Function: dcrypt_initialize
  • Search path: moduledir
  • Name list or pattern: libdcrypt_<backend>.so
  • Name parameter source: "openssl" only, as default and also set explicitly by some callers.
  • Module type: singular
  • Plugin modules:
  • Non-plugin modules: libdcrypt_openssl (if BUILD_DCRYPT_OPENSSL)
  • Calling executables: bin/doveadm, various (via auth/db-oauth2.c and plugins mail-crypt, var-expand-crypt)
  • Notes: Used to be switchable with gnutls.
  • Proposed action: Similar to #5, remove load point and statically call dcrypt_openssl_init (conditional on DCRYPT_BUILD_OPENSSL? but that's a make variable, not a preprocessor one).

14.

  • File: lib-storage/mail-storage-service.c
  • Function: mail_storage_service_load_modules
  • Search path: user_set->mail_plugin_dir, default moduledir
  • Name list or pattern: user_set->mail_plugins, default none
  • Name parameter source: n/a
  • Module type: Mail
  • Plugin modules: many
  • Non-plugin modules: unknown
  • Calling executables: bin/doveadm, libexec/doveadm-server, libexec/imap-urlauth-worker, libexec/imap, libexec/indexer-worker, libexec/dovecot-lda, libexec/lmtp, libexec/pop3, libexec/submission, libexec/script-login, libexec/quota-status (from quota plugin), possibly more via mail-crypt plugin.
  • Notes: Presumed to be the "main entrance", where the chosen mail plugins get loaded by most executables that do such loading.
  • Proposed action: See #8.

In trying to track down the various "unknowns" above for non-plugin modules (which at this point look most likely to be simply "none"), a troublesome question came up that I can't yet account for. The SQL drivers, which as noted can be configured as either builtins or modules following the libdriver_<dbname> pattern, didn't come up under any of these load points; yet there's no indication in the docs that they'd have to be loaded explicitly by listing them in a mail_plugins setting. The proposed action would be like #4, to remove the plugin option so they're either builtin or absent, but why did the exhaustive search fail to discover their existence?

As for what's the minimum work that must be done now to get the desired modules working, many of the "remove load point" items could be postponed as cleanups for another day, although I don't expect them to be too difficult anyway. The SSL parts could be left broken as they are. Looking at #8 and #14, it begins to appear that generic replacement of the modules API isn't really an alternative as much as an additional layer on top of what's required here anyway. For instance, such a replacement would need to reproduce the concept of "loading all modules of a given type i.e. from a given directory or naming pattern", in a context where there are no such filesystem directories to obtain a list of .so file names to scan. Thus I'm inclined more than ever not to bother.

  1. The jury's out on Perl; it seems fine so far but I do see some modules with .a files in /gales/pkg/perl/arch/auto/ and I lack the familiarity to readily determine if they're truly working. [^]
  2. Except for memory corruption attacks causing control to jump into library code unexpectedly, thus broadening the attacker's options for escalation, true, there is that "advantage", this is C after all. That risk is worsened by shared libraries, broad ones like libc especially : because they're shared between processes they have to be immutable, thus they have to be mapped into each process address space in full, unlike static libraries where only the compilation units actually referenced by a given executable need be linked into it. But that's OK, see, because shared libraries allow for address space scrambling, and we know how content scrambling worked out for DVDs, right? [^]
  3. I have no idea why include code for unloading or deinitializing modules, in the proper order and everything. Surely you could simply restart any running daemons to apply a change to the module configuration, and don't you have to do that anyway? [^]
  4. I removed module_dir_init from my search list upon finding it takes an array of loaded module objects, not their names, so tracing it would not be as informative as to which modules those are. [^]

11 Comments »

  1. Thanks for such a thorough write up.

    Ok, re SQL, noted.

    I say let's try postponing the remove load point items on the first pass so as to move ahead with minimal changes. If it turns out the changes are needed, we know what to do. What's your inclination though ?

    Comment by Robinson Dorion — 2022-10-27 @ 14:46

  2. You're welcome.

    I don't think the removals will generally come up as hard necessities; it's more that the code becomes dead weight, slowing down any codebase-wide operations like searching (and compiling, though at least that's just a matter of machine time), as well as local operations like reading. A compromise would be to remove just the condemned plugins/modules as proposed but leaving the load points, since it would be mostly a file or directory level removal, rather than removing code within a file and possibly cascading up to the callers.

    I guess your argument is basically, we did the hard part of finding what can be safely killed and setting it all down, but so far it's not even worth the expenditure of ammunition to follow through. I'll demur on expressing an inclination for now, but can certainly start with the required parts like answering the SQL drivers question and taking the actions for 8, 10 and 14.

    Comment by Jacob Welsh — 2022-10-27 @ 17:14

  3. SQL drivers mystery solved: while they're installed to the top-level moduledir, they're symlinked into both auth and dict subdirs, thereby falling under the catch-all load points 2, 4 and 9.

    However, I do not think it would be accurate to classify them as either Authentication or Dictionary modules, because they do not by themselves implement any new auth method or dict backend. Rather, they implement an internal SQL API (as in lib-sql/sql-api.c et al, with its own virtual method dispatch table and all) which is used in turn by the builtin auth/db-sql.c on the auth side and ...apparently nothing on the dict side.

    Thus, despite lacking a load point of their own I think it's fair to consider them a distinct module type: SQL Driver, and I stand by the proposed action: "like #4, to remove the plugin option so they're either builtin or absent".

    The bleeding stanched from this hole in my model, I now believe the non-plugin modules have been fully listed, on the strength of a "grep 'module_.*LTLIBRARIES' -r src" not turning up anything new outside of plugins. I will update the article to strike out those cautiously listed as "unknown".

    Comment by Jacob Welsh — 2022-10-28 @ 02:07

  4. It's looking like some degree of "static modules API" implementation is going to be required in any case; the runtime configurability of which plugins to load is compounded by the division (entirely spurious IMHO) between loading and initialization hinted at in footnote iv. Case in point, load point #8 stores the loaded module object array at a global "mail_storage_service_modules" pointer; initialization of those modules then is nowhere to be found in the doveadm subtree but possibly done at some other time by lib-storage/mail-storage-service.c. Module objects track their initialization status so that it's done only once, meaning the code may be assuming it can safely call module_dir_init multiple times on the same modules, so it may be unsafe to replace those with direct calls to the specific module initialization functions.

    The approach will likely be a hybrid, enumerating the specific modules at the load points as proposed, thereby getting them linked into the necessary binaries, perhaps using a MODULE_LOAD_STATIC macro of some sort to specify the name and init/deinit routines, or a virtual method table structure to define them within each module itself. Then the remaining module object methods would continue to be used as-is.

    Yet another complication (undocumented) is that a module can also have a "preinit" routine which if present is called at load time. The old-stats plugin (lib90_old_stats_plugin) has the only instance of it among the plugins; unsure about non-plugin modules so far. Its use there is to open /proc/self/io, for later gathering counts of read/write syscalls and total bytes. I hadn't encountered this interface before so looked it up in proc(5), which reports its origin as Linux kernel 2.6.20 and notes:

    In the current implementation, things are a bit racy on 32-bit systems: if process A reads process B's /proc/[pid]/io while process B is updating one of these 64-bit counters, process A could see an intermediate result.

    IOW, it can sometimes report nonsense values. Needless to say, I'm unimpressed and would sooner take it out (which looks quite simple) than implement the preinit mechanism just for this.

    Comment by Jacob Welsh — 2022-10-28 @ 23:17

  5. Indeed from the git log it looks like that preinit hook was specifically added for the old-stats plugin, likely for permissions reasons (privileges being dropped at a later stage so /proc/self/io is no longer readable).

    Comment by Jacob Welsh — 2022-10-28 @ 23:25

  6. Re #12,

    Can one module extend both mail (imap etc) and lmtp processes, or are these properly regarded as distinct module types?

    doc/example-config/conf.d/20-lmtp.conf illustrates a mail_plugins setting within a "protocol lmtp" block, describing it as a "space separated list of plugins to load (default is global mail_plugins)." This strongly suggests that they're the same type of module, just that you can load different ones for the different protocols.

    Comment by Jacob Welsh — 2022-10-29 @ 00:35

  7. Re #2,

    There's another load point at auth_module_load, called by passdb_preinit, userdb_preinit and mech_register_init, which seems entirely redundant with this one, loading specific modules that would be already covered by the blanket loading here.

    what's up there is that the blanket load point is using a filter function passed through a settings struct to exclude any module names starting with authdb_ or mech_ (that is, all of the listed non-plugin modules), stating that those are "lazily loaded" i.e. by the mentioned auth_module_load.

    I'd say then that the case is even clearer for removing the blanket load point which in fact covers only the var-expand-crypt plugin, while gssapi/ldap/imap auth options could perhaps still be kept with minimal extra effort given the way things are shaping up anyway.

    Comment by Jacob Welsh — 2022-10-29 @ 05:04

  8. Continuing re #2,

    gssapi, ldap and lua modules may be initialized specially via auth/passdb.c (passdbs_init).

    This is approximately correct (to be precise, gssapi is done in auth/mech.c (mech_init)). What this means is that those modules (libauthdb_ldap, libauthdb_lua and libmech_gssapi) are already set up to work builtin, much like libdict_ldap in #4. Thus I'm thinking they can be left in. This leaves libauthdb_imap, which is built unconditionally and has no builtin variant, so I might go ahead and remove it along with the load point; it looks fairly self-contained so this should be easy. Then again, it could still be sustained using the new static module structure.

    Re #9,

    It would seem that auth modules can add functionality both to the "auth" daemon process (handling authentication requests) and to doveadm (perhaps configuring users & passwords).

    Nonetheless I can't find how doveadm would be at all affected by auth modules, at least the authdb and mech ones on the table. Thus I'd go ahead and remove the load point but leave the auth modules alone except as covered in #2.

    5, 7 and 13 are threatening to turn into further time sinks. Besides that, the removals are going smoothly enough.

    Comment by Jacob Welsh — 2022-11-02 @ 04:56

  9. [...] progressed in porting Dovecot last week. Jacob's first step was producing an article to clearly structure the findings from the prior work. With this firm ground to stand on, we [...]

    Pingback by The Dovecot reports: how we came to forking a major email server « Fixpoint — 2023-04-06 @ 23:59

  10. [...] got it to pass its config test and start up after some adjustments. it does have this suspiciously dovecot-like 'modules' setup where it expects to dynamically load, as yet unclear if the modules will thus work at all or [...]

    Pingback by Grand reopening of #jwrd, the IRC channel « Fixpoint — 2023-05-13 @ 00:32

  11. [...] Covers 8, 10, 12 and 14 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/ [...]

    Pingback by JWRD Dovecot initial release, aka version 2.4.0 « Fixpoint — 2023-06-26 @ 04:40

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by MP-WP. Copyright Jacob Welsh.