![]() | Before we begin If you came here following a link promising "A Plan For Global Templates", you should probably read the introduction explaining why "global templates" is a hard term to define. Instead this will specifically be a plan for Scribunto modules. We already have four (or five) different template mechanisms. With the principle of "remove one to add one", the path to a simple and maintainable platform is to create strong systems that can fully replace prior systems and then retire the prior systems. Instead of multiplying the work by trying to fix all of the existing mechanisms, this proposal specifically targets the most powerful of the template mechanisms, Scribunto, and my claim is that templates using any of the other mechanisms should be rewritten to use Scribunto if they are intended to be "global". We'll return to this point at the end of this proposal, but from this point forward we are going to focus on Scribunto. |
This proposal is based on the Wikimania 2025 presentation "Imagining a Future for Scribunto": video, slides.
Scribunto is the primary template system used in Wikimedia projects since its introduction in 2013. Scribunto was intended to curb runaway template complexity and separate the content and presentation layer.[1] Although Scribunto was successful at improving template evaluation performance, its other goals are largely unrealized. Scribunto provides a secure sandbox for modules written in Lua 5.1, which has been end-of-life since February 2012.[2] Scribunto has limited localization support, which has spawned ad-hoc solutions such as Module:TNT and the Global templates proposal on MediaWiki. Further, Scribunto doesn't support modern software development processes such as code review and fork/merge version control.
Before we can build any new functionality into Scribunto, we need to address the fact that the Lua 5.1 sandbox is unsustainable. In addition to security concerns, the obsolete Lua version creates barriers to entry for editors since modern textbooks, university courses, or educational materials don't document this version of Lua. We need to begin to move to a supported version of Lua and create the infrastructure required to enable further upgrades on a regular schedule. Fortunately, the Parsoid project has created much of the needed infrastructure: it uses a canary server running "the next version" and tools ("round trip" and "visual diff" testing, in the case of Parsoid) to compare the rendering of tens of thousands of project pages between the "previous" and "next" versions of the software-under-test. Production is only transitioned to the "next" version once confidence is reached that the impact on our project content is acceptable.[3]
Although it is technically possible in many cases to rewrite Lua code such that it functions identically on both Lua 5.1 and a newer Lua release, more conservatively we can assume that there will be some pages for which explicit versioning is required: either separate versions of the module for different Lua versions will be needed or we'll need to explicitly mark that a module is/is not compatible with a specific Lua version.
Update the Scribunto extension to support Mutli-Content Revisions: support "script content" in multiple slots, and add a single new "script version index" slot which provides compatibility information for one or more "script content" slots. Thus we could have a module with a single "script content" slot which is marked as supporting all Lua versions in the "script version index" slot; a module with multiple "script content" slots each of which support a single Lua version as recorded in the "script version index" slot; or other variations.
Additional opportunities: move Scribunto documentation and tests from subpages to separate slots,[4] and add TemplateData information for Scribunto modules in its own slot (T56140, T54607).
Scribunto calls the component responsible for executing module code an "engine", and already supports multiple engines—just not simultaneously. Scribunto ships with support for running code both in an embedded sandbox (the luasandbox
PHP extension) and by shelling out to a command-line Lua binary.
In order to pragmatically migrate code from Lua 5.1 to a newer Lua version, we will need to be able to run multiple Lua engines simultaneously: an editor will want to be able to test with the newer Lua version in production while we still use Lua 5.1 for most renders. One difficulty here is that manner in which luasandbox
embeds Lua as a PHP extension does not support embedding multiple versions of Lua in the same PHP binary.[5]
On the other hand, it is possible to cross-compile newer versions of Lua to run in a Lua 5.1 engine, using a compiler written in Lua and run under Scribunto itself. The page User:Cscott/Ideas/Improved for-loops for Lua constitutes a live demo; although it doesn't loudly advertise the fact that its source language is Lua 5.4 as opposed to Lua 5.1, the first few lines of the source code document this fact. Similarly, it ought to be possible to cross-compile Lua 5.1 to execute under a newer Lua runtime. Cross-compilation may not be 100% accurate, but it may provide additional compatibility aids that can be combined with the canary approach described above to identify and correct edge cases where cross-compilation is not 100% faithful. This could limit the number of pages which need explicit porting.[6]
Target Lua 5.5 for "the next version of Lua", since as of this writing (2025-08-11) it is expected to be released imminently. For early development and testing, Lua 5.4 may be used as the target, but aligning with Lua 5.5 will prevent Scribunto from being immediately obsoleted and ensure that the latest documentation, learning materials, books, etc will be applicable to Scribunto on WMF projects.
Add data to the "script version information" slot which allows associating "compilers" written in Lua to code stored in certain slots, with defaults for certain versions specified by site configuration.
Initially deploy Scribunto to production with a 5.1 luasandbox
and a 5.5-to-5.1 cross-compiler attached to "Lua 5.5" modules. This allows editors to test "5.5-compatible" versions of their modules in production. Any modules with no compatibility explicitly marked will run in "5.1 default mode", although an extension like Extension:ParserMigration can be used to allow individual editor opt in to "5.5 default mode".
Then deploy a canary Scribunto with a 5.5 luasandbox
, a 5.1-to-5.5 cross-compiler for compatibility, and with modules without explicit compatibility markers running in "5.5" mode. This allows us to run Parsoid-style tests on live content between production and the canary, highlighting any pages/modules which display differently when running under the canary configuration. As with the Parsoid Read Views Confidence Framework, we can continue improving the cross-compiler and porting modules until we have confidence that we can flip the switch to run the "canary" configuration in production.
Additional opportunities: the cross-compiler mechanism can be used to run modules written in JavaScript (T61101, demo at User:Cscott/LuaTurtle) or multi-lingual Lua (Wikimania 2019 lightning talk, video) using a Lua runtime engine. In theory we could even switch to a high-performance JavaScript or WebAssembly runtime engine in the future and cross-compile Lua to run on that engine.
Although separation of code and presentation was part of the original design goals for Scribunto, current Scribunto modules still contain a large amount of wikitext and other presentation layer strings and markup embedded into the module. This impairs localization, since localizable strings are embedded inside code, and also raises a barrier to entry since editors who may wish only to adjust presentation details are faced with the task of understanding the Scribunto logic layer in order to do so.
It is not strictly necessary to address this deficiency in order to make the Lua sandbox sustainable or provide "global templates", but because many templates will likely be at least partially rewritten in order to port them to Lua 5.5 or (as future steps provide) improve localization or development process, I recommend addressing the structural issues with the Scribunto API at this step so that rewrites can use the cleaner mechanics.
Expose Scribunto input and output arguments as the Parsoid PFragment type, which allows structured data to be losslessly passed between Scribunto modules and other structured-data producers, such as Wikifunctions. Additional parser functions are added as structured data constructors, for example {{#kv:name1=value1|name2=value2}}
creates a structured list of key/value pairs.[7] In connection with special semantics for the argument named ...
,[7] this simplifies template argument handling[8] and provides a base for the localization of named arguments presented in the next step.
Add a "wikitext" MCR slot to Scribunto modules. For modules which opt-in to the new mechanism, the output of a Scribunto module is a structured data object instead of a wikitext string, and the contents of the "wikitext" slot are parsed using the data object as the "data context"; the result is the module output.[9] This achieves the code and presentation separation envisioned in the original Scribunto design, and again lays the groundwork for localization of the presentation layer in the next step.
Additional opportunities: VisualEditor support for the wikitext slot to allow "no Lua or wikitext knowledge needed" editing of the presentation layer of Scribunto modules. Migrate existing complex wikitext templates to the "wikitext slot" of a "no code" Scribunto module to utilize the localization and development process improvements of later steps in this proposal, and to eventually allow deprecation and removal of some of the legacy template systems. Support TemplateStyles for the module wikitext, further unifying template systems. Support an easy mechanism to use a SPARQL query as the "data context" to facilitate Wikidata-powered templates.
There are four aspects of Scribunto modules which require localization: the module name, the module Lua code (embedded function and variable names, comments, etc), the named arguments to the {{#invoke}}
of the Scribunto module, and any embedded wikitext or other content in the code or presentation layer. There have been a number of prior proposals to address template localization, including Module:TNT and the Global templates proposal on MediaWiki.
We will handle localization of the module name in the next step. Localization of the Lua code itself can be addressed with the cross-compiler mechanism from step 2, as demonstrated in the French example from this live demo.[10]
MediaWiki supports ~400 languages, so it is not practical to allocate a slot per language[11]. Instead a single slot is used to store localizations for all messages used by a Scribunto module, in a standard message bundle format.
Scribunto's default argument passing mechanism from {{#invoke}}
[12] is updated to support localization of named arguments, using the parser target language[13] to select the appropriate argument names from the message bundle.[14]
A Scribunto method is added to localize a message from the bundle for use in code, and a corresponding parser function is added which can be used in the "wikitext" slot for modules with proper code/presentation separation.
Scribunto does not support modern software development processes. The simple "file-based" version control system of MediaWiki's article space was superceded for software when CVS replaced RCS in the 1990s. Module code written on-wiki lacks fork/merge support, code review, and continuous integration testing. Further, certain common templates and modules are replicated on hundreds of different wiki projects without any built-in mechanism to keep them in sync.
Attempts to address this include proposals to create a specific repository for global templates[15] and tools like Synchronizer. The Synchronizer tool in particular already provides a solid basis for synchronization of templates and modules across wikis, embodying a workflow favored by editors which gives explicit manual control of the synchronization process. This prevents an update to one copy from invisibly breaking uses on hundreds of other wikis; instead local admins are empowered to control the distribution of updates on their own wikis. Wikidata is used to map template names between wikis, in the same manner used by interlanguage links.
Currently Synchronizer requires specification of a particular wiki as the "source" wiki for a sync operation. It should be extended to allow specification of a git repository[16] and branch as the "source" for a sync, and convert individual files in git into the various module slots (including the localization message bundle). This allows use to use the existing translatewiki integration with repositories[17] to localize synchronized templates as well, with translation updates appearing as git commits pushed to the repository.
Synchronizer should be extended to function in the complementary direction as well, taking a wiki with local changes and turning the local changes into a git patch against a specific branch in the source repository. In order for this to work, a sync from git should store the "last sync point" (commit hash) in an MCR slot, so that the gerrit patch or gitlab pull request uses the correct parent commit as a basis. For MediaWiki users without a corresponding gerrit/gitlab/github ID, the patches will be uploaded anonymously as done with our github pull request integration.
When this is complete module developers can opt-in to a full modern development process, as used for many other foundation projects, including translatewiki integration for localization. Local admins are still in control of applying upstream changes to their local wikis, still able to make local changes (and create branches in the code repository for said changes), and the process of upstreaming their local changes is facilitated to help minimize drift.
Additional opportunities: the WMF Continuous Integration pipeline (gerrit/jenkins/zuul or gitlab) can be integrated with the "tests" slot of a Scribunto module, so that Scribunto tests are run against any commit before and after merge. Synchronizer may also be extended to handle dependent groups of templates/modules instead of synchronizing template/modules one-by-one. (In particular, having a bundle of common templates that can get synced at once is a common request from Incubator users.) Synchronizer may also benefit from the addition of a "language-centric" alternate UX, where multiple templates are shown for a single language instead of multiple languages shown for a single template. This alternate UX would be more useful to a local wiki admin, who is concerned with synchronizing their local wiki and may not feel comfortable applying code changes to wikis other than their own.
To reduce barriers to entry for new editors, the Scribunto API should be gradually unified with the other platform APIs in JavaScript and PHP, so that method names and class structures are substantially the same across platform. This may entail the creation of a formal IDL mapping spec (such as the PHP mapping for WebIDL in IDLeDOM) so that method, class, and argument names are semi-automatically kept in sync across platforms and languages.
Scribunto has an HTML-creation library; it should probably be replaced by a DOM and SVG library to facilitate the creation of structured content.
This proposal outlines a plan for Scribunto which addresses its current maintainability crisis, more fully fulfills its original vision of code/presentation separation, allows full localization of templates, and modernizes the code development process used in creating and maintaining Scribunto modules.
Scribunto would become the second major feature using the Multi-Content Revision system, helping actualize the potential of MCR and pave the way for further feature use. Slots would be added for code content, a script version index with provisions for cross-compilation, a message bundle, a wikitext presentation layer, a git commit hash for the last synchronization point, and, as additional optional features, documentation, tests, and template styles. Existing modules would be migrated to a supported, maintainable, and editor-friendly Lua version using the same proven and repeatable process as used for the migration to Parsoid. Localization and development processes would similarly build on the proven processes used by translatewiki and gerrit/gitlab/github, respectively.
In addition, the necessary infrastructure would be present for more radical updates in the future: expanding the scope of VisualEditor to support many template use cases; supporting JavaScript as either a Scribunto module language, a Scribunto engine runtime, or both; fully localizing Lua at the comment and API level for a true multilingual code environment; further integration with Continuous Integration and other modern code development tools; and turning existing "no code" templates into Scribunto modules to apply the localization and other benefits of this proposal and eventually reduce the number of distinct template mechanisms in core.
Using the taxonomy defined by Some thoughts on Global Templates, so called, let's evaluate this proposal. As described in "before you begin" at the head of this page, this proposal focuses on templates defined by Scribunto, and adds a mechanism to port "regular wikitext" transclusions and "parser functions" transclusions to Scribunto in step 3. This covers a large number of the template use cases. This proposal doesn't directly address style consistency, page layout, semantic annotations, or work flow, although synchronized Scribunto modules could be used for these, as they currently are.[18] From the translator perspective, the localization mappings of module name (step 5) and module arguments (step 4) facilitate translating content which uses modules from one wiki to another.
Moving on to "global", there are three aspects defined in the taxonomy:
The proposal does not require a standalone "template" wiki,[15] although a standalone wiki could be created, for example to facilitate community-building. Although this would likely move primary development of modules to the "template" wiki, we expect that distribution of modules to other wikis would still be done via git to ensure control by local admins of the update timing. We don't recommend building specific code-development tools into a standalone wiki; those already exist in the various open source code forges and we should reuse that work rather than reinvent it.
The taxonomy also described "types". We expect input and output types for Scribunto modules to be defined by TemplateData (T395968, T55413) which is the third "type" defined in the taxonomy, although at a technical level providing for PFragment input arguments and output (step 3) facilitates passing structured data. In particular, it is expected that the Wikifunctions ZObject will eventually become a PFragment subtype, which will provide access to other types in the taxonomy. Other types and semantic information (for example a Wikidata item selector, or Wikidata-based constraints or annotations on values) can be incorporated into TemplateData, and are not directly part of the module/template system.
Finally, the "theory of change" embodied by this proposal is moderate/incrementalist. Each step in this proposal is intended as a modest change to an existing product (Scribunto) which doesn't immediately break existing uses. However, community participation is required to adopt the new framework. In the first steps, we need community support to transition modules from Lua 5.1 to Lua 5.5, and we expect a mix of collaboration with community (especially around issues which can be identified by the Linter extension) and a modest amount of foundation resources devoted to porting key modules. This migration/linter model has been previously used with success in the HTML5 Tidy effort and the Parsoid Read Views project.
Further progress toward module localization requires community participation to port existing modules to the new framework and to establish git repositories for key translatable modules. Unfortunately the needs of minority-language Wikipedias, which are most benefited by translatable and reusable modules, are not often visible to editors on English and German Wikipedia, where much template/module development occurs. We have made an effort to make porting modules to the new framework low-friction, but one can expect evangelism and foundation resources (focused edit-a-thons, for example) will be needed to shift the window such that localizable modules are considered standard, not the exception. Once established and accepted by the community, we can apply the migration/linter model to fully convert existing modules and ultimately deprecate other templating mechanisms. Ultimately all newly-created modules will come pre-equipped with translation message bundles, a wikitext slot, and documentation marked for translation, and requesting a gerrit or gitlab repository for the module is a matter of a few clicks when the module is first desired for use on a different wiki.
luasandbox
and a "slow version" by shelling out to a command-line binary. Interoperability between Lua versions (for example calling a Lua 5.1 module from a Lua 5.5 module) would be more difficult in this scenario, but execution fidelity would be improved. This may be a good option if we believe we will be able to convert all modules to the new version of Lua before flipping the switch to use it in production and thus that we will never have to support more than a single version in production, since SRE may be reluctant to run the shellbox Lua in production but willing to run it on an internal canary server.{{#media}}
for style consistency (T318433#10397735) and a Page Description slot for layout (T112991). We propose to add semantic information for modules to TemplateData (T395968, T55413).