MediaWiki deployment

Translatewiki.net uses oregano to deploy MediaWiki. Oregano is simple deployment tool. Oregano is documented in a blog post.

There is a script called twn-update-all that updates MediaWiki core, extensions and composer dependencies and updates the localisation cache. It does not run schema changes (update.php). We run the master branch of MediaWiki and extensions. For this reason the deployer should be aware of the latest MediaWiki changes and whether they require configuration or schema changes. The script automatically uses sudo -u betawiki and will fail if you don't have access.

The oregano commands must be run with sudo -u betawiki. oregano itself is in path and it must be run in the /srv/mediawiki directory. It is recommended to add alias b for the sudo command. After deploying a new version, please monitor messages from Rakkaus in #translatewiki on Libera.Chat IRC and be ready to rollback in case of issues. If you notice new PHP notices or warnings, those should be investigated and reported to Phabricator where applicable. You can also inspect the log file in /www/translatewiki.net/logs/error_php directly.

Deployment window

Regular deployment window checklist:

• Check error logs before production update.
• Prepare deployment plan.
• (If root) update packages: apt update, apt upgrade.
• Update production environment but do not deploy (do not create and release a tag). See deployment example below.
• Test on canary while checking error logs.
• If everything is OK, deploy and monitor error logs.
• Check #affects-translatewiki.net to see if issues are resolved (e.g. no more seen in logs), or need updating

Report errors as issues in Phabricator with #affects-translatewiki.net tag.

Full deployment example

cd /srv/mediawiki
# Update
b twn-update-all
b twn-update-database
# Test on canary
# Deploy and release
b oregano tag
b oregano deploy

How to see the difference against deployed code

If you want to see how the working directory differs from the currently deployed code, you can run

cd /srv/mediawiki
diff -urx .git targets/production workdir

You can use this before deployment to confirm that the working directory is clean, on manual deployment (e.g. single extension) to confirm only expected changes are deployed, or on full deployment to inspect what has changed.

How to test code changes before deploying

It is possible to test code changes before deploying them to all users with oregano.

You can enable the gadget "Add button to toggle canary mode (see MediaWiki deployment)." to get a link to toggle canary mode added to your user tool menu.

You know that canary mode is enabled if you see orange background stripes and a "You are using a canary" message near the page title. This varies a bit depending on the skin and which page you are on, so for example on Timeless the colored stripes are only visible at the very bottom, and the message is not visible on our main page.

You can now test your changes. Once you are finished, do remember to disable canary mode.

This method is not suitable for testing configuration changes, such as adding new message groups. These changes still apply immediately, and can cause "split brain" type of issues that may degrade performance or pollute caches.

JobQueue processor is not automatically restarted until the actual deployment. If you need to check something with the JobQueue, you need to restart it manually using sudo systemctl restart mw-jobrunner.

How to handle resolve conflicts during update

We occasionally cherry-pick patches or do local patches for various reasons. Although we try to keep those at minimum, updating the code may fail due to a merge conflict caused by some of these patches. You will notice this in the output of the command and often also L10N cache rebuild will fail because of fatal errors due to those conflicts.

Here is how to get rid of a conflicting patch (i.e. because an updated version was merged to master):

cd /srv/mediawiki/workdir/extensions/SomeExtension
b git rebase --abort
b git rebase -i # Drop the affected patches

Other times the patches might still be needed, in which case you need to solve the merge conflicts :(

How to rollback a deployment

If the deployment causes issues, such as bringing the site down, or some feature not working, or lot of log spam, it is recommended that you revert the deployment. This command reverts the site to the previously deployed version:

b oregano rollback

You cannot roll back group configuration changes! See the section about group configuration changes for more information.

After the immediate issues are averted, it is time to investigate what went wrong, fix it, and think if this can be prevented in the future.

How to do a configuration change

Submit a patch to the translatewiki repository in Gerrit. Get the patch merged. Self-merges are not forbidden, but discouraged.

When you merge configuration changes, do also deploy them in timely manner. Otherwise, the next person who deploys will push them out. If those changes cause trouble, you surely won't get a present from Santa Claus ;)

Do note that twn-update-all also updates config.

How to check MediaWiki version

Some configuration changes (such as removing language configuration for languages that have been exported to core for the first time) need a specific commit from MediaWiki core to have be deployed on Translatewiki.net before it's safe to deploy those configuration changes. To check which MediaWiki version is deployed, you can use:

version

Changes to MediaWiki configuration

This section applies to all files in mw-config directory.

Follow the instructions for a full deployment, but use twn-update-config instead as follows:

cd /srv/mediawiki
# Update
twn-update-config
# Test on canary
# Deploy and release
b oregano tag
b oregano deploy

Configuration files

• PrivateSettings.php and DatabaseSettings.php: Keys, passwords and other things which should not public. This file is only editable locally on the server.
• LanguageSettings.php and FallbackSettings.php: Settings related to configuration of available translation languages
• PermissionSettings.php: Most $wgGroupPermission things go here
• TranslateSettings.php: Settings related to the Translate extensions
• TranslatewikiSettings.php: Everything else

Changes to group configuration

The updated group configuration will be applied immediately! If the configuration depends on code change, be extremely sure that that code is first deployed to production with oregano or you will bring the site down.

How to handle issues related to group configuration updates

1. Go to cd /home/betawiki/config and edit the broken configuration with text editor (b nano ...php, or b git revert HEAD.
2. Commit the working code locally with b git commit -av
3. Submit a fix to Gerrit and get it merged
4. Run cd /home/betawiki/config, b git fetch and b git rebase -i origin/master – remove your fixup commit to avoid merge conflicts. If you still get conflicts, b git rebase --abort should help to get back to the previous state.

Updating only specific extensions

If you need to update one extension, for example UniversalLanguageSelector:

cd /srv/mediawiki/workdir/extensions/UniversalLanguageSelector
b git pull --rebase
cd /srv/mediawiki/workdir
b composer -n update
php maintenance/rebuildLocalisationCache.php --threads=4
cd /srv/mediawiki
b oregano tag
b oregano deploy

For the common case of updating Translate, UniversalLanguageSelector, TwnMainPage and cldr, there is a shortcut twn-update-maintained that does all of the above for these extensions and prints a list of changes in each extension.

Remember to run twn-update-database manually if needed.

Handling logspam and other issues

During log triage or canary testing you may find things that are broken. Those issues should be reported to Phabricator. You can check the #affects-translatewiki.net tag if the issue is already known. If the issue is there, perform a general search in Phabricator to see if the issue has been reported. If it is, add the #affects-translatewiki.net to it. If it isn't, file a new issue with as many details as possible, but not exposing private information like IP addresses. Add the #affects-translatewiki.net tag and other relevant tags to it.

Discuss with the co-deployers whether the issue is severe enough that it needs a workaround or even halting the deployment.