MediaWiki deployment

From translatewiki.net
Jump to navigation Jump to search

Translatewiki.net uses oregano to deploy MediaWiki. Oregano is simple deployment tool. Oregano is documented in a blog post.

There is a script called twn-update-all that updates MediaWiki core, extensions and composer dependencies and updates the localisation cache. It does not run schema changes (update.php). We run the master branch of MediaWiki and extensions. For this reason the deployer should be aware of the latest MediaWiki changes and whether they require configuration or schema changes. The script automatically uses sudo -u betawiki and will fail if you don't have access.

The oregano commands must be run with sudo -u betawiki. oregano itself is in path and it must be run in the /srv/mediawiki directory. It is recommended to add alias b for the sudo command. After deploying a new version, please monitor messages from Rakkaus in #mediawiki-i18n and be ready to rollback in case of issues. If you notice new PHP notices or warnings, those should be investigated and reported to Phabricator where applicable. You can also inspect the log file in /www/translatewiki.net/logs/error_php directly.

Deployment window

I (Nike) usually perform the following tasks during the regular deployment window:

  • Check error logs before production update.
  • Update packages: apt update, apt upgrade.
  • Update production environment but do not deploy (do not create and release a tag). See deployment example below.
  • Test on canary while checking error logs.
  • If everything is OK, deploy and monitor error logs.
  • Check #affects-translatewiki.net to see if issues are resolved (e.g. no more seen in logs), or need updating

Report errors as issues in Phabricator with #affects-translatewiki.net tag.

Full deployment example

cd /srv/mediawiki
# Update
twn-update-all
twn-update-database
# Test on canary
# Deploy and release
b oregano tag
b oregano deploy

How to test code changes before deploying

It is possible to test code changes before deploying them to all users with oregano. For example in Chrome it is done like this

  1. Go to translatewiki.net
  2. Open developer tools
  3. Open tab Application
  4. On the right column, under heading Storage, find item Cookies and expand it
  5. Open item https://translatewiki.net from the expanded list
  6. Find an empty row, right click on it, and choose Add new
  7. Write env as name, x as value, you can leave the rest as defaults
  8. Reload the page

You know that you did this correctly if you see orange background stripes and a "You are using a canary" message near the page title. This varies a bit depending on the skin and which page you are on, so for example on Timeless the colored stripes are only visible at the very bottom, and the message is not visible on our main page.

You can now test your changes. Once you are finished, do remember to delete the cookie.

This method is not suitable for testing configuration changes, such as adding new message groups. These changes still apply immediately, and can cause "split brain" type of issues that may degrade performance or pollute caches.
JobQueue processor is not automatically restarted until the actual deployment. If you need to check something with the JobQueue, you need to restart it manually using sudo systemctl restart mw-jobrunner.

How to handle resolve conflicts during update

We occasionally cherry-pick patches or do local patches for various reasons. Although we try to keep those at minimum, updating the code may fail due to a merge conflict caused by some of these patches. You will notice this in the output of the command and often also L10N cache rebuild will fail because of fatal errors due to those conflicts.

Here is how to get rid of a conflicting patch (i.e. because an updated version was merged to master):

cd /srv/mediawiki/workdir/extensions/SomeExtension
b git rebase --abort
b git rebase -i # Drop the affected patches

Other times the patches might still be needed, in which case you need to solve the merge conflicts :(

How to rollback a deployment

If the deployment causes issues, such as bringing the site down, or some feature not working, or lot of log spam, it is recommended that you revert the deployment. This command reverts the site to the previously deployed version:

b oregano rollback
You cannot roll back configuration changes! See the section about configuration changes for more information.

After the immediate issues are averted, it is time to investigate what went wrong, fix it, and think if this can be prevented in the future.

How to do a configuration change

  1. Submit a patch to the translatewiki repository in Gerrit.
  2. Get the patch merged. Self-merges are not forbidden, but discouraged.
  3. On translatewiki.net shell, run twn-update-config. This will deploy the config change immediately.
  4. Test the change on the site and monitor the error log as above.

Optionally, you can also send a out a message to IRC to keep other admins in sync. It works like this udpcast Your message here.

The updated configuration will be applied immediately! If the configuration depends on code change (such as adding a new extensions), be extremely sure that that code is first deployed to production with oregano or you will bring the site down. Do note that twn-update-all also updates config.
When you merge configuration changes, do also deploy them in timely manner. Otherwise, the next person who updates configuration will pull your approved changes too. If those changes cause trouble, you surely won't get a present from Santa Claus

Configuration files

  • PrivateSettings.php and DatabaseSettings.php: Keys, passwords and other things which should not public. This file is only editable locally on the server.
  • LanguageSettings.php and FallbackSettings.php: Settings related to configuration of available translation languages
  • PermissionSettings.php: Most $wgGroupPermission things go here
  • TranslateSettings.php: Settings related to the Translate extensions
  • TranslatewikiSettings.php: Everything else

How to handle issues related to configuration updates

  1. Go to cd /home/betawiki/config and edit the broken configuration with text editor (b nano ...php, or b git revert HEAD.
  2. Commit the working code locally with b git commit -av
  3. Submit a fix to Gerrit and get it merged
  4. Run cd /home/betawiki/config, b git fetch and b git rebase -i origin/master – remove your fixup commit to avoid merge conflicts. If you still get conflicts, b git rebase --abort should help to get back to the previous state.

Updating a single extension

If you need to update one extension, for example UniversalLanguageSelector:

cd /srv/mediawiki/workdir/extensions/UniversalLanguageSelector # Make sure to go to the correct directory
b git pull --rebase
cd /srv/mediawiki/workdir
php maintenance/rebuildLocalisationCache.php --threads=4
cd /srv/mediawiki
b oregano tag
b oregano deploy