Upgrade Mastodon

Up to: Tech WG

Stages

Steps to be followed for merging glitch + upstream stable release + instance's stable custom features

• Sync the glitch-clean-sync branch to glitch-soc/mastodon:main: Open the branch page and click "sync fork"
  • The purpose of this branch is solely to serve as a replica of the parent glitch-soc/mastodon:main branch and so glitch-clean-sync is a locked read-only branch by design to protect it from force-pushes or merges from other branches.

• Create a new branch called custom-glitch(insert version number) from the glitch-clean-sync branch. For example, if we are upgrading to v4.2, the branch will be custom-glitch4.2

git checkout origin/glitch-clean-sync
git checkout -b custom-glitch4.2

• Merge dev-stable to the custom glitch branch

git merge origin/dev-stable

• All the stable features stay on the dev-stable branch. To ensure that this branch stays clean, branch protections have been enabled.
• Any features that have passed all the tests, have no errors (including linting errors), received approval from at least 1 reviewer, and are ready to be merged should be merged from the feature branch to the dev-stable branch in a separate process. See Mastodon/Hacking for important guidelines for contributing custom features to dev-stable.
  • Briefly: dev-stable branch should ideally have no merge conflicts if all the steps to keep the branch clean and stable have been followed.
    • If there are merge conflicts, try and identify which features caused merge conflicts. Then request the feature developer to :
      • merge upstream (glitch-soc:main) to conflicting feature branches
      • create a PR to merge conflicting feature branches with dev-stable
      • ensure that all checks pass, there are no linting errors, and review the code to ensure that it does what the feature is intended to do before merging to dev-stable

History: Steps that were followed for merging glitch + 4.2RC + our instance's features

all branches refer to those in NeuromatchAcademy/mastodon unless specified otherwise.

• Sync the glitch-soc-main branch to glitch-soc/mastodon:main: Open the branch page and click "sync fork"
  • If there are any local changes that make a conflict, they can be discarded - this branch is supposed to just be a local copy of the upstream main branch that doesn't reflect our local changes, we will handle that in the next step.

• Create a pull request from glitch-soc-main to merge-upstream
  • Resolve conflicts, ensuring that changes we have made locally are preserved when possible.
  • If it is not possible to preserve a hack, eg. because the relevant feature has been deprecated, update the corresponding feature page and include that in the pull request description
  • Don't try and fix failing tests here - that would edit the glitch-soc-main branch, which will then be overwritten on the next fork sync. NOTE from Manisha -- The glitch-soc-main branch had commits from merge-upstream even after syncing the the branch. So I created a new branch called glitch-clean-sync from glitch-soc's main branch. Let's keep this branch clean and not merge anything to this branch. Only use the sync feature. I also tried to follow the rest of the process but it seemed a bit complicated/unclear. I've tried to simplify it in a separate section above for glitch+4.2+our instance's features
• Merge dev into merge-upstream if necessary
  • It seems like this is the way glitch-soc does it, merge all from upstream masto into a branch from dev, then merge dev into the branch to ensure all glitch modifications go on top of the upstream? not exactly sure -jonny
• Fix any failing tests in the merge-upstream branch
  • Ruby Tests, One and Two step db migrations must pass, but linting failing is fine and normal
• If any changes were made after merging dev, Create a pull request from merge-upstream to dev. Otherwise just merge merge-upstream to dev
  • We do this additional step before merging to main because we might have collected some hacks or additional code in dev from some other branches before deploying (and generally we want to play the upstream over dev)
• Merge dev to main

Deploying

• Make a backup on the linode dashboard
• Check for any release-specific instructions and follow those! If you are doing an upgrade through several versions, make sure you follow those too!
  • Base masto: https://github.com/mastodon/mastodon/releases
  • Glitch docs (mostly just the same as base masto): https://glitch-soc.github.io/docs/

Otherwise, in general...

Switch to the mastodon user and change to the masto directory

sudo su mastodon 
cd ~/live

Fetch changes

git pull

Install dependencies

bundle install
yarn install --frozen-lockfile

Run pre-deployment database migrations

RAILS_ENV=production SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rails db:migrate

Precompile assets

RAILS_ENV=production bundle exec rails assets:precompile

Restart services (you may have to switch back to your original user with exit because the mastodon user cannot sudo for obvious reasons)

sudo systemctl reload mastodon-web
sudo systemctl restart mastodon*

Run post-deployment database migrations

RAILS_ENV=production bundle exec rails db:migrate

Problems

(and solutions)

Precompilation Failed

You might see something like this:

mastodon@localhost:~/live$ RAILS_ENV=production bundle exec rails assets:precompile
DEPRECATION WARNING: Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2.

Check the Rails upgrade guide at https://guides.rubyonrails.org/upgrading_ruby_on_rails.html#new-activesupport-cache-serialization-format
for more information on how to upgrade.
 (called from <main> at /home/mastodon/live/config/environment.rb:7)
Compiling...
Compilation failed:
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/home/mastodon/live/node_modules/babel-plugin-lodash/node_modules/@babel/types/lib/validators/generated/index.js:2740:35)
    at PluginPass.Program (/home/mastodon/live/node_modules/babel-plugin-lodash/lib/index.js:102:44)

and assume that the compilation fails because of the deprecation error that follows immediately after "compilation failed:" but it is not so.

This is an OOM error, see: https://github.com/glitch-soc/mastodon/issues/2423

To remedy:

1) Switch back to your user (from the mastodon user) that has root permissions and stop the ElasticSearch process. Return to the mastodon user.

# previously...
# sudo su mastodon

exit # assuming you're the mastodon user
sudo systemctl stop elasticsearch

# return to mastodon user
sudo su mastodon

2) Since disrupting the asset precompilation may leave you in a state where webpack thinks it has successfully compiled, you need to clobber the previous assets

cd ~/live
RAILS_ENV=production bundle exec rails assets:clobber

3) Proceed with precompilation...

RAILS_ENV=production bundle exec rails assets:precompile

4) Remember to restart elasticsearch afterwards!

exit
sudo systemctl start elasticsearch

Discord

sneakers-the-rat#techwg-ops23-05-20 05:46:32

Tech WG/Ops Diary#23-05-19: We are attempting to Upgrade Mastodon to 4.1.2 and also merge in Exclusive Lists, wish us luck.

We begin with what is probably the worst way to do it based on the conversation in <#1049184335514828860> , specifically:

• main -> dev,
• upstream -> dev,
• dev -> pr,
• pr -> dev,
• dev -> main

https://discord.com/channels/1049136631065628772/1049184335514828860/1109355378178789396

mannazsci#techwg-ops23-10-01 07:44:40

Tech WG/Ops Diary#23-10-01 I am ready to deploy v4.2 but I encountered a few issues while following the process mentioned in Upgrade_Mastodon. The glitch-soc-main branch had commits from merge-upstream even after syncing the the branch. So I created a new branch called glitch-clean-sync from glitch-soc's main branch and added protections to this branch. This let's us keep this branch clean and not merge any custom features to this branch. We'd only use the sync feature. I also tried to follow the rest of the process but it seemed a bit complicated/unclear. I've tried to simplify it in a separate section above for glitch+4.2+our instance's features. I've also created a clean and stable dev-stable branch for custom features that we add. I've added branch protections here as well and PRs will require reviews on this branch.