Mastodon/Debugging
Up to Mastodon/Tech WG
Rails
To see the most recent rails logs:
sudo journalctl -u mastodon-web -e
Sidekiq
Do the same thing, substituting mastodon-web for any of the relevant sidekiq services:
| Service Name | Queues | Threads |
|---|---|---|
mastodon-sidekiq-default |
default, ingress, pull, push | 25 |
mastodon-sidekiq-ingress |
ingress, default, push, pull | 25 |
mastodon-sidekiq-push |
push, pull, default, ingress | 25 |
mastodon-sidekiq-pull |
pull, push, default, ingress | 25 |
mastodon-sidekiq-mailers |
mailers | 5 |
mastodon-sidekiq-scheduler |
scheduler | 5 |
See Also
Cooldown
so re: Stoplight and Cooldowns, Sidekiq#Cooldown tries to deliver something 16 times ( https://github.com/NeuromatchAcademy/mastodon/blob/eb24c0ad07c4137517e6bd37ebcc99d6e2b86797/app/workers/activitypub/delivery_worker.rb#L11 ) the delay rises exponentially (^4) with each retry. So eg by the 10th retry we're delaying an average of 208 minutes, and by 16 we're at 1365 (22 hours).
That delay uses sidekiq's `sidekiq_retry_in` method, which applies to each delivery task (ie. each status we're trying to push), but there is also an additional control flow tool Stoplight ( https://blog.bolshakov.dev/stoplight/ ) that applies per inbox URL (rather than per job). You start in a good (green) state. Each failure counts towards a threshold (10), after which it halts all jobs matching that inbox (red). After the cooldown period (60 seconds) it flips into a "yellow" state: if the next job succeeds, it flips back to green. If it fails, it goes immediately back to red.
That configuration seems sorta... pointless to me? if it only kicks in after 10 failures, then it'll only be halting after really long delays, right? it seems like that should be a high threshold with like a really long cooldown to me - if we haven't been able to deliver like 200 messages, then cooldown for like 6 hours (i'm not sure if receiving a message clears the stoplight)