140
Lemmy.world status update 2023-07-05 - Lemmy.world
lemmy.worldAnother day, another update. More troubleshooting was done today. What did we
do: - Yesterday evening @phiresky@phiresky@lemmy.world
[https://lemmy.world/u/phiresky] did some SQL troubleshooting with some of the
lemmy.world admins. After that, phiresky submitted some PRs to github. -
@cetra3@lemmy.ml [https://lemmy.ml/u/cetra3] created a docker image containing
3PR’s: Disable retry queue [https://github.com/LemmyNet/lemmy/pull/3468], Get
follower Inbox Fix [https://github.com/LemmyNet/lemmy/pull/3482], Admin Index
Fix [https://github.com/LemmyNet/lemmy/pull/3483] - We started using this image,
and saw a big drop in CPU usage and disk load. - We saw thousands of errors per
minute in the nginx log for old clients trying to access the websockets (which
were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws. -
We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue
with replying to DMs - We found that the many 502-errors were caused by an issue
in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an
upstream to be dead. As a workaround we can either 1.) Only use 1 container or
2.) set proxy_next_upstream timeout; in nginx. Currently we’re running with 1
lemmy container, so the 502-errors are completely gone so far, and because of
the fixes in the Lemmy code everything seems to be running smooth. If needed we
could spin up a second lemmy container using the proxy_next_upstream timeout;
workaround but for now it seems to hold with 1. Thanks to @phiresky@lemmy.world
[https://lemmy.world/u/phiresky] , @cetra3@lemmy.ml [https://lemmy.ml/u/cetra3]
, @stanford@discuss.as200950.com [https://discuss.as200950.com/u/stanford],
@db0@lemmy.dbzer0.com [https://lemmy.dbzer0.com/u/db0] ,
@jelloeater85@lemmy.world [https://lemmy.world/u/jelloeater85] ,
@TragicNotCute@lemmy.world [https://lemmy.world/u/TragicNotCute] for their help!
And not to forget, thanks to @nutomic@lemmy.ml [https://lemmy.ml/u/nutomic] and
@dessalines@lemmy.ml [https://lemmy.ml/u/dessalines] for their continuing hard
work on Lemmy! And thank you all for your patience, we’ll keep working on it!
Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after
implementing the new Lemmy docker image with the PRs.
[https://lemmy.world/pictrs/image/166fc6d9-972d-4ff2-aa3a-b2ecbbb90cd5.png]
lemmy.world announcing to public that this is installed on their production server: https://lemmy.world/post/1061471
My instance server is now getting over 10 comments a minute delivered by lemmy.world, federation outbound is going much better.