We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn’t work.
I’ll create an issue for the devs and retry once it’s fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…
Thank you for the transparancy!
I love post like this, another reason of why this instance is so good
Thanks for all your hard work @ruud@lemmy.world. It’s not easy!
UI: 0.18.0 BE: 0.18.1-rc.1 👍
seems to be working for me, I was already logged on
deleted by creator
We have one! This was tested on a test instance before
Cool, great minds think alike 😁
Yeah we have a test instance, but not sure if we could test with this kind of load…
is that users can’t login anymore
Ouch, that’s bad :(
Thank you for your work, as always :)
I mean that’s a Tuesday on Reddit lmao.
Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?
Because of a few things, really anoying bugs and the Jerboa app not working properly with older versions
For example the frontpage is no longer being constantly filled with random posts when you set the filter to “active” or “top day”.
I suspect there will be a large influx of new users in two days, and that having Jerboa not working on .world might cause a few issues
Reasons I can think of:
-
The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.
-
Sorting is broken pre 0.18, new posts keeps flowing in.
-
Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)
Try wefwef.app on mobile
(no I’m not a bot account shilling I just really like the app)
If there was a way to adjust the font sizes it would be 100% perfect. Damn! I’m surprised I hadn’t heard of it before. Thanks internet stranger!
Jerboa works fine with lemmy world, it just gives a warning and crashes on occasion. Not an issue to use it though.
Well you’re a lucky one. Jerboa has been unusable for me and other folks post v0.34 since the first rc release of 0.18. I finally managed to downgrade to 0.34 and login after lots of black magic. It used to crash instantly, even on 0.34.
That being said, good luck to lemmy.world admins and thanks a lot for the open comms !
That sounds like Bethesda’s definition of “works fine”
With the last couple updates jerboa has got to a point that I probably won’t even move to sync when it comes. I’m pretty happy with it. But if Lemmy Is Fun ever happens, I’m out.
I’m of a similar mind: I’m happy with Jerboa, yet also looking forward to Slide.
Yes, if you are already logged in. If you are not logged in, or get logged out you will not be able to log back in until the server version is above 0.18
“crashes on occasion”
Unfortunately, the occasion seems to be at least once every five minutes for me. thisisfine.jpg This is not fine and definitely a major issue. Especially as we get closer to July 1. After this news from lemmy.world, I will have to switch, until Sync arrives. Update: Liftoff is fantastic, give it a try!
Have you tried this fork? https://lemmy.ca/post/899342
For me at least, I’m unable to change the community scope or sorting method. The only thing I can use Jerboa for is “local” and sorted by “active”. It isn’t totally unusable, but I would much rather be able to switch to subscribed and hot.
I had the same bug. A full restart of my phone seems to have fixed it.
Why the hell does Jerboa even limit support to 0.18? That’s so stupid.
Connect for Lemmy works better.
Is it open source?
I don’t know but I installed and older version of Jerboa from F-Droid store. I couldn’t login from the most recent version of Jerboa because of this so it was pretty much useless for me. Working fine so far.
Likely because Jebora is written as a side project of the devs of lemmy so it’s always going to be cutting edge
I understand that. I’m not complaining about the quality. I’m just confused from a technical perspective why it doesn’t support rollback or older server versions in the event of…this.
Fixing the issues with posts appearing and scrolling the page the backend needed some pretty drastic rewrites. I’m not super familiar, but from what I’ve seen of the code, it would be a decent amount more work to support both versions.
Would it have been worth it? Yes, but it wasn’t anticipated that devs would stay on 0.0.17 for more than a day or two. With the time it takes for app stores to update, servers would have been updated before lemmy 0.0.45 was updated for the vast majority of users. At most, it would be a day or two instead of a week or two.
-
Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?
A Release Candidate is supposed to be past beta testing already and in a state of no major bugs.
deleted by creator
UI at 0.18.0 and backend at .1rc1
Yep we tried bringing both the UI and the backend to 0.18.1-rc1 but that gave issues. Lemmy devs told us to try this combination and that seems to work. 🤞
Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting… its all bugged.
Its maybe not that big a deal because Lemmy.world “has the most users”, so in some regards its https://lemmy.ca’s loss but… we need to restore reliable federation… especially before the July 1st rush IMO.
The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.
Ah that explains it. Someone posted a cool photo to my community from lemmy.ca but didn’t interact further. Looks like my comment didn’t even show up on their end.
Anyway, thanks to everyone working on the issue. I know these things aren’t easy.
Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!
Dessalines:
Nutomic:
Oh god, so much coffee… it’s also thrilling to grasp a sense of what’s going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter’s recommendation algorithm being open-sourced)
Yeah they’ve been hard at work all month. But it’s also okay if things aren’t ready in time. Most of the people who matter are already here.
Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit
Agreed. I’m here to stay with the content from the last couple weeks. 3 day poop embargoes alone is worth it. And recognizing users because the community is still small enough is SO different from they who shall not be named.
As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.
We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.
Although since we seem to be rolling more than a ship in a storm, I think a proportion of lemmings would end up hospitalised for alcohol poisoning.
@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?
With all the performance issues with 0.18.1, it’s highlighted that there needs to be a way to stress test these updates before applying them to the main instance.
Yes, considering that. But we’ll need people to use that when we will do testing…
Well, you have at least one person who would be willing to use it ;D
One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.
Light the beacons! lemmy.world calls for aid!
I’m not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance’s server and swapping a “beta” server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy’s architecture allows for it.
Well that doesn’t differ much from what I do. I just copy the files to a second directory and test with that. Easy rollback. Downside is, that all data is lost between upgrade and rollback, which will be the same in the scenario you suggest.
Ah, that’s fair. Best of luck either way. This is the tough part of admin work. FWIW, the stability we’ve had thus far has been pretty impressive!
Jerboa crashes constantly for me now, doesn’t even let me view anything. JSYK!
You might want to try https://wefwef.app
There’s a fix the dev has just released. See reviews section on the app. He said it’ll take a few hours to show in the playstore.
This is good news! I haven’t been able to try Jerboa fully yet since I use lemmy.world. I’m not seeing this review section you’re talking about but I’ll be on the lookout for an update. Thanks for sharing!
Sorry, my comment wasn’t very clear. On the app page within the playstore, if you scroll to the bottom and read the reviews you’ll see the developer has replied to say a fix will be released today. I’m yet to see it though.
Seems to be working fine!
Seems 0.18 is up… and…
OH NO ! The interface changed ! Bring it back ! Bring it back… /s
Just kidding… It’s looking good, so it’s time to explore a little and find all the new stuff.
Nice work @Rudd, thanks for all the hard work.
Suddenly my Jerboa won’t change from local and I can’t change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.
What you’re describing is an issue introduced with Jerboa 0.0.36-alpha and has been fixed in today released 0.0.37-alpha.
My current session works now with 0.18.0 + 0.18.1.rc1
Oh. Good luck with the next try!