Astiga website seems to be broken
_BrightDarkness_
Member
in Bugs
I cannot access play.asti.ga really good. I get "Bad Gateway" a bunch of times when accessing the site.
Sync doesn't work and I cannot stream my music. I tried on different browsers and my phone, all the same.
Did someone spill coffee over the servers?
Comments
Looking at this right now. I've fixed the other issue with the cover art, I think, and I'm beginning to wonder if this is related because a health check, prior to the artwork fix, was breaking and causing restarts of the server. I'll keep monitoring this.
Thanks for reporting!
I'm still seeing periodic `Bad Gateway` messages. So I'm continuing to look into this...
I noticed that loading stuff in general is also more slow than usual. It's not only on the website either. Tempo is also slow to grab images and data, although it does so eventually and more reliable than the website. I would expect it to be somewhere in the underlying infrastructure perhaps? You have just been changing quite some stuff with a release a few days ago and multiple database maintenance's after all.
I also have seen another
MEDIA_ELEMENT_ERROR: Format erroralready again on the website, so the errors aren't entirely solved with the image fix either. Music does stream however, that error didn't block that.Yes, I agree. I've been checking the usual suspects - disk usage was quite high on one server which has been fixed, and I initially thought this was the cause, but the errors are recurring now... still looking.
We think the "Bad Gateway" problems are fixed now. It turned out to be a problem in the way our servers talk to each other... essentially how they keep a record of what servers are running and so on. Stale data was retained by one of the servers which meant it was forwarding requests, getting a bad response, and showing that as "Bad Gateway".
This is resolved for now, although requires a little more investigation later to prevent further occurrences.
I also think the sync is broken. I can't tell if it works in the background or not. There is no log and no feedback when syncing.
On play.asti.ga/open-log, it shows this message:
Another issue that still persists is speed. I have a folder on Dropbox "Playlist 1" and when accessing it from the phone app, it will take about 1-3 minutes to load the files while being on the page "Contacting server, please wait..." It was taking about 20 seconds before the changes.
I also get this error message on some albums under play.asti.ga/album
404
Oops! Album not found.
We could not find the album you were looking for.
I'm currently trying to forcibly re-sync my library to maybe resolve this. There are also a bunch of duplicate albums for me.
What works for me?
Hey. I might think the 404 Oops! Album not found. might be due to special characters in the album name.
One of my album names is "#5", and in the search bar "#" = "%23" for the ASCII Encoding Reference.
There might be a problem matching to the correct path of the album?
%20 for space resolves to a real space and shows the album, but not for #.
Even though the "," is shown as %2 in the search bar, it still shows the album contents correctly. Idk
Problem might be related to that?
Edit: Also check this https://imgur.com/a/ToXOupS
Thanks for this; unfortunately I think they are separate issues.
It looks like a sync has run through since you posted. A lot of files were added - over 31,000, but I noticed it was a force refresh. Can you confirm? Do you still see the error if you perform a manual sync?
Do you see the same performance problems on the Web app?
Yeah, the album thing looks like an encoding issue. We'll look into that.
Yes. I force refreshed. This was a manual sync. No log is being displayed by manually syncing. Message of the log on page play.asti.ga/open-log after the sync has finished: "The last synchronisation was on 09-11-2024 13:29:38; this is over six days ago and so we have purged the log."
Using the browser Firefox on Windows 11; performance seems to be faster and more responsive than the android app. Although, opening a folder with over 700 files still takes a little bit. (Approx 20 seconds on the Firefox browser, Win11)
Speed is still an issue. Loading times are still high. It takes Tempo much longer than usual to grab data, and in same cases it times out (or takes so long I've given up, not sure). For example when trying to load a playlist of about 200 songs. Edit: I eventually got it after 20+ minutes, but obviously that's not very user friendly. Oddly enough a 100 song playlist took 1 minute "only". This all used to be pretty instant, or at least quick enough not to bother.
While actually timing out instead of waiting may be on the app, the speed issue seems to be on Astiga. The official app is equally slow.
I haven't tested on PC since you fixed the gateway error, but I'd expect it to be there too. Are the
MEDIA_ELEMENT_ERROR: Format errorand the likes also connected to the gateway or more likely connected to whatever causes the long loading times?I think the "Format error" issues were down to the Bad Gateway - where the server wasn't providing the audio stream.
I think slowness is not related to that. I'll be looking at this on Monday morning too, I hope that's ok.
No problem with me. Stuff works for the most part, noticeably slower but as long as I don't try to load too much at once doable.
An update on progress on the above items...
Hash mark issue
A fix for this is going into testing now.
Sync logs
We're looking into this; we think it's a logic error rather than relating to performance.
Storage pages (on the Web)
@_BrightDarkness_ I've spotted a folder in the logs where it (currently) takes about 5s to download info about a folder with 3,000 files. I'm not observing anything as long as 20s.
Can you confirm this is still occurring on the Web version, and/or the app(s)? If so, could you please let us know the URL (Web) or page (if on the app) you are requesting via info@asti.ga ?
App performance loading playlists
Note this has been an issue for a while that we're seeking to address soon. However, this might not be the same issue.
@Cambionn I'm noticing a playlist that takes 30s to load from your account. I'm looking into this now.
General sluggishness
Can anyone give specific areas of the app or website where this happens?
We've observed a slow query on the top level page that lists recent albums etc - we can fix that.
We've noticed image load times are slow and some places where we were making additional queries where we didn't need to, so we're fixing them.
Other than that, it would be helpful to have specific pages that are slow (feel free to email us if you'd rather do it privately: info@asti.ga ).
@gravelld Thanks for the detailed update!
Playlists are not the only part of my issue btw, although it's the most noticeable as I don't have many other places that load 200 tracks.
For example, the home screen often doesn't load data, or does so slowly. Folder browsing similarly struggles to grab data, as well as just normal artist and album pages. It seems to be app-wide in general. At times I also get a message the server isn't reachable, with it being back again shortly after. I expect it's telling me that due to taking too long to load. It is however not always slow, right now it works as instantly as I'm used to. I will keep an eye on it, so I can give you hopefully some timestamps of when it slowed down so you can check the logs.
This all only started after the last database maintenance/release. Because the website had issues with the gateway and such, I can hardly tell if those are fixed when those where or not. The website seems fine now but I haven't used it much due to the weekend and me only working in the morning then going out.
Podcasts interestingly enough seem fairly unaffected in Tempo.
Can you just clarify if you're talking about the app or the Web site? You use terms like "home screen" which could apply to either... You say later "The website seems fine now" but I don't know if you mean wrt the Bad Gateway messages only.
"Bad Gateway" is definitely fixed now.
Home screen, as in the home screen of the Tempo app.
For example, the home screen often doesn't load data, or does so slowly. Folder browsing similarly struggles to grab data, as well as just normal artist and album pages. It seems to be app-wide in general. At times I also get a message the server isn't reachable, with it being back again shortly after. I expect it's telling me that due to taking too long to load. It is however not always slow, right now it works as instantly as I'm used to. I will keep an eye on it, so I can give you hopefully some timestamps of when it slowed down so you can check the logs.
This is all about the Tempo app. The official Astiga app has similar issues, so it seems to linger at least in anything outside of the web interface.
Because the website had issues with the gateway and such, I can hardly tell if those are fixed when those where or not. The website seems fine now but I haven't used it much due to the weekend and me only working in the morning then going out.
Only this is about the website (play.asti.ga). I know the gateway issues are fixed. I however do not know which issues that I had on the website came from that and which may still be present as I haven't been using the website that much yet. So I simply cannot tell if there are still issues there. I will have to get back to that after using it some more. But so far, with the limited use done so far, it seems it's working fine.
Thanks @Cambionn . I was experimenting with that playlist that takes 30s to load... I can't actually see anything that has changed in the latest release that would affect it. Have those playlists always loaded quicker prior to this release? It's about 200 tracks.
It definitely was faster, yes. As said, the other day I was waiting for 20 minutes for that playlist. If you only see 30 seconds, it may have been simply another case than the one you're experimenting with. But it may also mean something went wrong earlier on before it logs on your side.
Since Tempo hasn't had any updates, and the issue is also in the official app, I would expect the issue is on your side.
Although I must admit, after that one time waiting 20 minutes I haven't had it that extreme anymore, while it happened more often before that. That was on Saturday. Since it's still a bit slower than usual, but nothing too bad. I would say in the seconds indeed, not minutes. So it may already be partly solved, but time will have to tell. But why it happened? No clue.
I'll try with Tempo today, see what I observe.
The twenty minute case may be the Bad Gateway issue also. The app retries failed requests, and it may have simply been retrying over and over.
There are clear inefficiencies in the way this code works - fixing it will take some time, but it was ever thus (hence my link to the other thread above, for extremely large playlists).
I looked at Tempo today... I thought it a bit sluggish although I didn't think it was that bad...
Worst thing I noticed was the track list taking a while to download for an album. This was taking about 5s. It's made worse by the fact that Tempo appears to make the call twice (maybe it's timing out and trying again).
The main underlying cause of the slowness there is the same as I observed with the playlists - we are very inefficient with the way we process data before passing it back to the client. I'm working on some ways to improve this. Again, this hasn't changed, code-wise, for this release. But it's possible some DB level change has made it slower (although there's been nothing directly related).
Sounds plausible to me. I haven't seen the extremes anymore, just a noticeable few seconds. If I see those extremes again I'll let you know. Otherwise I'll wait for the optimisation changes.
We made another discovery. Our infrastructure allows for geographic transparency, and we run several servers. It so happens that currently Astiga is running in Helsinki, which has added latency to our connection between the web site and database. This is significant because the way Astiga was written is to use a lot of small queries - this assumes low overheads. The Subsonic layer is especially prone to this - for example for each track in a playlist we end up making seven separate database calls. You can see how that scales 😉
We're going to change our infrastructure this weekend, removing our Helsinki server and co-locating everything in the same datacentre. Plus we'll probably change the code in the long term.
Sadly speed the issue still seems to be there. At 22:27 (Dutch time) I tried to load my playlist of 203 song in Tempo. I timed for 200 seconds, but nothing showed up. Gave up after that.
At 22:34 I tried it with the official app. It loaded, but took 40 seconds. Still noticeable slower than before, but less bad than Tempo.
Yep - hopefully moving the server today.
The server has now been moved. Any improvement @Cambionn ?
Definitely! At least it seems so for now. If I run into it again I'll let you know.
Accessing https://play.asti.ga/ right now, getting this message: "Service Unavailable"
Edit: Never mind, it works now. (10 seconds later)
Loading up my playlist now takes about 10-20 seconds. Definitely a lot faster. Great!
Thanks! @_BrightDarkness_ Yes, we saw a brief period of "Service Unavailable" too - we're looking into that. Probably the infrastructure auto configuring itself and temporarily losing connections... but obviously we'd like it to not do that.