Astiga website seems to be broken

I cannot access play.asti.ga really good. I get "Bad Gateway" a bunch of times when accessing the site.

Sync doesn't work and I cannot stream my music. I tried on different browsers and my phone, all the same.

Did someone spill coffee over the servers?

«1

Comments

  • gravelldgravelld Administrator

    Looking at this right now. I've fixed the other issue with the cover art, I think, and I'm beginning to wonder if this is related because a health check, prior to the artwork fix, was breaking and causing restarts of the server. I'll keep monitoring this.

    Thanks for reporting!

  • gravelldgravelld Administrator

    I'm still seeing periodic `Bad Gateway` messages. So I'm continuing to look into this...

  • CambionnCambionn Member
    edited November 8

    I noticed that loading stuff in general is also more slow than usual. It's not only on the website either. Tempo is also slow to grab images and data, although it does so eventually and more reliable than the website. I would expect it to be somewhere in the underlying infrastructure perhaps? You have just been changing quite some stuff with a release a few days ago and multiple database maintenance's after all.

    I also have seen another MEDIA_ELEMENT_ERROR: Format error already again on the website, so the errors aren't entirely solved with the image fix either. Music does stream however, that error didn't block that.

  • gravelldgravelld Administrator

    Yes, I agree. I've been checking the usual suspects - disk usage was quite high on one server which has been fixed, and I initially thought this was the cause, but the errors are recurring now... still looking.

  • gravelldgravelld Administrator

    We think the "Bad Gateway" problems are fixed now. It turned out to be a problem in the way our servers talk to each other... essentially how they keep a record of what servers are running and so on. Stale data was retained by one of the servers which meant it was forwarding requests, getting a bad response, and showing that as "Bad Gateway".

    This is resolved for now, although requires a little more investigation later to prevent further occurrences.

  • I also think the sync is broken. I can't tell if it works in the background or not. There is no log and no feedback when syncing.

    On play.asti.ga/open-log, it shows this message:

    The last synchronisation was on 09-11-2024 09:49:45; this is over six days ago and so we have purged the log.
    

    Another issue that still persists is speed. I have a folder on Dropbox "Playlist 1" and when accessing it from the phone app, it will take about 1-3 minutes to load the files while being on the page "Contacting server, please wait..." It was taking about 20 seconds before the changes.

    I also get this error message on some albums under play.asti.ga/album

    404

    Oops! Album not found.

    We could not find the album you were looking for.

    I'm currently trying to forcibly re-sync my library to maybe resolve this. There are also a bunch of duplicate albums for me.


    What works for me?

    1. Loading the website or pages on play.asti.ga. (No Bad Gateway anymore)
    2. Streaming, browsing (some) albums and music.
  • edited November 9

    Hey. I might think the 404 Oops! Album not found. might be due to special characters in the album name.

    One of my album names is "#5", and in the search bar "#" = "%23" for the ASCII Encoding Reference.

    There might be a problem matching to the correct path of the album?

    %20 for space resolves to a real space and shows the album, but not for #.

    Even though the "," is shown as %2 in the search bar, it still shows the album contents correctly. Idk

    Problem might be related to that?

    Edit: Also check this https://imgur.com/a/ToXOupS

  • gravelldgravelld Administrator

    Thanks for this; unfortunately I think they are separate issues.

    It looks like a sync has run through since you posted. A lot of files were added - over 31,000, but I noticed it was a force refresh. Can you confirm? Do you still see the error if you perform a manual sync?

    Do you see the same performance problems on the Web app?

    Yeah, the album thing looks like an encoding issue. We'll look into that.

  • edited November 9

    Yes. I force refreshed. This was a manual sync. No log is being displayed by manually syncing. Message of the log on page play.asti.ga/open-log after the sync has finished: "The last synchronisation was on 09-11-2024 13:29:38; this is over six days ago and so we have purged the log."

    Using the browser Firefox on Windows 11; performance seems to be faster and more responsive than the android app. Although, opening a folder with over 700 files still takes a little bit. (Approx 20 seconds on the Firefox browser, Win11)

  • CambionnCambionn Member
    edited November 9

    Speed is still an issue. Loading times are still high. It takes Tempo much longer than usual to grab data, and in same cases it times out (or takes so long I've given up, not sure). For example when trying to load a playlist of about 200 songs. Edit: I eventually got it after 20+ minutes, but obviously that's not very user friendly. Oddly enough a 100 song playlist took 1 minute "only". This all used to be pretty instant, or at least quick enough not to bother.

    While actually timing out instead of waiting may be on the app, the speed issue seems to be on Astiga. The official app is equally slow.

    I haven't tested on PC since you fixed the gateway error, but I'd expect it to be there too. Are the MEDIA_ELEMENT_ERROR: Format error and the likes also connected to the gateway or more likely connected to whatever causes the long loading times?

    Post edited by Cambionn on
  • gravelldgravelld Administrator

    I think the "Format error" issues were down to the Bad Gateway - where the server wasn't providing the audio stream.

    I think slowness is not related to that. I'll be looking at this on Monday morning too, I hope that's ok.

  • No problem with me. Stuff works for the most part, noticeably slower but as long as I don't try to load too much at once doable.

  • gravelldgravelld Administrator
    edited November 11

    An update on progress on the above items...

    Hash mark issue

    A fix for this is going into testing now.

    Sync logs

    We're looking into this; we think it's a logic error rather than relating to performance.

    Storage pages (on the Web)

    @_BrightDarkness_ I've spotted a folder in the logs where it (currently) takes about 5s to download info about a folder with 3,000 files. I'm not observing anything as long as 20s.

    Can you confirm this is still occurring on the Web version, and/or the app(s)? If so, could you please let us know the URL (Web) or page (if on the app) you are requesting via info@asti.ga ?

    App performance loading playlists

    Note this has been an issue for a while that we're seeking to address soon. However, this might not be the same issue.

    @Cambionn I'm noticing a playlist that takes 30s to load from your account. I'm looking into this now.

    General sluggishness

    Can anyone give specific areas of the app or website where this happens?

    We've observed a slow query on the top level page that lists recent albums etc - we can fix that.

    We've noticed image load times are slow and some places where we were making additional queries where we didn't need to, so we're fixing them.

    Other than that, it would be helpful to have specific pages that are slow (feel free to email us if you'd rather do it privately: info@asti.ga ).

  • CambionnCambionn Member
    edited November 11

    @gravelld Thanks for the detailed update!

    Playlists are not the only part of my issue btw, although it's the most noticeable as I don't have many other places that load 200 tracks.

    For example, the home screen often doesn't load data, or does so slowly. Folder browsing similarly struggles to grab data, as well as just normal artist and album pages. It seems to be app-wide in general. At times I also get a message the server isn't reachable, with it being back again shortly after. I expect it's telling me that due to taking too long to load. It is however not always slow, right now it works as instantly as I'm used to. I will keep an eye on it, so I can give you hopefully some timestamps of when it slowed down so you can check the logs.

    This all only started after the last database maintenance/release. Because the website had issues with the gateway and such, I can hardly tell if those are fixed when those where or not. The website seems fine now but I haven't used it much due to the weekend and me only working in the morning then going out.

    Podcasts interestingly enough seem fairly unaffected in Tempo.

  • gravelldgravelld Administrator

    Can you just clarify if you're talking about the app or the Web site? You use terms like "home screen" which could apply to either... You say later "The website seems fine now" but I don't know if you mean wrt the Bad Gateway messages only.

    "Bad Gateway" is definitely fixed now.

  • CambionnCambionn Member
    edited November 11

    Home screen, as in the home screen of the Tempo app.

    For example, the home screen often doesn't load data, or does so slowly. Folder browsing similarly struggles to grab data, as well as just normal artist and album pages. It seems to be app-wide in general. At times I also get a message the server isn't reachable, with it being back again shortly after. I expect it's telling me that due to taking too long to load. It is however not always slow, right now it works as instantly as I'm used to. I will keep an eye on it, so I can give you hopefully some timestamps of when it slowed down so you can check the logs.

    This is all about the Tempo app. The official Astiga app has similar issues, so it seems to linger at least in anything outside of the web interface.

    Because the website had issues with the gateway and such, I can hardly tell if those are fixed when those where or not. The website seems fine now but I haven't used it much due to the weekend and me only working in the morning then going out.

    Only this is about the website (play.asti.ga). I know the gateway issues are fixed. I however do not know which issues that I had on the website came from that and which may still be present as I haven't been using the website that much yet. So I simply cannot tell if there are still issues there. I will have to get back to that after using it some more. But so far, with the limited use done so far, it seems it's working fine.

  • gravelldgravelld Administrator

    Thanks @Cambionn . I was experimenting with that playlist that takes 30s to load... I can't actually see anything that has changed in the latest release that would affect it. Have those playlists always loaded quicker prior to this release? It's about 200 tracks.

  • CambionnCambionn Member
    edited November 11

    It definitely was faster, yes. As said, the other day I was waiting for 20 minutes for that playlist. If you only see 30 seconds, it may have been simply another case than the one you're experimenting with. But it may also mean something went wrong earlier on before it logs on your side.

    Since Tempo hasn't had any updates, and the issue is also in the official app, I would expect the issue is on your side.

    Although I must admit, after that one time waiting 20 minutes I haven't had it that extreme anymore, while it happened more often before that. That was on Saturday. Since it's still a bit slower than usual, but nothing too bad. I would say in the seconds indeed, not minutes. So it may already be partly solved, but time will have to tell. But why it happened? No clue.

  • gravelldgravelld Administrator

    I'll try with Tempo today, see what I observe.

    The twenty minute case may be the Bad Gateway issue also. The app retries failed requests, and it may have simply been retrying over and over.

    There are clear inefficiencies in the way this code works - fixing it will take some time, but it was ever thus (hence my link to the other thread above, for extremely large playlists).

  • gravelldgravelld Administrator
    edited November 12

    I looked at Tempo today... I thought it a bit sluggish although I didn't think it was that bad...

    Worst thing I noticed was the track list taking a while to download for an album. This was taking about 5s. It's made worse by the fact that Tempo appears to make the call twice (maybe it's timing out and trying again).

    The main underlying cause of the slowness there is the same as I observed with the playlists - we are very inefficient with the way we process data before passing it back to the client. I'm working on some ways to improve this. Again, this hasn't changed, code-wise, for this release. But it's possible some DB level change has made it slower (although there's been nothing directly related).

    Post edited by gravelld on
  • Sounds plausible to me. I haven't seen the extremes anymore, just a noticeable few seconds. If I see those extremes again I'll let you know. Otherwise I'll wait for the optimisation changes.

  • gravelldgravelld Administrator
    edited November 13

    We made another discovery. Our infrastructure allows for geographic transparency, and we run several servers. It so happens that currently Astiga is running in Helsinki, which has added latency to our connection between the web site and database. This is significant because the way Astiga was written is to use a lot of small queries - this assumes low overheads. The Subsonic layer is especially prone to this - for example for each track in a playlist we end up making seven separate database calls. You can see how that scales 😉

    We're going to change our infrastructure this weekend, removing our Helsinki server and co-locating everything in the same datacentre. Plus we'll probably change the code in the long term.

  • CambionnCambionn Member
    edited November 16

    Sadly speed the issue still seems to be there. At 22:27 (Dutch time) I tried to load my playlist of 203 song in Tempo. I timed for 200 seconds, but nothing showed up. Gave up after that.

    At 22:34 I tried it with the official app. It loaded, but took 40 seconds. Still noticeable slower than before, but less bad than Tempo.

    Post edited by Cambionn on
  • gravelldgravelld Administrator

    Yep - hopefully moving the server today.

  • gravelldgravelld Administrator

    The server has now been moved. Any improvement @Cambionn ?

  • Definitely! At least it seems so for now. If I run into it again I'll let you know.

  • edited November 16

    Accessing https://play.asti.ga/ right now, getting this message: "Service Unavailable"

    Edit: Never mind, it works now. (10 seconds later)

  • edited November 16

    Loading up my playlist now takes about 10-20 seconds. Definitely a lot faster. Great!

  • gravelldgravelld Administrator

    Thanks! @_BrightDarkness_ Yes, we saw a brief period of "Service Unavailable" too - we're looking into that. Probably the infrastructure auto configuring itself and temporarily losing connections... but obviously we'd like it to not do that.

  • Hey! Astiga seems to be running well.

    I have one issue left:

    I still can't access some albums as it says the album doesn't exist. You said it looks like an encoding issue. The album with the # is now accessible.

    I have albums starting with "." and it spits out "Oops! Page not found." on https://play.asti.ga/albums. https://play.asti.ga/albums/. redirects to the same link without the dot.

Sign In or Register to comment.