Is there a limit to how many tracks that Astiga can sync? OneDrive throttling and 429 errors

teamnteamn Member
edited June 2022 in Astiga web player

Hi there,

I recently tried to sync my OneDrive. I purchased premium just in case to help support because I realise I have a lot of music stored there so it would be unreasonable for me to sync for free.

I started the sync at this time:

2022-06-08T07:41:39 START 2022-06-08 07:41:39

Then I soon get an email saying its finished:

2022-06-09T03:03:29 END   2022-06-09 03:03:29

However I think it didn't sync all the tracks.

I looked over the log.txt I received in an email - the log was 1MB or so (I think this might be quite small for the volume of files). I checked the web player and I can see not all songs are added. I looked at the log and it looks like only 3 or 4 of the 21 root folders had any songs added.

For reference my onedrive has something like 3.2TB of music (222k files 27k folders, yes this is a ridiculous amount)

Am i doing something wrong or does astiga not really support this much music? i was hoping i could use this service to save me from writing my own app to play music from onedrive :(

Post edited by gravelld on

Answers

  • neasanneasan Member

    This happened to me when I had to re-sync my collection recently, the alphabet on my Artists page looked very strange 😂

    I'm premium as well so I just forced another sync and it was fine

  • gravelldgravelld Administrator

    @teamn Do you see these errors in the logs:

    Client error: `GET https://graph.microsoft.com/v1.0/me/drives` resulted in a `429 Too Many Requests` response:
{"error":{"code":"activityLimitReached","message":"The request has been throttled","innerError":{"code":"throttledRequest","innerError":{"code":"quota"},"date":"2022-06-08T07:48:43","request-id":"9b9f06ae-1c98-452e-a225-9190c88a84f1","client-request-id":"9b9f06ae-1c98-452e-a225-9190c88a84f1"}}}

    This means, at the time, we were making too many requests of OneDrive.

    I did some more research and found these pages:

    In short:

    • For now, try again!
    • I'll take a look to make sure we're following best practice. I can see we are performing retries and back-off.

    Thanks for upgrading to premium btw 🙂

  • teamnteamn Member
    edited June 2022

    Hi @gravelld

    Thank you, that is insightful! Indeed, the log does contain many of those messages.

    From the Github post I also note someone's comment:

    • Tenants can be "hard locked" which means all of your requests will be throttled. There is no way to deduce programmatically when this occurs.

    There is a possibility that could be happening to me with my Microsoft 365 tenancy. For good measure I'll close the OneDrive client and anything else that uses OneDrive/SPO APIs until the sync finishes - I'll try again now 🙂

    Thanks for the service so far - playing songs from the storage/file explorer does work great anyway (although I'll avoid it until the sync finishes just in case!).

  • gravelldgravelld Administrator

    Thanks - yeah, playing from the file explorer will incur further request "cost", so might be best to wait...

    Do you have a reason you think your own tenancy might be exhausted?

  • teamnteamn Member

    I think it's a mixture of me syncing with my phone, 3 PCs, using scripts on remote servers to download huge video files from s3 buckets to onedrive (using a tool called rclone) then reencoding and syncing to onedrive again.

    I see that once asatiga has "scanned the folders" it generates lots of API download requests - I think downloading and uploading are probably "heavier" api request that make it more likely to trigger the throttling. Looking at OneDrive/Office activity logs I can see how many are generated: about 3-4.5k per hour

    https://i.imgur.com/WhXs4d2.png


    The sync is still getting much further than it was before (I've had a browser tab open all day watching the log), so fingers crossed it might be finished the next day

  • gravelldgravelld Administrator

    This is really helpful, thanks. I think it might be better if we surfaced this error more obviously, and not have it buried in the logs.

    I'm going to edit the title of this thread slightly to include specific reference to Onedrive throttling, hope that's ok.

  • gravelldgravelld Administrator

    We've added further improvements to our handling of "Too Many Requests" messages from OneDrive. We now use the Retry-After header for folder requests as well as file requests.

Sign In or Register to comment.