Is there a limit to how many tracks that Astiga can sync? OneDrive throttling and 429 errors
I recently tried to sync my OneDrive. I purchased premium just in case to help support because I realise I have a lot of music stored there so it would be unreasonable for me to sync for free.
I started the sync at this time:
2022-06-08T07:41:39 START 2022-06-08 07:41:39
Then I soon get an email saying its finished:
2022-06-09T03:03:29 END 2022-06-09 03:03:29
However I think it didn't sync all the tracks.
I looked over the log.txt I received in an email - the log was 1MB or so (I think this might be quite small for the volume of files). I checked the web player and I can see not all songs are added. I looked at the log and it looks like only 3 or 4 of the 21 root folders had any songs added.
For reference my onedrive has something like 3.2TB of music (222k files 27k folders, yes this is a ridiculous amount)
Am i doing something wrong or does astiga not really support this much music? i was hoping i could use this service to save me from writing my own app to play music from onedrive :(
This happened to me when I had to re-sync my collection recently, the alphabet on my Artists page looked very strange 😂
I'm premium as well so I just forced another sync and it was fine
@teamn Do you see these errors in the logs:
This means, at the time, we were making too many requests of OneDrive.
I did some more research and found these pages:
Thanks for upgrading to premium btw 🙂
Thank you, that is insightful! Indeed, the log does contain many of those messages.
From the Github post I also note someone's comment:
There is a possibility that could be happening to me with my Microsoft 365 tenancy. For good measure I'll close the OneDrive client and anything else that uses OneDrive/SPO APIs until the sync finishes - I'll try again now 🙂
Thanks for the service so far - playing songs from the storage/file explorer does work great anyway (although I'll avoid it until the sync finishes just in case!).
Thanks - yeah, playing from the file explorer will incur further request "cost", so might be best to wait...
Do you have a reason you think your own tenancy might be exhausted?
I think it's a mixture of me syncing with my phone, 3 PCs, using scripts on remote servers to download huge video files from s3 buckets to onedrive (using a tool called rclone) then reencoding and syncing to onedrive again.
I see that once asatiga has "scanned the folders" it generates lots of API download requests - I think downloading and uploading are probably "heavier" api request that make it more likely to trigger the throttling. Looking at OneDrive/Office activity logs I can see how many are generated: about 3-4.5k per hour
The sync is still getting much further than it was before (I've had a browser tab open all day watching the log), so fingers crossed it might be finished the next day
This is really helpful, thanks. I think it might be better if we surfaced this error more obviously, and not have it buried in the logs.
I'm going to edit the title of this thread slightly to include specific reference to Onedrive throttling, hope that's ok.