Support for non-AWS S3 storage?

ElwellElwell Member
in Storage

Hi, I'm trying to connect astiga to our large Ceph cluster at work, however it seems intent on prefixing the bucketname in front of the URL, which we don't support (bucket names are not in DNS and won't be)

ie I'm trying to connect to https://ceph.example.com with bucketname mybucket, but I get

Error executing "ListObjectsV2" on "https://mybucket.ceph.example.com/?list-type=2&delimiter=%2F&prefix="; AWS HTTP error: cURL error 6: Could not resolve host: mybucket.ceph.example.com (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)
Go back to the initial directory

Any suggestions for how I can configure this storage in Astiga?

Comments

  • gravelldgravelld Administrator

    Thanks for trying! By default, the SDK we use (the default AWS one) prepends the bucket name to endpoint URLs.

    However, there is a way of asking that this isn't done, but it requires a code change on our end (and maybe some more UI work to configure this). I've added a ticket, but it's not scheduled for the next milestone.

    I guess the only other workaround is some sort of proxy on your side to redirect the requests, but I wouldn't blame you for discounting that as a lot of work!

  • ElwellElwell Member

    Thanks. I also tried using the 'path' method (https://ceph.example.com/mybucket/) but the add storage dialogue still required a value in bucket name

  • gravelldgravelld Administrator
    edited July 2023

    Ok, it looks like we do perform what is required (`'use_path_style_endpoint' => true`) but only when the endpoint contains "oraclecloud.com"!

    I've hacked something together to do the same thing for your domain. Can you test it to see if it works? Unfortunately I don't have access to your credentials so I can't test if myself (if you can you could set up a bucket for me to test in).

  • ElwellElwell Member

    Thanks!

    Storage successfully added which contains a small set of tracks, but indexing fails as thats still trying bucket prefixed URL


    2023-07-14T18:33:25 START 2023-07-14 18:33:25

    2023-07-14T18:33:25 GET  /

    2023-07-14T18:33:29 ERROR  Error executing "ListObjectsV2" on "https://mybucket.ceph.example.com/?list-type=2&delimiter=%2F&prefix="; AWS HTTP error: cURL error 6: Could not resolve host: mybucket.ceph.example.com (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

    (Url redacted, but timestamp correct)

    Playing directly from play.asti.ga/storage/ is working and retrieving metadata tho.

  • gravelldgravelld Administrator

    Thanks. Really sorry for the slow reply on this. I've made a change so sync should work... any good?

  • ElwellElwell Member

    Yep!

    Now to start filling up my storage quota. Thanks!

  • gravelldgravelld Administrator

    👍🏼

Sign In or Register to comment.