Need to move about 10 terabytes of data. Here's the scenario:

• Data is currently with a vendor that we partner with. Only export option is for us to pull data out with APIs.
• Need to move this data to a customer's S3 instance for backup purposes.

Basically, I'm kind of torn because we don't want to have customers to write an integration, but i'm thinking of some options:

1. Python script to download the 50TB locally, and then push up to customer's S3 bucket using something like rclone using their credentials. Would have to do in batches obviously. Don't love the idea of making this a 2-step process though.
2. Write a generic script for customer to run somewhere to pull data from vendor into their S3 bucket using our api as a proxy. This is quite messy because we'd have to also join their data with some of our internal metadata (we don't want to grab all).

Imagine data is something like this:

Vendor

• mixed customer data
• 1000 files or so related to 1 customer
• 500 files for customer that we don't need (abandoned/soft deleted/etc).

API's to download from there ^{^}

Our system

• links objects to the data above. helps us know which we actually need to pull over (which 1000, not all 1500)