Hey all, so I'm pretty new to AWS/ S3, but I was wondering what the best (i.e fastest) way to upload hundreds to thousands of files to S3 is. For context, my application is written in C# using the AWS S3 SDK package.

Some more context: I'm generating hundreds to thousands of tiny png images from a single (massive) tiff input image using GDAL, so called tiles to then be able to display them on a map (using leaflet). Now, since processing one file takes a long time (5-10 minutes) I'm tasked with containerizing the application to be able to orchestrate it across tens if not hundreds of containers since the application needs to process literal thousands of tiffs. The generated output is structured in directories akin to the following:

- outDir
  - 0
    - 0.png
  - 1
    - 0.png
    - 1.png

and so on, about 20 sub-directories with each containing (exponentially) more files. Now, after this generation has finished, I need to synchronize the output, and for that I need to get it all in one place, back on the S3 object storage, but what's the best way of doing that? The entire thing is a few megabytes, but made of around hundreds if not thousands of files (in testing, averaging about 900 files), and as far as I can tell I can't directly upload a folder and all it's children at once, meaning I'd need to make about 900 separate API calls, which seems ridiculous, so my current plan of action is to zip it up and send it as a single file to reduce API load, is there something I'm missing? Or does anyone have a better idea?