FiFo and AWS s3


It would be nice if AWS s3 could be used as a backend for datasets and backups. So far the attempts with this have been rather fruitless. There seem to be two main problems:

1) In the GZ the certificates are not properly validated (reported by ipalready taken, ask for details)
2) In zones the certificates do work, however the upload fails after some time.

To elaborate on 2) FiFo uses multi part uploads streaming a number of chunks at the same time to optimize the process. It seems that AWS dies with a socket timeout after ~ 25 chunks (tested with 4MB chunks and 1 MB chunks). Furthermore the final part of the multipart upload, confirming the 'complete' of all parts never works.

My knowledge of AWS is very limited and I have no idea how to debug that, neither is this a top priority just a nice to have. If anyone wants to step in on this and see how to get things working they'd be very welcome.

the s3 library fifo uses is erlcoud [1], wrapped by fifo_s3[2] abstracting the multipart layer.





Heinz N. Gies


Heinz N. Gies



Fix versions

Affects versions