DataFlex
DataFlex
S3 Upload the Parts for a Multipart Upload
See more Amazon S3 (new) Examples
This example uploads a large file in parts. The multipart upload needs to have been first initiated prior to uploading the parts.See http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html for more information about uploading parts.
Chilkat DataFlex Downloads
Use ChilkatAx-win32.pkg
Procedure Test
Boolean iSuccess
Handle hoXmlInit
String sUploadId
String sFileToUploadPath
Integer iPartSize
Handle hoFac
Integer iNumParts
String sPartsListFile
Handle hoPartsListXml
Handle hoRest
Boolean iBTls
Integer iPort
Boolean iBAutoReconnect
Variant vAuthAws
Handle hoAuthAws
Integer iPartNumber
Handle hoSbPartNumber
Boolean iBPartAlreadyUploaded
Integer iNumUploadedParts
Variant vXRec0
Handle hoXRec0
Variant vFoundRec
Handle hoFoundRec
Variant vFileStream
Handle hoFileStream
String sResponseStr
String sEtag
Variant vXPart
Handle hoXPart
String sTemp1
Integer iTemp1
Boolean bTemp1
Move False To iSuccess
// In the 1st step for uploading a large file, the multipart upload was initiated
// as shown here: Initiate Multipart Upload
// Other S3 Multipart Upload Examples:
// Complete Multipart Upload
// Abort Multipart Upload
// List Parts
// When we initiated the multipart upload, we saved the XML response to a file. This
// XML response contains the UploadId. We'll begin by loading that XML and getting
// the Upload ID.
Get Create (RefClass(cComChilkatXml)) To hoXmlInit
If (Not(IsComObjectCreated(hoXmlInit))) Begin
Send CreateComObject of hoXmlInit
End
Get ComLoadXmlFile Of hoXmlInit "s3_multipart_uploads/initiate.xml" To iSuccess
If (iSuccess <> True) Begin
Showln "Did not find the initiate.xml XML file."
Procedure_Return
End
Get ComGetChildContent Of hoXmlInit "UploadId" To sUploadId
Showln "UploadId = " sUploadId
// When uploading parts, we need to keep an XML record of each part number
// and its corresponding ETag, which is received in the response for each part.
// There can be up to 10000 parts, numbered 1 to 10000.
// After all parts have been uploaded, the final step will be to complete
// the multipart upload (see Complete Multipart Upload)
// In this example, the large file we want to upload is somethingBig.zip
Move "s3_multipart_uploads/somethingBig.zip" To sFileToUploadPath
// The minimum allowed part size is 5MB (5242880 bytes). The last part can be smaller because
// it will contain the remainder of the file. (This minimum is enforced by the AWS service.)
// We'll use the minimum allowed part size for this example.
Move 5242880 To iPartSize
// Let's use Chilkat's FileAccess API to examine the file to be uploaded. We'll get the size
// of the file and find out how many parts will be needed, including the final "partial" part.
Get Create (RefClass(cComCkFileAccess)) To hoFac
If (Not(IsComObjectCreated(hoFac))) Begin
Send CreateComObject of hoFac
End
Get ComOpenForRead Of hoFac sFileToUploadPath To iSuccess
// How many parts will there be if each part is 5242880 bytes?
Get ComGetNumBlocks Of hoFac iPartSize To iNumParts
Showln "numParts = " iNumParts
Send ComFileClose To hoFac
// Imagine that we may be running this for the 1st time, or maybe we already
// attempted to upload parts, and something failed. Maybe there was a network problem
// the resulted in not all parts getting uploaded. We'll write this code so that if run again,
// it will upload whatever parts haven't yet been uploaded.
// We'll keep a partsList.xml file to record the parts that have already been successfully
// uploaded. If this file does not yet exist, we'll create it..
Move "s3_multipart_uploads/partsList.xml" To sPartsListFile
Get Create (RefClass(cComChilkatXml)) To hoPartsListXml
If (Not(IsComObjectCreated(hoPartsListXml))) Begin
Send CreateComObject of hoPartsListXml
End
Get ComFileExists Of hoFac sPartsListFile To bTemp1
If (bTemp1 = True) Begin
Get ComLoadXmlFile Of hoPartsListXml sPartsListFile To iSuccess
End
// Make sure the top-level tag is "CompleteMultipartUpload"
Set ComTag Of hoPartsListXml To "CompleteMultipartUpload"
// --------------------------------------
// Before entering the loop to upload parts,
// setup the REST object with AWS authentication,
// and make the initial connection.
Get Create (RefClass(cComChilkatRest)) To hoRest
If (Not(IsComObjectCreated(hoRest))) Begin
Send CreateComObject of hoRest
End
// Connect to the Amazon AWS REST server.
Move True To iBTls
Move 443 To iPort
Move True To iBAutoReconnect
Get ComConnect Of hoRest "s3.amazonaws.com" iPort iBTls iBAutoReconnect To iSuccess
// ----------------------------------------------------------------------------
// Important: For buckets created in regions outside us-east-1,
// there are three important changes that need to be made.
// See Working with S3 Buckets in Non-us-east-1 Regions for the details.
// ----------------------------------------------------------------------------
// Provide AWS credentials for the REST call.
Get Create (RefClass(cComChilkatAuthAws)) To hoAuthAws
If (Not(IsComObjectCreated(hoAuthAws))) Begin
Send CreateComObject of hoAuthAws
End
Set ComAccessKey Of hoAuthAws To "AWS_ACCESS_KEY"
Set ComSecretKey Of hoAuthAws To "AWS_SECRET_KEY"
Set ComServiceName Of hoAuthAws To "s3"
Get pvComObject of hoAuthAws to vAuthAws
Get ComSetAuthAws Of hoRest vAuthAws To iSuccess
// Set the bucket name via the HOST header.
// In this case, the bucket name is "chilkat100".
Set ComHost Of hoRest To "chilkat100.s3.amazonaws.com"
// --------------------------------------
Move 1 To iPartNumber
Get Create (RefClass(cComChilkatStringBuilder)) To hoSbPartNumber
If (Not(IsComObjectCreated(hoSbPartNumber))) Begin
Send CreateComObject of hoSbPartNumber
End
While (iPartNumber <= iNumParts)
Showln "---- " iPartNumber " ----"
// This cumbersome way of converting an integer to a string is because
// Chilkat examples are written in a script that is converted to many programming languages.
// At this time, the translator does not have integer-to-string code generation capability..
Send ComClear To hoSbPartNumber
Get ComAppendInt Of hoSbPartNumber iPartNumber To iSuccess
Move False To iBPartAlreadyUploaded
// If there are no children, then the XML is empty and no parts have yet been uploaded.
Get ComNumChildren Of hoPartsListXml To iNumUploadedParts
If (iNumUploadedParts > 0) Begin
// If some parts have been uploaded, check to see if this particular part was already upload.
// If so, then it can be skipped.
// Position ourselves at the 1st record.
Get ComGetChild Of hoPartsListXml 0 To vXRec0
If (IsComObject(vXRec0)) Begin
Get Create (RefClass(cComChilkatXml)) To hoXRec0
Set pvComObject Of hoXRec0 To vXRec0
End
Get ComGetAsString Of hoSbPartNumber To sTemp1
Get ComFindNextRecord Of hoXRec0 "PartNumber" sTemp1 To vFoundRec
If (IsComObject(vFoundRec)) Begin
Get Create (RefClass(cComChilkatXml)) To hoFoundRec
Set pvComObject Of hoFoundRec To vFoundRec
End
Get ComLastMethodSuccess Of hoXRec0 To bTemp1
If (bTemp1 = True) Begin
Move True To iBPartAlreadyUploaded
Showln "Part " iPartNumber " was previously uploaded."
Get ComGetXml Of hoFoundRec To sTemp1
Showln sTemp1
Send Destroy of hoFoundRec
End
Send Destroy of hoXRec0
End
// If this part was not already uploaded, we need to upload.
// Also update the partsListXml and save as each part is successfully uploaded.
If (iBPartAlreadyUploaded = False) Begin
Showln "Uploading part " iPartNumber " ..."
// Setup the stream source for the large file to be uploaded..
Get Create (RefClass(cComChilkatStream)) To hoFileStream
If (Not(IsComObjectCreated(hoFileStream))) Begin
Send CreateComObject of hoFileStream
End
Set ComSourceFile Of hoFileStream To sFileToUploadPath
// The Chilkat Stream API has features to make uploading a parts
// of a file easy. Indicate the part size by setting the SourceFilePartSize
// property.
Set ComSourceFilePartSize Of hoFileStream To iPartSize
// Our HTTP start line to upload a part will look like this:
// PUT /ObjectName?partNumber=PartNumber&uploadId=UploadId HTTP/1.1
// Set the query params. We'll need partNumber and uploadId.
// Make sure the query params from previous iterations are clear.
Get ComClearAllQueryParams Of hoRest To iSuccess
Get ComGetAsString Of hoSbPartNumber To sTemp1
Get ComAddQueryParam Of hoRest "partNumber" sTemp1 To iSuccess
Get ComAddQueryParam Of hoRest "uploadId" sUploadId To iSuccess
// Upload this particular file part.
// Tell the fileStream which part is being uploaded.
// Our partNumber is 1-based (the 1st part is at index 1), but the fileStream's SourceFilePart
// property is 0-based. Therefore we use partNumber-1.
Set ComSourceFilePart Of hoFileStream To iPartNumber - 1
// Because the SourceFilePart and SourceFilePartSize properties are set, the stream will
// will provide just that part of the file.
Get pvComObject of hoFileStream to vFileStream
Get ComFullRequestStream Of hoRest "PUT" "/somethingBig.zip" vFileStream To sResponseStr
Get ComLastMethodSuccess Of hoRest To bTemp1
If (bTemp1 <> True) Begin
Get ComLastErrorText Of hoRest To sTemp1
Showln sTemp1
Procedure_Return
End
Get ComResponseStatusCode Of hoRest To iTemp1
If (iTemp1 <> 200) Begin
// Examine the request/response to see what happened.
Get ComResponseStatusCode Of hoRest To iTemp1
Showln "response status code = " iTemp1
Get ComResponseStatusText Of hoRest To sTemp1
Showln "response status text = " sTemp1
Get ComResponseHeader Of hoRest To sTemp1
Showln "response header: " sTemp1
Showln "response body: " sResponseStr
Showln "---"
Get ComLastRequestStartLine Of hoRest To sTemp1
Showln "LastRequestStartLine: " sTemp1
Get ComLastRequestHeader Of hoRest To sTemp1
Showln "LastRequestHeader: " sTemp1
Procedure_Return
End
// OK, this part was uploaded..
// The response will have a 0-length body. The only information we need is the
// ETag response header field.
Get ComResponseHdrByName Of hoRest "ETag" To sEtag
// It should be present, but just in case there was no ETag header...
Get ComLastMethodSuccess Of hoRest To bTemp1
If (bTemp1 <> True) Begin
Showln "No ETag response header found!"
Get ComResponseHeader Of hoRest To sTemp1
Showln "response header: " sTemp1
Procedure_Return
End
// We need to add record to the partsListXml.
// The record will look like this:
// <Part>
// <PartNumber>PartNumber</PartNumber>
// <ETag>ETag</ETag>
// </Part>
Get ComNewChild Of hoPartsListXml "Part" "" To vXPart
If (IsComObject(vXPart)) Begin
Get Create (RefClass(cComChilkatXml)) To hoXPart
Set pvComObject Of hoXPart To vXPart
End
Send ComNewChildInt2 To hoXPart "PartNumber" iPartNumber
Send ComNewChild2 To hoXPart "ETag" sEtag
Send Destroy of hoXPart
Get ComSaveXml Of hoPartsListXml sPartsListFile To iSuccess
If (iSuccess <> True) Begin
Get ComLastErrorText Of hoPartsListXml To sTemp1
Showln sTemp1
Procedure_Return
End
Showln "-- Part " iPartNumber " uploaded. ---------------------"
End
Move (iPartNumber + 1) To iPartNumber
Loop
Showln "Finished. All parts uploaded."
End_Procedure