(DataFlex) FTP Large File Upload

Demonstrates how to use the LargeFileUpload method introduced in v9.5.0.58.

The LargeFileUpload method is the same as PutFile, but designed to work around the following potential problem associated with an upload that is extremely large.

FTP uses two TCP (or TLS) connections: a control connection to submit commands and receive replies, and a data connection for actual file transfers. It is the nature of FTP that during a transfer the control connection stays completely idle. Many routers and firewalls automatically close idle connections after a certain period of time. Worse, they often don't notify the user, but just silently drop the connection.

For FTP, this means that during a long transfer the control connection can get dropped because it is detected as idle, but neither client nor server are notified. When all data has been transferred, the server assumes the control connection is alive and it sends the transfer confirmation reply.

Likewise, the client thinks the control connection is alive and it waits for the reply from the server. But since the control connection got dropped without notification, the reply never arrives and eventually the connection will timeout.

The Solution: LargeFileUpload uploads the file in chunks, where each chunk appends to the remote file. This way, each chunk is a separate FTP upload that does not take too long to complete. The chunkSize specifies the number of bytes to upload in each chunk. The size should be based on the amount of memory available (because each chunk will reside in memory as it's being uploaded), the transfer rate, and the total size of the file being uploaded. For example, if a 4GB file is uploaded, and the chunkSize is set to 1MB (1,048,576 bytes), then 4000 separate chunks would be required. This is likely not a good choice for chunkSize. A more appropriate chunkSize might be 20MB, in which case the upload would complete in 200 separate chunks. The application would temporarily be using a 20MB buffer for uploading chunks. The tradeoff is between the number of chunks (the more chunks, the larger the overall time to upload), the amount of memory that is reasonable for the temporary buffer, and the amount of time required to upload each chunk (if the chunk size is too large, then the problem described above is not solved).

Chilkat ActiveX Downloads ActiveX for 32-bit and 64-bit Windows

Use ChilkatAx-win32.pkg

Procedure Test
    Handle hoFtp
    Boolean iSuccess
    String sLocalPath
    String sRemoteFilename
    Integer iChunkSize
    String sTemp1

    // This example requires the Chilkat API to have been previously unlocked.
    // See Global Unlock Sample for sample code.

    Get Create (RefClass(cComChilkatFtp2)) To hoFtp
    If (Not(IsComObjectCreated(hoFtp))) Begin
        Send CreateComObject of hoFtp
    End

    Set ComHostname Of hoFtp To "ftp.someFtpServer.com"
    Set ComUsername Of hoFtp To "my-ftp-login"
    Set ComPassword Of hoFtp To "my-ftp-password"

    // Connect and login to the FTP server.
    Get ComConnect Of hoFtp To iSuccess
    If (iSuccess <> True) Begin
        Get ComLastErrorText Of hoFtp To sTemp1
        Showln sTemp1
        Procedure_Return
    End

    // Change to the remote directory where the file will be uploaded.
    Get ComChangeRemoteDir Of hoFtp "junk" To iSuccess
    If (iSuccess <> True) Begin
        Get ComLastErrorText Of hoFtp To sTemp1
        Showln sTemp1
        Procedure_Return
    End

    Move "c:/temp/veryLargeFile.dat" To sLocalPath
    Move "veryLargeFile.dat" To sRemoteFilename
    // Upload in chunks of 10 million bytes.
    Move 10000000 To iChunkSize

    Get ComLargeFileUpload Of hoFtp sLocalPath sRemoteFilename iChunkSize To iSuccess
    If (iSuccess <> True) Begin
        Get ComLastErrorText Of hoFtp To sTemp1
        Showln sTemp1
        Procedure_Return
    End

    Get ComDisconnect Of hoFtp To iSuccess

    Showln "Large File Uploaded!"


End_Procedure