(Go) FTP Large File Upload

Demonstrates how to use the LargeFileUpload method introduced in v9.5.0.58.

The LargeFileUpload method is the same as PutFile, but designed to work around the following potential problem associated with an upload that is extremely large.

FTP uses two TCP (or TLS) connections: a control connection to submit commands and receive replies, and a data connection for actual file transfers. It is the nature of FTP that during a transfer the control connection stays completely idle. Many routers and firewalls automatically close idle connections after a certain period of time. Worse, they often don't notify the user, but just silently drop the connection.

For FTP, this means that during a long transfer the control connection can get dropped because it is detected as idle, but neither client nor server are notified. When all data has been transferred, the server assumes the control connection is alive and it sends the transfer confirmation reply.

Likewise, the client thinks the control connection is alive and it waits for the reply from the server. But since the control connection got dropped without notification, the reply never arrives and eventually the connection will timeout.

The Solution: LargeFileUpload uploads the file in chunks, where each chunk appends to the remote file. This way, each chunk is a separate FTP upload that does not take too long to complete. The chunkSize specifies the number of bytes to upload in each chunk. The size should be based on the amount of memory available (because each chunk will reside in memory as it's being uploaded), the transfer rate, and the total size of the file being uploaded. For example, if a 4GB file is uploaded, and the chunkSize is set to 1MB (1,048,576 bytes), then 4000 separate chunks would be required. This is likely not a good choice for chunkSize. A more appropriate chunkSize might be 20MB, in which case the upload would complete in 200 separate chunks. The application would temporarily be using a 20MB buffer for uploading chunks. The tradeoff is between the number of chunks (the more chunks, the larger the overall time to upload), the amount of memory that is reasonable for the temporary buffer, and the amount of time required to upload each chunk (if the chunk size is too large, then the problem described above is not solved).

Chilkat Go Downloads Go Package for Windows, MacOS, Linux, Alpine Linux, Solaris

    // This example requires the Chilkat API to have been previously unlocked.
    // See Global Unlock Sample for sample code.

    ftp := chilkat.NewFtp2()

    ftp.SetHostname("ftp.someFtpServer.com")
    ftp.SetUsername("my-ftp-login")
    ftp.SetPassword("my-ftp-password")

    // Connect and login to the FTP server.
    success := ftp.Connect()
    if success != true {
        fmt.Println(ftp.LastErrorText())
        ftp.DisposeFtp2()
        return
    }

    // Change to the remote directory where the file will be uploaded.
    success = ftp.ChangeRemoteDir("junk")
    if success != true {
        fmt.Println(ftp.LastErrorText())
        ftp.DisposeFtp2()
        return
    }

    localPath := "c:/temp/veryLargeFile.dat"
    remoteFilename := "veryLargeFile.dat"
    // Upload in chunks of 10 million bytes.
    chunkSize := 10000000

    success = ftp.LargeFileUpload(localPath,remoteFilename,chunkSize)
    if success != true {
        fmt.Println(ftp.LastErrorText())
        ftp.DisposeFtp2()
        return
    }

    success = ftp.Disconnect()

    fmt.Println("Large File Uploaded!")

    ftp.DisposeFtp2()