Java

Efficiently Process a Huge XML File

See more XML Examples

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).

Note: This example requires Chilkat v9.5.0.80 or greater.

Chilkat2 Java

import com.chilkatsoft.chilkat2.ChilkatException;
import com.chilkatsoft.chilkat2.FileAccess;
import com.chilkatsoft.chilkat2.StringBuilder;
import com.chilkatsoft.chilkat2.Xml;

public class ChilkatExample {

    public static void main(String[] args) {
        // This example shows a way to efficiently process a gigantic XML file -- one that may be too large
        // to fit in memory.
        //
        // Two types of XML parsers exist: DOM parsers and SAX parsers.

        // A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
        // and the application has the luxury of interacting with the XML in a convenient, random-access
        // way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
        // huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

        // A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.
        // Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
        //
        // The technique described here is a hybrid.  It streams the XML file as unstructured text
        // to extract fragments that are individually treated as separate XML documents loaded into
        // the Chilkat Xml parser.
        //
        // For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
        //
        // <Transactions>
        //     <Transaction id="1">
        //          ...
        //     </Transaction>
        //     <Transaction id="2">
        //          ...
        //     </Transaction>
        //     <Transaction id="3">
        //          ...
        //     </Transaction>
        // ...
        // </Transactions>

        // In the following code, each <Transaction ...> ... </Transaction>
        // is extracted and loaded separately into an Xml object, where it can be manipulated
        // independently.  The entire XML file is never entirely loaded into memory.

        FileAccess fac = new FileAccess();

        try {
            fac.openForRead("qa_data/xml/transactions.xml");
        } catch (ChilkatException e) {
            System.out.println(e.getLastErrorText());
            return;
        }

        Xml xml = new Xml();
        StringBuilder sb = new StringBuilder();
        boolean firstIteration = true;
        int retval = 1;
        int numTransactions = 0;

        // The begin marker is "XML tag aware".  If the begin marker begins with "<"
        // and ends with ">", then it is assumed to be an XML tag and it will also match
        // substrings where the ">" can be a whitespace char.
        String beginMarker = "<Transaction>";
        String endMarker = "</Transaction>";

        while (retval == 1) {
            sb.clear();
            // The retval can have the following values:
            // 0: No more fragments exist.
            // 1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
            // -1: Error.
            retval = fac.readNextFragment(firstIteration, beginMarker, endMarker, "utf-8", sb);
            firstIteration = false;

            if (retval == 1) {
                numTransactions = numTransactions + 1;
                xml.loadSb(sb, true);
                // Your application may now do what it needs with this particular XML fragment...
            }
        }

        if (retval < 0) {
            System.out.println(fac.getLastErrorText());
        }

        System.out.println("numTransactions: " + numTransactions);
    }
}