Xbase++

Efficiently Process a Huge XML File

See more XML Examples

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).

Note: This example requires Chilkat v9.5.0.80 or greater.

Xbase++

LOCAL nSuccess
LOCAL oFac
LOCAL oXml
LOCAL oSb
LOCAL nFirstIteration
LOCAL nRetval
LOCAL nNumTransactions
LOCAL cBeginMarker
LOCAL cEndMarker

nSuccess := 0

//  This example shows a way to efficiently process a gigantic XML file -- one that may be too large
//  to fit in memory.  
//  
//  Two types of XML parsers exist: DOM parsers and SAX parsers.

//  A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
//  and the application has the luxury of interacting with the XML in a convenient, random-access
//  way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
//  huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

//  A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.  
//  Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
//  
//  The technique described here is a hybrid.  It streams the XML file as unstructured text
//  to extract fragments that are individually treated as separate XML documents loaded into
//  the Chilkat Xml parser.
//  
//  For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
//  
//  <Transactions>
//      <Transaction id="1">
//           ...
//      </Transaction>
//      <Transaction id="2">
//           ...
//      </Transaction>
//      <Transaction id="3">
//           ...
//      </Transaction>
//  ...
//  </Transactions>

//  In the following code, each <Transaction ...> ... </Transaction>
//  is extracted and loaded separately into an Xml object, where it can be manipulated
//  independently.  The entire XML file is never entirely loaded into memory.

oFac := CreateObject("Chilkat.FileAccess")

nSuccess := oFac:OpenForRead("qa_data/xml/transactions.xml")
IF (nSuccess == 0)
    ? oFac:LastErrorText
    oFac:destroy()
    RETURN
ENDIF

oXml := CreateObject("Chilkat.Xml")
oSb := CreateObject("Chilkat.StringBuilder")
nFirstIteration := 1
nRetval := 1
nNumTransactions := 0

//  The begin marker is "XML tag aware".  If the begin marker begins with "<"
//  and ends with ">", then it is assumed to be an XML tag and it will also match
//  substrings where the ">" can be a whitespace char.
cBeginMarker := "<Transaction>"
cEndMarker := "</Transaction>"

DO WHILE nRetval == 1
    oSb:Clear()
    //  The retval can have the following values:
    //  0: No more fragments exist.
    //  1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
    //  -1: Error.
    nRetval := oFac:ReadNextFragment(nFirstIteration, cBeginMarker, cEndMarker, "utf-8", oSb)
    nFirstIteration := 0

    IF (nRetval == 1)
        nNumTransactions := nNumTransactions + 1
        nSuccess := oXml:LoadSb(oSb, 1)
        //  Your application may now do what it needs with this particular XML fragment...
    ENDIF

ENDDO

IF (nRetval < 0)
    ? oFac:LastErrorText
ENDIF

? "numTransactions: " + Str(nNumTransactions)

oFac:destroy()
oXml:destroy()
oSb:destroy()