(Visual FoxPro) Efficiently Process a Huge XML File

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).

Note: This example requires Chilkat v9.5.0.80 or greater.

Chilkat ActiveX Downloads ActiveX for 32-bit and 64-bit Windows

LOCAL loFac
LOCAL lnSuccess
LOCAL loXml
LOCAL loSb
LOCAL lnFirstIteration
LOCAL lnRetval
LOCAL lnNumTransactions
LOCAL lcBeginMarker
LOCAL lcEndMarker

* This example shows a way to efficiently process a gigantic XML file -- one that may be too large
* to fit in memory.  
* 
* Two types of XML parsers exist: DOM parsers and SAX parsers.

* A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
* and the application has the luxury of interacting with the XML in a convenient, random-access
* way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
* huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

* A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.  
* Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
* 
* The technique described here is a hybrid.  It streams the XML file as unstructured text
* to extract fragments that are individually treated as separate XML documents loaded into
* the Chilkat Xml parser.
* 
* For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
* 
* <Transactions>
*     <Transaction id="1">
*          ...
*     </Transaction>
*     <Transaction id="2">
*          ...
*     </Transaction>
*     <Transaction id="3">
*          ...
*     </Transaction>
* ...
* </Transactions>

* In the following code, each <Transaction ...> ... </Transaction>
* is extracted and loaded separately into an Xml object, where it can be manipulated
* independently.  The entire XML file is never entirely loaded into memory.

* For versions of Chilkat < 10.0.0, use CreateObject('Chilkat_9_5_0.FileAccess')
loFac = CreateObject('Chilkat.FileAccess')

lnSuccess = loFac.OpenForRead("qa_data/xml/transactions.xml")
IF (lnSuccess = 0) THEN
    ? loFac.LastErrorText
    RELEASE loFac
    CANCEL
ENDIF

* For versions of Chilkat < 10.0.0, use CreateObject('Chilkat_9_5_0.Xml')
loXml = CreateObject('Chilkat.Xml')
* For versions of Chilkat < 10.0.0, use CreateObject('Chilkat_9_5_0.StringBuilder')
loSb = CreateObject('Chilkat.StringBuilder')
lnFirstIteration = 1
lnRetval = 1
lnNumTransactions = 0

* The begin marker is "XML tag aware".  If the begin marker begins with "<"
* and ends with ">", then it is assumed to be an XML tag and it will also match
* substrings where the ">" can be a whitespace char.
lcBeginMarker = "<Transaction>"
lcEndMarker = "</Transaction>"

DO WHILE lnRetval = 1
    loSb.Clear()
    * The retval can have the following values:
    * 0: No more fragments exist.
    * 1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
    * -1: Error.
    lnRetval = loFac.ReadNextFragment(lnFirstIteration,lcBeginMarker,lcEndMarker,"utf-8",loSb)
    lnFirstIteration = 0

    IF (lnRetval = 1) THEN
        lnNumTransactions = lnNumTransactions + 1
        lnSuccess = loXml.LoadSb(loSb,1)
        * Your application may now do what it needs with this particular XML fragment...
    ENDIF

ENDDO

IF (lnRetval < 0) THEN
    ? loFac.LastErrorText
ENDIF

? "numTransactions: " + STR(lnNumTransactions)

RELEASE loFac
RELEASE loXml
RELEASE loSb