(PowerBuilder) Efficiently Process a Huge XML File

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).

Note: This example requires Chilkat v9.5.0.80 or greater.

Chilkat ActiveX Downloads ActiveX for 32-bit and 64-bit Windows

integer li_rc
oleobject loo_Fac
integer li_Success
oleobject loo_Xml
oleobject loo_Sb
integer li_FirstIteration
integer li_Retval
integer li_NumTransactions
string ls_BeginMarker
string ls_EndMarker

// This example shows a way to efficiently process a gigantic XML file -- one that may be too large
// to fit in memory.  
// 
// Two types of XML parsers exist: DOM parsers and SAX parsers.

// A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
// and the application has the luxury of interacting with the XML in a convenient, random-access
// way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
// huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

// A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.  
// Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
// 
// The technique described here is a hybrid.  It streams the XML file as unstructured text
// to extract fragments that are individually treated as separate XML documents loaded into
// the Chilkat Xml parser.
// 
// For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
// 
// <Transactions>
//     <Transaction id="1">
//          ...
//     </Transaction>
//     <Transaction id="2">
//          ...
//     </Transaction>
//     <Transaction id="3">
//          ...
//     </Transaction>
// ...
// </Transactions>

// In the following code, each <Transaction ...> ... </Transaction>
// is extracted and loaded separately into an Xml object, where it can be manipulated
// independently.  The entire XML file is never entirely loaded into memory.

loo_Fac = create oleobject
// Use "Chilkat_9_5_0.FileAccess" for versions of Chilkat < 10.0.0
li_rc = loo_Fac.ConnectToNewObject("Chilkat.FileAccess")
if li_rc < 0 then
    destroy loo_Fac
    MessageBox("Error","Connecting to COM object failed")
    return
end if

li_Success = loo_Fac.OpenForRead("qa_data/xml/transactions.xml")
if li_Success = 0 then
    Write-Debug loo_Fac.LastErrorText
    destroy loo_Fac
    return
end if

loo_Xml = create oleobject
// Use "Chilkat_9_5_0.Xml" for versions of Chilkat < 10.0.0
li_rc = loo_Xml.ConnectToNewObject("Chilkat.Xml")

loo_Sb = create oleobject
// Use "Chilkat_9_5_0.StringBuilder" for versions of Chilkat < 10.0.0
li_rc = loo_Sb.ConnectToNewObject("Chilkat.StringBuilder")

li_FirstIteration = 1
li_Retval = 1
li_NumTransactions = 0

// The begin marker is "XML tag aware".  If the begin marker begins with "<"
// and ends with ">", then it is assumed to be an XML tag and it will also match
// substrings where the ">" can be a whitespace char.
ls_BeginMarker = "<Transaction>"
ls_EndMarker = "</Transaction>"

do while li_Retval = 1
    loo_Sb.Clear()
    // The retval can have the following values:
    // 0: No more fragments exist.
    // 1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
    // -1: Error.
    li_Retval = loo_Fac.ReadNextFragment(li_FirstIteration,ls_BeginMarker,ls_EndMarker,"utf-8",loo_Sb)
    li_FirstIteration = 0

    if li_Retval = 1 then
        li_NumTransactions = li_NumTransactions + 1
        li_Success = loo_Xml.LoadSb(loo_Sb,1)
        // Your application may now do what it needs with this particular XML fragment...
    end if

loop

if li_Retval < 0 then
    Write-Debug loo_Fac.LastErrorText
end if

Write-Debug "numTransactions: " + string(li_NumTransactions)


destroy loo_Fac
destroy loo_Xml
destroy loo_Sb