(Delphi ActiveX) Efficiently Process a Huge XML File

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).

Note: This example requires Chilkat v9.5.0.80 or greater.

Chilkat for Delphi Downloads Chilkat ActiveX DLL for Delphi Chilkat non-ActiveX DLL for Delphi * The examples here use the ActiveX DLL.

uses
    Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
    Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, Chilkat_TLB;

...

procedure TForm1.Button1Click(Sender: TObject);
var
fac: TCkFileAccess;
success: Integer;
xml: TChilkatXml;
sb: TChilkatStringBuilder;
firstIteration: Integer;
retval: Integer;
numTransactions: Integer;
beginMarker: WideString;
endMarker: WideString;

begin
// This example shows a way to efficiently process a gigantic XML file -- one that may be too large
// to fit in memory.  
// 
// Two types of XML parsers exist: DOM parsers and SAX parsers.

// A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
// and the application has the luxury of interacting with the XML in a convenient, random-access
// way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
// huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

// A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.  
// Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
// 
// The technique described here is a hybrid.  It streams the XML file as unstructured text
// to extract fragments that are individually treated as separate XML documents loaded into
// the Chilkat Xml parser.
// 
// For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
// 
// <Transactions>
//     <Transaction id="1">
//          ...
//     </Transaction>
//     <Transaction id="2">
//          ...
//     </Transaction>
//     <Transaction id="3">
//          ...
//     </Transaction>
// ...
// </Transactions>

// In the following code, each <Transaction ...> ... </Transaction>
// is extracted and loaded separately into an Xml object, where it can be manipulated
// independently.  The entire XML file is never entirely loaded into memory.

fac := TCkFileAccess.Create(Self);

success := fac.OpenForRead('qa_data/xml/transactions.xml');
if (success = 0) then
  begin
    Memo1.Lines.Add(fac.LastErrorText);
    Exit;
  end;

xml := TChilkatXml.Create(Self);
sb := TChilkatStringBuilder.Create(Self);
firstIteration := 1;
retval := 1;
numTransactions := 0;

// The begin marker is "XML tag aware".  If the begin marker begins with "<"
// and ends with ">", then it is assumed to be an XML tag and it will also match
// substrings where the ">" can be a whitespace char.
beginMarker := '<Transaction>';
endMarker := '</Transaction>';

while retval = 1 do
  begin
    sb.Clear();
    // The retval can have the following values:
    // 0: No more fragments exist.
    // 1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
    // -1: Error.
    retval := fac.ReadNextFragment(firstIteration,beginMarker,endMarker,'utf-8',sb.ControlInterface);
    firstIteration := 0;

    if (retval = 1) then
      begin
        numTransactions := numTransactions + 1;
        success := xml.LoadSb(sb.ControlInterface,1);
        // Your application may now do what it needs with this particular XML fragment...
      end;
  end;

if (retval < 0) then
  begin
    Memo1.Lines.Add(fac.LastErrorText);
  end;

Memo1.Lines.Add('numTransactions: ' + IntToStr(numTransactions));
end;