CODEWITHSUNDEEP
xmldelphidatamodule
Leo
Leo

Reputation: 1819

How can I search a large XML data set?

I have a DataModule with XML and I need do a search...

Unfortunately there are more than 300,000 records and I can't make a loop to check one-by-one.

Is it possible to make a query without using a database?

Is there another solution?

Upvotes: 4

Views: 942

Answers (6)

mjn
mjn

Reputation: 36664

For a SAX parser for Delphi, check this Stackoverflow question:

Is there a SAX Parser for Delphi and Free Pascal?

Upvotes: 0

Despatcher
Despatcher

Reputation: 1725

You don't say how you are implementing the datasource. I have used TClientDataSet connected via TXMLTransformProvider (OK not for 300K records) but for a few thousand. and simply setting the filter and filtered properties seems to "Query" it just fine...

Or have I missed something?

Upvotes: 1

Alister
Alister

Reputation: 6837

I think it is probably important to ask Why are you using XML to store 300k records?. As XML is not the most efficient format to manipulate data with.

If you're stuck with XML then you might be best to read the XML file into some sort of database (you might get away with an in memory table, but then again you might run out of memory). I think if you use a TXMLDocument object to load the XML file into you'll either have a serious performance issue or run out of memory (I had trouble when I was playing with a 250k record xml file awhile back).

You might be able to use the MSXML DOM directly (you can probably import the type library) or use SAX which will allow you to parse it sequentially, neither of which I have had much experience with.

Upvotes: 2

David Freeman
David Freeman

Reputation:

There are a number of in-memory databases that may be useful. At least you could then index and query the data as required. One I know of is from components4developers.com. David

Upvotes: 1

ChrisW
ChrisW

Reputation: 56123

You can search using something like XPath: however, that would just mean that the XPath implementation does the searching on your behalf (which doesn't necessarily improve performance).

Upvotes: 3

Mason Wheeler
Mason Wheeler

Reputation: 84640

XML's fine for small amounts of information, but for a dataset that big, a relational database is really the only sane choice, especially if you need to be able to query it.

Upvotes: 3

Related Questions