CODEWITHSUNDEEP
c#linq-to-json
Allan Xu
Allan Xu

Reputation: 9298

Need to run LINQ queries on several hundred JSON files in child folders

Using c#:

I have a few hundreds of JSON files in nested folders in file system. I need to run LINQ queries within the data in files and find the JSON files that their JSON data matches certain crieria.

I can simly serialize all the JSON files in a List, then run my LINQ query on the array. However the approach takes lots of memory since I am reading all data from disk.

Is there any way to run my LINQ query on JSON files in file system without loading all of them in memory?

Upvotes: 2

Views: 466

Answers (2)

Alen.Toma
Alen.Toma

Reputation: 4870

Ok noSql will not work for you, but here i found a diffrent solution that you could use.

Insert the files in sql db then you could simply do a select stats on them.

Here is an one way of doing it 

-- Load file contents into a variable
SELECT @json = BulkColumn
 FROM OPENROWSET (BULK 'C:\JSON\Books\book.json', SINGLE_CLOB) as j

-- Load file contents into a table 
SELECT BulkColumn
 INTO #temp 
 FROM OPENROWSET (BULK 'C:\JSON\Books\book.json', SINGLE_CLOB) as j

And using Json_Value to read

SELECT FirstName, LastName,
JSON_VALUE(jsonInfo,'$.info.address[0].town') AS Town
FROM #temp
WHERE JSON_VALUE(jsonInfo,'$.info.address[0].state') LIKE 'US%'
ORDER BY JSON_VALUE(jsonInfo,'$.info.address[0].town')

Here is how to import json files

https://learn.microsoft.com/en-us/sql/relational-databases/json/import-json-documents-into-sql-server?view=sql-server-2017

And here is how to do a where sats in them.

https://learn.microsoft.com/en-us/sql/t-sql/functions/json-value-transact-sql?view=sql-server-2017

Upvotes: 0

verbal
verbal

Reputation: 171

You should be able to stream the data as described in the following posts or something similar. This should help with the memory issues. How to parse huge JSON file as stream in Json.NET?, Parsing large json file in .NET

Upvotes: 2

Related Questions