CODEWITHSUNDEEP
hl7-fhirhapi-fhir
JoyfulPanda
JoyfulPanda

Reputation: 1047

Looking for a faster way to export Resources in HAPI FHIR

Following the documentation on $export API (link), it is possible to export FHIR data, e.g. Patient, Observation, etc. However, this process is very slow (take many days) if the Resources to be exported are substantial in size. Querying database, serializing, writing down the Resources into NDJSON files, then communicate the data over Internet, etc. is an expensive process.

Is there any other way to export FHIR data? E.g. a more efficient technique? low-level exporting (some customized programs to export data directly from the database on the server)? Or perhaps increase the server capacity (CPU, memory, process priority/multi-threading configuration, etc.)

Feel free to suggest whatever solutions. The ultimate goal is to have NDJSON files (after the export) in order to ingest them into other 3rd party data-warehouses for further analysis.

Environment

Upvotes: 1

Views: 813

Answers (1)

Mirjam Baltus
Mirjam Baltus

Reputation: 2299

If you have access to the database, you can write whatever program you want to get the export done, to any format you would like - provided you know how to convert the FHIR data. Or maybe you can use existing database tools that can dump data to files.

If you really need to go through the REST API, you can also request the resources with regular searches and write a program that iterates over the result Bundles to get all the data out and into NDJSON files. However, this will lead to a lot of calls and network load. One of the reasons for $export is to prevent clogging the system/network, and have the server be available for other requests while exporting. Maybe after the initial export you can use the $export parameters to only export changed data, which should be quicker than the whole set.

Upvotes: 0

Related Questions