Efficient Incremental Update Strategy for Large Datasets in Azure Blob Storage and Azure Search

Question

I'm working on a project that involves storing and syncing large datasets using Azure Blob Storage and Azure AI Search. I'm facing performance challenges, and I’d love some advice on best practices for handling this efficiently.

Current Approach:

Storage: Data is stored in a flattened JSON structure in Azure Blob Storage.

Sync Process:

• Retrieve JSON blobs and sort by the Last-Modified timestamp in metadata.
• Sync only the top 1000 updated records due to API limitations.
• Save the last synced timestamp to optimize future operations.

Challenges:

• No Built-in Sorting: Azure Blob Storage doesn’t support server-side sorting by metadata, requiring local sorting.
• Performance Issues: Fetching metadata for all blobs is inefficient with large datasets (hundreds of gigabytes).
• Timestamp Consistency: Ensuring consistent sync when blobs share identical timestamps.

Questions:

• How can I efficiently handle large datasets to perform incremental updates without retrieving all metadata?
• Are there alternative strategies or tools for managing sorting and pagination?
• How can I ensure synchronization consistency when timestamps are identical?

What I've Tried:

• Searching online for strategies but found limitations in existing solutions.
• Tested some Azure services but faced scalability issues.

Any insights, tools, or strategies would be greatly appreciated!