How can I load a large dataset into memory on ASP.NET Core?

Question

Context:

• I am building an API to support a search query suggestion feature.
  • For example, a GET request to "/api/values/micro" will return suggestions such as "microwave", "micromanager", etc..
• I am using ASP.NET Core to construct this API (hosted on Azure).
• I am going to use a trie data structure to quickly fetch search results.
• I have a file with about 7 million lines of data. Each line is the title of a wikipedia article.
• I am going to seed this trie with data from this file.
• Instead of embedding this data file into my Visual Studio project and deploying it to Azure, I want to upload this data file to Azure blob storage.
• At initialization time, I need my ASP.NET Core API to fetch this file from blob storage and load it into memory as a trie so that subsequent HTTP requests can retrieve data from this trie quickly (makes no sense to do this for each request).

Question:

How can I load this data at initialization time, before HTTP requests start coming in? Do I do it in my controller's constructor? In my Startup.cs? I would appreciate some guidance here.

Thanks!

Answer 1

I would do this in Configure method. Code should be something like this, then webapp will be available until data are pre-loaded.

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
    {
        using (var serviceScope = app.ApplicationServices.GetRequiredService<IServiceScopeFactory>().CreateScope())
        {
            //service should has been configured in ConfigureServices step
            //Create method to check if data loaded. If not then load them.
            serviceScope.ServiceProvider.GetService<ApplicationDbContext>().CheckDataLoaded();
        }
        //more steps here..

Answer 2

If you want to ensure that this data is loaded before calling any code in your controllers, then you could put it in the Startup class which is used for bootstrapping your application.