CODEWITHSUNDEEP
c#sql-serverdatabase
cj devin
cj devin

Reputation: 1375

SQLServer:How to speed up Select and Insert query in loops of statements using c#

I am trying to insert into table from array of Json as well as select records from SQL Server DB table. When executing the below method it is almost taking more than 10 minutes.

public async Task CreateTableAsync(string formsJson, string connectionString)
{
    SqlConnection con = new SqlConnection(connectionString);
    List<FormsJson> listOfformsJson = JsonConvert.DeserializeObject<List<FormsJson>>(formsJson);
    foreach (var form in listOfformsJson)
    {
        string formId = Guid.NewGuid().ToString();
        //insert into forms Table
        string formQuery = "insert into Forms([FormId]) values(@FormId)";
        using (var cmd = new SqlCommand(formQuery, con))
        {
            cmd.CommandTimeout = 120;
            //Pass values to Parameters
            cmd.Parameters.AddWithValue("@FormId", formId);
            if (con.State == System.Data.ConnectionState.Closed)
            {
                con.Open();
            }
            cmd.ExecuteNonQuery();
        }

        //relationship between forms and ETypes,get all the eTypes and fill
        foreach (var typeOf in form.TypeOf)
        {
            //get all the eTypeIds for this typeof field
            string query = "select Id from ETypes Where TypeOf = @typeOf";
            List<string> eTypeIdList = new List<string>();
            using (var sqlcmd = new SqlCommand(query, con))
            {
                sqlcmd.CommandTimeout = 120;
                //Pass values to Parameters
                sqlcmd.Parameters.AddWithValue("@typeOf", typeOf);
                if (con.State == System.Data.ConnectionState.Closed)
                {
                    con.Open();
                }
                SqlDataReader sqlDataReader = sqlcmd.ExecuteReader();
                while (sqlDataReader.Read())
                {
                    string eTypeId = sqlDataReader[0].ToString();
                    eTypeIdList.Add(eTypeId);
                }
                sqlDataReader.Close();
            }
            //insert into Forms ETypes Relationship
            string fe_query = "";
            foreach (var eTypeId in eTypeIdList)
            {
                fe_query = "insert into Forms_ETypes([Form_Id],[EType_Id]) values (@Form_Id,@EType_Id)";
                if (con.State == System.Data.ConnectionState.Closed)
                {

                    con.Open();
                }
                using (var fesqlcmd = new SqlCommand(fe_query, con))
                {
                    fesqlcmd.CommandTimeout = 120;
                    //Pass values to Parameters
                    fesqlcmd.Parameters.AddWithValue("@Form_Id", formId);
                    fesqlcmd.Parameters.AddWithValue("@EType_Id", eTypeId);
                    fesqlcmd.ExecuteNonQuery();
                }
            }
        }
    }
}

Outer foreach(...listofformsJson) loop more than hundreds of records.
And same for the inner loop around hundreds of rows.

In between commandTimeout, keeping the open connection with server statements. Any help to optimize the time and remove/add ADO statements.

Upvotes: 1

Views: 1135

Answers (1)

mjwills
mjwills

Reputation: 23819

The primary issue here is that you are pulling all of the data out of the database and then, row by row, inserting it back in. This is not optimal from the database's point of view. It is great at dealing with sets - but you are treating the set as lots of individual rows. Thus it becomes slow.

From a set-based standpoint, you have only two statements that you need to run:

  1. Insert the Forms row
  2. Insert the Forms_ETypes rows (as a set, not one at a time)

1) should be what you have now:

insert into Forms([FormId]) values(@FormId)

2) should be something like:

insert Forms_ETypes([Form_Id],[EType_Id]) SELECT @FormId, Id from ETypes Where TypeOf IN ({0});

using this technique to pass in your form.TypeOf values. Note this assumes you have fewer than 500 entries in form.TypeOf. If you have many (e.g. greater than 500) then using a UDT is a better approach (note some info on UDTs suggest that you need to use a stored proc, but that isn't the case).

This will enable you to run just two SQL statements - the first, then the second (vs possibly thousands with your current solution).

This will save time for two reasons:

  1. The database engine didn't need to pass the data over the wire twice (from your DB server to your application, and back again).
  2. You enabled the database engine to do a large set based operation, rather than lots of smaller operations (with latency due to the request-response nature of the loop).

Upvotes: 3

Related Questions