How to use multi threading in Quartz.net

Question

I am planning to use quartz.net for processing files.

• I will receive 1 or more files every hour and each file will have 1000’s of rows for different countries.
• I want to read those rows, validate, process and insert it into multiple tables.
• I want to pick 1 file and for each distinct country in that file I like to create a thread. Such that 1 thread will handle all data for 1 country.
• At a given time there should not be more than 5 threads.

Now how do I define this in quartz.net? The below is the code I have in which I am going through each file by file and each row by row and I am not doing any multithreading

Scheduling a job

var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "MyScheduler";
properties["quartz.threadPool.threadCount"] = "5";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
IScheduler s = sf.GetScheduler();
if (!s.IsStarted)
    s.Start();

var jobKey = new JobKey("UniqueJobName", "BatchProcess");
if (s.GetJobDetail(jobKey) != null)
    return "Error! Already running";

IJobDetail jobDetail = JobBuilder.Create<CountryProcessJob>()
    .WithIdentity(jobKey)
    .Build();

ITrigger trigger = TriggerBuilder.Create()
    .WithIdentity("UniqueTriggerName", "BatchProcess")
    .StartAt(DateTime.Now.AddSeconds(1))
    .Build();

s.ScheduleJob(jobDetail, trigger);

Job

public class CountryProcessJob : IJob
{
    public void Execute(IJobExecutionContext context)
    {
        While() // TillAllFilesAreProcessed
        {
        // Read A File
            While() //for each row in file
            {
                // validate
                // Process
                // Insert
            }
        }
    }
}

Should I have a main Job to loop through the Files one at time and then with in the Job should I define multiple Jobs for processing each country? Is this how to achieve multi threading in quartz?

Schedule the main Job

var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "MyScheduler";
properties["quartz.threadPool.threadCount"] = "5";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
IScheduler s = sf.GetScheduler();
if (!s.IsStarted)
    s.Start();

var jobKey = new JobKey("UniqueJobName", "BatchProcess");
if (s.GetJobDetail(jobKey) != null)
    return "Error! Already running";

IJobDetail jobDetail = JobBuilder.Create<FileProcessJob>()
    .WithIdentity(jobKey)
    .Build();

ITrigger trigger = TriggerBuilder.Create()
    .WithIdentity("UniqueTriggerName", "BatchProcess")
    .StartAt(DateTime.Now.AddSeconds(1))
    .Build();

s.ScheduleJob(jobDetail, trigger);

Main Job

public class FileProcessJob : IJob
{
    public void Execute(IJobExecutionContext context)
    {
        while() // TillAllFilesAreProcessed
        {
        // Read A File
            foreach(var eachCountry in  file) //for each row in file
            {
                // Create a Job
                var jobKey = new JobKey("UniqueCountryJobName", "BatchProcess");
                if (s.GetJobDetail(jobKey) != null)
                    return "Error! Already running";

                IJobDetail jobDetail = JobBuilder.Create<CountryProcessJob>()
                    .WithIdentity(jobKey)
                    .Build();

                ITrigger trigger = TriggerBuilder.Create()
                    .WithIdentity("UniqueCountryTriggerName", "BatchProcess")
                    .StartAt(DateTime.Now.AddSeconds(1))
                    .Build();

                s.ScheduleJob(jobDetail, trigger);              
            }
        }
    }
}

Create multiple Jobs for each country

public class CountryProcessJob : IJob
{
    public void Execute(IJobExecutionContext context)
    {
        // For each row for that country 
        // validate
        // Process
        // Insert
    }
}

So should I create multiple Jobs to implement multiple thread to run at same time? Please help me to run 5 concurrent threads processing each distinct country with in a file.

Answer 1

You can create a separate static class, and call it in the execute method. An static class have a fixed space in the memory.

public class ReadCountry : IJob
{
    public void Execute(IJobExecutionContext context)
    {
       CountryProcessJob.DoIt();
    }
}

CountryProcessJob class

public static class CountryProcessJob
    {
        public static void DoIt()
        {
            While() // TillAllFilesAreProcessed
            {
            // Read A File
                While() //for each row in file
                {
                    // validate
                    // Process
                    // Insert
                }
            }
        }
    }

Answer 2

Quartz.Net Jobs run in a seperate thread. so if you want to configure the max nuber of threads take a look at this answer: https://stackoverflow.com/a/4108795/745011