CODEWITHSUNDEEP
javascriptmeteor
torayeff
torayeff

Reputation: 9702

Running background tasks in Meteor.js

This is my scenario:

1. Scrape some data every X minutes from example.com
2. Insert it to Mongodb database
3. Subscribe for this data in Meteor App.

Because, currently I am not very good at Meteor this is what I am going to do:

1. Write scraper script for example.com in Python or PHP.
2. Run script every X minutes with cronjob.
3. Insert it to Mongodb.

Is it possible to do it completely with Meteor without using Python or PHP? How can I handle task that runs every X minutes?

Upvotes: 0

Views: 849

Answers (2)

Max Savin
Max Savin

Reputation: 252

I can suggest Steve Jobs, my new package for scheduling background jobs in Meteor.

You can use the register, replicate, and remove actions 

// Register the job

Jobs.register({ 
    dataScraper: function (arg) {
        var data = getData()

        if (data) {
            this.replicate({
                in: {
                    minutes: 5
                }
            });

            this.remove(); // or, this.success(data)
        } else {
            this.reschedule({
                in: {
                    minutes: 5
                }
            })
        }
    }
})

// Schedule the job to run on Meteor startup
// `singular` ensures that there is only pending job with the same configuration

Meteor.startup(function () {
    Jobs.run("dataScraper", {
        in: {
            minutes: 5
        }, 
        singular: true
    })
})

Depending on your preference, you can store the result in the database, as part of the jobs history, or remove it entirely.

Upvotes: 1

mritz_p
mritz_p

Reputation: 3098

There are Cron like systems such as percolate:synced-cron for Meteor. There, you could register a job using Later.js syntax similar to this example taken from the percolate:synced-cron readme file:

SyncedCron.add({
  name: 'Crunch some important numbers for the marketing department',
  schedule: function(parser) {
    // parser is a later.parse object
    return parser.text('every 2 hours');
  }, 
  job: function() {
    var numbersCrunched = CrushSomeNumbers();
    return numbersCrunched;
  }
});

If you want to rely on an OS level cron job, you could just provide an HTTP endpoint in your Meteor.js application that you could then access through curl at the chosen time.

Upvotes: 4

Related Questions