CODEWITHSUNDEEP
phpmysqlasynchronousscalabilitymessage-queue
William Entriken
William Entriken

Reputation: 39253

Asynchronous messaging in PHP/MySQL?

I'm working on image hosting website (a competitor to Gallery) and need to step up my game. New photos are uploaded or placed on the server via FTP and are indexed (quick), then they are later thumbnailed (slow).

  1. The most simple implementation is to allow users to upload, then index and thumbnail while they wait (slow web page load). An admin can log in to index and then thumbnail other files that were manually uploaded (slow page load that refreshes and can take hours).

  2. The implementation I have now is user uploads cause indexing while they wait, and sometimes (randomly) the whole site indexes to piggyback on top of a normal pageload (cause additional marginal delay for the user). Pages on the site refer to the URL where the thumbnail should be if it existed. If the user requests a nonexistent thumbnail it is created while they wait ("lazy thumbnailing").

The benefit of 2 is that on a multicore system, all cores are used. The downside is that the first time a page of new photos loads 30x50 MB is needed and the page times out, with some images completing and others working on subsequest loads.

SO THE QUESTION IS what is the right way to implement task processing here? Bonus points if it scales to multiple servers on a shared database.

One idea follows (may be drivel):

I thought about making a job table in the DB: 

id INT NOT NULL AUTO_INCREMENT, 
priority INT NOT NULL, 
worker INT, 
workstarted DATETIME, 
func CHAR(10) NOT NULL, 
args VARCHAR NOT NULL, 
PRIMARY KEY(id),
INDEX(priority),
UNIQUE(func,args) -- prevent duplicate jobs if user does F5

Then when work needs to get done, insert a row and create a worker thread (if enough threads don't already exist). Then wait for the response and proceed if the job was completed successfully, or timeout after 5 seconds. If there was a timeout, use AJAX to get the content to the user when it is available.

Things I don't know how to do in this approach are: how to coordinate if enough worker threads exist (use a table and a heart beat?), how the worker sends the response (delete the job when complete and write to another table if failed?), how the requestor waits for the response efficiently or times out (anything better than polling the DB?). Depending on the approach to these, the system can scale with workers running on different servers.

Upvotes: 2

Views: 2006

Answers (1)

N.B.
N.B.

Reputation: 14061

I suggest using ZeroMQ for your problem.

It's a socket framework with bindings in multiple languages (including PHP). You can develop a system of workers without the use of the database, and you can scale the work across multiple nodes (physical machines that do the work).

Here's an introduction to the 0MQ.

Here's the documentation with PHP examples.

Here's the 0MQ guide at github.

Subjective opinion: it simply rocks to use 0MQ. It's incredibly fast and there's tons of examples, combined with an excellent guide.

Upvotes: 3

Related Questions