Long php process hangs the serve. Bad project design?

Question

Hi I need suggestions for my research project.

I'm building a database that read RSS feeds produced by google alerts and saves the results in a database for latter categorization. I'm using Wordpress and pods framework to handle the database and UI.

I have 4 objects (pods) with their own tables:

• Resources, it's the site data taken from an alert feed.
• Sources, it's the domain of the site, like stackoverflow.com
• Feeds, with the alert query and rss url.
• Topics, the main topics under which the other objects are categorized.

The program flow is shortly this:

• For every topic take the feeds.
• For every feed load the the rss xml.
• For every entry URL in the rss, if newer than last check, control if the domain is already saved in the Sources object.
• If the source it's present check if the URL is saved in the Resource object.
• If the resource is present (that is we already have the data for this URL) add the current topic and feed of the loop to the resource (if not present).
• If the resource is not present save the resource with some data and the current topic and feed.
• If the source is not present save the source with the current topic.

This way I will have a bunch of resources with linked feeds and topics, and the relative sources with their topics.

The problem is that the data goes up to the hundreds very fast and in one month I already reached more than 1500 records for resources.

So now every time I run the script, since for every new entry it has to compare it with all the previous ones, the script sistematically hangs.

So I need a way to make it more efficient or to avoid the problem splitting the process.

Since the script is called via Ajax, I thought this flow would work:

• Ask the server for the topics/feeds structure.
• For every feed in every topic ask the server to load the XML and pass it back has an array.
• Then in the front end for every entry send a compare and save call.

Of course the drawback is that I will got plenty of calls.

Another technique I heard about is to flush the data during the server process, since I understood this should trick the server time limit to reset. But I'm not sure I understood well.

Of course the best solution would be to rebuild everything using more specific code instead than two general purpose abstraction layers. But I'm really short in time!

Edit: code here https://github.com/bakaburg1/overseer

Answer 1

I think you're loading too much data at once. The whole point of AJAX is to load data a bit at a time, on-demand as you need it. It doesn't really matter that you have a lot of HTTP calls to the server as long as the scripts you're running to return data don't hang like yours is doing.

I would suggest questioning whether you really need to return all this data at once. If you think you do, a front-end redesign is probably in order. Don't load so much data at once on a single page that it takes forever to load. If you do, caching and other things will help, but it will never truly fix the problem.