CODEWITHSUNDEEP
multithreadingperlhttp
john doe
john doe

Reputation: 403

Thread "tagging" in Perl for async HTTP request

I know there are other classes I can use, however I want to use LWP. The accepted answer here (How do I make parallel HTTP requests in Perl, and receive them back in order?) does what I want however I need to "tag" the threads so that I can match the request with the response. I have modified the code to the following:

#Setup and query the DB:

#hold all response\id pairs
my @response_bundles; 
while (my @row = $query->fetchrow_array()) {
                my $id = $row[0];
                my $url = $row[1];

                push @threads, async {
                        my @container_hash;
                        $container_hash[0] = @row;
                        $container_hash[1] = $ua->get($url); 

                        push @response_bundles, @container_hash;
                };
}       

#Once this loop terminates all threads are done
for my $thread (@threads) {
                $thread->join;
}

#Since all threads are completed we can now sort through the response bundles
for my $response_bundle (@response_bundles) {
                print "test\n";
}

My goal is to kick off a bunch of HTTP requests and store their ($id, $response) pair in an array. I push them all into an async process and the sub in async{} SHOULD be doing this (but it's not). I loop through the thread array and once that is complete all threads should be done. I then go through my bundle and do things however the "print test" is never firing. Am I thinking about this wrong?

EDIT:

Per the comment I tried returning the value but this does not work either

while (my @row = $query->fetchrow_array()) {
                my $id = $row[0];
                my $url = $row[1];

                push @threads, async {
                        my @container_hash;
                        $container_hash[0] = @row;
                        $container_hash[1] = $ua->get($url); 

                        return @container_hash;
                };
}       

#Once this loop terminates all threads are done
for my $thread (@threads) {
                my @container;
                @container = $thread->join;
                print $container[1];
}

Upvotes: 1

Views: 189

Answers (1)

gangabass
gangabass

Reputation: 10666

You need to return the data you need from your thread so main program can process it:

while (my @row = $query->fetchrow_array()) {
                my $id = $row[0];
                my $url = $row[1];

                push @threads, async {
                        my @container_hash;
                        $container_hash[0] = \@row; #I think you need this
                        $container_hash[1] = $ua->get($url); 

                        return \@container_hash; #and here we return reference to our data
                };
}       

#Once this loop terminates all threads are done
for my $thread (@threads) {

                my $container = $thread->join;
                print $container->[1], "\n"; #this is a reference that's why we use ->
}

Upvotes: 1

Related Questions