Laravel 504 gateway timeout

Question

I have a script in Laravel platform which process thousands of records , and it's timing out issue. I want to avoid update settings on server like php.ini file.

Is there any way to chunk all records in small pieces and use them in further processing until all records will be used.

I have thousands of records which are getting Price from another server.

So,

1-10 records works fine.

1-100 records works fine.

More than 500 records script stuck and gives error like below.

405 nginx server not allowed

My Route is like :- example.com/updateAllKeywords

Below code is getting keyword to fetch keywords'price from eBay API

        $keywordCollection = Keywords::all();

        Keywords::chunk(100, function ($keywordCollection) use($request,$keywordCollection,$defaultStartDate,$ignoredItems,$response) {
            foreach ($keywordCollection->toArray() as $keyword_result) {
                $response = $this->apirequest($request, $keyword_result['id'], $keyword_result, $defaultStartDate, $ignoredItems);
            }
        });

This is api request function:

 /**
 * Call Finding API for Active List items 
 *
 * @return API Response
 */
public function apirequest(Request $request, $id, $keyword_result, $defaultStartDate, $ignoredItems) {

    try {

        /* Request */
        $request = new Types\FindItemsAdvancedRequest();

        /* 1) default 100 items being displayed without pagination */
        $request->paginationInput = new Types\PaginationInput();
        $request->paginationInput->entriesPerPage = Config::get('constants.paginationoptions.entries_per_page');

        /* 2) ItemFilter for three conditions NEW ,NEWOTHER,USED */
        $itemFilterForConditions = new Types\ItemFilter();
        $itemFilterForConditions->name = 'Condition';
        $itemFilterForConditions->value = ['1000', '1500', '3000'];

        /* 3) ItemFilter for hide duplicate items */
        $itemFilterduplicate = new Types\ItemFilter();
        $itemFilterduplicate->name = 'HideDuplicateItems';
        $itemFilterduplicate->value[] = 'true';

        /* 4) pull items before given date / days */
        $itemFilterStartDateFrom = new Types\ItemFilter();
        $itemFilterStartDateFrom->name = 'StartTimeFrom';
        $itemFilterStartDateFrom->value[] = date('Y-m-d\TH:i:s\Z', mktime(0, 0, 0, date('m'), date('d') - $defaultStartDate['defaultStartDate'], date('y')));

        /* Exclude Keywords/MPN by set Priced Percentage */
        if (!empty($keyword_result['price']) && ($defaultStartDate['exclusionPercentage'] > 0)) {
            $percentageAmount = $keyword_result['price'] * ($defaultStartDate['exclusionPercentage'] / 100);
            $percentageAmount = floor($percentageAmount).substr($percentageAmount-floor($percentageAmount),1,2+1);
            $itemFilterPriced = new Types\ItemFilter();
            $itemFilterPriced->name = 'MinPrice';
            $itemFilterPriced->value[] = $percentageAmount;
        }

        $itemFilter = (isset($itemFilterPriced)) ? [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom, $itemFilterPriced] : [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom];

        $request->itemFilter = $itemFilter;

        /* 5) Request KEYWORD/MPN to filter Items Active Lists */
        $searchKeyword = (!empty($keyword_result['exclusions'])) ? $keyword_result['keyword'] . " -" . "(" . trim($keyword_result['exclusions']) . ")" : $keyword_result['keyword'];
        $request->keywords = $searchKeyword;

        /* 6) Request For sorting Items by Lowest To Highest Itemprice Price + Shipping Price */
        $request->sortOrder = 'PricePlusShippingLowest';

        /* 7) Request For Keyword Search Including the Item Descriptions,Title and Subtitle */
        $request->descriptionSearch = ($defaultStartDate['advancedExclusions'] == 1) ? true : false;

        $service = $this->index();

        /* Response returned by $request */
        $response = $this->apiresponse($request, $service, $id, $keyword_result['keyword'], $ignoredItems);

        return $response;

    } catch (\Exception $e) {
        echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
        exit;
    }
}


public function apiresponse($request, $service, $id, $keyword, $ignoredItems) {

            try {

                $response = $service->findItemsAdvanced($request);

                /* fetch all ignored Items */

                $newItems = array();
                $newOtherItems = array();
                $usedItems = array();
                $errors = "";

                if ($response->ack == "Success") {
                    $results = $response->searchResult->item;

                    ob_start();

                    foreach ($results as $item) {

                        if ($item->listingInfo->buyItNowAvailable == '1') {
                            $itemPrice = $item->listingInfo->buyItNowPrice->value;
                        } elseif (($item->listingInfo->listingType == 'Auction') && ($item->sellingStatus->currentPrice->value == 0)) {
                            $itemResponse = $this->getItemDetails($item->itemId);
                            $itemPrice = isset($itemResponse) ? $itemResponse : 0;
                        } else {
                            $itemPrice = $item->sellingStatus->currentPrice->value;
                        }

                        /* check for ignored items */
                        if (!in_array($item->itemId, $ignoredItems)) {

                            /* check for NEW Condition (conditionId = 1000) */
                            if ($item->condition->conditionId == '1000') {
                                $newItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for NEW OTHER Condition (conditionId = 1500) */ elseif ($item->condition->conditionId == '1500') {
                                $newOtherItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for USED Condition (conditionId = 3000) */ elseif ($item->condition->conditionId == '3000') {
                                $usedItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                        }

                        ob_flush();
                    }

                    $newItems = array_slice($newItems, 0, 20);
                    $newOtherItems = array_slice($newOtherItems, 0, 20);
                    $usedItems = array_slice($usedItems, 0, 20);

                    $insertItems = array_merge($newItems, $newOtherItems, $usedItems);

                    /* clean up items */
                    Items::where('keywordId', $id)->delete();

                    /* save pulled items */
                    $items = Items::insert($insertItems);
                } elseif ($response->ack == "Failure") {
                    foreach ($response->errorMessage->error as $error) {
                        $errors .= "%s: %s\n\n" . $error->severity === Enums\ErrorSeverity::C_ERROR ? '<b>Error</b>&nbsp;' : '<b>Warning</b>&nbsp;' . $error->message;
                    }
                }

                /* save API log fot the keyword */
                $this->setapilog($request, $response, $keyword);

                if (!empty($errors)) {
                    echo json_encode(array('ack' => 'Failure', 'msg' => $errors));
                    exit;
                } else {
                    return true;
                }
            } catch (\Exception $e) {
                echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
                exit;
            }
        }

Any information on this would be greatly appreciated. Thanks!

Answer 1

Create this has a job Laravel Queues Put your function in the handle method.

$response = $this->apiRequest(...);

Dispatch your job from say your controller

foreach ($keywordCollection as $key => $keywordResult) {
   ProcessJob::dispatch($keywordResult)->delay($key * 2); 
}

This way you are implementing exponential backoff such that as your array increases you increase your wait time to make the next api request.

Answer 2

@kinjal jethva

Problem:

You are using Database Query into foreach loop. You are calling your apirequest() methods into foreach loop.

When yourr $keywords array have 500 records.

You are calling

    $keyword_result => 500 times. 

    $defaultStartDate = 500 times

In total you are calling your 1000 database query in one single http request.

Solution:

Calling $defaultStartDate inside your apirequest() function is totally useless. Instead call it outside and pass value as a parameter. So it will call once.

$keyword_result can be called better way.

$defaultStartDate = Settings::select('defaultStartDate', 
'exclusionPercentage', 'advancedExclusions')->where('currentEnv', 1)->get()-
>first()->toArray();

$keywordValues = array_keys($keywords);

$keywordCollection = Keywords::whereIn('id', $keywordValues)-
>select('keyword', 'exclusions', 'price')->first()->toArray();

foreach ($keywordCollection as $keyword_result) {

$response = $this->apirequest($request, $keyword_result->id, 
$keyword_result, $defaultStartDate);

}

/**
  * Call Finding API for Active List items
  *
  * @return API Response
  */
function apirequest(Request $request, $id, $keyword_result, $defaultStartDate)
{
 // your code here.
}

This is just a structure of code you can use.