Laravel get memory_limit error, when create folder from database

Question

I Want to create folders based on user_id inside /laravel/public/file/FolderUserXXX. If I have 1000 record user in database, i want to create 1000 folder inside the path (I need to create those folders because my server gets down and folders disappear).

I have created a job to execute that and I tried it in local with small data (is working) but in production I always get error Allowed memory size of xxxxxxxxxxxxx bytes exhausted (tried to allocate xxxxxxxx bytes). I add memory_limit until 4GB in php.ini but still get the problem.

Here is my code in app/Console/Commands

scenario:

createfolder job for user with status >= 3 and remark_restore is null. On every successful creation remark_restore will change to 1. I tried to limit 1 data per job and run every minute, but I still get error.

But if I change memory_limit to -1 only to run this job and then change it back to previous value, will be there any impact in the future for my web?

Any suggestion on how to prevent the memory_limit error?

public function handle()
{
    try {
        ini_set('max_execution_time', 300000);
        ini_set('memory_limit','4096M');
        $take = 1;

        $make_folder = User::where('status', '>=', '3')->wherenull('remark_restore')->limit($take)->get();

        foreach ($make_folder as $make_folder) {
            $path = public_path().'/file/'.$make_folder->id;

            if(!(file_exists($path))) {
                $data = User::where('id', $make_folder->id)->update(array('remark_restore' => '1' ));
                File::makeDirectory($path, $mode = 0777, true, true);
            }
        }

        $count = User::where('status', '>=', '3')->wherenull('remark_restore')->count();

        echo 'Pending : '.$count;
    }
    catch (\Exception $e) {
        \Log::warning($e);
    }
}

Answer 1

Okay, you are confusing things...

A correct approach here is to dispatch a job for each folder instead of having a single job to do every creation.

So, you should have a Job like CreateFolderForUser and in it have something like:

class CreateFolderForUser
{
    protected $user;

    public function __construct(User $user)
    {
        $this->user = $user;
    }

    public function handle()
    {
        try {
            $path = public_path("/file/{$this->user->id}");

            if(!(file_exists($path))) {
                File::makeDirectory($path, 0777, true, true);

                $this->user->remark_restore = 1;
                $this->user->save();
            }
        }
        catch (\Exception $e) {
            \Log::warning($e);
        }
    }
}

See that I have removed everything about iterating users, you just need to do this job that is "Create a folder for this user", nothing else...

Also, see that I have moved the remark_restore update after creating the directory, because on your previous code, if the creation fails, you have already updated the user's remark_restore to 1 but that is false, you had an error...

---

Now you should have a Scheduler that runs code or do it in a command so the scheduler calls that command every 5 minutes or whatever you want to. This code will see the database and dispatch this job with each user you want to have a folder created.

public function dispatchUserFolderCreation()
{
    $take = 1000;
    $users = User::where('status', '>=', '3')
        ->whereNull('remark_restore')
        ->limit($take)
        ->chunk(100, function (Collection $users) {
            foreach ($users as $user) {
                dispatch(new CreateFolderForUser($user));
            }
        });
}

You could dynamically pass the $take you want, if you are using a command, so the default is 1000 but if you want less or more (do not send 10000, do it in batches of 2000, not more), you can do it.