403 Forbidden error is received on file upload

Question

I'm trying to upload a file in my Laravel 9 application. On the localhost, it works just fine, however, the same action fails when I try it from my application running on an Amazon Linux EC2.

What I know:

• The action does not throw any exceptions, it just receives an HTTP 403 error. (I've enabled debug in my .env)
• I'm only receiving this error in places where I'm trying to upload an image. Other controllers work fine.

What I suspect the issue is:

• I believe the issue is with Apache/server configuration. I've added a dd() call in the first line of the update() function and it doesn't even get there. The 403 error is thrown before that. see below.

What I've done to troubleshoot:

• Made sure the S3 bucket policy is ok (the same controller works fine on localhost)
• Made sure the file permissions are ok (see screenshot below)
• Made sure the authorize() function inside the request class returns true
• looked at the error logs and don't see anything relevant in there either.
• Made sure file_uploads = On, upload_max_filesize = 4M and max_file_uploads = 20 inside php.ini

Here is what my controller looks like (the authorize() function inside UpdateContactRequest returns true):

    //https://myapp.com/contacts/1
    //App\Http\Controllers\ContactController
    public function update(UpdateContactRequest $request, Contact $contact)
    {
        dd($request); //This does not get executed. The 403 error happens before reaching this. 

        $contact->update($request->all());

        $this->uploadAvatar($request, $contact);

        Flash::success('Contact updated successfully.');

        return redirect(route('dealers.contacts.index', $contact->dealer->id));
    }

Here is the uploadAvatar function I use to do the upload:

    private function uploadAvatar(Request $request, Contact $contact)
    {
        if ($request->hasFile('avatar')) {
            try {
                $contact
                    ->addMediaFromRequest('avatar')
                    ->sanitizingFileName(function ($fileName) {
                        return strtolower(str_replace(['#', '/', '\\', ' '], '-', $fileName));
                })
                ->toMediaCollection('avatars');
            } catch (\Spatie\MediaLibrary\MediaCollections\Exceptions\FileUnacceptableForCollection $e) {
                Flash::error($e->getMessage());
            }
        }
    }

How the routes are defined:

Route::get('/', function () {
    return view('welcome');
});

Route::get('/test', App\Http\Controllers\TestController::class);
Route::get('/embed-iframe/{uuid}', [App\Http\Controllers\EmbedController::class, 'iframe']);
Route::get('/embed-js/{uuid}', [App\Http\Controllers\EmbedController::class, 'js']);
Auth::routes();

Route::middleware('admin')->group(function () {
    Route::get('/home', [App\Http\Controllers\HomeController::class, 'index'])->name('home');
    Route::resource('dealers', App\Http\Controllers\DealerController::class);
    Route::post('refreshDealerCRMData', [App\Http\Controllers\DealerController::class, 'refreshCRMData']);
    Route::post('loadCRMView', [App\Http\Controllers\DealerController::class, 'loadCRMView']);
    Route::resource('cms', App\Http\Controllers\CmsController::class);
    Route::resource('crms', App\Http\Controllers\CrmController::class);
    Route::resource('leads', App\Http\Controllers\LeadController::class);
    Route::resource('contacts', App\Http\Controllers\ContactController::class);
    Route::resource('attachment-categories', App\Http\Controllers\AttachmentCategoryController::class);
    Route::resource('CRMAttachments', App\Http\Controllers\CRMAttachmentController::class);
    Route::resource('dealers.leads', App\Http\Controllers\DealerLeadController::class)->scoped([
        'dealers' => 'dealer.id',
        'leads' => 'lead.id',
    ]);
    Route::resource('dealers.contacts', App\Http\Controllers\DealerContactController::class)->scoped([
        'dealers' => 'dealer.id',
        'contacts' => 'contact.id',
    ]);
});

Here is a screenshot of my root directory (/var/www/html):

Here are my virtual hosts defined in /etc/httpd/conf/httpd.conf

<VirtualHost *:443>
 ServerAdmin user@domain.com
 ServerName myapp.com
 ServerAlias www.myapp.com
 DocumentRoot /var/www/html/public
 <Directory /var/www/html>
  Options Indexes FollowSymLinks MultiViews
  AllowOverride All
  Require all granted
 </Directory>
</VirtualHost>

<VirtualHost *:80>
  ServerAdmin user@domain.com
  ServerName myapp.com
  ServerAlias www.myapp.com
  DocumentRoot /var/www/html/public
  <Directory /var/www/html>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>

Here is my IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket",
                "arn:aws:s3:::mybucket/*"
            ]
        }
    ]
}

Here is a screenshot of the exact error being thrown:

How can I further troubleshoot this when no actual exception is thrown?

Answer 1

I'm guessing this 403 issue you were experiencing was having place in a shared hosting. When you create the symlink with php artisan storage:link it contains the full absolute path to the storage/app/public folder. This is what probably causes the 403 response by the server.

The solution is to create a symlink with a relative path from the project root like this

ln -s ../storage/app/public public/storage

Answer 2

For other users who run into mysterious 403 Errors using AWS, here is an approach to debugging this and problems like it.

First, verify if your request is even hitting your application. An excellent way to do this is to try to dd() in the AppServiceProvider. In this case, it would not have returned anything, which is your clue that it is outside your application. If it would have been inside, continue debugging using dd() through your application or logging into the Laravel log.

The next step is ensuring you are hitting the server—two approaches for this when using AWS.

Using Sampled Requests View

Aws has a thorough walk-through for viewing a sample of web requests - this will allow you to view any incoming requests that AWS WAF has inspected and either allowed or blocked.

AWS Waf Logs

Additionally, when setting up new projects, I recommend always enabling AWS Waf Logs, allowing you to see which rule is the terminating rule blocking the request.

Both of these methods should be sufficient in debugging this type of error.

Answer 3

The issue was with an AWS WAF rule preventing HTTP post size from exceeding a certain number. Removed the rule, and it's now working again.