CODEWITHSUNDEEP
phphtmlpostupload
Broseph12
Broseph12

Reputation: 31

Overcome host's max file size limit for PHP/POST data when uploading files?

I'm with a fairly mediocre low-cost (shared) host at the moment (as it's all I can afford just now) and want to implement a very basic file upload feature on a page. I want to allow files of up to 100MB to be uploaded to the server but my free host limits the PHP_MAX_FILESIZE to 32MB and the POST_FILESIZE to 64MB.

Is there any way of overcoming this without requiring the user to split the larger files into smaller chunks? Either a JavaScript or Flash-based solution which could perhaps somehow route parts of the file through different HTTP requests or some other way to get around the 32MB limit?

Or are there any commands I can attempt to make which might over-ride the host's limits? I've already tried using .htaccess without success. EDIT: also tried ini_set. I'm thinking the only way is some kind of chunking/splitting/multi-streaming or another way to solve the inability to set higher PHP values.

Any suggestions are hugely appreciated. Thanks.

Upvotes: 3

Views: 4164

Answers (9)

RealityRipple
RealityRipple

Reputation: 67

You can bypass any upload/POST limitation using nothing more than PHP and JS:

JS:

  1. Retrieve the user-selected file from an input[type=file] element as a Blob.
  2. Split the Blob up into chunks your server can handle.
  3. Upload each Blob, with identifying information.

PHP:

  1. Check and combine the split Blobs back into a single file.

Note: Blobs are used exclusively because just about every other method of storing data in the browser fails on files in the 2-to-4 GB range (depending on the browser).

Here's an example that provides a script with your server's limit, splits a file up into nearly-equally-sized chunks under that limit, and then recombines them server-side. It will reject any files that are small enough to be uploaded normally.

index.html:

<!doctype html>
<html lang="en">
 <head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <title>Large File Upload</title>
  <script src="lfu.js"></script>
 </head>
 <body>
  <table id="tblUpload">
   <tr><td><label for="txtName">Name: </label></td><td><input type="text" id="txtName"></td></tr>
   <tr><td><label for="txtUpload">File: </label></td><td><input type="file"id="txtUpload"></td></tr>
   <tr><td colspan="2"><button type="button" id="cmdUpload">Upload</button></td></tr>
  </table>
 </body>
</html>

lfu.js:

const chunkFrac = 7 / 8;
let chunkSize = 1024 * 1024 * 5;

async function init() {
 await loadChunkLimit();
 document.getElementById('cmdUpload').addEventListener('click', upload);
 document.getElementById('txtUpload').addEventListener('change', setName);
}

function setName() {
 const fList = document.getElementById('txtUpload').files;
 if (fList.length !== 1)
  return;
 document.getElementById('txtName').value = fList[0].name;
}

async function upload() {
 const fList = document.getElementById('txtUpload').files;
 if (fList.length !== 1)
  alert('Please select a single file to upload.');
 document.getElementById('tblUpload').setAttribute('style', 'display: none;');
 const bTest = new Blob([fList[0]]);
 splitAndSend(bTest);
}

async function splitAndSend(blob) {
 const chunks = Math.ceil(blob.size / chunkSize);
 const bPerCh = Math.ceil(blob.size / chunks);

 const tblStatus = document.createElement('table');

 tblStatus.setAttribute('id', 'tblStatus');
  const trFile = document.createElement('tr');
   const tdFileLabel = document.createElement('td');
    const lblLoad = document.createElement('label');
    lblLoad.textContent = 'Uploading File: ';
   tdFileLabel.appendChild(lblLoad);
  trFile.appendChild(tdFileLabel);
   const tdFileBar = document.createElement('td');
    const pbLoad = document.createElement('progress');
    pbLoad.setAttribute('id', 'pbLoad');
    pbLoad.setAttribute('value', '0')
    pbLoad.setAttribute('max', chunks);
   tdFileBar.appendChild(pbLoad);
  trFile.appendChild(tdFileBar);
   const tdFilePct = document.createElement('td');
    const lblPercent = document.createElement('span');
    lblPercent.textContent = ' 0%';
   tdFilePct.appendChild(lblPercent);
  trFile.appendChild(tdFilePct);
 tblStatus.appendChild(trFile);

  const trChunk = document.createElement('tr');
   const tdChunkLabel = document.createElement('td');
    const lblChunkLoad = document.createElement('label');
    lblChunkLoad.textContent = 'Uploading Chunk: ';
   tdChunkLabel.appendChild(lblChunkLoad);
  trChunk.appendChild(tdChunkLabel);
   const tdChunkBar = document.createElement('td');
    const pbChunkLoad = document.createElement('progress');
    pbChunkLoad.setAttribute('id', 'pbChunkLoad');
    pbChunkLoad.setAttribute('value', '0')
    pbChunkLoad.setAttribute('max', '1');
   tdChunkBar.appendChild(pbChunkLoad);
  trChunk.appendChild(tdChunkBar);
   const tdChunkPct = document.createElement('td');
    const lblChunkPercent = document.createElement('span');
    lblChunkPercent.setAttribute('id', 'lblChunkPercent');
    lblChunkPercent.textContent = ' Initializing...';
   tdChunkPct.appendChild(lblChunkPercent);
  trChunk.appendChild(tdChunkPct);
 tblStatus.appendChild(trChunk);

  const trTime = document.createElement('tr');
   const tdTimeLabel = document.createElement('td');
    const lblTimeLabel = document.createElement('label');
    lblTimeLabel.textContent = 'ETA: ';
   tdTimeLabel.appendChild(lblTimeLabel);
  trTime.appendChild(tdTimeLabel);
   const tdTime = document.createElement('td');
   tdTime.setAttribute('colspan', '2');
    const lblTime = document.createElement('span');
    lblTime.setAttribute('id', 'lblTime');
    lblTime.textContent = '...';
   tdTime.appendChild(lblTime);
  trTime.appendChild(tdTime);
 tblStatus.appendChild(trTime);

  const trSpeed = document.createElement('tr');
   const tdSpeedLabel = document.createElement('td');
    const lblSpeedLabel = document.createElement('label');
    lblSpeedLabel.textContent = 'Speed: ';
   tdSpeedLabel.appendChild(lblSpeedLabel);
  trSpeed.appendChild(tdSpeedLabel);
   const tdSpeed = document.createElement('td');
   tdSpeed.setAttribute('colspan', '2');
    const lblSpeed = document.createElement('span');
    lblSpeed.setAttribute('id', 'lblSpeed');
    lblSpeed.textContent = '0 B/s';
   tdSpeed.appendChild(lblSpeed);
  trSpeed.appendChild(tdSpeed);
 tblStatus.appendChild(trSpeed);

 document.body.appendChild(tblStatus);

 const uuid = crypto.randomUUID();
 for (let i = 0; i < chunks; i++) {
  pbLoad.setAttribute('value', i);
  lblPercent.textContent = ' ' + (Math.floor(i / chunks * 1000) / 10) + '%';
  pbChunkLoad.setAttribute('value', 0);
  pbChunkLoad.setAttribute('max', 1);
  lblChunkPercent.textContent = ' Sending...';
  const start = i * bPerCh;
  let end = start + bPerCh;
  if (end > blob.size)
   end = blob.size;
  const bData = blob.slice(start, end, 'application/octet-stream');
  const r = await post(document.getElementById('txtName').value, uuid, i, chunks, bData, bPerCh, blob.size);
  pbLoad.setAttribute('value', i + 1);
  lblPercent.textContent = ' ' + (Math.floor((i + 1) / chunks * 1000) / 10) + '%';
  if (!r) {
   document.body.removeChild(tblStatus);
   alert('Failed to upload chunk ' + (i + 1) + ' of ' + chunks + '!');
   document.getElementById('tblUpload').removeAttribute('style');
   return;
  }
 }
 document.body.removeChild(tblStatus);
 alert('Complete!');
 document.getElementById('tblUpload').removeAttribute('style');
 document.getElementById('txtName').value = '';
 document.getElementById('txtUpload').value = '';
}

function byteSize(b) {
 if (b < 1024)
  return b + ' B';
 if (b < 1024 * 1024)
  return Math.floor(b * 10 / 1024) / 10 + ' KB';
 if (b < 1024 * 1024 * 1024)
  return Math.floor(b * 10 / (1024 * 1024)) / 10 + ' MB';
 if (b < 1024 * 1024 * 1024 * 1024)
  return Math.floor(b * 10 / (1024 * 1024 * 1024)) / 10 + ' GB';
 return Math.floor(b * 10 / (1024 * 1024 * 1024 * 1024)) / 10 + ' TB';
}

function timeLen(t) {
 const m = 60;
 const h = m * 60;
 const d = h * 24;
 const w = d * 7;
 let r = '';
 if (t >= w) {
  r = Math.floor(t / w) + 'w';
  t %= w;
 }
 if (t >= d) {
  r+= Math.floor(t / d) + 'd';
  t %= w;
 }
 if (t >= h) {
  r += Math.floor(t / h) + 'h';
  t %= h;
 }
 if (t >= m) {
  r += Math.floor(t / m) + 'm';
  t %= m;
 }
 r += t + 's';
 return r;
}

async function loadChunkLimit() {
 const szs = await get('limit.php');
 if (szs === false)
  return;
 if (!szs.hasOwnProperty('limit'))
  return;
 if (isNaN(szs.limit))
  return;
 chunkSize = Math.floor(szs.limit * chunkFrac);
}

function get(path) {
 return new Promise(
  function(resolve) {
   const req = new XMLHttpRequest();
   req.open('GET', path);
   req.addEventListener('load', function(ev){
    if (req.readyState !== 4)
     return;
    try
    {
     const jRet = JSON.parse(req.response);
     if (!jRet)
      resolve(false);
     else
      resolve(jRet);
    }
    catch(ex)
    {
     console.log(ex);
     resolve(false);
    }
   });
   req.addEventListener('error', function(ev){
    resolve(false);
    console.log(ev);
   });
   req.send();
  }
 )
}

function post(fname, uuid, idx, ct, data, chLen, alLen) {
 return new Promise(
  function(resolve) {
   const pbChunkLoad = document.getElementById('pbChunkLoad');
   const lblChunkPercent = document.getElementById('lblChunkPercent');
   const lblTime = document.getElementById('lblTime');
   const lblSpeed = document.getElementById('lblSpeed');
   const fData = new FormData();
   fData.append('uuid', uuid);
   fData.append('chunk', idx);
   fData.append('chunks', ct);
   fData.append('files', data, fname);
   const req = new XMLHttpRequest();
   req.open('POST', 'lfu.php');
   req.addEventListener('load', function(ev){
    if (req.readyState !== 4)
     return;
    pbChunkLoad.setAttribute('value', 0);
    pbChunkLoad.setAttribute('value', 1);
    lblChunkPercent.textContent = ' Sent!';
    try
    {
     const jRet = JSON.parse(req.response);
     if (!jRet.success)
     {
      alert('Server Error:\n\n' + jRet.error);
      resolve(false);
      return;
     }
     resolve(true);
    }
    catch(ex)
    {
     console.log(ex);
     resolve(false);
    }
   });
   let lastAmt = 0;
   let lastChk = 0;
   req.upload.addEventListener('progress', function(ev){
    const now = Date.now();
    if (pbChunkLoad.getAttribute('max') != ev.total)
     pbChunkLoad.setAttribute('max', ev.total);
    pbChunkLoad.setAttribute('value', ev.loaded);
    lblChunkPercent.textContent =  ' ' + Math.floor(ev.loaded / ev.total * 100) + '%';
    if (lastAmt === 0) {
     lastAmt = ev.loaded;
     lastChk = now;
     return;
    }
    if (now - lastChk < 3000)
     return;
    const speed = (ev.loaded - lastAmt) / ((now - lastChk) / 1000);
    lastAmt = ev.loaded;
    lastChk = now;
    const done = (idx * chLen + ev.loaded);
    lblSpeed.textContent = byteSize(speed) + '/s';
    lblTime.textContent = timeLen(Math.floor((alLen - done) / speed));
   });
   req.addEventListener('error', function(ev){
    resolve(false);
    console.log(ev);
   });
   req.send(fData);
  }
 );
}

window.addEventListener('load', init);

limit.php:

<?php
 header('Content-Type: application/json');

 function szToB($sSize) {
  $sSuffix = strtoupper(substr($sSize, -1));
  if (!in_array($sSuffix, array('P','T','G','M','K')))
   return intval($sSize, 10); 
  $r = intval(substr($sSize, 0, -1), 10);
  switch ($sSuffix) {
   case 'P':
    $r *= 1024;
   case 'T':
    $r *= 1024;
   case 'G':
    $r *= 1024;
   case 'M':
    $r *= 1024;
   case 'K':
    $r *= 1024;
    break;
  }
  return $r;
 }

 $post = szToB(ini_get('post_max_size'));
 $upload = szToB(ini_get('upload_max_filesize'));
 $overhead = 1024;
 if ($upload < $post - $overhead)
  $limit = $upload;
 else
  $limit = $post - $overhead;

 $out = array();
 $out['limit'] = $limit;
 echo json_encode($out, JSON_FORCE_OBJECT | JSON_PRETTY_PRINT);
?>

lfu.php:

<?php
 function goodUUID($uuid) {
  $id = preg_replace('/[^0-9a-f]/', '', $uuid);
  if (strlen($id) !== 32)
   return false;
  $id = substr($id, 0, 8).'-'.substr($id, 8, 4).'-'.substr($id, 12, 4).'-'.substr($id, 16, 4).'-'.substr($id, 20);
  return ($id === $uuid);
 }

 function tmpName($id, $idx) {
  return 'raw-'.$id.'-'.$idx.'.bin';
 }

 function superClean($message) {
  if ($message === null)
   return null;
  if (!is_string($message))
   return null;
  return preg_replace('/[^A-Za-z0-9\.]/', '', $message);
 }

 if (!array_key_exists('uuid', $_POST))
  die(json_encode(['success'=> false, 'error' => 'FILE_UUID']));
 $uuid = $_POST['uuid'];
 if (!goodUUID($uuid))
  die(json_encode(['success'=> false, 'error' => 'FILE_UUID']));

 if (!array_key_exists('chunks', $_POST))
  die(json_encode(['success'=> false, 'error' => 'CHUNK_COUNT']));
 $chunks = intval($_POST['chunks'], 10);
 if ($chunks < 2)
  die(json_encode(['success'=> false, 'error' => 'CHUNK_COUNT']));

 if (!array_key_exists('chunk', $_POST))
  die(json_encode(['success'=> false, 'error' => 'CHUNK_NUMBER']));
 $chunk = intval($_POST['chunk'], 10);
 if ($chunk < 0 || $chunk > $chunks - 1)
  die(json_encode(['success'=> false, 'error' => 'CHUNK_NUMBER']));

 if (!array_key_exists('files', $_FILES) || empty($_FILES['files']))
  die(json_encode(['success'=> false, 'error' => 'FILE_UPLOAD']));

 $fileErr = UPLOAD_ERR_OK;
 if (array_key_exists('error', $_FILES['files']))
  $fileErr = $_FILES['files']['error'];
 if ($fileErr !== UPLOAD_ERR_OK) {
  switch ($fileErr) {
   case UPLOAD_ERR_INI_SIZE:   die(json_encode(['success'=> false, 'error'=> 'UPLOAD_INI_SIZE']));
   case UPLOAD_ERR_FORM_SIZE:  die(json_encode(['success'=> false, 'error'=> 'UPLOAD_FORM_SIZE']));
   case UPLOAD_ERR_PARTIAL:    die(json_encode(['success'=> false, 'error'=> 'UPLOAD_PARTIAL']));
   case UPLOAD_ERR_NO_FILE:    die(json_encode(['success'=> false, 'error'=> 'UPLOAD_NO_FILE']));
   case UPLOAD_ERR_NO_TMP_DIR: die(json_encode(['success'=> false, 'error'=> 'UPLOAD_NO_TMP_DIR']));
   case UPLOAD_ERR_CANT_WRITE: die(json_encode(['success'=> false, 'error'=> 'UPLOAD_CANT_WRITE']));
   case UPLOAD_ERR_EXTENSION:  die(json_encode(['success'=> false, 'error'=> 'UPLOAD_EXTENSION']));
  }
  die(json_encode(['success'=> false, 'error'=> 'UPLOAD_UNKNOWN: '.$fileErr]));
 }

 if (!array_key_exists('name', $_FILES['files']))
  die(json_encode(['success'=> false, 'error' => 'FILE_UPLOAD']));
 $destName = superClean($_FILES['files']['name']);
 if (strlen($destName) < 1)
  die(json_encode(['success'=> false, 'error' => 'FILE_NAME']));

 if (!array_key_exists('tmp_name', $_FILES['files']))
  die(json_encode(['success'=> false, 'error' => 'FILE_UPLOAD']));
 $tmpFile = $_FILES['files']['tmp_name'];
 if (filesize($tmpFile) < 1)
  die(json_encode(['success'=> false, 'error' => 'FILE_SIZE']));

 if (!move_uploaded_file($tmpFile, 'out/'.tmpName($uuid, $chunk)))
  die(json_encode(['success'=> false, 'error'=> 'CHUNK_ACCESS']));

 for ($i = 0; $i < $chunks; $i++)
 {
  if (!file_exists('out/'.tmpName($uuid, $i)))
   die(json_encode(['success'=> true, 'complete' => false]));
 }

 try
 {
  for ($i = 0; $i < $chunks; $i++)
  {
   $ch = file_get_contents('out/'.tmpName($uuid, $i));
   file_put_contents('out/'.$destName, $ch, FILE_APPEND);
   $ch = null;
  }
 }
 catch (Exception $ex)
 {
  die(json_encode(['success'=> false, 'error'=> 'CHUNK_COMBINE']));
 }

 for ($i = 0; $i < $chunks; $i++)
 {
  unlink('out/'.tmpName($uuid, $i));
 }

 die(json_encode(['success'=> true, 'complete' => true]));
?>

Public-facing script caveats:

  • The example code uses a client-side generated UUID to identify files. The server does check the UUID's validity, but there may be some security weakness behind letting the client send this value.
  • The settable file name is stripped of invalid characters with the regular expression /[^A-Za-z0-9\.]/, which blocks a lot of otherwise safe characters. Expand this to your uses, or better yet, save files with server-side generated names.
  • There is no cleanup for partial uploads in this code. I would suggest a cron job to clean up unused chunks after a set amount of time, based on the newest file date between the chunks.
  • There is no integrity verification system implemented. This is because of file size limits - While blob can safely handle extremely large files, all browsers have file-size limits when it comes to ArrayBuffer and FileReader interactions, which are required for hashing with Subtle Crypto. I tried a hand-made SHA-1 class that would let me feed 64-byte chunks in piece by piece from the Blob, and while it works, it's incredibly slow when dealing with large files. The bottleneck is in converting the Blob into a usable array of bytes (or DWORDs); even if you pre-convert a decent chunk at a time, the amount of time it takes is unacceptable. Maybe some day Subtle Crypto will be changed to allow a Blob to be passed to the digest function.
  • The limit.php file lets the client know what the chunk size should be by determining the max upload size and removing 1 KB of overhead. 1024 bytes is probably technically overkill, even for the multipart/form-data encoding. I also only use 7/8ths of this limit, just to give the system some breathing room, and to reduce issues caused by timeouts and non-focused windows. It's also not really very efficient to check the limit every time the script runs; if you know what your server is capable of, it'd be better to make chunkSize a constant. Smaller chunk sizes may be desired for connectivity reasons, as well.
  • Add authentication to this upload process, or suffer the consequences.

Upvotes: 0

Semptrion
Semptrion

Reputation: 41

Abstract:

1) Read the file before to sending it (catch into onsubmit event); split and send the chunks as textarea fields.

2) In the server side, recover the chunks, and make one single file.

Proposal:

Depending on the environment in which your script runs and where the file resides, your options include the following:

 XMLHttpRequest

object (for reading files available via URLs on your website)

 FileSystemObject

(if you use Windows Scripting Host or Internet Explorer in a trusted environment) a "helper" Java applet that reads a file or URL for your script.

(Extract from http://www.javascripter.net/faq/reading2.htm)

If ok, remove the input file element of the form.

Then, split the string into many chunks.

mySplitResult = myReadedDocument.split( 1024 * 1024 ); // 1M each

That make an array of pieces of your document:

Add the values into the form (remember set the same name with [] to all the new controls). Assume that form id is 'myForm':

formName = document.getElementById('myForm');

mySplitResult.forEach(function(item) {     
    ta = document.createElement('textarea');
    ta.appendChild(document.createTextNode(item))
    ta.setAttribute("name", "chunk[]" );
    formName.appendChild(ta);

});

In the server side, you can reconstruct the chunks and save as a file.

<?php
    $chunks = $_POST['chunk'];
    $fileContent = implode( '', $chunks );
    file_put_content( 'dirname/filename.foo', $fileContent );
?>

The weight of success is that you can read the file on the client side.

Good luck!

Upvotes: 0

Bailey Parker
Bailey Parker

Reputation: 15905

You could have your users upload to an outside site. Then give you the URL from that outside site. If you have enough space, you can circumvent how long the outside site keeps the file by downloading it to a directory in your site. It's certainly not the best option, but your users will have a bigger upload quota and probably faster upload speeds (shared servers and speed mix like oil and water).

Upvotes: 0

Hardik
Hardik

Reputation: 536

if your host allows java applet then in sourceforge there is already package for it. which allows you to dw file via java applet from users machine to your host via small packages. it works because applet handles the file upload code in users machine and at server side you will receive small chunks which you can bind later and any file size can be uploaded.

i found the link here it is, http://sourceforge.net/projects/juploader/

Upvotes: 1

Halcyon
Halcyon

Reputation: 57693

Since ini_set doesn't seem to be working, you could try and set it via the .htaccess file. I'm not sure about the exact syntax but it's something involving php_flag. I wouldn't be surprised if this doesn't work either.

From my experience choosing a good host based on their advertising is impossible. I know of no other way than to simply try out a bunch and hope your run across one that isn't super retentive.

Upload limits are a common problem. If it's common for your customers to upload very large files then perhaps it would be wise to look into some other hosting plan. If it's not very common at all you could just have them send you the file so you can FTP it. Sometimes the best solution is the simplest solution.

Upvotes: 0

Darm
Darm

Reputation: 5659

You can use Flash. Start with this: http://soenkerohde.com/2010/01/chunk-file-upload/

OR

use https://github.com/moxiecode/plupload

Upvotes: 2

kapitanluffy
kapitanluffy

Reputation: 1267

if your 'free host' already limits you, there is nothing you can do about it. try reading

http://www.php.net/manual/en/ini.core.php#ini.post-max-size

and

http://www.php.net/manual/en/ini.core.php#ini.upload-max-filesize

here is where you can set it during runtime (ini_set) or not

http://www.php.net/manual/en/configuration.changes.modes.php

i suggest you just do multiple file uploads. 100 mb right? are you planning to host videos and movies? try looking for a better paid host rather than free ones :)

Upvotes: 1

Damien Pirsy
Damien Pirsy

Reputation: 25445

Some hosts allow using a local copy of php.ini (mine does, for example), so you could change parameters at will. Keep in mind that 100MB each file can rapidly bloat your host, expecially if it's not top of the category, so be careful.

Upvotes: 0

CrayonViolent
CrayonViolent

Reputation: 32517

might also possibly be able to use ini_set('upload_max_filesize','100M');

But I have a sneaking suspicion that your host might not be happy with you trying to circumvent their limit...

Upvotes: 1

Related Questions