CODEWITHSUNDEEP
functionazureffmpegcpu-usageserverless
The Lemon
The Lemon

Reputation: 1389

ffmpeg azure function consumption plan low CPU availability for high volume requests

I am running an azure queue function on a consumption plan; my function starts an FFMpeg process and accordingly is very CPU intensive. When I run the function with less than 100 items in the queue at once it works perfectly, azure scales up and gives me plenty of servers and all of the tasks complete very quickly. My problem is once I start doing more than 300 or 400 items at once, it starts fine but after a while the CPU slowly goes from 80% utilisation to only around 10% utilisation - my functions cant finish in time with only 10% CPU. This can be seen in the image shown below. Does anyone know why the CPU useage is going lower the more instances my function creates? Thanks in advance Cuan

edit: the function is set to only run one at a time per instance, but the problem exists when set to 2 or 3 concurrent processes per instance in the host.json

edit: the CPU drops get noticeable at 15-20 servers, and start causing failures at around 60. After that the CPU bottoms out at an average of 8-10% with individuals reaching 0-3%, and the server count seems to increase without limit (which would be more helpful if I got some CPU with the servers)

Thanks again, Cuan.

I've also added the function code to the bottom of this post in case it helps.

live metrics cpu

CPU useageg

using System.Net;
using System;
using System.Diagnostics;
using System.ComponentModel;

public static void Run(string myQueueItem, TraceWriter log)
{
    log.Info($"C# Queue trigger function processed a request: {myQueueItem}");
    //Basic Parameters
        string ffmpegFile = @"D:\home\site\wwwroot\CommonResources\ffmpeg.exe";
        string outputpath = @"D:\home\site\wwwroot\queue-ffmpeg-test\output\";
        string reloutputpath = "output/";
        string relinputpath = "input/";
        string outputfile = "video2.mp4";
        string dir =  @"D:\home\site\wwwroot\queue-ffmpeg-test\";

    //Special Parameters

        string videoFile = "1 minute basic.mp4";
        string sub = "1 minute sub.ass";
    //guid tmp files

        // Guid g1=Guid.NewGuid();
        // Guid g2=Guid.NewGuid();
        // string f1 = g1 + ".mp4";
        // string f2 = g2 + ".ass";
        string f1 = videoFile;
        string f2 = sub;
    //guid output - we will now do this at the caller level
        string g3 = myQueueItem;
        string outputGuid = g3+".mp4";
    //get input files
    //argument
        string tmp = subArg(f1, f2, outputGuid );
    //String.Format("-i \"" + @"input/tmp.mp4" + "\" -vf \"ass = '" + sub + "'\" \"" + reloutputpath +outputfile + "\" -y");
    log.Info("ffmpeg argument is: "+tmp);


    //startprocess parameters
    Process process = new Process();
    process.StartInfo.FileName = ffmpegFile;
    process.StartInfo.Arguments =  tmp;
    process.StartInfo.UseShellExecute = false;
    process.StartInfo.RedirectStandardOutput = true;
    process.StartInfo.RedirectStandardError = true;
    process.StartInfo.WorkingDirectory = dir;
    //output handler

    process.OutputDataReceived += new DataReceivedEventHandler(
        (s, e) => 
        { 
            log.Info("O: "+e.Data);
        }
    );
    process.ErrorDataReceived += new DataReceivedEventHandler(
        (s, e) => 
        { 
            log.Info("E: "+e.Data);
        }
    );
    //start process
    process.Start();
    log.Info("process started");
    process.BeginOutputReadLine();
    process.BeginErrorReadLine();
    process.WaitForExit();
}
public static void getFile(string link, string fileName, string dir, string relInputPath){
    using (var client = new WebClient()){
        client.DownloadFile(link, dir + relInputPath+ fileName);
        }

}
public static string subArg(string input1, string input2, string output1){
    return String.Format("-i \"" + @"input/" +input1+ "\" -vf \"ass = '" + @"input/"+input2 + "'\" \"" + @"output/" +output1 + "\" -y");

}

Upvotes: 0

Views: 568

Answers (1)

The Lemon
The Lemon

Reputation: 1389

When you use the D:\home directory you are writing to the virtual function, which means each instance has to continually try to write to the same spot as the functions run which causes the massive I/O block. Instead writing to D:\local and then sending the finished file somewhere else solves that issue, this way rather than each instance constantly writing to a location they only write when completed, and write to a location designed to handle high throughput.

The easiest way I could find to manage the input and output after writing to D:\local was just to hook up the function to an azure storage container and handle the ins and outs that way. Doing so made the average CPU stay at 90-100% for upwards of 70 concurrent Instances.

Upvotes: 2

Related Questions