CODEWITHSUNDEEP
javascriptjqueryhtmlcanvas
Darryl Huffman
Darryl Huffman

Reputation: 2589

Synchronised Audio and Canvas

The Problem

I am trying to time the frames of a canvas to be synchronised with audio playing in the background.
When I use window.requestAnimationFrame you get an erratic number of frames within a second across different browsers/computers. I also tried setInterval(function(){ //code }, 1); which was closer to perfect than window.requestAnimationFrame but it still didn't get the job done correctly.

Here's how my code is working.

The setInterval fires every millisecond, then checks to see what objects should be spawning at x time, and it also clears and updates the canvas. Also, I'm using milliseconds to calculate the velocity of objects so that they reach a position on the canvas at the correct x time.

Is there any better way of doing this?
I'll list my code below, and a link to my current CodePen so you can see it in action. (click the canvas area to start the animation there is music in the background to be synchronised with, but it wont start until you click)

Sonar Song on CodePen

var Queues = [
    [1, '#FFFFFF', 'line', true, [20,20], [80,80], 500, 950],
    [2, '#FFFFFF', 'line', true, [80,80], [55,20], 950, 1200]
];

var playing = false;


$(document).click(function(){
    if(playing == false) {
        var audio = document.getElementById("audio");
        audio.currentTime = 0;
        audio.play();
        apparatus();
        playing = true;
    } else {
        var audio = document.getElementById("audio");
        audio.pause();
        playing = false;
        $('body canvas').remove();
    }

});

function apparatus() {

    var canvas, context, toggle, time = 0, vidtime = 0;

    var points = [];

    init();
    function init() {

        canvas = document.createElement( 'canvas' );
        context = canvas.getContext( '2d' );
        canvas.width = $(document).width();
        canvas.height = $(document).height();
        document.body.appendChild( canvas );

    }

    var point = function(options) {
        this.position = {}, this.end_position = {}, this.distance = {}, this.velocity = {}, this.time = {};
        points.push(this);

        // TIMING
        this.time.start = options[6];
        this.time.end = options[7];

        // VECTORS
        this.position.x = options[4][0] * (canvas.width / 100);
        this.position.y = options[4][1] * (canvas.height / 100);
        this.end_position.x = options[5][0] * (canvas.width / 100);
        this.end_position.y = options[5][1] * (canvas.height / 100);

        this.distance.x = Math.abs(this.position.x - this.end_position.x);
        this.distance.y = Math.abs(this.position.y - this.end_position.y);

        if(this.position.x > this.end_position.x) {
            this.velocity.x = -Math.abs(this.distance.x / (this.time.end - this.time.start)); 
        } else {
            this.velocity.x = Math.abs(this.distance.x / (this.time.end - this.time.start)); 
        }
        if(this.position.y > this.end_position.y) {
            this.velocity.y = -Math.abs(this.distance.y / (this.time.end - this.time.start)); 
        } else {
            this.velocity.y = Math.abs(this.distance.y / (this.time.end - this.time.start)); 
        }

        // STYLING
        this.style = options[2];
        this.color = options[1];

        //-- STYLING / STYLE TYPES
        if(this.style == 'line') {
        }
    }

    point.prototype.draw = function() {
        this.position.x += this.velocity.x;
        this.position.y += this.velocity.y;

        context.fillStyle = this.color;
        context.beginPath();
        context.arc( this.position.x, this.position.y, 10, 0, Math.PI * 2, true );
        context.closePath();
        context.fill();
    }

    setInterval(function(){
        time++;
        console.log(time);
        context.fillStyle = '#000000';
        context.fillRect(0, 0,  $(document).width(),  $(document).height());
        for(var i = 0; i < Queues.length; i++) {
            if(time == Queues[i][6]) {
                new point(Queues[i]);
            }
            if(time == Queues[i][7]) {
                console.log('la');
            }
        }
        for(var i = 0; i <= points.length; i++) {
            if(points[i] != null) {
                points[i].draw();
            }
        }
    }, 1);
}

Upvotes: 4

Views: 1655

Answers (1)

Blindman67
Blindman67

Reputation: 54069

Syncing

You can not rely on setInterval or setTimeout as they will only respond as close as possible to the requested time and thus are very unreliable for syncing things like animation and time.

You need to use a consistent time to sync with. You also have to make your code adapt to being late.

Date and requestAnimationFrame

First off getting the time. JavaScript provides the time in milliseconds via the Date object.

The following will return the current time in millisecond since some past time (I don't know when 1452798031458 ms ago)

var currentTime = new Date().valueOf();

Or if you use window.requestAnimationFrame you get the time as the first argument

//  main update loop
function update(time){
    // time is the millisecond time 
    requestAnimationFrame(update); // request new frame
}
update(new Date().valueOf);

If you want something to happen at a set time you must consider the time between opportunities to test the time as this can vary.

Regular event timing

Let try a timing event based on the frame time.

First some variables to track the times

var frameTime;     // time between calls to update
var lastFrameTime;  // the time of the perviouse call to update
var eventTime;      // the event start time that we want to stay in sync with
var timeInterval = 1000; // when we want sync events

Then to start the event we want to stay in sync with.

function startSound(){
     sound.play(); // assuming you have the sound.
     eventTime = new Date().valueOf(); // now
}

The update function that will do the correct thing as close as possible to the timing we want.

function update(time){
    frameTime = time - lastFrameTime; // how long since the last call to update     
    var timeSinceEvent = time-eventTime; // get the time since the event started
    var timeSinceLastSync = timeSinceEvent % timeInterval;

    // Several options here so explained below

    requestAnimationFrame(update); // request new frame
    lastFrameTime = time; // remember the last frame time
}
startSound(); // start playing
lastFrameTime = new Date().valueOf;
update(new Date().valueOf);

Now you have some options. Human perception is not that great at separating visual FX at anything under 1/10th of a second, Human hearing is very good at separating events in time (I get annoyed when the is a 10ms delay in playing guitar over a recording) so you have to consider what you are presenting.

Visual events

Visually is the simplest. You simply fire your snyc event as close as you can after the sync time.

if(timeSinceLastSync > timeInterval){
     // do the visual event
     eventTime += timeInterval;  // update the event time to the time this
                                 // event was to have happened.
}

This will fire the visual event within the frame time 1/60 (min) second. You will notice that I add to eventTime. I do not get the current time because the current time could be as much as 30ms out and if we add even a ms error it will quickly accumulate.

Audio, high precision events

For audio events we preempt the event time by looking at the last frame time

// make sure we are near the next event time. this is to avoid firing the event 
// twice. Once on the frame before and once on the frame after.
if((timeInterval - timeSinceLastSync) >  timeInterval/2){ 
    // will the time to the new sync event be less than half the last frame time
    // or have we passed that time.
    if((timeInterval - timeSinceLastSync) < frameTime || timeSinceLastSync > timeInterval ){
        // fire the synced event.
        eventTime += timeInterval;  // update the event time to the time this
                                 // event was or will happened.
    }
}

What we do here is use the time that the last frame took to complete and assume that the current frame will take the same time. We then find how long until the sync event and if that time is less than half the frame time then fire the event. Or we may have passed the sync event time. Either way we need to fire the event now.

Then again just update the current sync event time by adding.

This method will get you as close as you can to the time you want to fire the events. (within 1/120 of a second assuming 60fps)

Improvements

You can improve on this. JavaScript is blocking so some things you do need to wait till your update function has ended. You can also track the time you spend in the update function by getting the time just before you exit the function. You will then have an estimation of how long till the code you request to fire will happen and can include that in the calculations.Typically the time between frames is 16ms, but your update function may only run for 8ms, so triggering an event will happen 8ms before the next frame.

Performance and microseconds

There is also a micro second (1/1,000,000 of a second) timers on some browsers called performance

// assume no performance API
var usePerformance = false
// at the start of you code check if its available
if(typeof performance === "function"){  // is performance available
    usePerformance = true;
}

To avoid having to test for the performance API each time you need the time create separate function using either API

var updateP = function(){
}
// normal update function
var updateN = function(){
}
// set the correct update function
var update;
if(usePerformance){
    update = updateP;
}else{
    update = updateN;
}

Using performance

To use the performance API 

var now = performance.now(); // returns the time in ms with a fractional
                             // representing the microseconds.

Time between events

var n = performance.now();
console.log(performance.now()-n);  // returns 0.021 depending on the system speed

Hope this helps.

Upvotes: 3

Related Questions