FFmpeg: desynced audio when using xfade and acrossfade together

Question

I have a problem when using xfade and acrossfade together to concatenate 12 videos together and keep the audio. When I leave out the audio stream and just apply the xfade filter, the video encodes just fine. Otherwise the transition hangs and the audio starts to early.

Here is my filter_complex:

[0:v][1:v]xfade=transition=pixelize:duration=0.5:offset=3.042[V1];
[0:a][1:a]acrossfade=d=0.5[A1];
[V1][2:v]xfade=transition=pixelize:duration=0.5:offset=6.084[V2];
[A1][2:a]acrossfade=d=0.5[A2];
[V2][3:v]xfade=transition=pixelize:duration=0.5:offset=9.108[V3];
[A2][3:a]acrossfade=d=0.5[A3];
[V3][4:v]xfade=transition=pixelize:duration=0.5:offset=23.619[V4];
[A3][4:a]acrossfade=d=0.5[A4];
[V4][5:v]xfade=transition=pixelize:duration=0.5:offset=38.13[V5];
[A4][5:a]acrossfade=d=0.5[A5];
[V5][6:v]xfade=transition=pixelize:duration=0.5:offset=52.641[V6];
[A5][6:a]acrossfade=d=0.5[A6];
[V6][7:v]xfade=transition=pixelize:duration=0.5:offset=67.152[V7];
[A6][7:a]acrossfade=d=0.5[A7];
[V7][8:v]xfade=transition=pixelize:duration=0.5:offset=70.176[V8];
[A7][8:a]acrossfade=d=0.5[A8];
[V8][9:v]xfade=transition=pixelize:duration=0.5:offset=84.687[V9];
[A8][9:a]acrossfade=d=0.5[A9];
[V9][10:v]xfade=transition=pixelize:duration=0.5:offset=99.198[V10];
[A9][10:a]acrossfade=d=0.5[A10];
[V10][11:v]xfade=transition=pixelize:duration=0.5:offset=113.709[video];
[A10][11:a]acrossfade=d=0.5[audio];  

The transition duration is 0.5 seconds and the input length as read out from ffprobe (format duration):

0: 3.542
1: 3.542
2: 3.524
3: 15.011
4: 15.011
5: 15.011
6: 15.011
7: 3.524
8: 15.011
9: 15.011
10: 15.011
11: 15.011

As far as I understood, the offset of the xfade filter should be the video duration plus the previous offset minus transition duration, right? So what's going wrong here?

        3.542 - 0.5 = 3.042
3.042 + 3.542 - 0.5 = 6.084
6.084 + 3.524 - 0.5 = 9.108
...

Answer 1

Try the next variant:

#!/bin/bash
ffmpeg -hide_banner -i "test01.mkv" -i "test02.mkv" -i "test03.mkv" -filter_complex "
[0:v][1:v]xfade=transition=pixelize:duration=0.5:offset=3.042[1v];
[1v][2:v]xfade=transition=pixelize:duration=0.5:offset=6.084[2v];

[0:a]atrim=0:3.542[0a];
[1:a]atrim=0:3.542[1a];
[2:a]apad,atrim=0:3.524[2a];

[0a][1a]acrossfade=d=0.5[01a];
[01a][2a]acrossfade=d=0.5[02a]
" -map [2v] -map [02a] -c:v h264_nvenc -cq 23 -c:a aac -q:a 4 -y output.mkv

maybe, need to decrease time to align video and audio.

[update 1] added apad (Thanks to Gyan)

it was too lazy to do it manually:

#!/bin/bash
LST=($(ls -1 test*.mkv))
TOT=${#LST[*]}
f="${LST[0]}"
DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"
INP=("-i" "$f")

PDV="[0:v]"
PDC="[0a]"
FLA="[0:a]aresample=async=1,apad,atrim=0:${DUR}${PDC};"

OFS=$(echo $DUR -0.5 | bc -l)
echo $f $DUR $OFS

for (( i=1; i<=$(( $TOT -2 )); i++ )); do
  f="${LST[$i]}"
  INP+=("-i" "$f")
  DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"

  FLV+="${PDV}[${i}:v]xfade=transition=pixelize:duration=0.5:offset=${OFS}"
  PDV="[${i}v]"
  FLV+="${PDV};"

  PDA="[${i}a]"
  FLA+="[${i}:a]aresample=async=1,apad,atrim=0:${DUR}${PDA};"

  FLC+="${PDC}[${i}a]acrossfade=d=0.5"
  PDC="[0${i}a]"
  FLC+="${PDC};"

  OFS=$(echo $OFS + $DUR -0.5 | bc -l)
  echo $f $DUR $OFS
done

f="${LST[-1]}"
INP+=("-i" "$f")
DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"

FLV+="${PDV}[${i}:v]xfade=transition=pixelize:duration=0.5:offset=${OFS}"
PDV="[${i}v]"
FLV+="${PDV};"

PDA="[${i}a]"
FLA+="[${i}:a]aresample=async=1,apad,atrim=0:${DUR}${PDA};"

FLC+="${PDC}[${i}a]acrossfade=d=0.5"
PDC="[0${i}a]"
FLC+="${PDC}"

echo $f $DUR
echo $FLV
echo $FLA
echo $FLC
echo ""
ffmpeg "${INP[@]}" -filter_complex "$FLV $FLA $FLC" -map $PDV -map $PDC -c:v h264_nvenc -cq 20 -c:a aac -q:a 4 -y output.mp4 -hide_banner

[update 2] added aresample=async=1

[update 3] just code optimization

Answer 2

Thanks for your help.

I found out, when I output the video and audio streams in two seperate files (mp4/m4a) and then merging them together, all is in sync. But why can't I process it in one command?

Here is my current solution based on @Баяр Гончикжапов answer.

#!/bin/bash
LST=($(ls -1 ./test/*.mp4))
TOT=${#LST[*]}
f="${LST[0]}"
DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"
OFS=$(echo $DUR -0.5 | bc -l)
echo $f $DUR $OFS
INP=("-i" "$f")
FCT=1
PDV="[${FCT}v]"
FLV="[0:v][1:v]xfade=transition=pixelize:duration=0.5:offset=${OFS}${PDV}"
PDA="[0a]"
FLA="[0:a]apad,atrim=0:${DUR}${PDA};"
PDC="[01a]"
FLC="[0a][1a]acrossfade=d=0.5${PDC}"

for (( i=1; i<=$(( $TOT -2 )); i++ )); do
  f="${LST[$i]}"
  INP+=("-i" "$f")
  DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"
  OFS=$(echo $OFS + $DUR -0.5 | bc -l)
  echo $f $DUR $OFS

  ((FCT++))
  FLV+=";${PDV}[${FCT}:v]xfade=transition=pixelize:duration=0.5:offset=${OFS}"
  PDV="[${FCT}v]"
  FLV+="${PDV}"

  PDA="[${i}a]"
  FLA+="[${i}:a]apad,atrim=0:${DUR}${PDA};"

  FLC+=";${PDC}[${FCT}a]acrossfade=d=0.5"
  PDC="[0${FCT}a]"
  FLC+="${PDC}"
done
f="${LST[-1]}"
INP+=("-i" "$f")
DUR="$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f")"
PDA="[${i}a]"
FLA+="[${i}:a]atrim=0:${DUR}${PDA};"
echo $f $DUR
echo $FLV
echo ""
echo $FLA
echo ""
echo $FLC
echo ""
ffmpeg "${INP[@]}" -filter_complex "$FLV" -map $PDV -c:v h264_nvenc -cq 20 -an -y output.mp4 -hide_banner
ffmpeg "${INP[@]}" -filter_complex "$FLA $FLC" -map $PDC -c:a aac -q:a 4 -y output.m4a -hide_banner
ffmpeg -y -hide_banner -i output.mp4 -i output.m4a -c copy final.mp4