VMAF picture quality evaluation – a model of measuring compression loss

What gets measured gets managed.

Video Multimethod Assessment Fusion (VMAF) is an objective full-reference video quality metric developed by Netflix in cooperation with the University of Southern California.

From time to time, we have need to evaluate the picture quality loss during compression and transmission, it turns out VMAF is a good tool. The note is to explain how to use ffmpeg to get VMAF score based on a same reference video. VMAF score will represent the picture quality during compressing procedure comparing to Reference source video.

ffmpeg will call a filter “libvmaf” calculating VMAF between compressed/distorted video and reference one, then it gives a VMAS score of each frame in a CSV file, here is an example if 2 videos are frame to frame (pixel to pixel) matching:

ffmpeg -v info -threads 0 -i …/Compressed.ts -i …/Reference.ts -an -filter_complex [0:v] settb=AVTB,setpts=PTS-STARTPTS,scale=1920x1080,yadif=1:-1:1[distorted]; [1:v] settb=AVTB,setpts=PTS-STARTPTS,null,yadif=1:-1:1[reference]; [distorted][reference]libvmaf=n_threads=32:log_fmt=csv:log_path=…/vmaf.csv:feature=name=psnr|name=float_ssim:eof_action=endall -f null –

in the output …/vmaf.csv file, we will have picture quality data, column of “vmaf” contains each frame’s VMAF score

By analysis VMAF column, we can get their mean to see an overall VMAF score or using tool like PLOT to get a graph to see which segment the video has terrible distortion.

We can also generate a side-by-side video to check visual distortion, command be used:

ffmpeg -y -v panic -threads 0 -i …/Compressed.ts -i …/Reference.ts -an -filter_complex [0:v]trim=start=55.12,settb=AVTB,setpts=PTS-STARTPTS,crop=iw/2:ih:0:0,scale=960:1080,yadif=1:-1:1,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text=%{pts\\:flt}sec \\(%{pict_type}\\):x=10:y=10:fontsize=40:fontcolor=white:borderw=2:bordercolor=black,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text=frame %{frame_num}:x=10:y=50:fontsize=40:fontcolor=white:borderw=2:bordercolor=black,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text=…/Compressed.ts:x=10:y=90:fontsize=40:fontcolor=white:borderw=2:bordercolor=black[left]; ??[1:v]trim=start=56.76,settb=AVTB,setpts=PTS-STARTPTS,crop=iw/2:ih:ow:0,scale=960:1080,yadif=1:-1:1,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text=%{pts\\:flt}sec \\(%{pict_type}\\):x=10:y=10:fontsize=40:fontcolor=white:borderw=2:bordercolor=black,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text=frame %{frame_num}:x=10:y=50:fontsize=40:fontcolor=white:borderw=2:bordercolor=black,drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf:text= …/Reference.ts:x=10:y=90:fontsize=40:fontcolor=white:borderw=2:bordercolor=black[right]; ??[left][right]hstack=inputs=2[vid] -map [vid] -c:v libx264 -r 25 -g 25 -preset slow -crf 18 …/pq_half_sidebyside.mp4

The output file …/pq_half_sidebyside.mp4 will have left side being compressed/distorted and right-side being Reference.

From screenshot above you will see blur on ground on left side as well as characters, this is introduced by compressing.

There are a few challenges on above model:

• All is assumed that both Reference and compressed/distorted video are frame-to-frame accurate, this is normally impossible when you recording 2 video streams.

Therefore we need to find out the right start point of both videos, then the offset values(in second) can be passed into filter “trim”

ffmpeg -v info -threads 0 -i …/Compressed.ts -i …/Reference.ts -an -filter_complex [0:v]trim=start=55.12,settb=AVTB,setpts=PTS-STARTPTS,scale=1920x1080,yadif=1:-1:1[distorted]; [1:v]trim=start=56.76,settb=AVTB,setpts=PTS-STARTPTS,null,yadif=1:-1:1[reference]; [distorted][reference]libvmaf=n_threads=32:log_fmt=csv:log_path=…/vmaf.csv:feature=name=psnr|name=float_ssim:eof_action=endall -f null –

• ffmpeg need to be compiled with option of “–enable-libvmaf –enable-version3”, and you need to build and install vmaf first:

cd ~/ffmpeg_sources && \
git clone https://github.com/Netflix/vmaf.git && \
cd ~/ffmpeg_sources/vmaf/libvmaf && \
meson build --buildtype release && \
sudo ninja -vC build && \
sudo ninja -vC build install