13 KiB
Table of Contents
AV1-vs-HEVC
Introduction
In this project, the objective is to assess the performance of the AV1 codec in comparison to the HEVC codec. HEVC has gained widespread acceptance and usage, supplanting the AVC codec, whose sole remaining advantage lies in its dramatically high compatibility. The intended use case here is home cinema. While file size is a consideration, image quality takes precedence. If, over the course of the tests, it becomes evident that the results achievable with AV1 lead to smaller files but at the expense of lower image quality, this would be a decisive criterion against its adoption.
The primary motivation for transitioning from HEVC to AV1 can be succinctly summarized with the following quote from Wikipedia:
Like VP9, but unlike H.264 (AVC) and H.265 (HEVC), AV1 has a royalty-free licensing model that does not hinder adoption in open-source projects.
To conduct the tests, initially, 24 test samples were provided, closely resembling real cinematic content (very closely), yet, of course, free from any copyright concerns and presented in BluRay quality. A test course was devised for these samples to navigate. It is worth mentioning that all tests are conducted with a bit depth of 10 and the pixel format "yuv420p10le".
The goal of the tests is to determine the parameters for the AV1 encoder (SVT-AV1) that yield comparable or even superior results to the HEVC encoder (x265). The evaluation incorporates the following metrics:
- Time required for encoding
- File size compared to the source
- Video quality assessed through the VMAF score
Test environment
All tests were conducted on fairly average consumer-grade hardware:
Hardware
- AMD Ryzen 7 3700X @ factory clock speed
- 32 GiB DDR4 RAM @ 3600MHz
Software
ffmpeg -i input.mkv -an -c:v libx265 -preset slow -crf 21 -x265-params "aq-mode=1:qcomp=0.7:vbv-bufsize=9000:vbv-maxrate=9000:no-sao=1:no-strong-intra-smoothing=1:keyint=240:min-keyint=24" -pix_fmt yuv420p10le -y output.mkv
Sample creation
As the source files were extensive movie files and only the video track was relevant for this test, they were stripped of all unnecessary tracks and reduced to a five-minute excerpt (starting from the timestamp 00:30:00 to avoid any kind of intros).
ffmpeg -ss 00:30:00 -i input.mkv -t 00:05:00 -vcodec copy -an -sn -dn -map_chapters -1 -y sample01.mkv
HEVC encoding
Firstly, the samples undergo encoding in HEVC, utilizing the following parameters, which have proven effective in numerous practical applications:
ffmpeg -i input.mkv -an -c:v libx265 -preset slow -crf 21 -x265-params "aq-mode=1:qcomp=0.7:vbv-bufsize=9000:vbv-maxrate=9000:no-sao=1:no-strong-intra-smoothing=1:keyint=240:min-keyint=24" -pix_fmt yuv420p10le -y output.mkv
AV1 encoding
Following that, all samples will be encoded using the AV1 codec. In terms of presets, the initial approach will adhere to the recommendations provided by the Alliance for Open Media:
Presets between 4 and 6 offer what many people consider a reasonable trade-off between quality and encoding time.
For the selection of CRF (Constant Rate Factor) values, we begin with the default setting in HandBrake - 30. To gain a clearer perspective, the value is then iteratively decreased by 1 until reaching a value of 10.
This results in the following command:
ffmpeg -i input.mkv -an -c:v libsvtav1 -preset <PRESET> -crf <CRF> -svtav1-params tune=0 -pix_fmt yuv420p10le -g 245 -y output.mkv
HEVC results
As mentioned above, all samples were initially encoded with HEVC. Now we have a baseline that we should aim to at least meet, if not surpass:
| Metric | Mean | Median |
|---|---|---|
| Encoding Time | 632 sec | 603 sec |
| Filesize Percentage | 16.36 pct | 13.02 pct |
| VMAF Score | 87.94 pct | 92.03 pct |
AV1 results
Below are the results of all Preset-CRF combinations in a comprehensive table:
| Preset | CRF | Encoding Duration (% of HEVC, Mean) | Encoding Duration (% of HEVC, Median) | Filesize (% of sample, Mean) | Filesize (% of sample, Median) | VMAF score (as %, Mean) | VMAF score (as %, Median) |
|---|---|---|---|---|---|---|---|
| 4 | 10 | 245.86 pct | 239.07 pct | 43.6 pct | 32.78 pct | 89.3 pct | 93.03 pct |
| 4 | 11 | 232.04 pct | 218.68 pct | 37.91 pct | 28.11 pct | 89.18 pct | 92.87 pct |
| 4 | 12 | 219.84 pct | 206.97 pct | 33.33 pct | 24.46 pct | 89.07 pct | 92.73 pct |
| 4 | 13 | 207.88 pct | 197.28 pct | 29.23 pct | 21.43 pct | 88.95 pct | 92.58 pct |
| 4 | 14 | 197.81 pct | 189.86 pct | 25.82 pct | 19.13 pct | 88.82 pct | 92.44 pct |
| 4 | 15 | 189.6 pct | 181.38 pct | 23.06 pct | 17.14 pct | 88.7 pct | 92.31 pct |
| 4 | 16 | 181.49 pct | 174.23 pct | 20.52 pct | 15.79 pct | 88.57 pct | 92.17 pct |
| 4 | 17 | 174.94 pct | 169.99 pct | 18.53 pct | 14.42 pct | 88.45 pct | 92.07 pct |
| 4 | 18 | 170.21 pct | 167.05 pct | 16.82 pct | 13.56 pct | 88.34 pct | 91.97 pct |
| 4 | 19 | 164.46 pct | 164.8 pct | 15.19 pct | 12.71 pct | 88.21 pct | 91.88 pct |
| 4 | 20 | 154.19 pct | 155.1 pct | 13.8 pct | 12.04 pct | 88.09 pct | 91.79 pct |
| 4 | 21 | 151.72 pct | 153.83 pct | 12.8 pct | 11.52 pct | 87.99 pct | 91.73 pct |
| 4 | 22 | 149.27 pct | 150.66 pct | 11.66 pct | 10.82 pct | 87.81 pct | 91.66 pct |
| 4 | 23 | 146.89 pct | 147.33 pct | 10.71 pct | 10.28 pct | 87.76 pct | 91.59 pct |
| 4 | 24 | 143.7 pct | 142.33 pct | 9.66 pct | 9.6 pct | 87.61 pct | 91.51 pct |
| 4 | 25 | 139.27 pct | 136.57 pct | 8.52 pct | 8.43 pct | 87.42 pct | 91.41 pct |
| 4 | 26 | 135.39 pct | 133.79 pct | 7.61 pct | 7.58 pct | 87.24 pct | 91.31 pct |
| 4 | 27 | 130.99 pct | 129.51 pct | 6.86 pct | 6.89 pct | 87.06 pct | 91.22 pct |
| 4 | 28 | 127.6 pct | 125.21 pct | 6.26 pct | 6.38 pct | 86.89 pct | 91.13 pct |
| 4 | 29 | 124.03 pct | 120.86 pct | 5.71 pct | 5.88 pct | 86.71 pct | 91.03 pct |
| 4 | 30 | 120.65 pct | 116.5 pct | 5.28 pct | 5.52 pct | 86.54 pct | 90.94 pct |
| 5 | 10 | 99.18 pct | 96.81 pct | 40.36 pct | 28.44 pct | 89.16 pct | 92.94 pct |
| 5 | 11 | 95.47 pct | 93.03 pct | 35.42 pct | 25.04 pct | 89.05 pct | 92.8 pct |
| 5 | 12 | 91.88 pct | 91.2 pct | 31.23 pct | 22.35 pct | 88.92 pct | 92.66 pct |
| 5 | 13 | 88.84 pct | 87.65 pct | 27.71 pct | 20.2 pct | 88.81 pct | 92.53 pct |
| 5 | 14 | 86.25 pct | 85.03 pct | 24.69 pct | 18.44 pct | 88.69 pct | 92.41 pct |
| 5 | 15 | 83.47 pct | 81.8 pct | 22.05 pct | 16.47 pct | 88.56 pct | 92.27 pct |
| 5 | 16 | 80.91 pct | 78.84 pct | 19.8 pct | 14.89 pct | 88.44 pct | 92.14 pct |
| 5 | 17 | 78.63 pct | 76.36 pct | 17.92 pct | 13.95 pct | 88.31 pct | 92.02 pct |
| 5 | 18 | 77.0 pct | 75.22 pct | 16.26 pct | 13.09 pct | 88.19 pct | 91.91 pct |
| 5 | 19 | 75.25 pct | 74.32 pct | 14.88 pct | 12.4 pct | 88.08 pct | 91.83 pct |
| 5 | 20 | 66.89 pct | 66.54 pct | 13.68 pct | 11.85 pct | 87.95 pct | 91.73 pct |
| 5 | 21 | 66.16 pct | 65.55 pct | 12.62 pct | 11.31 pct | 87.85 pct | 91.66 pct |
| 5 | 22 | 65.07 pct | 64.12 pct | 11.65 pct | 10.77 pct | 87.73 pct | 91.58 pct |
| 5 | 23 | 64.59 pct | 63.41 pct | 10.81 pct | 10.18 pct | 87.63 pct | 91.53 pct |
| 5 | 24 | 63.71 pct | 62.48 pct | 9.91 pct | 9.31 pct | 87.5 pct | 91.45 pct |
| 5 | 25 | 62.31 pct | 61.05 pct | 8.67 pct | 8.23 pct | 87.3 pct | 91.35 pct |
| 5 | 26 | 61.16 pct | 59.69 pct | 7.76 pct | 7.46 pct | 87.12 pct | 91.26 pct |
| 5 | 27 | 60.03 pct | 57.58 pct | 7.03 pct | 6.84 pct | 86.93 pct | 91.16 pct |
| 5 | 28 | 59.29 pct | 55.95 pct | 6.43 pct | 6.32 pct | 86.76 pct | 91.07 pct |
| 5 | 29 | 58.31 pct | 54.92 pct | 5.93 pct | 5.96 pct | 86.59 pct | 90.99 pct |
| 5 | 30 | 58.13 pct | 54.92 pct | 5.44 pct | 5.6 pct | 86.45 pct | 90.97 pct |
| 6 | 10 | 72.86 pct | 71.36 pct | 44.88 pct | 31.55 pct | 89.23 pct | 93.05 pct |
| 6 | 11 | 69.82 pct | 67.28 pct | 38.84 pct | 27.29 pct | 89.1 pct | 92.89 pct |
| 6 | 12 | 66.99 pct | 64.9 pct | 34.07 pct | 24.18 pct | 88.98 pct | 92.73 pct |
| 6 | 13 | 64.73 pct | 62.62 pct | 30.2 pct | 21.9 pct | 88.86 pct | 92.59 pct |
| 6 | 14 | 62.76 pct | 60.43 pct | 27.04 pct | 19.99 pct | 88.75 pct | 92.46 pct |
| 6 | 15 | 61.11 pct | 58.85 pct | 24.37 pct | 17.63 pct | 88.64 pct | 92.34 pct |
| 6 | 16 | 59.3 pct | 56.95 pct | 21.82 pct | 16.1 pct | 88.5 pct | 92.2 pct |
| 6 | 17 | 57.65 pct | 55.41 pct | 19.75 pct | 14.67 pct | 88.38 pct | 92.07 pct |
| 6 | 18 | 56.61 pct | 55.04 pct | 17.94 pct | 13.62 pct | 88.26 pct | 91.96 pct |
| 6 | 19 | 55.33 pct | 54.14 pct | 16.41 pct | 12.86 pct | 88.14 pct | 91.87 pct |
| 6 | 20 | 49.21 pct | 47.51 pct | 15.03 pct | 12.29 pct | 88.01 pct | 91.76 pct |
| 6 | 21 | 48.4 pct | 46.66 pct | 13.86 pct | 11.71 pct | 87.9 pct | 91.69 pct |
| 6 | 22 | 47.86 pct | 46.02 pct | 12.79 pct | 11.15 pct | 87.79 pct | 91.61 pct |
| 6 | 23 | 47.39 pct | 45.38 pct | 11.88 pct | 10.7 pct | 87.68 pct | 91.53 pct |
| 6 | 24 | 46.74 pct | 44.8 pct | 10.93 pct | 10.18 pct | 87.56 pct | 91.46 pct |
| 6 | 25 | 45.87 pct | 43.8 pct | 9.78 pct | 9.25 pct | 87.39 pct | 91.37 pct |
| 6 | 26 | 45.09 pct | 42.88 pct | 8.75 pct | 8.23 pct | 87.21 pct | 91.25 pct |
| 6 | 27 | 44.13 pct | 42.01 pct | 7.85 pct | 7.47 pct | 87.01 pct | 91.16 pct |
| 6 | 28 | 43.46 pct | 41.46 pct | 7.05 pct | 6.83 pct | 86.82 pct | 91.06 pct |
| 6 | 29 | 42.65 pct | 40.48 pct | 6.44 pct | 6.3 pct | 86.63 pct | 90.95 pct |
| 6 | 30 | 42.19 pct | 39.84 pct | 5.87 pct | 5.81 pct | 86.48 pct | 90.92 pct |
Furthermore, there are three graphics that illustrate the results more effectively. It becomes evident, for instance, that as the CRF value decreases (across all three presets), the spread in filesizes (mean vs. median) increases.
Looking for candidates
After collecting all the results, an analysis was conducted to determine which AV1 Preset-CRF combinations are at least on par with HEVC. The criteria for this assessment were straightforward and are presented here in pseudocode:
filesize_mean_av1 <= filesize_mean_hevc
filesize_median_av1 <= filesize_median_hevc
vmaf_score_mean_av1 >= vmaf_score_mean_hevc
vmaf_score_median_av1 >= vmaf_score_median_hevc
It turned out that not a single Preset-CRF combination met all the criteria, and in this specific scenario, AV1 is not on par with HEVC. Only when allowing a 20% margin in file size, there is one (!) Preset-CRF combination in AV1 (Preset 4 - CRF 17) whose VMAF score can match HEVC. However, with these settings, the average encoding time also increases significantly by 75%.
Conclusion
The results, with their raw numbers, clearly indicate that the AV1 codec is currently not suitable for the intended purpose mentioned in the introduction. Among all combinations of preset and CRF, there wasn't a single one (!) that could compete with the results of HEVC. While the differences between the results of HEVC and AV1 are not drastic (although a 75% increase in encoding time is noteworthy), one must simply weigh the importance of different criteria. Whether the primary use case is the web or home cinema plays a significant role in this decision. Since the initial focus of the test was on finding the better candidate for home cinema, HEVC must be considered the superior option.