AV1-vs-HEVC/templates/README.md.tmpl

---
gitea: none
include_toc: true
---
# AV1-vs-HEVC

## Introduction
In this project, the objective is to assess the performance of the [AV1](https://en.wikipedia.org/wiki/Av1) codec in comparison to the [HEVC](https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding) codec.
HEVC has gained widespread acceptance and usage, supplanting the [AVC](https://en.wikipedia.org/wiki/Advanced_Video_Coding) codec, whose sole remaining advantage lies in its dramatically high compatibility.
The intended use case here is home cinema. While file size is a consideration, image quality takes precedence. If, over the course of the tests, it becomes evident that the results achievable with AV1 lead to smaller files but at the expense of lower image quality, this would be a decisive criterion against its adoption.

The primary motivation for transitioning from HEVC to AV1 can be succinctly summarized with the following quote from Wikipedia:

> Like VP9, but unlike H.264 (AVC) and H.265 (HEVC), AV1 has a royalty-free licensing model that does not hinder adoption in open-source projects.

To conduct the tests, initially, 24 test samples were provided, closely resembling real cinematic content (very closely), yet, of course, free from any copyright concerns and presented in BluRay quality. A test course was devised for these samples to navigate. It is worth mentioning that all tests are conducted with a bit depth of 10 and the pixel format "yuv420p10le".

The goal of the tests is to determine the parameters for the AV1 encoder ([SVT-AV1](https://gitlab.com/AOMediaCodec/SVT-AV1)) that yield comparable or even superior results to the HEVC encoder ([x265](https://bitbucket.org/multicoreware/x265_git/)). The evaluation incorporates the following metrics:

- **Time** required for encoding
- **File size** compared to the source
- **Video quality** assessed through the [VMAF](https://en.wikipedia.org/wiki/Video_Multimethod_Assessment_Fusion) score

## Test environment
All tests were conducted on fairly average consumer-grade hardware:

### Hardware
* AMD Ryzen 7 3700X @ factory clock speed
* 32 GiB DDR4 RAM @ 3600MHz

### Software
* Arch Linux
* [Kernel](https://archlinux.org/packages/core/x86_64/linux/): 6.6.8-arch1-1
* [ffmpeg](https://archlinux.org/packages/extra/x86_64/ffmpeg/): 2:6.1-3
* [svt-av1](https://archlinux.org/packages/extra/x86_64/svt-av1/): 1.8.0-1
* [x265](https://archlinux.org/packages/extra/x86_64/x265/): 3.5-3

```shell
ffmpeg -i input.mkv -an -c:v libx265 -preset slow -crf 21 -x265-params "aq-mode=1:qcomp=0.7:vbv-bufsize=9000:vbv-maxrate=9000:no-sao=1:no-strong-intra-smoothing=1:keyint=240:min-keyint=24" -pix_fmt yuv420p10le -y output.mkv
```

## Sample creation
As the source files were extensive movie files and only the video track was relevant for this test, they were stripped of all unnecessary tracks and reduced to a five-minute excerpt (starting from the timestamp 00:30:00 to avoid any kind of intros).

```shell
ffmpeg -ss 00:30:00 -i input.mkv -t 00:05:00 -vcodec copy -an -sn -dn -map_chapters -1 -y sample01.mkv
```

## HEVC encoding
Firstly, the samples undergo encoding in HEVC, utilizing the following parameters, which have proven effective in numerous practical applications:

```shell
ffmpeg -i input.mkv -an -c:v libx265 -preset slow -crf 21 -x265-params "aq-mode=1:qcomp=0.7:vbv-bufsize=9000:vbv-maxrate=9000:no-sao=1:no-strong-intra-smoothing=1:keyint=240:min-keyint=24" -pix_fmt yuv420p10le -y output.mkv
```

## AV1 encoding
Following that, all samples will be encoded using the AV1 codec. In terms of presets, the initial approach will adhere to the [recommendations](https://gitlab.com/AOMediaCodec/SVT-AV1/-/blob/master/Docs/Ffmpeg.md#example-2-encoding-for-personal-use) provided by the Alliance for Open Media:

> Presets **between 4 and 6** offer what many people consider a reasonable trade-off between quality and encoding time.

For the selection of CRF (Constant Rate Factor) values, we begin with the default setting in [HandBrake](https://handbrake.fr/) - **30**. To gain a clearer perspective, the value is then iteratively decreased by 1 until reaching a value of **10**.

This results in the following command:

```shell
ffmpeg -i input.mkv -an -c:v libsvtav1 -preset <PRESET> -crf <CRF> -svtav1-params tune=0 -pix_fmt yuv420p10le -g 245 -y output.mkv
```

## HEVC results
As mentioned above, all samples were initially encoded with HEVC. Now we have a baseline that we should aim to at least meet, if not surpass:

| Metric              | Mean    | Median  |
| ------------------- | ------- | ------- |
| Encoding Time       | {{ aggregated_metrics.hevc.encoding_time.mean }} sec | {{ aggregated_metrics.hevc.encoding_time.median }} sec |
| Filesize Percentage | {{ aggregated_metrics.hevc.filesize_percentage.mean }} pct  | {{ aggregated_metrics.hevc.filesize_percentage.median }} pct  |
| VMAF Score          | {{ aggregated_metrics.hevc.vmaf_score.mean }} pct  | {{ aggregated_metrics.hevc.vmaf_score.median }} pct  |

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/{{ diagrams.hevc.encoding_time }}"><img src="results/diagrams/{{ diagrams.hevc.encoding_time }}" width="600" height="450" /></a>

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/{{ diagrams.hevc.filesize_percentage }}"><img src="results/diagrams/{{ diagrams.hevc.filesize_percentage }}" width="600" height="450" /></a>

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/{{ diagrams.hevc.vmaf_score }}"><img src="results/diagrams/{{ diagrams.hevc.vmaf_score }}" width="600" height="450" /></a>

## AV1 results
Below are the results of all Preset-CRF combinations in a comprehensive table:

| Preset | CRF | Encoding Duration (% of HEVC, Mean) | Encoding Duration (% of HEVC, Median) | Filesize (% of sample, Mean) | Filesize (% of sample, Median) |  VMAF score (as %, Mean) | VMAF score (as %, Median) |
| ------ | --- | ----------------------------------- | ------------------------------------- | ---------------------------- | ------------------------------ | ------------------------ | ------------------------- |
{% for preset, crfs in aggregated_metrics.av1.items() -%}
{% for crf, metrics in crfs.items() -%}
| {{ preset }} | {{ crf }} | {{ metrics.encoding_time_percentage.mean }} pct | {{ metrics.encoding_time_percentage.median }} pct | {{ metrics.filesize_percentage.mean }} pct | {{ metrics.filesize_percentage.median }} pct | {{ metrics.vmaf_score.mean }} pct | {{ metrics.vmaf_score.median }} pct |
{% endfor %}{% endfor %}

Furthermore, there are three graphics that illustrate the results more effectively. It becomes evident, for instance, that as the CRF value decreases (across all three presets), the spread in filesizes (mean vs. median) increases.

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/av1_encoding_time_percentages.png"><img src="results/diagrams/av1_encoding_time_percentages.png" width="600" height="450" /></a>

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/av1_filesize_percentages.png"><img src="results/diagrams/av1_filesize_percentages.png" width="600" height="450" /></a>

<a href="AV1-vs-HEVC/src/branch/main/results/diagrams/av1_vmaf_scores.png"><img src="results/diagrams/av1_vmaf_scores.png" width="600" height="450" /></a>

## Looking for candidates
After collecting all the results, an analysis was conducted to determine which AV1 Preset-CRF combinations are at least on par with HEVC. The criteria for this assessment were straightforward and are presented here in pseudocode:
```python
filesize_mean_av1 <= filesize_mean_hevc
filesize_median_av1 <= filesize_median_hevc
vmaf_score_mean_av1 >= vmaf_score_mean_hevc
vmaf_score_median_av1 >= vmaf_score_median_hevc
```
It turned out that not a single Preset-CRF combination met all the criteria, and in this specific scenario, AV1 is not on par with HEVC.
Only when allowing a 20% margin in file size, there is one (!) Preset-CRF combination in AV1 (Preset 4 - CRF 17) whose VMAF score can match HEVC.
However, with these settings, the average encoding time also increases significantly by 75%.

## Conclusion
The results, with their raw numbers, clearly indicate that the AV1 codec is currently not suitable for the intended purpose mentioned in the introduction.
Among all combinations of preset and CRF, there wasn't a single one (!) that could compete with the results of HEVC.
While the differences between the results of HEVC and AV1 are not drastic (although a 75% increase in encoding time is noteworthy), one must simply weigh the importance of different criteria.
Whether the primary use case is the web or home cinema plays a significant role in this decision.
Since the initial focus of the test was on finding the better candidate for home cinema, HEVC must be considered the superior option.