add opencv benchmark #711

Dan-Flores · 2025-06-04T14:37:27Z

This PR updates the changes in #674, which added a benchmark for decoding with the OpenCV library.

Changes in this PR:

Removed iteration through available backends
Removed API/ABI compatibility checks
Added FFMPEG as default backend for OpenCV
Added option to select stream_index in TorchCodecPublic, TorchAudioDecoder

Benchmark 1: nasa_13013.mp4 (320x180)

The file "nasa_13013.mp4" has multiple video streams. This can be seen by calling ffprobe test/resources/nasa_13013.mp4.
For accurate benchmarking, we should ensure the same video streams are being decoded. This was not true for the original benchmark in add opencv benchmark #674, hence the unexpected results.
- OpenCV decoder with FFMPEG backend does not support setting the video stream, height, or width parameters.
- The updated benchmark below specifies a stream_index to TorchCodecPublic and TorchAudioDecoder.

python benchmarks/decoders/benchmark_decoders.py --decoders torchcodec_public:seek_mode=approximate+stream_index=0,torchcodec_public:seek_mode=exact+stream_index=0,opencv,torchaudio:stream_index=0 --min-run-seconds 40

[--------------- video=/home/danielflores3/github/Dan-Flores/benchmarks/decoders/../../test/resources/nasa_13013.mp4 h264 480x270, 13.013s 29.97002997002997fps ---------------]
                                                             |  decode 10 uniform frames  |  decode 10 random frames  |  first 1 frames  |  first 10 frames  |  first 100 frames
1 threads: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
      OpenCV[backend=FFMPEG]                                 |            32.6            |            32.5           |        9.9       |        10.9       |        26.6      
      TorchAudio:stream_index=0                              |            93.3            |            88.6           |       10.5       |        11.9       |        31.4      
      TorchCodecPublic:seek_mode=approximate+stream_index=0  |            35.3            |            31.7           |       10.3       |        11.1       |        25.3      
      TorchCodecPublic:seek_mode=exact+stream_index=0        |            37.1            |            33.5           |       12.3       |        13.1       |        27.9      

Times are in milliseconds (ms).

Benchmark 2: mandelbrot_1920x1080_120s.mp4

To generate a video for this benchmark, the command below was used.
- The generated video file has one video stream to ensure the decoders select the same video stream.

ffmpeg -y -f lavfi -i mandelbrot=s=1920x1080 -t 120 -c:v h264 -r 60 -g 600 -pix_fmt yuv420p mandelbrot_1920x1080_120s.mp4

python benchmarks/decoders/benchmark_decoders.py --decoders torchcodec_public:seek_mode=approximate,torchcodec_public:seek_mode=exact,opencv,torchaudio --min-run-seconds 40  --video-paths mandelbrot_1920x1080_120s.mp4

[----------------------------------------------------- video=mandelbrot_1920x1080_120s.mp4 h264 1920x1080, 120.0s 60.0fps -----------------------------------------------------]
                                                             |  decode 10 uniform frames  |  decode 10 random frames  |  first 1 frames  |  first 10 frames  |  first 100 frames
1 threads: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
      OpenCV[backend=FFMPEG]                                 |          15146.3           |          15064.6          |       50.8       |        80.1       |       363.6      
      TorchAudio:stream_index=0                              |           6944.1           |           8415.5          |       57.5       |       100.3       |       555.6      
      TorchCodecPublic:seek_mode=approximate+stream_index=0  |           6804.0           |           5428.5          |       50.3       |        83.4       |       449.2      
      TorchCodecPublic:seek_mode=exact+stream_index=0        |           6936.3           |           5591.3          |      188.8       |       223.1       |       592.1      

Times are in milliseconds (ms).

Interpreting the Results:

The updated results for benchmarking nasa_13013.mp4 on the lower resolution stream shows similar performance by TorchCodec and OpenCV.
The results of benchmarking the high resolution video show varied results.
- OpenCV's decoder sequentially decodes frames. This approach results in lower performance when decoding random and uniform frames, but may be more optimized for decoding frames in order, due to the grab() and retrieve() functions.
- TorchCodec decoders perform significantly better than OpenCV at decoding random and uniform frames, but are slower than OpenCV when decoding a high number of sequential frames.
  - TorchCodec in exact mode sees a large performance decrease when decoding few sequential frames due to performing a full scan of the file.
  - TorchCodec in approximate mode maintains higher performance by avoiding the full scan.

NicolasHug

Thanks for the PR @Dan-Flores ! It looks good, I shared a few comments below. As we just discussed offline, it might be worth checking the output frames for validity, to make sure that the all the decoders we're benchmarking are returning similar frames.

NicolasHug · 2025-06-05T15:32:52Z

benchmarks/decoders/benchmark_decoders_library.py

+        self._print_each_iteration_time = False
+
+    def decode_frames(self, video_file, pts_list):
+        import cv2


It's best to only import in __init__ and then store the module as a self.cv2 attibute. Otherwise, we'd be benchmarking the import statement when calling the decoding method, and it may add noise to the results.

benchmarks/decoders/benchmark_decoders_library.py

NicolasHug · 2025-06-05T15:39:44Z

benchmarks/decoders/benchmark_decoders_library.py

+        import cv2
+
+        frames = [
+            cv2.resize(frame, (width, height))


We should note that openCV doesn't apply antialias, while the rest of the decode_and_resize() methods apply antialias by default. That makes these methods less comparable.

I would suggest not to expose decode_and_resize for openCV if we can, so as to prevent any confusion, but if we need to expose it for technical reasons then let's at least add a comment pointing out this discrepancy about antialiasing.

CC @scotts as it's relevant to the whole "resize inconsistency chaos"

decode_and_reize() is needed to generate data for the README, so I've left a comment here.

NicolasHug

Thanks for the PR @Dan-Flores ! It looks good, I shared a few comments below. As we just discussed offline, it might be worth checking the output frames for validity, to make sure that the all the decoders we're benchmarking are returning similar frames.

NicolasHug

Thanks for the PR @Dan-Flores ! It looks good, I shared a few comments below. As we just discussed offline, it might be worth checking the output frames for validity, to make sure that the all the decoders we're benchmarking are returning similar frames.

Dan-Flores · 2025-06-06T20:02:09Z

Thanks for the PR @Dan-Flores ! It looks good, I shared a few comments below. As we just discussed offline, it might be worth checking the output frames for validity, to make sure that the all the decoders we're benchmarking are returning similar frames.

cc @NicolasHug - It seems OpenCV only supports certain frame resolutions, so the benchmark on nasa13013.mp4 was using a smaller resolution, and was not a fair comparison (see image below). I was unfortunately unable to modify the VideoCapture's resolution - OpenCV's documentation on these properties suggests that modifying them can fail unexpectedly.

I ran the benchmark using a generated mandelbrot video at 1920x1080, which OpenCV is able to support.

NicolasHug · 2025-06-09T09:57:25Z

Thanks for checking @Dan-Flores !

I think what's happening with nasa_13013.mp4 is simply that opencv is decoding a different stream! We can see the streams with ffprobe:

~/dev/torchcodec  » ffprobe test/resources/nasa_13013.mp4                                        
...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test/resources/nasa_13013.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.76.100
  Duration: 00:00:13.06, start: 0.000000, bitrate: 412 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 320x180 [SAR 1:1 DAR 16:9], 71 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, stereo, fltp, 72 kb/s (default)
    Metadata:
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Subtitle: mov_text (tx3g / 0x67337874), 0 kb/s (default)
    Metadata:
      handler_name    : SubtitleHandler
  Stream #0:3[0x4](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 480x270 [SAR 1:1 DAR 16:9], 128 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:4[0x5](eng): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:5[0x6](eng): Subtitle: mov_text (tx3g / 0x67337874), 0 kb/s (default)
    Metadata:
      handler_name    : SubtitleHandler

the mp4 containers has 2 video streams: one with 320x180 resolution and one with 480x270 resolution. OpenCV decodes the first one while we decode the second one, because it's what ffmpeg considers to be the "best" stream.
So to get a fair comparison we just need to tell opencv to decode the same stream.

And BTW, we probably want to double check we were decoding the same streams with our other benchmark backends (torchaudio etc.), but that can be done separately.

Dan-Flores · 2025-06-10T20:31:34Z

benchmarks/decoders/benchmark_decoders_library.py

@@ -828,7 +829,7 @@ def run_benchmarks(
        # are using different random pts values across videos.
        random_pts_list = (torch.rand(num_samples) * duration).tolist()

-        for decoder_name, decoder in decoder_dict.items():
+        for decoder_name, decoder in sorted(decoder_dict.items(), key=lambda x: x[0]):


This change was added to make it easier to compare benchmarks, I'm open to alternative approaches to sorting or removing it entirely.

jinhohwang-meta and others added 6 commits June 3, 2025 13:05

add opencv benchmark

ea4bcb5

reflect comments

beebaf2

Remove backend checking from OpenCV benchmark

109a38b

Remove backend checking from OpenCV benchmark

a6f95f0

Set ffmpeg as default backend for OpenCV

1a64027

Remove benchmarks.png

6ad479c

facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 4, 2025

Dan-Flores marked this pull request as ready for review June 4, 2025 19:04

NicolasHug reviewed Jun 5, 2025

View reviewed changes

Remove duplicate imports, return tensor

1854f5f

Dan-Flores added 3 commits June 10, 2025 10:07

Add stream_index option to TorchCodecPublic decoder

dc75277

sort before plotting

65f750f

Add stream_index to TorchAudioDecoder

df508df

Dan-Flores commented Jun 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add opencv benchmark #711

add opencv benchmark #711

Dan-Flores commented Jun 4, 2025 •

edited

Loading

Uh oh!

NicolasHug left a comment

Uh oh!

NicolasHug Jun 5, 2025

Uh oh!

Uh oh!

NicolasHug Jun 5, 2025

Uh oh!

Dan-Flores Jun 6, 2025

Uh oh!

NicolasHug left a comment

Uh oh!

NicolasHug left a comment

Uh oh!

Dan-Flores commented Jun 6, 2025

Uh oh!

NicolasHug commented Jun 9, 2025 •

edited

Loading

Uh oh!

Dan-Flores Jun 10, 2025

Uh oh!

Uh oh!

add opencv benchmark #711

Are you sure you want to change the base?

add opencv benchmark #711

Conversation

Dan-Flores commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes in this PR:

Benchmark 1: nasa_13013.mp4 (320x180)

Benchmark 2: mandelbrot_1920x1080_120s.mp4

Interpreting the Results:

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

NicolasHug Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NicolasHug Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Dan-Flores Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

Dan-Flores commented Jun 6, 2025

Uh oh!

NicolasHug commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dan-Flores Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Dan-Flores commented Jun 4, 2025 •

edited

Loading

NicolasHug commented Jun 9, 2025 •

edited

Loading