Hardware encoding Linux Radeon Navi10 5600XT

Hello I’d really like to get hardware encoding working on this fantastic video editing suite.

Issue Summary - Hardware Encoding Linux vaapi AMD Radeon 5600 XT

Whenever I try to export using hardware encoding on either h264 or hevc I receive the error pasted below.

I have enabled h264_vaapi and hevc_vaapi in the Configure Hardware Encoding options and checked Use Hardware Encoder.

Shotcut error (from export job log)

[AVHWDeviceContext @ 0x7f9098019e40] Opened VA display via X11 display :0.
[AVHWDeviceContext @ 0x7f9098019e40] libva: VA-API version 1.9.0
[AVHWDeviceContext @ 0x7f9098019e40] libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[AVHWDeviceContext @ 0x7f9098019e40] Failed to initialise VAAPI connection: -1 (unknown libva error).
Failed to create VAAPI device.
[consumer avformat] Failed to initialize VA-API: -5
[h264_vaapi @ 0x7f9098001d80] A hardware frames reference is required to associate the encoding device.
Failed with exit code 11
Failed with exit code 11


As you can see hardware encoding is available under vaapi with multiple “VAEntrypointEncSlice” entries. Harware encoding also works well under ffmpeg (see below).

vainfo: VA-API version: 1.9 (libva 2.9.1)
vainfo: Driver version: Mesa Gallium driver 20.2.3 for AMD Radeon RX 5600 XT (NAVI10, DRM 3.39.0, 5.9.10-arch1-1, LLVM 11.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

ffmpeg works!

The strange thing is using vaapi under ffmpeg the hardware encoding works without issues. It’s super fast too!

ffmpeg -init_hw_device vaapi=amd:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device amd -filter_hw_device amd -i input.mp4 -vf 'format=nv12|vaapi,hwupload' -y -c:v hevc_vaapi -qp:v 21 output.mp4


0c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev ca)

System specs

  • Arch Linux (latest)
  • GPU AMD Radeon RX 5600 XT (NAVI10, DRM 3.39.0, 5.9.10-arch1-1, LLVM 11.0.0)
  • gnome desktop with mutter window manager and wayland session

I would appreciate any ideas on what else I can do to troubleshoot this.

OK I’ve fixed this - hardware encoding now works!

Hardware Encoding Fix

Adding the LIBVA environment variable enables hardware encoding and autodetect.
$ LIBVA_DRIVER_NAME=radeonsi shotcut

QT5 on Wayland

$QT_QPA_PLATFORM=wayland LIBVA_DRIVER_NAME=radeonsi shotcut

However using hevc_vaapi results in a file size twice as large as compared with using libx265.

1 Like

This topic was automatically closed after 90 days. New replies are no longer allowed.