Can't get hardware encoding to work on Linux with Radeon RX 6700 XT

I’ve been using shotcut for a few years now and absolutely love it. I started out with other video editors, but shotcut has given me the best experience so far. For the last few years, I used Nvidia GTX 1080 Ti as my GPU and for hardware encoding in shotcut without problem and it has been great. However, I recently switched to AMD Radeon RX 6700 XT and thought that I could still benefit from hardware encoding via the VA-API. Unfortunately, I’m having problems so I hope someone can point me in the right direction.

Description of problem:

If I click on “Configure” next to “Use hardware encoder” and click on “detect”, I get a message “Nothing found”. In the logs, this is what I see:

[Info   ] <MainWindow::showStatusMessage> "Detecting hardware encoders..." 
[Info   ] <EncodeDock::detectHardwareEncoders> checking for "h264_nvenc" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "/usr/bin/ffmpeg" ("-hide_banner", "-f", "lavfi", "-i", "color=s=640x360", "-frames", "1", "-an", "-c:v", "h264_nvenc", "-f", "rawvideo", "pipe:") 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Input #0, lavfi, from 'color=s=640x360':" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Duration: N/A, start: 0.000000, bitrate: N/A" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Stream mapping:" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_nvenc))" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Press [q] to stop, [?] for help" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "[h264_nvenc @ 0x564d1f84d100] Cannot load libcuda.so.1" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Conversion failed!" 
[Info   ] <EncodeDock::detectHardwareEncoders> checking for "hevc_nvenc" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "/usr/bin/ffmpeg" ("-hide_banner", "-f", "lavfi", "-i", "color=s=640x360", "-frames", "1", "-an", "-c:v", "hevc_nvenc", "-f", "rawvideo", "pipe:") 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Input #0, lavfi, from 'color=s=640x360':" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Duration: N/A, start: 0.000000, bitrate: N/A" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Stream mapping:" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "  Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (hevc_nvenc))" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Press [q] to stop, [?] for help" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "[hevc_nvenc @ 0x5611ed7ec100] Cannot load libcuda.so.1" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Conversion failed!" 
[Info   ] <EncodeDock::detectHardwareEncoders> checking for "h264_vaapi" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "/usr/bin/ffmpeg" ("-hide_banner", "-f", "lavfi", "-i", "color=s=640x360", "-frames", "1", "-an", "-init_hw_device", "vaapi=vaapi0:,connection_type=x11", "-filter_hw_device", "vaapi0", "-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", "-f", "rawvideo", "pipe:") 
[Debug  ] <EncodeDock::detectHardwareEncoders> "[AVHWDeviceContext @ 0x557c39fd9b00] No VA display found for any default device." 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Device creation failed: -22." 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Failed to set value 'vaapi=vaapi0:,connection_type=x11' for option 'init_hw_device': Invalid argument" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Error parsing global options: Invalid argument" 
[Info   ] <EncodeDock::detectHardwareEncoders> checking for "hevc_vaapi" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "/usr/bin/ffmpeg" ("-hide_banner", "-f", "lavfi", "-i", "color=s=640x360", "-frames", "1", "-an", "-init_hw_device", "vaapi=vaapi0:,connection_type=x11", "-filter_hw_device", "vaapi0", "-vf", "format=nv12,hwupload", "-c:v", "hevc_vaapi", "-f", "rawvideo", "pipe:") 
[Debug  ] <EncodeDock::detectHardwareEncoders> "[AVHWDeviceContext @ 0x560c333deb00] No VA display found for any default device." 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Device creation failed: -22." 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Failed to set value 'vaapi=vaapi0:,connection_type=x11' for option 'init_hw_device': Invalid argument" 
[Debug  ] <EncodeDock::detectHardwareEncoders> "Error parsing global options: Invalid argument" 
[Info   ] <MainWindow::showStatusMessage> "Nothing found" 

vainfo shows:

type $ vainfo 
libva info: VA-API version 1.13.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_13
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.13 (libva 2.13.0)
vainfo: Driver version: Mesa Gallium driver 21.3.9 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.46.0, 5.18.5-100.fc35.x86_64, LLVM 13.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

So I think it is supposed to work?

I am able to run ffmpeg with hardware acceleration successfully:

$ ffmpeg -init_hw_device vaapi=amd:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device amd -filter_hw_device amd -i input.mp4 -vf 'format=nv12|vaapi,hwupload' -y -c:v hevc_vaapi -qp:v 21 output.mp4
ffmpeg version 4.4.2 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librav1e --enable-librtmp --enable-librubberband --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libglslang --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.76.100
  Duration: 00:00:54.38, start: 0.000000, bitrate: 18603 kb/s
  Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 18339 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 257 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[hevc_vaapi @ 0x55d9bc803380] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.76.100
  Stream #0:0(und): Video: hevc (Main) (hev1 / 0x31766568), vaapi_vld(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 60 fps, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc58.134.100 hevc_vaapi
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc58.134.100 aac
frame= 3262 fps= 62 q=-0.0 Lsize=  240292kB time=00:00:54.35 bitrate=36213.6kbits/s speed=1.03x    
video:239377kB audio:854kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.025359%
[aac @ 0x55d9bc7a18c0] Qavg: 462.621

Details about my system:

Shotcut version: 21.08.03

inxi machine output:

System:
  Host: hp-z840 Kernel: 5.18.5-100.fc35.x86_64 arch: x86_64 bits: 64
    Desktop: MATE v: 1.26.0 Distro: Fedora release 35 (Thirty Five)
Machine:
  Type: Desktop System: Hewlett-Packard product: HP Z840 Workstation v: N/A
    serial: XXXXXXXXX
  Mobo: Hewlett-Packard model: 2129 v: 1.01 serial: N/A
    UEFI: Hewlett-Packard v: M60 v02.57 date: 05/21/2021
Memory:
  RAM: total: 251.7 GiB used: 16.75 GiB (6.7%)
  Array-1: capacity: 512 GiB slots: 8 EC: Multi-bit ECC
  Device-1: CPU0-DIMM1 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-2: CPU0-DIMM2 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-3: CPU0-DIMM3 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-4: CPU0-DIMM4 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-5: CPU0-DIMM5 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-6: CPU0-DIMM6 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-7: CPU0-DIMM7 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-8: CPU0-DIMM8 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Array-2: capacity: 512 GiB slots: 8 EC: Multi-bit ECC
  Device-1: CPU1-DIMM1 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-2: CPU1-DIMM2 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-3: CPU1-DIMM3 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-4: CPU1-DIMM4 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-5: CPU1-DIMM5 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-6: CPU1-DIMM6 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-7: CPU1-DIMM7 type: DDR4 size: 16 GiB speed: 2400 MT/s
  Device-8: CPU1-DIMM8 type: DDR4 size: 16 GiB speed: 2400 MT/s
CPU:
  Info: 2x 14-core model: Intel Xeon E5-2680 v4 bits: 64 type: MT MCP SMP
    cache: L2: 2x 3.5 MiB (7 MiB)
  Speed (MHz): avg: 1247 min/max: 1200/3300 cores: 1: 1199 2: 1198 3: 1198
    4: 1200 5: 1198 6: 1198 7: 1198 8: 1198 9: 1198 10: 1202 11: 1198 12: 1757
    13: 1198 14: 1198 15: 1197 16: 1199 17: 1198 18: 1198 19: 1199 20: 1198
    21: 1198 22: 1198 23: 1844 24: 1198 25: 2097 26: 1198 27: 1198 28: 1198
    29: 1198 30: 1198 31: 1198 32: 1197 33: 1198 34: 1197 35: 1197 36: 1197
    37: 1197 38: 1197 39: 1197 40: 1476 41: 1197 42: 1198 43: 1197 44: 1198
    45: 1197 46: 1197 47: 1197 48: 1197 49: 1197 50: 1297 51: 1198 52: 1197
    53: 1471 54: 1197 55: 1198 56: 1198
Graphics:
  Device-1: AMD Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M]
    driver: amdgpu v: kernel
  Display: x11 server: X.Org v: 1.20.14 driver: X: loaded: amdgpu
    gpu: amdgpu resolution: 3840x2160~60Hz
  OpenGL: renderer: AMD Radeon RX 6700 XT (NAVY_FLOUNDER DRM 3.46.0
    5.18.5-100.fc35.x86_64 LLVM 13.0.0)
    v: 4.6 Mesa 21.3.9
Audio:
  Device-1: Intel C610/X99 series HD Audio driver: snd_hda_intel
  Device-2: AMD Navi 21/23 HDMI/DP Audio driver: snd_hda_intel
  Device-3: Logitech Yeti X type: USB
    driver: hid-generic,snd-usb-audio,usbhid
  Sound Server-1: ALSA v: k5.18.5-100.fc35.x86_64 running: yes
  Sound Server-2: PipeWire v: 0.3.52 running: yes
Network:
  Device-1: Intel Ethernet I218-LM driver: e1000e
  IF: eno1 state: up speed: 1000 Mbps duplex: full mac: a0:8c:fd:f0:47:f4
  Device-2: Intel I210 Gigabit Network driver: igb
  IF: enp10s0 state: down mac: a0:8c:fd:f0:47:f5
  IF-ID-1: virbr0 state: down mac: 52:54:00:8e:ff:84
RAID:
  Device-1: data type: zfs status: ONLINE level: linear raw: size: 3.72 TiB
    free: 3.62 TiB zfs-fs: size: 3.6 TiB free: 3.5 TiB
  Components: Online: 1: 2: 3: 4:
  Device-2: data2 type: zfs status: ONLINE level: raidz2-0 raw:
    size: 63.7 TiB free: 63.5 TiB zfs-fs: size: 42.27 TiB free: 42.17 TiB
  Components: Online: 1: sda 2: sdb 3: sdc 4: sdd 5: sde 6: sdf 7: sdg
Drives:
  Local Storage: total: raw: 67.63 TiB usable: 53.55 TiB
    used: 250.05 GiB (0.5%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB256HBHQ-000L7
    size: 238.47 GiB
  ID-2: /dev/nvme1n1 vendor: Samsung model: MZVLB1T0HBLR-000L2
    size: 953.87 GiB
  ID-3: /dev/nvme2n1 vendor: Samsung model: MZVLB1T0HBLR-000L7
    size: 953.87 GiB
  ID-4: /dev/nvme3n1 vendor: Samsung model: MZVLB1T0HBLR-000L7
    size: 953.87 GiB
  ID-5: /dev/nvme4n1 vendor: Samsung model: MZVLB1T0HBLR-000L7
    size: 953.87 GiB
  ID-6: /dev/sda vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-7: /dev/sdb vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-8: /dev/sdc vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-9: /dev/sdd vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-10: /dev/sde vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-11: /dev/sdf vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
  ID-12: /dev/sdg vendor: Hitachi model: HUH721010AL4200 size: 9.1 TiB
Partition:
  ID-1: / size: 236.89 GiB used: 42.98 GiB (18.1%) fs: btrfs
    dev: /dev/nvme0n1p3
  ID-2: /boot size: 973.4 MiB used: 274.9 MiB (28.2%) fs: ext4
    dev: /dev/nvme0n1p2
  ID-3: /boot/efi size: 598.8 MiB used: 6.7 MiB (1.1%) fs: vfat
    dev: /dev/nvme0n1p1
  ID-4: /home size: 3.6 TiB used: 103.37 GiB (2.8%) fs: zfs
    logical: data/home
Swap:
  ID-1: swap-1 type: zram size: 8 GiB used: 0 KiB (0.0%) dev: /dev/zram0

I did do some searching before posting, and I did see this thread:

However, using the LIBVA_DRIVER_NAME=radeonsi environment variable did not help and showed exactly the same symptoms as described.

Any help would be very much appreciated. If more information needs to be provided, I am happy to do so, but I hope I didn’t miss anything above.

You are using a too old version of Shotcut. That problem was fixed in v22.01. It may or may not work if you manually configure it to use vaapi instead of using detection.

@shotcut Thank you for the reply. I forgot to mention it, but when i manually configured it to use vaapi, the export created a blank/black video file with audio only, and no video.

I will see about trying v22.01 or newer.

Just a follow up. Used version 22.06.23 and it just worked. Sorry, I should have tried that first, but my Linux distro only included the version I first posted about. Got the latest portable Linux tarball and everything just worked! Thanks!

1 Like

This topic was automatically closed after 90 days. New replies are no longer allowed.