The following is a summary of debugging strategies I developped during my struggles with encoding H.264/H.265 using Vulkan Video. It is intended to be a brief and helpful guide for others suffering under Vulkan Video, more specifically with the setting of video session parameters.
A few months ago I was given the task to integrate an open source Vulkan video encoder into our renderer. I had never touched Vulkan before (I vaguely knew to feel fear towards it) and knew nothing about video compression, namely the H.264 and H.265/HEVC codecs.
Streaming real-time XR to the web sounds great, until latency ruins the experience.
In this post, we explain how we made server-side rendering not just usable, but comfortable in XR, using a few clever tricks to smooth out the delay and keep everything feeling grounded, even when the network isn’t.
In the end of 2023 Vulkan released Vulkan Video extensions which allowed for GPU-accelerated encoding of H.264 and H.265 video. This was bleeding edge stuff: these extensions provided a GPU-agnostic solution for video compression which did not exist previously. At 3dverse, our renderer was dependent on NVIDIA solely because of video encoding, so this new solution would finally make us GPU-agnostic!
However. To this day, there is very limited information online to help someone implement anything with these extensions. The following is a semi-rageful but triumphant account of working with Vulkan Video from the vantage point of an engine programmer who had no previous experience with Vulkan, or any video compression codecs.
