Streaming real-time XR to the web sounds great, until latency ruins the experience.
In this post, we explain how we made server-side rendering not just usable, but comfortable in XR, using a few clever tricks to smooth out the delay and keep everything feeling grounded, even when the network isn’t.
In the end of 2023 Vulkan released Vulkan Video extensions which allowed for GPU-accelerated encoding of H.264 and H.265 video.
This was bleeding edge stuff: these extensions provided a GPU-agnostic solution for video compression which did not exist previously.
At 3dverse, our renderer was dependent on NVIDIA solely because of video encoding, so this new solution would finally make us GPU-agnostic!
However. To this day, there is very limited information online to help someone implement anything with these extensions.
The following is a semi-rageful but triumphant account of working with Vulkan Video from the vantage point of an engine programmer who had no previous experience with Vulkan, or any video compression codecs.
All these will be part of the blog post summary.