Debugging Strategies for Vulkan Video

The following is a summary of debugging strategies I developped during my struggles with encoding H.264/H.265 using Vulkan Video. It is intended to be a brief and helpful guide for others suffering under Vulkan Video, more specifically with the setting of video session parameters. This is part 3 to the adventure I started writing about here.

---

Most of my issues in getting Vulkan Video to encode H.264/H.265 were related to the setting of session parameters (VPS, SPS, PPS or VUI). More specifically, vkGetEncodedVideoSessionParametersKHR was returning VK_OUT_OF_HOST_MEMORY, which was Vulkan's cryptic reaction to a wrongly set parameter. To debug these issues, I incrementally gained a high level understanding of how the H.264/H.265 codecs functioned.

However, to speed up the process, I also developped debugging strategies that helped me narrow down on the issue in a bulldozer brute-force get-the-job-done way. Of course as software engineers, we all know what it is to “get code working without understanding what it does”. Maybe some of us are more guilty of this than others. It is definitely not my mantra; I think it is important to know to a certain extent what is going on behind the scenes of the lines of code you write. But you have to pick and choose your battles.

I knew I probably didn’t have the motivation to spend months learning about video compression by means of an austere 700 page PDF before even touching any code.

This is the 700 page-long PDF ITU-T Rec. H.264 (08/2010) Advanced video coding for generic audiovisual services. Good luck trying to avoid this document if you want to learn anything about H.264. I was not able to find documentation that provided a friendly and resourceful dive into H.264, just the spec.

The 700 page-long sequel ITU-T Rec. H.265 (08/2021) High efficiency video coding.

What I did instead with the dreaded session parameters was try to gain at least some superficial understanding of what they were as I went. I also deployed the following brute-force debugging strategies, which I recommend to any others who are struggling.

First Helpful Realization

There is a way to narrow down which parameter set is causing issues with vkGetEncodedVideoSessionParametersKHR. First off, this is the general code which precedes a vkGetEncodedVideoSessionParametersKHR call:

// (code is simplified for readability)

VkVideoEncodeH265SessionParametersGetInfoKHR h265_params_get_info;
h265_params_get_info.writeStdVPS = VK_TRUE;
h265_params_get_info.writeStdSPS = VK_TRUE;
h265_params_get_info.writeStdPPS = VK_TRUE;

// This is how I pass the parameters to vkGetEncodedVideoSessionParametersKHR
VkVideoEncodeSessionParametersGetInfoKHR params_get_info;
// These are the actual parameters in question
params_get_info.videoSessionParameters = params;
params_get_info.pNext = &h265_params_get_info;

// This is what returned VK_OUT_OF_HOST_MEMORY in my case
const auto result = vkGetEncodedVideoSessionParametersKHR(
			vulkan_device, &params_get_info,
			&feedback_info, encoded_params_buffer.size(), encoded_params_buffer.data());

Say I wanted to check if it was the VPS specifically that was causing the issue. I could do this:

VkVideoEncodeH265SessionParametersGetInfoKHR h265_params_get_info;
h265_params_get_info.writeStdVPS = VK_TRUE;
h265_params_get_info.writeStdSPS = VK_FALSE;
h265_params_get_info.writeStdPPS = VK_FALSE;

Now vkGetEncodedVideoSessionParametersKHR would only try to encode the VPS and if I kept getting the same error, then it was safe to point the finger at the VPS.

I’ll outline my game plans, as they came to me. It was game plan 3 and 4 which were most instrumental in getting me to the finish line.

Game Plan 1

My first game plan consisted of:

1. Narrow down which parameter set(s) contained invalid parameters using the trick outlined above.

2. Zero intialize the problematic parameter set.

   // Zero initialize. All parameters = 0.
   StdVideoH265VideoParameterSet vps = {};
   // Is this an acceptable configuration of VPS? Does vkGetEncodedVideoSessionParametersKHR return VK_SUCCCESS? No? Go to 3.

3. Construct a minimally set version of the parameter set until vkGetEncodedVideoSessionParametersKHR returned VK_SUCCESS.

StdVideoH265VideoParameterSet vps = {};
vps.flags.vps_temporal_id_nesting_flag = 1;
// Is this an acceptable configuration of VPS? No? Keep trying filling other parameters.

I started doing this, but unfortunately it was not great, because I was also looking up information on all the parameters as I went to make sure I was putting in accepted values. It was more time consuming. It allowed me to narrow down the problematic parameter set, but not the problematic parameter.

Game Plan 2

1. Go through NVIDIA's official vk_video_samples and try to copy the parameter initialization values as much as possible.

If I copied everything to the T then this would presumably work. This game plan was tedious and inefficient as the code in vk_video_samples is not that easy to parse through. It sets many parameters but ideally in my implementation I would keep it more minimal. I didn’t want to copy a spaghetti code situation. It was also not the most accurate way of “copying” as not all parameters were hard coded, rather they were calculated or dependent on user program args. This lead me to game plan 3.

Game Plan 3

This is the game plan that helped me get H.265 working. The following synthesizes my strategy (in a helpful TLDR way) that I elaborate on in my second article of this series, H.265 Vulkan Video Encoding.

1. Create a Vulkan API trace of my application encoding a video
2. Create a Vulkan API trace of NVIDIA's official vk_video_samples encoding the same video
3. Compare each API trace (do a diff) with some tool like Diff Checker. Specifically do a diff of the vkGetEncodedVideoSessionParametersKHR call to have a crystal clear view of what parameters were being provided to that Vulkan call in both applications
4. For each parameter that was set differently, do a deep dive on it:
   1. Try to gain an understanding of what that parameter is (helpful to use AI for this, I used Claude.ai)
   2. Ctrl+F the parameter within the 700 page long H.265 PDF which outlined the accepted values of that parameter. Was my provided value unaccepted?
   3. If it makes ostensible sense to just copy what vk_video_samples provided, then just copy it otherwise change it to an accepted value that makes sense

Game Plan 4

This is the game plan that helped me get H.264 working, which I wrote about in H.264 Vulkan Video Encoding.

1. Create a Vulkan API trace of my application encoding a video
2. Post it on Vulkan Discord video channel and beg for help. There are a lot of helpful people on there