Vulkan testing with VkRunner

Submitted by Neil Roberts on April 4, 2018, 10:56 p.m.

Details

Reviewer None
Submitted April 4, 2018, 10:56 p.m.
Last Updated Nov. 3, 2018, 7:04 a.m.
Revision 4

Cover Letter(s)

Revision 1
      This series sets up a framework for testing Vulkan in Piglit. The main
thing is to import a tool like shader_runner called VkRunner. There
are also 5 actual tests which mainly serve as examples.

The code for VkRunner was developed as a standalone project here:

https://github.com/igalia/vkrunner

If this branch gets merged it would probably make sense to abandon
that repo and have Piglit as the canonical source for the code.

Eventually we could probably get a lot of testing fairly cheaply by
modifying some of the generated_tests scripts to also generate Vulkan
tests. It might even be worth semi-automatically converting existing
GL shader tests using something similar to the script on the
GL_ARB_gl_spirv branch.

For a general overview of VkRunner you might like to take a look at
this blog post:

https://blogs.igalia.com/nroberts/2018/04/03/vkrunner-a-shader-test-tool-for-vulkan/

Neil Roberts (8):
  util: Add a PIGLIT_STRINGIFY macro
  util: Add a piglit_fatal utility
  util: Add a libutil_vulkan library
  util: Move the half float functions from util-gl to util
  framework: Add a Vulkan platform
  Import VkRunner
  framework: Add a vulkan tests profile
  vulkan: Add some tests for glsl450 builtin functions using doubles

 CMakeLists.txt                                     |    7 +
 cmake/piglit_util.cmake                            |    2 +-
 cmake/target_api/CMakeLists.txt                    |    4 +
 cmake/target_api/vulkan/CMakeLists.txt             |   10 +
 framework/test/vk_shader_test.py                   |   39 +
 tests/CMakeLists.txt                               |    4 +
 tests/util/CMakeLists.vulkan.txt                   |   15 +
 tests/util/piglit-util-gl.c                        |   90 --
 tests/util/piglit-util-gl.h                        |   29 -
 tests/util/piglit-util-vulkan.c                    |    1 +
 tests/util/piglit-util.c                           |  105 ++
 tests/util/piglit-util.h                           |   32 +
 tests/vulkan.py                                    |   30 +
 tests/vulkan/CMakeLists.txt                        |    1 +
 .../glsl450/face-forward-double.vk_shader_test     |   88 ++
 tests/vulkan/glsl450/frexp-double.vk_shader_test   |   61 +
 tests/vulkan/glsl450/isinf-double.vk_shader_test   |   81 +
 tests/vulkan/glsl450/reflect-double.vk_shader_test |   55 +
 tests/vulkan/glsl450/refract-double.vk_shader_test |   88 ++
 tests/vulkan/vkrunner/CMakeLists.txt               |    1 +
 tests/vulkan/vkrunner/CMakeLists.vulkan.txt        |   46 +
 tests/vulkan/vkrunner/README.md                    |  125 ++
 tests/vulkan/vkrunner/main.c                       |  160 ++
 tests/vulkan/vkrunner/make-formats.py              |   93 ++
 tests/vulkan/vkrunner/vr-allocate-store.c          |  181 +++
 tests/vulkan/vkrunner/vr-allocate-store.h          |   48 +
 tests/vulkan/vkrunner/vr-buffer.c                  |   87 ++
 tests/vulkan/vkrunner/vr-buffer.h                  |   73 +
 tests/vulkan/vkrunner/vr-config.c                  |  117 ++
 tests/vulkan/vkrunner/vr-config.h                  |   49 +
 tests/vulkan/vkrunner/vr-error-message.c           |   41 +
 tests/vulkan/vkrunner/vr-error-message.h           |   35 +
 tests/vulkan/vkrunner/vr-feature-offsets.c         |   92 ++
 tests/vulkan/vkrunner/vr-feature-offsets.h         |   39 +
 tests/vulkan/vkrunner/vr-flush-memory.c            |   52 +
 tests/vulkan/vkrunner/vr-flush-memory.h            |   37 +
 tests/vulkan/vkrunner/vr-format-table.h            | 1580 ++++++++++++++++++++
 tests/vulkan/vkrunner/vr-format.c                  |  289 ++++
 tests/vulkan/vkrunner/vr-format.h                  |   86 ++
 tests/vulkan/vkrunner/vr-list.c                    |   89 ++
 tests/vulkan/vkrunner/vr-list.h                    |  122 ++
 tests/vulkan/vkrunner/vr-pipeline.c                |  668 +++++++++
 tests/vulkan/vkrunner/vr-pipeline.h                |   54 +
 tests/vulkan/vkrunner/vr-script.c                  |  948 ++++++++++++
 tests/vulkan/vkrunner/vr-script.h                  |  146 ++
 tests/vulkan/vkrunner/vr-subprocess.c              |   61 +
 tests/vulkan/vkrunner/vr-subprocess.h              |   34 +
 tests/vulkan/vkrunner/vr-test.c                    |  569 +++++++
 tests/vulkan/vkrunner/vr-test.h                    |   40 +
 tests/vulkan/vkrunner/vr-vbo.c                     |  637 ++++++++
 tests/vulkan/vkrunner/vr-vbo.h                     |   77 +
 tests/vulkan/vkrunner/vr-vk-core-funcs.h           |    1 +
 tests/vulkan/vkrunner/vr-vk-device-funcs.h         |   56 +
 tests/vulkan/vkrunner/vr-vk-instance-funcs.h       |    9 +
 tests/vulkan/vkrunner/vr-vk.c                      |  148 ++
 tests/vulkan/vkrunner/vr-vk.h                      |   56 +
 tests/vulkan/vkrunner/vr-window.c                  |  678 +++++++++
 tests/vulkan/vkrunner/vr-window.h                  |   73 +
 58 files changed, 8319 insertions(+), 120 deletions(-)
 create mode 100644 cmake/target_api/vulkan/CMakeLists.txt
 create mode 100644 framework/test/vk_shader_test.py
 create mode 100644 tests/util/CMakeLists.vulkan.txt
 create mode 100644 tests/util/piglit-util-vulkan.c
 create mode 100644 tests/vulkan.py
 create mode 100644 tests/vulkan/CMakeLists.txt
 create mode 100644 tests/vulkan/glsl450/face-forward-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/frexp-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/isinf-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/reflect-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/refract-double.vk_shader_test
 create mode 100644 tests/vulkan/vkrunner/CMakeLists.txt
 create mode 100644 tests/vulkan/vkrunner/CMakeLists.vulkan.txt
 create mode 100644 tests/vulkan/vkrunner/README.md
 create mode 100644 tests/vulkan/vkrunner/main.c
 create mode 100755 tests/vulkan/vkrunner/make-formats.py
 create mode 100644 tests/vulkan/vkrunner/vr-allocate-store.c
 create mode 100644 tests/vulkan/vkrunner/vr-allocate-store.h
 create mode 100644 tests/vulkan/vkrunner/vr-buffer.c
 create mode 100644 tests/vulkan/vkrunner/vr-buffer.h
 create mode 100644 tests/vulkan/vkrunner/vr-config.c
 create mode 100644 tests/vulkan/vkrunner/vr-config.h
 create mode 100644 tests/vulkan/vkrunner/vr-error-message.c
 create mode 100644 tests/vulkan/vkrunner/vr-error-message.h
 create mode 100644 tests/vulkan/vkrunner/vr-feature-offsets.c
 create mode 100644 tests/vulkan/vkrunner/vr-feature-offsets.h
 create mode 100644 tests/vulkan/vkrunner/vr-flush-memory.c
 create mode 100644 tests/vulkan/vkrunner/vr-flush-memory.h
 create mode 100644 tests/vulkan/vkrunner/vr-format-table.h
 create mode 100644 tests/vulkan/vkrunner/vr-format.c
 create mode 100644 tests/vulkan/vkrunner/vr-format.h
 create mode 100644 tests/vulkan/vkrunner/vr-list.c
 create mode 100644 tests/vulkan/vkrunner/vr-list.h
 create mode 100644 tests/vulkan/vkrunner/vr-pipeline.c
 create mode 100644 tests/vulkan/vkrunner/vr-pipeline.h
 create mode 100644 tests/vulkan/vkrunner/vr-script.c
 create mode 100644 tests/vulkan/vkrunner/vr-script.h
 create mode 100644 tests/vulkan/vkrunner/vr-subprocess.c
 create mode 100644 tests/vulkan/vkrunner/vr-subprocess.h
 create mode 100644 tests/vulkan/vkrunner/vr-test.c
 create mode 100644 tests/vulkan/vkrunner/vr-test.h
 create mode 100644 tests/vulkan/vkrunner/vr-vbo.c
 create mode 100644 tests/vulkan/vkrunner/vr-vbo.h
 create mode 100644 tests/vulkan/vkrunner/vr-vk-core-funcs.h
 create mode 100644 tests/vulkan/vkrunner/vr-vk-device-funcs.h
 create mode 100644 tests/vulkan/vkrunner/vr-vk-instance-funcs.h
 create mode 100644 tests/vulkan/vkrunner/vr-vk.c
 create mode 100644 tests/vulkan/vkrunner/vr-vk.h
 create mode 100644 tests/vulkan/vkrunner/vr-window.c
 create mode 100644 tests/vulkan/vkrunner/vr-window.h
    
Revision 4
      Here is a second attempt at incorporating VkRunner into Piglit. This
time, instead of copying the source code into Piglit, it now just
executes the VkRunner executable is an external dependency. I think
there is now more interest in maintaining VkRunner as a separate repo
because it may later be integrated into other testing frameworks as
well such as CTS.

If the executable is not found then Piglit will automatically report
the test as skipped. It will normally be searched for in the PATH, but
its location can be overridden with the PIGLIT_VKRUNNER_BINARY
environment variable.

The VkRunner repo can be found here:
https://github.com/igalia/vkrunner

In the meantime since the last patch series, VkRunner has been
continuously developed and has gained features. Notably for Piglit, it
now works on Windows including building with Visual Studio. It has
also switched to using CMake for the build system.

Neil Roberts (2):
  framework: Add a vulkan tests profile
  vulkan: Add some tests for glsl450 builtin functions using doubles

 framework/test/piglit_test.py                 | 17 ++++
 tests/vulkan.py                               | 33 +++++++
 .../face-forward-double.vk_shader_test        | 88 +++++++++++++++++++
 .../glsl450/frexp-double.vk_shader_test       | 61 +++++++++++++
 .../glsl450/isinf-double.vk_shader_test       | 81 +++++++++++++++++
 .../glsl450/reflect-double.vk_shader_test     | 55 ++++++++++++
 .../glsl450/refract-double.vk_shader_test     | 88 +++++++++++++++++++
 7 files changed, 423 insertions(+)
 create mode 100644 tests/vulkan.py
 create mode 100644 tests/vulkan/glsl450/face-forward-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/frexp-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/isinf-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/reflect-double.vk_shader_test
 create mode 100644 tests/vulkan/glsl450/refract-double.vk_shader_test
    

Revisions