cts_gl.py: only generate tests for GL45-CTS

Submitted by Marek Olšák on Sept. 9, 2016, 1:10 p.m.

Details

Message ID 1473426656-14681-1-git-send-email-maraeo@gmail.com
State New
Headers show
Series "cts_gl.py: only generate tests for GL45-CTS" ( rev: 1 ) in Piglit

Not browsing as part of any series.

Commit Message

Marek Olšák Sept. 9, 2016, 1:10 p.m.
From: Marek Olšák <marek.olsak@amd.com>

Older versions don't work and there is no interest in fixing the official
gl45release CTS branch, so it's useless to add those forever-broken tests.

Using '-t gl45-cts' is annoying, because adding '-t cube' adds non-GL45
tests instead of filtering the already filtered list.
---
 tests/cts_gl.py | 18 ------------------
 1 file changed, 18 deletions(-)

Patch hide | download patch | download mbox

diff --git a/tests/cts_gl.py b/tests/cts_gl.py
index d650ffd..f647613 100644
--- a/tests/cts_gl.py
+++ b/tests/cts_gl.py
@@ -62,31 +62,13 @@  class DEQPCTSTest(deqp.DEQPBaseTest):
 
     @property
     def extra_args(self):
         return super(DEQPCTSTest, self).extra_args + \
             [x for x in _EXTRA_ARGS if not x.startswith('--deqp-case')]
 
 # Add all of the suites by default, users can use filters to remove them.
 profile = deqp.make_profile(  # pylint: disable=invalid-name
     itertools.chain(
         deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL30-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL31-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL32-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL33-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL40-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL41-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL42-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL43-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
-            deqp.gen_caselist_txt(_CTS_BIN, 'GL44-CTS-cases.txt', _EXTRA_ARGS)),
-        deqp.iter_deqp_test_cases(
             deqp.gen_caselist_txt(_CTS_BIN, 'GL45-CTS-cases.txt', _EXTRA_ARGS)),
     ),
     DEQPCTSTest)

Comments

Quoting Marek Olšák (2016-09-09 06:10:56)
> From: Marek Olšák <marek.olsak@amd.com>
> 
> Older versions don't work and there is no interest in fixing the official
> gl45release CTS branch, so it's useless to add those forever-broken tests.
> 
> Using '-t gl45-cts' is annoying, because adding '-t cube' adds non-GL45
> tests instead of filtering the already filtered list.

-t takes a regex so you could do something like -t 'gl45*cube'

That said I have no opinion one way or the other about this.

Dylan

> ---
>  tests/cts_gl.py | 18 ------------------
>  1 file changed, 18 deletions(-)
> 
> diff --git a/tests/cts_gl.py b/tests/cts_gl.py
> index d650ffd..f647613 100644
> --- a/tests/cts_gl.py
> +++ b/tests/cts_gl.py
> @@ -62,31 +62,13 @@ class DEQPCTSTest(deqp.DEQPBaseTest):
>  
>      @property
>      def extra_args(self):
>          return super(DEQPCTSTest, self).extra_args + \
>              [x for x in _EXTRA_ARGS if not x.startswith('--deqp-case')]
>  
>  # Add all of the suites by default, users can use filters to remove them.
>  profile = deqp.make_profile(  # pylint: disable=invalid-name
>      itertools.chain(
>          deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL30-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL31-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL32-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL33-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL40-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL41-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL42-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL43-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL44-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
>              deqp.gen_caselist_txt(_CTS_BIN, 'GL45-CTS-cases.txt', _EXTRA_ARGS)),
>      ),
>      DEQPCTSTest)
> -- 
> 2.7.4
> 
> _______________________________________________
> Piglit mailing list
> Piglit@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
On 09/09/16 15:10, Marek Olšák wrote:
> From: Marek Olšák <marek.olsak@amd.com>
> 
> Older versions don't work and there is no interest in fixing the official

Could you elaborate that older versions doesn't work? We are right now
focusing on gl44, and except some local patches that we needed to add in
order to get some tests working, it is going fine. Although perhaps you
are talking about the need to patch CTS to get those 44 tests working ...

> gl45release CTS branch, so it's useless to add those forever-broken tests.
> 
> Using '-t gl45-cts' is annoying, because adding '-t cube' adds non-GL45
> tests instead of filtering the already filtered list.
> ---
>  tests/cts_gl.py | 18 ------------------
>  1 file changed, 18 deletions(-)
> 
> diff --git a/tests/cts_gl.py b/tests/cts_gl.py
> index d650ffd..f647613 100644
> --- a/tests/cts_gl.py
> +++ b/tests/cts_gl.py
> @@ -62,31 +62,13 @@ class DEQPCTSTest(deqp.DEQPBaseTest):
>  
>      @property
>      def extra_args(self):
>          return super(DEQPCTSTest, self).extra_args + \
>              [x for x in _EXTRA_ARGS if not x.startswith('--deqp-case')]
>  
>  # Add all of the suites by default, users can use filters to remove them.
>  profile = deqp.make_profile(  # pylint: disable=invalid-name
>      itertools.chain(
>          deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL30-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL31-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL32-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL33-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL40-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL41-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL42-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL43-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
> -            deqp.gen_caselist_txt(_CTS_BIN, 'GL44-CTS-cases.txt', _EXTRA_ARGS)),
> -        deqp.iter_deqp_test_cases(
>              deqp.gen_caselist_txt(_CTS_BIN, 'GL45-CTS-cases.txt', _EXTRA_ARGS)),
>      ),
>      DEQPCTSTest)
>
On Sat, Sep 10, 2016 at 2:41 PM, Alejandro Piñeiro <apinheiro@igalia.com> wrote:
> On 09/09/16 15:10, Marek Olšák wrote:
>> From: Marek Olšák <marek.olsak@amd.com>
>>
>> Older versions don't work and there is no interest in fixing the official
>
> Could you elaborate that older versions doesn't work? We are right now
> focusing on gl44, and except some local patches that we needed to add in
> order to get some tests working, it is going fine. Although perhaps you
> are talking about the need to patch CTS to get those 44 tests working ...

Yeah, that's what I was talking about. It's pretty embarassing having
to apply 100 patches to non-conformant CTS just to make it work and be
conformant.

The idea is that, for most people, there is no practical reason to
have gl44 CTS support in piglit if the official branch remains broken
forever. Having a separate piglit profile just for gl44 (so that
everybody else can ignore it) would be OK though.

Marek
On 10/09/16 15:39, Marek Olšák wrote:
> On Sat, Sep 10, 2016 at 2:41 PM, Alejandro Piñeiro <apinheiro@igalia.com> wrote:
>> On 09/09/16 15:10, Marek Olšák wrote:
>>> From: Marek Olšák <marek.olsak@amd.com>
>>>
>>> Older versions don't work and there is no interest in fixing the official
>>
>> Could you elaborate that older versions doesn't work? We are right now
>> focusing on gl44, and except some local patches that we needed to add in
>> order to get some tests working, it is going fine. Although perhaps you
>> are talking about the need to patch CTS to get those 44 tests working ...
> 
> Yeah, that's what I was talking about. It's pretty embarassing having
> to apply 100 patches to non-conformant CTS just to make it work and be
> conformant.
> 

Ok, I understand now.

> The idea is that, for most people, there is no practical reason to
> have gl44 CTS support in piglit if the official branch remains broken
> forever. 

But for some people, like us, applying those patches are the only
solution, because we only have access to the tarball (that I assume that
is a tarball of the official branch).

Having said so, a) probably we are the exception and not the rule b)
there is no much difference between running gl44 and gl45, so this
change would not affect us too much.

> Having a separate piglit profile just for gl44 (so that
> everybody else can ignore it) would be OK though.

Something like a cts_gl44.py?

BR
On Sep 10, 2016 4:42 PM, "Alejandro Piñeiro" <apinheiro@igalia.com> wrote:
>
>
> On 10/09/16 15:39, Marek Olšák wrote:
> > On Sat, Sep 10, 2016 at 2:41 PM, Alejandro Piñeiro <apinheiro@igalia.com>
wrote:
> >> On 09/09/16 15:10, Marek Olšák wrote:
> >>> From: Marek Olšák <marek.olsak@amd.com>
> >>>
> >>> Older versions don't work and there is no interest in fixing the
official
> >>
> >> Could you elaborate that older versions doesn't work? We are right now
> >> focusing on gl44, and except some local patches that we needed to add
in
> >> order to get some tests working, it is going fine. Although perhaps you
> >> are talking about the need to patch CTS to get those 44 tests working
...
> >
> > Yeah, that's what I was talking about. It's pretty embarassing having
> > to apply 100 patches to non-conformant CTS just to make it work and be
> > conformant.
> >
>
> Ok, I understand now.
>
> > The idea is that, for most people, there is no practical reason to
> > have gl44 CTS support in piglit if the official branch remains broken
> > forever.
>
> But for some people, like us, applying those patches are the only
> solution, because we only have access to the tarball (that I assume that
> is a tarball of the official branch).
>
> Having said so, a) probably we are the exception and not the rule b)
> there is no much difference between running gl44 and gl45, so this
> change would not affect us too much.
>
> > Having a separate piglit profile just for gl44 (so that
> > everybody else can ignore it) would be OK though.
>
> Something like a cts_gl44.py?

Yes.

Marek
On 9 September 2016 at 14:10, Marek Olšák <maraeo@gmail.com> wrote:
> From: Marek Olšák <marek.olsak@amd.com>
>
> Older versions don't work and there is no interest in fixing the official
> gl45release CTS branch, so it's useless to add those forever-broken tests.
>
Surely you meant so say s/official gl45release CTS branch/official pre
gl45release CTS branches/, right ?

That aside, is the "there is no interest" official statement or your
assumption based on the patches needed to get it working ? Guess it's
a bit hard to get things going (push fixes to CTS) but not impossible.

Thanks
Emil
On Mon, Sep 12, 2016 at 3:33 PM, Emil Velikov <emil.l.velikov@gmail.com> wrote:
> On 9 September 2016 at 14:10, Marek Olšák <maraeo@gmail.com> wrote:
>> From: Marek Olšák <marek.olsak@amd.com>
>>
>> Older versions don't work and there is no interest in fixing the official
>> gl45release CTS branch, so it's useless to add those forever-broken tests.
>>
> Surely you meant so say s/official gl45release CTS branch/official pre
> gl45release CTS branches/, right ?

Yes, I use the branch, not the tarballs.

>
> That aside, is the "there is no interest" official statement or your
> assumption based on the patches needed to get it working ? Guess it's
> a bit hard to get things going (push fixes to CTS) but not impossible.

It's my assumption that is highly probable based on the fact that
there are a bazillion CTS patches being exchanged by mail instead of
pushing them to the repo.

Marek
On 12 September 2016 at 14:55, Marek Olšák <maraeo@gmail.com> wrote:
> On Mon, Sep 12, 2016 at 3:33 PM, Emil Velikov <emil.l.velikov@gmail.com> wrote:
>> On 9 September 2016 at 14:10, Marek Olšák <maraeo@gmail.com> wrote:
>>> From: Marek Olšák <marek.olsak@amd.com>
>>>
>>> Older versions don't work and there is no interest in fixing the official
>>> gl45release CTS branch, so it's useless to add those forever-broken tests.
>>>
>> Surely you meant so say s/official gl45release CTS branch/official pre
>> gl45release CTS branches/, right ?
>
> Yes, I use the branch, not the tarballs.
>
The focus is on the gl45release vs _pre_ gl45release wording. As-is
you mention there's no interest in fixing the GL45 branch, yet
cts_gl.py is updated to run _only_ GL45 tests. Which in itself is
contradictory.

>>
>> That aside, is the "there is no interest" official statement or your
>> assumption based on the patches needed to get it working ? Guess it's
>> a bit hard to get things going (push fixes to CTS) but not impossible.
>
> It's my assumption that is highly probable based on the fact that
> there are a bazillion CTS patches being exchanged by mail instead of
> pushing them to the repo.
>
My humble understanding is that patches/fixes get checked over the
weekly working group meetings. Things can be slow when there is little
documentation (short/missing commit messages anyone?) and/or there is
nobody to put them on the table.

So better yet, attend the respective meetings and help move things forward.

That said, this is not an objection against the patch. But more of
"strive to improve, don't call names" suggestion :-)

Regards,
Emil