Patchwork randr: check rotated virtual size limits correctly

login
register
mail settings
Submitter Aaron Plattner
Date May 11, 2011, 9:18 p.m.
Message ID <20110511141817.GA16757@soprano.nvidia.com>
Download mbox | patch
Permalink /patch/5427/
State Accepted, archived
Commit b6c7b9b2f39e970cedb6bc1e073f901e28cb0fa3
Headers show

Comments

Aaron Plattner - May 11, 2011, 9:18 p.m.
Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
the RandR path that cause RRSetScreenConfig requests to fail if the
size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
it compares the rotated screen dimensions to the unrotated limits,
which causes 90- and 270-degree rotation to fail unless your screen
happens to be square:

  X Error of failed request:  BadValue (integer parameter out of range for operation)
    Major opcode of failed request:  153 (RANDR)
    Minor opcode of failed request:  2 (RRSetScreenConfig)
    Value in failed request:  0x780
    Serial number of failed request:  14
    Current serial number in output stream:  14

Fix this by moving the check above the code that swaps the dimensions
based on the rotation.

Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
---
This patch is against server-1.10-branch.

 randr/rrscreen.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
Jeremy Huddleston - May 12, 2011, 1:05 p.m.
That looks like it does the right thing... (still... get a Tested-by:)

Reviewed-by: Jeremy Huddleston <jeremyhu@apple.com>

On May 11, 2011, at 7:18 AM, Aaron Plattner wrote:

> Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
> the RandR path that cause RRSetScreenConfig requests to fail if the
> size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
> it compares the rotated screen dimensions to the unrotated limits,
> which causes 90- and 270-degree rotation to fail unless your screen
> happens to be square:
> 
>  X Error of failed request:  BadValue (integer parameter out of range for operation)
>    Major opcode of failed request:  153 (RANDR)
>    Minor opcode of failed request:  2 (RRSetScreenConfig)
>    Value in failed request:  0x780
>    Serial number of failed request:  14
>    Current serial number in output stream:  14
> 
> Fix this by moving the check above the code that swaps the dimensions
> based on the rotation.
> 
> Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
> ---
> This patch is against server-1.10-branch.
> 
> randr/rrscreen.c |   12 ++++++------
> 1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/randr/rrscreen.c b/randr/rrscreen.c
> index 1bc1a9e..da6d48d 100644
> --- a/randr/rrscreen.c
> +++ b/randr/rrscreen.c
> @@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
>      */
>     width = mode->mode.width;
>     height = mode->mode.height;
> -    if (rotation & (RR_Rotate_90|RR_Rotate_270))
> -    {
> -	width = mode->mode.height;
> -	height = mode->mode.width;
> -    }
> -
>     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
> 	client->errorValue = width;
> 	free(pData);
> @@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
> 	return BadValue;
>     }
> 
> +    if (rotation & (RR_Rotate_90|RR_Rotate_270))
> +    {
> +	width = mode->mode.height;
> +	height = mode->mode.width;
> +    }
> +
>     if (width != pScreen->width || height != pScreen->height)
>     {
> 	int	c;
> -- 
> 1.7.4.1
> 
> _______________________________________________
> xorg-devel@lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>
Robert Hooker - May 23, 2011, 10:55 p.m.
On Thu, May 12, 2011 at 2:05 AM, Jeremy Huddleston <jeremyhu@apple.com> wrote:
> That looks like it does the right thing... (still... get a Tested-by:)
>
> Reviewed-by: Jeremy Huddleston <jeremyhu@apple.com>
>
> On May 11, 2011, at 7:18 AM, Aaron Plattner wrote:
>
>> Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
>> the RandR path that cause RRSetScreenConfig requests to fail if the
>> size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
>> it compares the rotated screen dimensions to the unrotated limits,
>> which causes 90- and 270-degree rotation to fail unless your screen
>> happens to be square:
>>
>>  X Error of failed request:  BadValue (integer parameter out of range for operation)
>>    Major opcode of failed request:  153 (RANDR)
>>    Minor opcode of failed request:  2 (RRSetScreenConfig)
>>    Value in failed request:  0x780
>>    Serial number of failed request:  14
>>    Current serial number in output stream:  14
>>
>> Fix this by moving the check above the code that swaps the dimensions
>> based on the rotation.
>>
>> Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
>> ---
>> This patch is against server-1.10-branch.
>>
>> randr/rrscreen.c |   12 ++++++------
>> 1 files changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/randr/rrscreen.c b/randr/rrscreen.c
>> index 1bc1a9e..da6d48d 100644
>> --- a/randr/rrscreen.c
>> +++ b/randr/rrscreen.c
>> @@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
>>      */
>>     width = mode->mode.width;
>>     height = mode->mode.height;
>> -    if (rotation & (RR_Rotate_90|RR_Rotate_270))
>> -    {
>> -     width = mode->mode.height;
>> -     height = mode->mode.width;
>> -    }
>> -
>>     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
>>       client->errorValue = width;
>>       free(pData);
>> @@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
>>       return BadValue;
>>     }
>>
>> +    if (rotation & (RR_Rotate_90|RR_Rotate_270))
>> +    {
>> +     width = mode->mode.height;
>> +     height = mode->mode.width;
>> +    }
>> +
>>     if (width != pScreen->width || height != pScreen->height)
>>     {
>>       int     c;
>> --
>> 1.7.4.1
>>
>> _______________________________________________
>> xorg-devel@lists.x.org: X.Org development
>> Archives: http://lists.x.org/archives/xorg-devel
>> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>>
>
> _______________________________________________
> xorg-devel@lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>

It works here. Reference:
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/740933

Tested-by: Robert Hooker <robert.hooker@canonical.com>
Tested-by: Kent Baxley <kent.baxley@canonical.com>
dude - May 27, 2011, 7:26 p.m.
Aaron Plattner <aplattner@...> writes:

> 
> Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
> the RandR path that cause RRSetScreenConfig requests to fail if the
> size is too large.  Unfortunately, when RandR 1.1 rotation is enabled



Hi, I have the same problem. Which file do I have to patch? Or which is the
correct command to make things running?

Patch

diff --git a/randr/rrscreen.c b/randr/rrscreen.c
index 1bc1a9e..da6d48d 100644
--- a/randr/rrscreen.c
+++ b/randr/rrscreen.c
@@ -910,12 +910,6 @@  ProcRRSetScreenConfig (ClientPtr client)
      */
     width = mode->mode.width;
     height = mode->mode.height;
-    if (rotation & (RR_Rotate_90|RR_Rotate_270))
-    {
-	width = mode->mode.height;
-	height = mode->mode.width;
-    }
-
     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
 	client->errorValue = width;
 	free(pData);
@@ -927,6 +921,12 @@  ProcRRSetScreenConfig (ClientPtr client)
 	return BadValue;
     }
 
+    if (rotation & (RR_Rotate_90|RR_Rotate_270))
+    {
+	width = mode->mode.height;
+	height = mode->mode.width;
+    }
+
     if (width != pScreen->width || height != pScreen->height)
     {
 	int	c;