[i-g-t] gitlab-CI: Force a UTF-8 local

Submitted by Daniel Vetter on Dec. 6, 2018, 6:27 p.m.

Details

Message ID 20181206182714.31393-1-daniel.vetter@ffwll.ch
State New
Series "gitlab-CI: Force a UTF-8 local"
Headers show

Commit Message

Daniel Vetter Dec. 6, 2018, 6:27 p.m.
python is brittle otherwise and loves to just blow up. Especially
when you have commits with non-ASCII characters, which gitlab CI
injects into the docker env.

Strangely we've only seen this go boom on the ninja test target
in the arm64 cross-compile. For paranoia roll it out everywhere.

Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 .gitlab-ci.yml | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

Patch hide | download patch | download mbox

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index efd01be0ea07..5c457ab0431b 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -22,6 +22,7 @@  stages:
 build:tests-fedora:
   stage: build
   script:
+    - export LANG=C.UTF-8
     - meson $MESON_OPTIONS build
     - ninja -C build
     - ninja -C build igt-gpu-tools-doc
@@ -33,6 +34,7 @@  build:tests-fedora-no-libunwind:
   stage: build
   script:
     - dnf remove -y libunwind-devel
+    - export LANG=C.UTF-8
     - meson -Dlibunwind=false build
     - ninja -C build
 
@@ -41,6 +43,7 @@  build:tests-fedora-oldest-meson:
   stage: build
   script:
     - dnf remove -y meson
+    - export LANG=C.UTF-8
     - pip3 install /usr/src/meson-*.tar.gz
     - meson $MESON_OPTIONS build
     - ninja -C build
@@ -50,6 +53,7 @@  build:tests-debian-meson:
   image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian:latest
   stage: build
   script:
+    - export LANG=C.UTF-8
     - meson $MESON_OPTIONS build
     - ninja -C build
     - cp build/tests/test-list.txt meson-test-list.txt
@@ -62,6 +66,7 @@  build:tests-debian-meson-arm64:
   stage: build
   script:
     - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
+    - export LANG=C.UTF-8
     - meson --cross-file meson-cross-arm64.txt build
     - ninja -C build
   artifacts:
@@ -85,7 +90,9 @@  test:ninja-test:
   dependencies:
     - build:tests-fedora
   stage: test
-  script: ninja -C build test
+  script:
+    - export LANG=C.UTF-8
+    - ninja -C build test
 
 test:ninja-test-arm64:
   image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian-arm64:latest
@@ -94,6 +101,7 @@  test:ninja-test-arm64:
   stage: test
   script:
     - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
+    - export LANG=C.UTF-8
     - env > build/envdump.txt
     - locale
     - locale -a
@@ -117,6 +125,7 @@  pages:
     - build:tests-fedora
   stage: deploy
   script:
+    - export LANG=C.UTF-8
     - ninja -C build igt-gpu-tools-doc
     - cp -r build/docs/reference/igt-gpu-tools/html public
   artifacts:

Comments

Petri Latvala Dec. 7, 2018, 10:50 a.m.
On Thu, Dec 06, 2018 at 07:27:14PM +0100, Daniel Vetter wrote:
> python is brittle otherwise and loves to just blow up. Especially
> when you have commits with non-ASCII characters, which gitlab CI
> injects into the docker env.
> 
> Strangely we've only seen this go boom on the ninja test target
> in the arm64 cross-compile. For paranoia roll it out everywhere.


Might I suggest a global



variables:
 LANG: "C.UTF-8"


using this feature that I just today learned about:
https://docs.gitlab.com/ee/ci/variables/README.html
Ville Syrjälä Dec. 7, 2018, 5:28 p.m.
On Thu, Dec 06, 2018 at 07:27:14PM +0100, Daniel Vetter wrote:
> python is brittle otherwise and loves to just blow up. Especially
> when you have commits with non-ASCII characters, which gitlab CI
> injects into the docker env.
> 
> Strangely we've only seen this go boom on the ninja test target
> in the arm64 cross-compile. For paranoia roll it out everywhere.
> 
> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
> ---
>  .gitlab-ci.yml | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index efd01be0ea07..5c457ab0431b 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -22,6 +22,7 @@ stages:
>  build:tests-fedora:
>    stage: build
>    script:
> +    - export LANG=C.UTF-8

LC_ALL?

>      - meson $MESON_OPTIONS build
>      - ninja -C build
>      - ninja -C build igt-gpu-tools-doc
> @@ -33,6 +34,7 @@ build:tests-fedora-no-libunwind:
>    stage: build
>    script:
>      - dnf remove -y libunwind-devel
> +    - export LANG=C.UTF-8
>      - meson -Dlibunwind=false build
>      - ninja -C build
>  
> @@ -41,6 +43,7 @@ build:tests-fedora-oldest-meson:
>    stage: build
>    script:
>      - dnf remove -y meson
> +    - export LANG=C.UTF-8
>      - pip3 install /usr/src/meson-*.tar.gz
>      - meson $MESON_OPTIONS build
>      - ninja -C build
> @@ -50,6 +53,7 @@ build:tests-debian-meson:
>    image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian:latest
>    stage: build
>    script:
> +    - export LANG=C.UTF-8
>      - meson $MESON_OPTIONS build
>      - ninja -C build
>      - cp build/tests/test-list.txt meson-test-list.txt
> @@ -62,6 +66,7 @@ build:tests-debian-meson-arm64:
>    stage: build
>    script:
>      - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
> +    - export LANG=C.UTF-8
>      - meson --cross-file meson-cross-arm64.txt build
>      - ninja -C build
>    artifacts:
> @@ -85,7 +90,9 @@ test:ninja-test:
>    dependencies:
>      - build:tests-fedora
>    stage: test
> -  script: ninja -C build test
> +  script:
> +    - export LANG=C.UTF-8
> +    - ninja -C build test
>  
>  test:ninja-test-arm64:
>    image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian-arm64:latest
> @@ -94,6 +101,7 @@ test:ninja-test-arm64:
>    stage: test
>    script:
>      - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
> +    - export LANG=C.UTF-8
>      - env > build/envdump.txt
>      - locale
>      - locale -a
> @@ -117,6 +125,7 @@ pages:
>      - build:tests-fedora
>    stage: deploy
>    script:
> +    - export LANG=C.UTF-8
>      - ninja -C build igt-gpu-tools-doc
>      - cp -r build/docs/reference/igt-gpu-tools/html public
>    artifacts:
> -- 
> 2.20.0.rc1
> 
> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev
Daniel Vetter Dec. 10, 2018, 9:45 a.m.
On Fri, Dec 07, 2018 at 07:28:16PM +0200, Ville Syrjälä wrote:
> On Thu, Dec 06, 2018 at 07:27:14PM +0100, Daniel Vetter wrote:
> > python is brittle otherwise and loves to just blow up. Especially
> > when you have commits with non-ASCII characters, which gitlab CI
> > injects into the docker env.
> > 
> > Strangely we've only seen this go boom on the ninja test target
> > in the arm64 cross-compile. For paranoia roll it out everywhere.
> > 
> > Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
> > ---
> >  .gitlab-ci.yml | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index efd01be0ea07..5c457ab0431b 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -22,6 +22,7 @@ stages:
> >  build:tests-fedora:
> >    stage: build
> >    script:
> > +    - export LANG=C.UTF-8
> 
> LC_ALL?

Implied by setting LANG. I checked by looking at the output of locale. I
honestly have no idea what should be done.
-Daniel

> 
> >      - meson $MESON_OPTIONS build
> >      - ninja -C build
> >      - ninja -C build igt-gpu-tools-doc
> > @@ -33,6 +34,7 @@ build:tests-fedora-no-libunwind:
> >    stage: build
> >    script:
> >      - dnf remove -y libunwind-devel
> > +    - export LANG=C.UTF-8
> >      - meson -Dlibunwind=false build
> >      - ninja -C build
> >  
> > @@ -41,6 +43,7 @@ build:tests-fedora-oldest-meson:
> >    stage: build
> >    script:
> >      - dnf remove -y meson
> > +    - export LANG=C.UTF-8
> >      - pip3 install /usr/src/meson-*.tar.gz
> >      - meson $MESON_OPTIONS build
> >      - ninja -C build
> > @@ -50,6 +53,7 @@ build:tests-debian-meson:
> >    image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian:latest
> >    stage: build
> >    script:
> > +    - export LANG=C.UTF-8
> >      - meson $MESON_OPTIONS build
> >      - ninja -C build
> >      - cp build/tests/test-list.txt meson-test-list.txt
> > @@ -62,6 +66,7 @@ build:tests-debian-meson-arm64:
> >    stage: build
> >    script:
> >      - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
> > +    - export LANG=C.UTF-8
> >      - meson --cross-file meson-cross-arm64.txt build
> >      - ninja -C build
> >    artifacts:
> > @@ -85,7 +90,9 @@ test:ninja-test:
> >    dependencies:
> >      - build:tests-fedora
> >    stage: test
> > -  script: ninja -C build test
> > +  script:
> > +    - export LANG=C.UTF-8
> > +    - ninja -C build test
> >  
> >  test:ninja-test-arm64:
> >    image: $CI_REGISTRY/$CI_PROJECT_PATH/igt-debian-arm64:latest
> > @@ -94,6 +101,7 @@ test:ninja-test-arm64:
> >    stage: test
> >    script:
> >      - export PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig/
> > +    - export LANG=C.UTF-8
> >      - env > build/envdump.txt
> >      - locale
> >      - locale -a
> > @@ -117,6 +125,7 @@ pages:
> >      - build:tests-fedora
> >    stage: deploy
> >    script:
> > +    - export LANG=C.UTF-8
> >      - ninja -C build igt-gpu-tools-doc
> >      - cp -r build/docs/reference/igt-gpu-tools/html public
> >    artifacts:
> > -- 
> > 2.20.0.rc1
> > 
> > _______________________________________________
> > igt-dev mailing list
> > igt-dev@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/igt-dev
> 
> -- 
> Ville Syrjälä
> Intel
Ville Syrjälä Dec. 10, 2018, 4:02 p.m.
On Mon, Dec 10, 2018 at 10:45:36AM +0100, Daniel Vetter wrote:
> On Fri, Dec 07, 2018 at 07:28:16PM +0200, Ville Syrjälä wrote:
> > On Thu, Dec 06, 2018 at 07:27:14PM +0100, Daniel Vetter wrote:
> > > python is brittle otherwise and loves to just blow up. Especially
> > > when you have commits with non-ASCII characters, which gitlab CI
> > > injects into the docker env.
> > > 
> > > Strangely we've only seen this go boom on the ninja test target
> > > in the arm64 cross-compile. For paranoia roll it out everywhere.
> > > 
> > > Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
> > > ---
> > >  .gitlab-ci.yml | 11 ++++++++++-
> > >  1 file changed, 10 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > index efd01be0ea07..5c457ab0431b 100644
> > > --- a/.gitlab-ci.yml
> > > +++ b/.gitlab-ci.yml
> > > @@ -22,6 +22,7 @@ stages:
> > >  build:tests-fedora:
> > >    stage: build
> > >    script:
> > > +    - export LANG=C.UTF-8
> > 
> > LC_ALL?
> 
> Implied by setting LANG. I checked by looking at the output of locale. I
> honestly have no idea what should be done.

I think LANG only provides the default for the all the LC_ env
variables if you haven't set anything at all, whereas LC_ALL
overrides all of them even if they're set.

unset LC_MESSAGES
unset LANG
unset LC_ALL
LANG=foo locale -> foo
LANG=foo LC_MESSAGES=bar locale -> bar
LANG=foo LC_MESSAGES=bar LC_ALL=hah -> hah