[v3] travis: add docker based cmake build job

Submitted by Andres Gomez on Jan. 29, 2018, 2:11 a.m.

Details

Message ID 20180129021117.9056-1-agomez@igalia.com
State Superseded
Headers show
Series "travis: add docker based cmake build job" ( rev: 3 ) in Piglit

Not browsing as part of any series.

Commit Message

Andres Gomez Jan. 29, 2018, 2:11 a.m.
Until now we were only running the python unit tests.

It seems desirable to also check that the CMake based build compiles
successfully. We do that now using docker.

The docker build can be tweaked with some environment variables and,
also, be stored in the docker hub if desired. Check the changes for
extra details regarding these variables.

v2: Removed other build possibilities other than just from inside
    Travis-CI, as suggested by Juan.
v3: Replaced the "RELEASE" parameter to create the docker image with
    "PARENT" and removed some unneeded documentation after v2, as
    suggested by Juan.

Cc: Dylan Baker <dylan@pnwbakers.com>
Cc: Juan A. Suarez <jasuarez@igalia.com>
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
---
 .travis.yml              | 47 ++++++++++++++++++++++++++++--
 docker/Rockerfile.piglit | 74 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 118 insertions(+), 3 deletions(-)
 create mode 100644 docker/Rockerfile.piglit

Patch hide | download patch | download mbox

diff --git a/.travis.yml b/.travis.yml
index b47829ff9..32a6b53e5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,5 +1,16 @@ 
+sudo: false
+os: linux
 language: python
-cache: pip
+cache:
+  - ccache
+  - pip
+services:
+  - docker
+
+env:
+  global:
+    - BUILD=pytest
+
 matrix:
   include:
     - python: 2.7
@@ -12,7 +23,37 @@  matrix:
       env: TOX_ENV="py35-{generator,noaccel,accel-nix,streams}"
     - python: 3.6
       env: TOX_ENV="py36-{generator,noaccel,accel-nix,streams}"
+    - env: BUILD=cmake
+
 install:
-    pip install tox
+  - |
+    if [[ $BUILD == pytest ]]; then
+      pip install tox
+    else
+      wget https://github.com/grammarly/rocker/releases/download/1.3.1/rocker-1.3.1-linux_amd64.tar.gz
+      tar xvf rocker-1.3.1-linux_amd64.tar.gz
+      rm rocker-1.3.1-linux_amd64.tar.gz
+    fi
+
+before_script:
+  - |
+    if [[ $BUILD != pytest ]]; then
+      mkdir -p -m777 ~/.ccache
+    fi
+
 script:
-    - tox -e $TOX_ENV
+  - |
+    if [[ $BUILD == pytest ]]; then
+      tox -e $TOX_ENV
+    else
+      ./rocker build -f docker/Rockerfile.piglit .
+    fi
+
+after_success:
+  - |
+    if [[ $BUILD != pytest ]]; then
+      if [[ -n $DOCKER_USERNAME && $TRAVIS_BRANCH == master ]]; then
+        docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD"
+        docker push ${DOCKER_IMAGE:-freedesktop/mesa}:piglit
+      fi
+    fi
diff --git a/docker/Rockerfile.piglit b/docker/Rockerfile.piglit
new file mode 100644
index 000000000..d325205b0
--- /dev/null
+++ b/docker/Rockerfile.piglit
@@ -0,0 +1,74 @@ 
+#
+# This builds Piglit.
+#
+# ~~~
+#  rocker build -f Rockerfile.piglit [--attach]                 \
+#    [--var TAG=piglit]           # piglit-17.0, piglit-13.0, ... \
+#    [--var PARENT=ubuntu:xenial] # ubuntu:xenial, freedesktop/mesa:17.3, ...
+# ~~~
+#
+# Environment variables that are used in the build:
+#  - DOCKER_IMAGE: name of the final image to be tagged (default: freedesktop/mesa)
+#  - MAKEFLAGS: flags to pass to make (e.g., "-j8")
+#  - CCACHE_DIR: ccache directory (default: ~/.ccache)
+#
+# To run
+#
+# ~~~
+#   mkdir -p -m777 ~/my_results_dir
+#   docker run --privileged --rm -t -v ~/my_results_dir:/results:Z  \
+#              -v /tmp/.X11-unix:/tmp/.X11-unix freedesktop/mesa:piglit
+# ~~~
+#
+
+{{ $image := (or .Env.DOCKER_IMAGE "freedesktop/mesa") }}
+{{ $parent_image := (or .PARENT "ubuntu:xenial") }}
+{{ $ccachedir := (or .Env.CCACHE_DIR "~/.ccache") }}
+
+FROM {{ $parent_image }}
+
+LABEL maintainer "Andres Gomez <agomez@igalia.com>"
+
+USER root
+
+ENV LC_ALL=C.UTF-8
+
+RUN apt-get update                                                      \
+  && apt-get -y --no-install-recommends install sudo gcc g++ ccache     \
+    git pkg-config bash-completion cmake                                \
+    libz-dev libpng-dev libgl-dev libegl1-mesa-dev libwaffle-dev        \
+    python3-setuptools python3-pip                                      \
+  && rm -fr /var/lib/apt/lists/*
+
+RUN pip3 install numpy six mako
+
+RUN getent passwd local > /dev/null || adduser --gecos "" local && passwd -d local && adduser local sudo
+
+USER local
+
+{{ if .Env.MAKEFLAGS }}
+ENV MAKEFLAGS={{ .Env.MAKEFLAGS }}
+{{ end }}
+
+WORKDIR /home/local
+
+MOUNT {{ $ccachedir }}:/home/local/.ccache:Z
+
+RUN sudo chown -R local:local /home/local/.ccache
+
+ENV PATH=/usr/lib/ccache:$PATH
+
+ADD . /home/local/piglit
+RUN sudo chown -R local:local /home/local/piglit
+
+WORKDIR /home/local/piglit
+
+ATTACH [ "/bin/bash" ]
+
+RUN cmake . && cmake --build .
+
+VOLUME /results
+
+{{ if .TAG }}
+TAG {{ $image }}:{{ .TAG }}
+{{ end }}

Comments

Andres Gomez <agomez@igalia.com> writes:

> Until now we were only running the python unit tests.
>
> It seems desirable to also check that the CMake based build compiles
> successfully. We do that now using docker.
>
> The docker build can be tweaked with some environment variables and,
> also, be stored in the docker hub if desired. Check the changes for
> extra details regarding these variables.
>
> v2: Removed other build possibilities other than just from inside
>     Travis-CI, as suggested by Juan.
> v3: Replaced the "RELEASE" parameter to create the docker image with
>     "PARENT" and removed some unneeded documentation after v2, as
>     suggested by Juan.
>
> Cc: Dylan Baker <dylan@pnwbakers.com>
> Cc: Juan A. Suarez <jasuarez@igalia.com>
> Signed-off-by: Andres Gomez <agomez@igalia.com>
> Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>

I'm definitely interested in extending our tests using Travis, but could
you add some comments explaining what this system does?  Do we do a
bunch of image build work (numpy/six/mako install) on each Travis build,
or does it get cached?  Do I need to do any setup in Travis in order for
this to take effect?
On Mon, 2018-01-29 at 14:06 -0800, Eric Anholt wrote:
> Andres Gomez <agomez@igalia.com> writes:
> 
> > Until now we were only running the python unit tests.
> > 
> > It seems desirable to also check that the CMake based build compiles
> > successfully. We do that now using docker.
> > 
> > The docker build can be tweaked with some environment variables and,
> > also, be stored in the docker hub if desired. Check the changes for
> > extra details regarding these variables.
> > 
> > v2: Removed other build possibilities other than just from inside
> >     Travis-CI, as suggested by Juan.
> > v3: Replaced the "RELEASE" parameter to create the docker image with
> >     "PARENT" and removed some unneeded documentation after v2, as
> >     suggested by Juan.
> > 
> > Cc: Dylan Baker <dylan@pnwbakers.com>
> > Cc: Juan A. Suarez <jasuarez@igalia.com>
> > Signed-off-by: Andres Gomez <agomez@igalia.com>
> > Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
> 
> I'm definitely interested in extending our tests using Travis, but could
> you add some comments explaining what this system does?  Do we do a
> bunch of image build work (numpy/six/mako install) on each Travis build,
> or does it get cached?  Do I need to do any setup in Travis in order for
> this to take effect?

Yes, I should have been already more verbose with explanations when
posting this patch in the first place.

First, thanks to your reply I've sent a new v4 version after finding a
couple of issues.

With this v4 version, you can check this travis build, which is where
I've tested the patch:
https://travis-ci.org/Igalia/piglit/builds/335657962

There you will see that, in addition to the usual 5 python builds
running the unit tests, there is an additional cmake build, this is the
docker based build:
https://travis-ci.org/Igalia/piglit/jobs/335657970

Without any special setup in Travis, the docker build will run with the
default options. This will be done on every run:
 * Download the parent ubuntu:xenial image.
 * Install building dependencies from APT.
 * Install additional Python dependencies (numpy/six/mako).
 * Build piglit.

The build is, however, quite quick since docker and travis have many of
the needed bits cached and it also makes use of ccache, but still this
process has to happen every time.

Now, this patch allows a couple of additional tweaks:
   1. You can select a different parent image by setting in Travis-CI the
      env variable DOCKER_PARENT. You could want to do this for different
      reasons. From the top of my head:
       * To speed up the build; using a parent image with the APT and
      Python dependencies already installed.
       * To create a build in which there are additional or alternative
      libraries installed; for example, at Igalia, we have the docker
      image "igalia/mesa:released-17.3.3.debug". This way, in case we
      want to later use the generated docker image to test piglit, we
      will be able to run the test suite against mesa-17.3.3 built with
      debug symbols.
   2. You can define a docker hub repository to which upload the generated
      build as "$DOCKER_IMAGE:$DOCKER_TAG". For that, you have to set in
      Travis-CI the following env variables:
       * DOCKER_IMAGE: The path to the docker hub repository you want to
      use. We use "igalia/mesa".
       * DOCKER_TAG: The specific tag for the newly created image. For
      example, you could use "piglit". Therefore, in our case, a new
      build will be uploaded as "igalia/mesa:piglit".
       * DOCKER_USERNAME: Your user to the docker hub.
       * DOCKER_PASSWORD: Your password for the docker hub.

Combining 1. and 2. you could have, for example, automatic builds of
piglit's master uploaded to your personal docker hub with a specific
mesa version installed. This is very useful for later running the test
suite. It is also a good way of providing a specific environment for
debugging/reproducing a bug and share with other developers.

I hope this clarifies further the aim of this integration.

PS: If this is accepted in piglit, my next step is to work in a patch
proposal for having a similar use case for mesa. This will ease the
combination of 1. and 2. that I was mentioning.