[spice-gtk,2/3] .gitlab-ci: Save artifacts for copr builds use and deploy

Submitted by Snir Sheriber on Sept. 9, 2019, 1:06 p.m.

Details

Message ID 20190909130630.8873-2-ssheribe@redhat.com
State New
Headers show
Series "Series without cover letter" ( rev: 1 ) in Spice

Not browsing as part of any series.

Commit Message

Snir Sheriber Sept. 9, 2019, 1:06 p.m.
deploy is done by triggering copr's webhook so copr will
get the artifacts and generate a build.
---

This is another suggestion to integrate deployment to copr
The flow is as follow:
- Successful gitlab-ci build will generate tarball and spec file
  as accessible artifacts
- once the linux build completed the deploy job will trigger the
  copr's project custom webhook
- Triggering the webhook will cause copr to execute its pre-defined
  custom source script which will download the artifacts and
  let the build in copr to begin

Three steps are needed in order to make this work:
1. having this two patches
2. set a custom source script in copr to just get the tarball and spec
   for example:
     # only getting the specfile and tarball artifacts, locate in outputdir and edit
     # release number
     curl -L --output artifacts.zip  "https://gitlab.freedektop.org/spice/spice-gtk/-/jobs/artifacts/master/download?job=fedora"
     unzip -j artifacts.zip '*.spec'
     unzip -j artifacts.zip '*.tar.xz'
     sed -i -E "s/(^Release:[[:space:]]*)([^%]*)/\1`date+'%Y%m%d%H%M.spice.latest'`/" *.spec
3. set in gitlab the COPR_ID_UUID environment variable to the package's
   <COPR_ID>/<COPR_UUID> (this variable needs to be set in this
   "<COPR_ID>/<COPR_UUID>" form and combination due to some gitlab
   environment variables limitations. This variable should be masked
   so it cannot be used by a malicious entity)


Pros and cons in regard to the previous suggestion
(https://patchwork.freedesktop.org/series/65881/)
PROS:
* Only minor changes are required
* .copr/Makefile is not needed
* gitlab-ci build artifacts are used, not need to build again
* less scripting hacks

CONS:
* gitlab + copr + code minor configurations are required
* customized configurations (non standard)
* The environment variable trick is a bit hacky

---
 .gitlab-ci.yml | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

Patch hide | download patch | download mbox

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e2d1c55..826e925 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -20,9 +20,12 @@  variables:
               mingw64-usbredir mingw32-usbredir
 
 fedora:
+  stage: build
   artifacts:
     paths:
       - build-*/meson-logs/*.txt
+      - build-default/meson-dist/spice-gtk*.tar.xz
+      - build-default/*.spec
     when: always
     expire_in: 1 week
 
@@ -33,6 +36,8 @@  fedora:
     - ninja -C build-spice-protocol install
 
   script:
+    # Use version format as follows [current-version].[num-of-commits-since-last-tag]
+    - echo $(git describe --match=v\* --abbrev=0 | sed "s/v// ; s/$/./")$(git rev-list $(git describe --abbrev=0)..HEAD | wc -l) > .tarball-version
     - meson --buildtype=release build-default --werror
     # Meson does not update submodules recursively
     - git submodule update --init --recursive
@@ -48,6 +53,7 @@  fedora:
     - ninja -C build-feat-disabled test
 
 windows:
+  stage: build
   artifacts:
     paths:
       - build-win64/meson-logs/*.txt
@@ -67,3 +73,20 @@  windows:
     - mkdir build-win64 && cd build-win64
     - mingw64-meson --buildtype=release -Dgtk_doc=disabled --werror
     - ninja install
+
+fedora:deploy:
+  stage: deploy
+  needs: ["fedora"]
+  only:
+    variables:
+    # Run this stage only if COPR_ID_UUID is defined
+    # COPR_ID_UUID is gitlab VARIABLE that should be set to this string: "<COPR_ID>/<COPR_UUID>"
+    - $COPR_ID_UUID
+  script:
+    # This is a custom webhook which can triger a custom source script (set in copr) that
+    # should just download the specfile and tarball from the artifacts url.
+    #  Create a copr's custom source script:
+    #   https://docs.pagure.org/copr.copr/custom_source_method.html#custom-source-method
+    #  Get artifacts created by this gitlab ci:
+    #   https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html#downloading-the-latest-artifacts
+    - curl -X POST https://copr.fedorainfracloud.org/webhooks/custom/$COPR_ID_UUID/spice-gtk/

Comments

> 
> deploy is done by triggering copr's webhook so copr will
> get the artifacts and generate a build.
> ---
> 
> This is another suggestion to integrate deployment to copr
> The flow is as follow:
> - Successful gitlab-ci build will generate tarball and spec file
>   as accessible artifacts
> - once the linux build completed the deploy job will trigger the
>   copr's project custom webhook
> - Triggering the webhook will cause copr to execute its pre-defined
>   custom source script which will download the artifacts and
>   let the build in copr to begin
> 
> Three steps are needed in order to make this work:
> 1. having this two patches
> 2. set a custom source script in copr to just get the tarball and spec
>    for example:
>      # only getting the specfile and tarball artifacts, locate in outputdir
>      and edit
>      # release number
>      curl -L --output artifacts.zip
>      "https://gitlab.freedektop.org/spice/spice-gtk/-/jobs/artifacts/master/download?job=fedora"
>      unzip -j artifacts.zip '*.spec'
>      unzip -j artifacts.zip '*.tar.xz'
>      sed -i -E
>      "s/(^Release:[[:space:]]*)([^%]*)/\1`date+'%Y%m%d%H%M.spice.latest'`/"
>      *.spec

Why you need this? I mean, is not good the version generated by the CI?

> 3. set in gitlab the COPR_ID_UUID environment variable to the package's
>    <COPR_ID>/<COPR_UUID> (this variable needs to be set in this
>    "<COPR_ID>/<COPR_UUID>" form and combination due to some gitlab
>    environment variables limitations. This variable should be masked
>    so it cannot be used by a malicious entity)
> 

I suppose it's used as a kind of security. Is the COPR_ID secret?

> 
> Pros and cons in regard to the previous suggestion
> (https://patchwork.freedesktop.org/series/65881/)
> PROS:
> * Only minor changes are required
> * .copr/Makefile is not needed
> * gitlab-ci build artifacts are used, not need to build again
> * less scripting hacks
> 
> CONS:
> * gitlab + copr + code minor configurations are required
> * customized configurations (non standard)

What do you mean with this? Which component configuration are you referring to?

> * The environment variable trick is a bit hacky
> 

Is more for security it seems to me.

I like this solution better than previous. Part of the comments above
would be good to go to the commit message.
Looks like this series is a bit RFC.

Where's patch 3/3 ?

> ---
>  .gitlab-ci.yml | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e2d1c55..826e925 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -20,9 +20,12 @@ variables:
>                mingw64-usbredir mingw32-usbredir
>  
>  fedora:
> +  stage: build
>    artifacts:
>      paths:
>        - build-*/meson-logs/*.txt
> +      - build-default/meson-dist/spice-gtk*.tar.xz
> +      - build-default/*.spec
>      when: always
>      expire_in: 1 week
>  
> @@ -33,6 +36,8 @@ fedora:
>      - ninja -C build-spice-protocol install
>  
>    script:
> +    # Use version format as follows
> [current-version].[num-of-commits-since-last-tag]
> +    - echo $(git describe --match=v\* --abbrev=0 | sed "s/v// ;
> s/$/./")$(git rev-list $(git describe --abbrev=0)..HEAD | wc -l) >
> .tarball-version
>      - meson --buildtype=release build-default --werror
>      # Meson does not update submodules recursively
>      - git submodule update --init --recursive
> @@ -48,6 +53,7 @@ fedora:
>      - ninja -C build-feat-disabled test
>  
>  windows:
> +  stage: build
>    artifacts:
>      paths:
>        - build-win64/meson-logs/*.txt
> @@ -67,3 +73,20 @@ windows:
>      - mkdir build-win64 && cd build-win64
>      - mingw64-meson --buildtype=release -Dgtk_doc=disabled --werror
>      - ninja install
> +
> +fedora:deploy:
> +  stage: deploy
> +  needs: ["fedora"]
> +  only:
> +    variables:
> +    # Run this stage only if COPR_ID_UUID is defined
> +    # COPR_ID_UUID is gitlab VARIABLE that should be set to this string:
> "<COPR_ID>/<COPR_UUID>"
> +    - $COPR_ID_UUID
> +  script:
> +    # This is a custom webhook which can triger a custom source script (set
> in copr) that

typo: triger -> trigger

> +    # should just download the specfile and tarball from the artifacts url.
> +    #  Create a copr's custom source script:
> +    #
> https://docs.pagure.org/copr.copr/custom_source_method.html#custom-source-method
> +    #  Get artifacts created by this gitlab ci:
> +    #
> https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html#downloading-the-latest-artifacts
> +    - curl -X POST
> https://copr.fedorainfracloud.org/webhooks/custom/$COPR_ID_UUID/spice-gtk/

Frediano
Hi,


On 9/9/19 6:54 PM, Frediano Ziglio wrote:
>> deploy is done by triggering copr's webhook so copr will
>> get the artifacts and generate a build.
>> ---
>>
>> This is another suggestion to integrate deployment to copr
>> The flow is as follow:
>> - Successful gitlab-ci build will generate tarball and spec file
>>    as accessible artifacts
>> - once the linux build completed the deploy job will trigger the
>>    copr's project custom webhook
>> - Triggering the webhook will cause copr to execute its pre-defined
>>    custom source script which will download the artifacts and
>>    let the build in copr to begin
>>
>> Three steps are needed in order to make this work:
>> 1. having this two patches
>> 2. set a custom source script in copr to just get the tarball and spec
>>     for example:
>>       # only getting the specfile and tarball artifacts, locate in outputdir
>>       and edit
>>       # release number
>>       curl -L --output artifacts.zip
>>       "https://gitlab.freedektop.org/spice/spice-gtk/-/jobs/artifacts/master/download?job=fedora"
>>       unzip -j artifacts.zip '*.spec'
>>       unzip -j artifacts.zip '*.tar.xz'
>>       sed -i -E
>>       "s/(^Release:[[:space:]]*)([^%]*)/\1`date+'%Y%m%d%H%M.spice.latest'`/"
>>       *.spec
> Why you need this? I mean, is not good the version generated by the CI?


Should be fine since the version is 
[current-version].[num-of-commits-since-last-tag]
I used it just for convenience so once it's installed naming will 
provide some build
info on the non-standard fedora\rhel pkg.


>
>> 3. set in gitlab the COPR_ID_UUID environment variable to the package's
>>     <COPR_ID>/<COPR_UUID> (this variable needs to be set in this
>>     "<COPR_ID>/<COPR_UUID>" form and combination due to some gitlab
>>     environment variables limitations. This variable should be masked
>>     so it cannot be used by a malicious entity)
>>
> I suppose it's used as a kind of security. Is the COPR_ID secret?

Yep


>
>> Pros and cons in regard to the previous suggestion
>> (https://patchwork.freedesktop.org/series/65881/)
>> PROS:
>> * Only minor changes are required
>> * .copr/Makefile is not needed
>> * gitlab-ci build artifacts are used, not need to build again
>> * less scripting hacks
>>
>> CONS:
>> * gitlab + copr + code minor configurations are required
>> * customized configurations (non standard)
> What do you mean with this? Which component configuration are you referring to?


Using the custom webhook (instead of the gitlab webhook) and
using the copr custom script instead of other method


>
>> * The environment variable trick is a bit hacky
>>
> Is more for security it seems to me.


Yes, but i found the combination of id/uuid a bit tricky


>
> I like this solution better than previous. Part of the comments above
> would be good to go to the commit message.
> Looks like this series is a bit RFC.
>
> Where's patch 3/3 ?


Oh, forget to to edit this, it is not needed
It's just another patch fixing spice-protocol clone address so it
will work on my private repo

Snir.


>
>> ---
>>   .gitlab-ci.yml | 23 +++++++++++++++++++++++
>>   1 file changed, 23 insertions(+)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index e2d1c55..826e925 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -20,9 +20,12 @@ variables:
>>                 mingw64-usbredir mingw32-usbredir
>>   
>>   fedora:
>> +  stage: build
>>     artifacts:
>>       paths:
>>         - build-*/meson-logs/*.txt
>> +      - build-default/meson-dist/spice-gtk*.tar.xz
>> +      - build-default/*.spec
>>       when: always
>>       expire_in: 1 week
>>   
>> @@ -33,6 +36,8 @@ fedora:
>>       - ninja -C build-spice-protocol install
>>   
>>     script:
>> +    # Use version format as follows
>> [current-version].[num-of-commits-since-last-tag]
>> +    - echo $(git describe --match=v\* --abbrev=0 | sed "s/v// ;
>> s/$/./")$(git rev-list $(git describe --abbrev=0)..HEAD | wc -l) >
>> .tarball-version
>>       - meson --buildtype=release build-default --werror
>>       # Meson does not update submodules recursively
>>       - git submodule update --init --recursive
>> @@ -48,6 +53,7 @@ fedora:
>>       - ninja -C build-feat-disabled test
>>   
>>   windows:
>> +  stage: build
>>     artifacts:
>>       paths:
>>         - build-win64/meson-logs/*.txt
>> @@ -67,3 +73,20 @@ windows:
>>       - mkdir build-win64 && cd build-win64
>>       - mingw64-meson --buildtype=release -Dgtk_doc=disabled --werror
>>       - ninja install
>> +
>> +fedora:deploy:
>> +  stage: deploy
>> +  needs: ["fedora"]
>> +  only:
>> +    variables:
>> +    # Run this stage only if COPR_ID_UUID is defined
>> +    # COPR_ID_UUID is gitlab VARIABLE that should be set to this string:
>> "<COPR_ID>/<COPR_UUID>"
>> +    - $COPR_ID_UUID
>> +  script:
>> +    # This is a custom webhook which can triger a custom source script (set
>> in copr) that
> typo: triger -> trigger
>
>> +    # should just download the specfile and tarball from the artifacts url.
>> +    #  Create a copr's custom source script:
>> +    #
>> https://docs.pagure.org/copr.copr/custom_source_method.html#custom-source-method
>> +    #  Get artifacts created by this gitlab ci:
>> +    #
>> https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html#downloading-the-latest-artifacts
>> +    - curl -X POST
>> https://copr.fedorainfracloud.org/webhooks/custom/$COPR_ID_UUID/spice-gtk/
> Frediano