Integrating a Cloudsmith repository with a Semaphore CI workflow

At Cloudsmith, we believe that packaging should be at the centre of any modern build and deployment process. In fact, we think that Continuous Packaging is the glue that ties Continuous Integration and Continuous Deployment or Delivery together.

So with that in mind, in this blog, we will take a walk through how easy it is to integrate Cloudsmith with a Semaphore CI workflow and push the artifacts and packages that you build to a private repository.

TL:DR – It’s super easy. You just need to add a block to a pipeline in a Semaphore CI workflow that installs the Cloudsmith CLI, and then uses the CLI to push your package. We will step through how you do that!

What is Semaphore CI?

Semaphore CI is a Continuous Integration and Continuous Deployment or Delivery platform.  It’s cloud-native, fast, easy to get started with and supports any platform or language.  You can create your workflow and pipelines by writing the YAML configuration files directly, committing them along with your source code, or using the intuitive Visual Builder that Semaphore includes.

Building a workflow from scratch is beyond the scope of this blog, but we have created a straightforward example workflow that we can walk through to see where Cloudsmith fits in and how to set things up.

Let’s get started!

When you first log into Semaphore, you can authorise it to connect to your GitHub account and then choose which repository you want to use to create a Semaphore Project.  Once you do this, Semaphore automatically installs the required hooks in your source repo so that new pushes can trigger your workflows on all or specific branches, or even for pull requests:

The next thing that we need to set up is to add our Cloudsmith API Key as a secret for this Semaphore project, as the Cloudsmith CLI requires the API Key to authenticate to Cloudsmith to push a package.  It’s easy to add a secret to a Semaphore project and then use it as an environment variable in a pipeline:

Remember ALWAYS to store secrets used in your build or deployment processes in a secrets manager (here were are using the built-in Semaphore secrets store) or encrypted in some form. Never, ever add secrets or credentials to your configuration files as plain text, or commit any in your source.

OK, that’s the basics of our project settings configured.  Now let’s take a look at the centrepiece of a Semaphore build and deployment process – the Workflow.

Semaphore Concepts

At this point, it’s probably worth running over a few of the terms we will come across when describing this workflow and what they mean.

First, you have the workflow itself, which may consist of multiple pipelines, perhaps one to run tests and another for deployment. Pipelines then consist of one or more blocks.  Blocks run on an agent, which defines the execution environment, and they also specify which Jobs need to run. Jobs themselves then specify the commands that you want to execute.

In our example workflow here, we just have two blocks – A “Build Package” block and a “Push Package” block:

This is how things look in the Semaphore Visual Builder, and we can also drill down into individual blocks here to see the jobs within them. For example, this is what the “Checkout and Build” Job from the “Build Package” block looks like:

As previously mentioned though, all configuration in Semaphore is specified in YAML files. The initial pipeline is sourced from .semaphore/semaphore.yml. So let’s go and take a look at that now and see what it contains.

The semaphore.yml file starts by declaring a name for the pipeline and the agent that defines the hardware and software environment that the blocks will use:

version: v1.0
name: Initial Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

Next we declare our blocks.  The first block we declare is the “Build Package” block:

- name: Build Package
    task:
      jobs:
        - name: Checkout and Build
          commands:
            - cache clear
            - checkout
            - make
            - fpm -f -s dir -t deb -v 1.3.1 -n cloudsmith-semaphore-demo .
            - cache store package cloudsmith-semaphore-demo_1.3.1_amd64.deb
      prologue:
        commands:
          - sudo apt-get update
          - sudo apt-get -y install ruby ruby-dev rubygems
          - gem install --no-document fpm

There are a few things to note here.  First, is that any commands listed in the prologue are executed before each job, and here we use these to install the required tooling that we need to for our project (in this case, the rubygem “fpm”, which we will use to create a Debian package once our source is built)

Then, the commands in our “Checkout and Build” job will run. Going through these one at a time, the first command clears the Semaphore cache, so any artifacts from previous runs of this pipeline will be removed.  Next, we check out our project source and run “make” to build our source. We then use “fpm” to package the results of our build into a Debian package, and finally, we store the built package in the Semaphore cache with the cache key “package”.

We need to use the Semaphore cache to pass files between blocks and jobs as each job runs in a separate, isolated machine that starts with a clean environment.

The second block that we declare is our “Push Package” block:

  - name: Push Package
    task:
      secrets:
        - name: Cloudsmith-API
      jobs:
        - name: Push
          commands:
            - cache restore package
            - cloudsmith push deb demo/examples-repo/debian/buster cloudsmith-semaphore-demo_1.3.1_amd64.deb
      prologue:
        commands:
          - sem-version python 3.7
          - pip install cloudsmith-cli

Things to note here are that we use the prologue to install the required tooling for the Block, namely the Cloudsmith CLI.  We also use the secret that we added to the project here, which sets up our Cloudsmith API Key as an Environment Variable when the job runs.

In the “Push” job,  we simply restore the package from the Semaphore cache. Then we use the “cloudsmith push” command with the Cloudsmith account called “demo”, our repository called “examples-repo”, and we are using “Debian” and “Buster” as the distribution and version for this package.

So putting that all together, our complete semaphore.yml file looks like this:

version: v1.0
name: Initial Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build Package
    task:
      jobs:
        - name: Checkout and Build
          commands:
            - cache clear
            - checkout
            - make
            - fpm -f -s dir -t deb -v 1.3.1 -n cloudsmith-semaphore-demo .
            - cache store package cloudsmith-semaphore-demo_1.3.1_amd64.deb
      prologue:
        commands:
          - sudo apt-get update
          - sudo apt-get -y install ruby ruby-dev rubygems
          - gem install --no-document fpm
  - name: Push Package
    task:
      secrets:
        - name: Cloudsmith-API
      jobs:
        - name: Push
          commands:
            - cache restore package
            - cloudsmith push deb demo/examples-repo/debian/buster cloudsmith-semaphore-demo_1.3.1_amd64.deb
      prologue:
        commands:
          - sem-version python 3.7
          - pip install cloudsmith-cli

That’s all we need for this job to push the package to our private Cloudsmith repository.

Now, because Semaphore configured our Github repository automatically when we initially set up the project, all that is left for us to do is make a change to our source, add and commit the change, and then “git push”:

This push then triggers the Semaphore workflow to start:

Our pipeline is now running, and we can see that the “Build Package” block is executing the “Checkout and Build” Job:

We can view a live tail of the logs from this block, and we can watch for the output of each command in the job:

The job completed all commands successfully, and now the “Push Package” block starts and executes the “Push” job:

Again, we can view the logs from this block, and we can see that the “cloudsmith push” command succeeded and pushed our package to our private Cloudsmith repository:

That’s everything complete. Our Semaphore workflow ran successfully, and the package we created in our build process was uploaded to Cloudsmith.  We can take a look now at our Cloudsmith repository and see our new package:

Summary

It is easy to add a private Cloudsmith repository to a workflow that you have in Semaphore CI. You don’t need to make sweeping changes to your build process. Cloudsmith and its universal package support means that, in reality, you are just a “cloudsmith push” away from adding your own central, single source of truth for your package management needs. In doing so, you are one step closer down the path of Continuous Packaging.

Continuous Integration, Continuous Packaging, Continuous Deployment / Delivery. The modern DevOps workflow, helping you secure and control your software supply chain.

Liked this article? Don\'t be selfish (:-), share with others:   Tweet