Jenkins Pipelines Integration

Quickstart

1. Create a new "secret text" credential containing a Phylum authentication token. The credential ID should be specified since it will be used in the Jenkinsfile (e.g., phylum-token).

2. Add the following "Phylum" stage to an existing Jenkinsfile declarative pipeline configuration:

    stage('Phylum') {
      agent {
        docker {
          image 'phylumio/phylum-ci:latest'
          alwaysPull true
        }
      }
      environment {
        // The environment variable must be named like
        // this but the credential ID can be different.
        PHYLUM_API_KEY = credentials('phylum-token')
      }
      steps {
        // The `--force-analysis` and `--all-deps` flags
        // are needed because this configuration has no
        // history available for computing changes.
        sh 'phylum-ci -vv --force-analysis --all-deps'
      }
    }

⚠️ INFO ⚠️

This is a limited configuration that provides the current state of all dependencies. It does not account for newly added or modified dependencies like would be found in a branch or pull request. Please see the rest of this documentation for more comprehensive configuration options.

Overview

Once configured for a repository as part of a multibranch pipeline project, the Jenkins Pipelines integration will provide analysis of project dependencies from manifests and lockfiles. This can happen as a result of a commit or a pull request (PR).

For PR pipelines, analyzed dependencies will include any that are added/modified in the PR. For branch pipelines, the analyzed dependencies will be determined by comparing dependency files in the branch to the default branch. All dependencies will be analyzed when the branch pipeline is run on the default branch.

Results are provided in the pipeline logs. The CI job will return an error (i.e., fail the build) if any of the analyzed dependencies fail to meet the established policy unless audit mode is specified. If one or more dependencies are still processing (no results available), then the logs will make that clear and the CI job will only fail if dependencies that have completed analysis results do not meet the active policy.

Prerequisites

Jenkins is supported for multibranch pipelines through the use of a Docker image. The prerequisites for using this image are:

• Access to the phylumio/phylum-ci Docker image
• A Phylum token with API access
  • Contact Phylum or register to gain access
    • See also phylum auth register command documentation
  • Consider using a bot or group account for this token
• Access to the Phylum API endpoints
  • That usually means a connection to the internet, optionally via a proxy

Configure Jenkinsfile

Phylum analysis of dependencies can be added to existing pipelines or on it's own with this minimal declarative pipeline configuration:

pipeline {
  agent none
  stages {
    stage('Phylum') {
      agent {
        docker {
          image 'phylumio/phylum-ci:latest'
          alwaysPull true
        }
      }
      environment {
        PHYLUM_API_KEY = credentials('phylum-token')
      }
      options {
        // This is optional but may save time since
        // a full checkout is needed later.
        skipDefaultCheckout()
      }
      steps {
        checkout scmGit(
          branches: [[name: '**']],
          extensions: [cleanBeforeCheckout()],
          // Change to match your repository URL and creds.
          userRemoteConfigs: [[
            credentialsId: 'CHANGEME',
            url: 'https://github.com/CHANGEME/CHANGEME.git'
          ]]
        )
        withCredentials([gitUsernamePassword(credentialsId: 'CHANGEME', gitToolName: 'Default')]) {
          sh 'phylum-ci -vv'
        }
      }
      post {
        always {
          // Cleaning the workspace ensures the git
          // checkout is valid for future runs.
          cleanWs()
        }
      }
    }
  }
}

This configuration contains a single stage named "Phylum" which executes from a custom Docker image. It will run for all pull requests and pushes to any branch. It provides debug output but otherwise does not override any of the phylum-ci arguments, which are all either optional or default to secure values. Let's take a deeper dive into each part of the configuration:

Docker image selection

Choose the Docker image tag to match your comfort level with image dependencies. latest is a "rolling" tag that will point to the image created for the latest released phylum-ci Python package. A particular version tag (e.g., 0.42.4-CLIv6.1.2) is created for each release of the phylum-ci Python package and should not change once published.

However, to be certain that the image does not change...or be warned when it does because it won't be available anymore...use the SHA256 digest of the tag. The digest can be found by looking at the phylumio/phylum-ci tags on Docker Hub or with the command:

# NOTE: The command-line JSON processor `jq` is used here for the sake of a one line example. It is not required.
❯ docker manifest inspect --verbose phylumio/phylum-ci:0.42.4-CLIv6.1.2 | jq .Descriptor.digest
"sha256:77b761ccef10edc28b0f009a40fbeab240bf004522edaaea05572dc3728b6ca6"

For instance, at the time of this writing, all of these tag references pointed to the same image:

      agent {
        docker {
          // NOTE: These are examples. Only one image line for `phylum-ci` is expected.
          //
          // Not specifying a tag means a default of `latest`
          image 'phylumio/phylum-ci'
          // Be more explicit about wanting the `latest` tag
          image 'phylumio/phylum-ci:latest'
          // Use a specific release version of the `phylum-ci` package
          image 'phylumio/phylum-ci:0.42.4-CLIv6.1.2'
          // Use a specific image with it's SHA256 digest
          image 'phylumio/phylum-ci@sha256:77b761ccef10edc28b0f009a40fbeab240bf004522edaaea05572dc3728b6ca6'

          // This option is useful when image is NOT specified by hash
          alwaysPull true
        }
      }

Only the last tag reference, by SHA256 digest, is guaranteed to not have the underlying image it points to change. If digest references are not used, it is recommended to enable the setting to always pull the image, especially if the "latest" tag is configured. That way, the latest changes will be used instead of the ones when the tag was first pulled.

The default phylum-ci Docker image contains git and the installed phylum Python package. It also contains an installed version of the Phylum CLI and all required tools needed for lockfile generation. An advantage of using the default Docker image is that the complete environment is packaged and made available with components that are known to work together.

One disadvantage to the default image is it's size. It can take a while to download and may provide more tools than required for your specific use case. Special slim tags of the phylum-ci image are provided as an alternative. These tags differ from the default image in that they do not contain the required tools needed for lockfile generation (with the exception of the pip tool). The slim tags are significantly smaller and allow for faster action run times. They are useful for those instances where no manifest files are present and/or only lockfiles are used.

Here are examples of using the slim image tags:

      agent {
        docker {
          // NOTE: These are examples. Only one image line for `phylum-ci` is expected.
          //
          // Use the most current release of *both* `phylum-ci` and the Phylum CLI
          image 'phylumio/phylum-ci:slim'
          // Use the `slim` image with a specific release version of `phylum-ci` and Phylum CLI
          image 'phylumio/phylum-ci:0.42.4-CLIv6.1.2-slim'

          // This option is useful when image is NOT specified by hash
          alwaysPull true
        }
      }

See the documentation for using Docker with Pipeline more information.

User-defined variables

A Phylum token with API access is required to perform analysis on project dependencies. Contact Phylum or register to gain access. See also phylum auth register command documentation and consider using a bot or group account for this token.

Provide the token value in a user-defined variable named PHYLUM_API_KEY, set at the stage level or higher. The value for this variable is sensitive and should be set as a secret text credential. Care should be taken to protect it appropriately.

      environment {
        // Variable must be named `PHYLUM_API_KEY`
        // but the credential ID can be different.
        PHYLUM_API_KEY = credentials('phylum-token')
      }

Git checkout step

The git version control system is used within the phylum-ci package to do things like determine if there was a dependency file change and, when specified, report on new dependencies only. Therefore, a full clone of the repository is required to ensure that the local working copy is always pristine and history is available to pull the requested information.

      steps {
        // A full checkout is needed to provide history and proper diffs.
        // A `checkout scm` step is not enough here.
        checkout scmGit(
          branches: [[name: '**']],
          extensions: [cleanBeforeCheckout()],
          // Change to match your repository URL and creds.
          userRemoteConfigs: [[
            credentialsId: 'CHANGEME',
            url: 'https://github.com/CHANGEME/CHANGEME.git'
          ]]
        )
      }

See the checkout step and checkout plugin documentation for more information.

phylum-ci step arguments

The arguments to the phylum-ci script are the way to exert control over the execution of the Phylum analysis. The phylum-ci script entry point has a number of arguments that are all optional and defaulted to secure values. To view the arguments, their description, and default values, run the script with --help output as specified in the Usage section of the top-level README.md or view the script options output for the latest release.

      steps {
        // This block is needed for change detection and using
        // authenticated git commands. Change the `credentialsId`.
        withCredentials([gitUsernamePassword(credentialsId: 'CHANGEME', gitToolName: 'Default')]) {
          // NOTE: These are examples. Only one `phylum-ci` entry is expected.
          //
          // Use the defaults for all the arguments.
          // The default behavior is to only analyze newly added dependencies
          // against the active policy set at the Phylum project level.
          sh 'phylum-ci'

          // Provide debug level output. Highly recommended.
          sh 'phylum-ci -vv'

          // Consider all dependencies in analysis results instead of just the newly added ones.
          // The default is to only analyze newly added dependencies, which can be useful for
          // existing code bases that may not meet established policy rules yet,
          // but don't want to make things worse. Specifying `--all-deps` can be useful for
          // casting the widest net for strict adherence to Quality Assurance (QA) standards.
          sh 'phylum-ci --all-deps'

          // Some lockfile types (e.g., Python/pip `requirements.txt`) are ambiguous in that
          // they can be named differently and may or may not contain strict dependencies.
          // In these cases it is best to specify an explicit path, either with the `--depfile`
          // option or in a `.phylum_project` file. The easiest way to do that is with the
          // Phylum CLI, using `phylum init` command (docs.phylum.io/cli/commands/phylum_init)
          // and committing the generated `.phylum_project` file.
          sh 'phylum-ci --depfile requirements-prod.txt'

          // Specify multiple explicit dependency file paths.
          sh 'phylum-ci --depfile requirements-prod.txt Cargo.toml path/to/dependency.file'

          // Exclude dependency files by gitignore-style pattern.
          sh 'phylum-ci --exclude "requirements-*.txt"'

          // Specify multiple exclusion patterns.
          sh 'phylum-ci --exclude "build.gradle" "tests/fixtures/"'
          sh 'phylum-ci \
              --exclude "/requirements-*.txt" \
              --exclude "build.gradle" "fixtures/"'

          // Force analysis for all dependencies in a manifest file. This is especially useful
          // for *workspace* manifest files where there is no companion lockfile (e.g., libraries).
          sh 'phylum-ci --force-analysis --all-deps --depfile Cargo.toml'

          // Perform analysis as part of a group-owned project.
          // A paid account is needed to use groups: https://phylum.io/pricing
          sh 'phylum-ci --group my_group'

          // Analyze all dependencies in audit mode, to gain insight without failing builds.
          sh 'phylum-ci --all-deps --audit'

          // Ensure the latest Phylum CLI is installed.
          sh 'phylum-ci --force-install'

          // Install a specific version of the Phylum CLI.
          sh 'phylum-ci --phylum-release 6.4.0 --force-install'

          // Mix and match for your specific use case.
          sh 'phylum-ci \
            -vv \
            --depfile requirements-dev.txt \
            --depfile requirements-prod.txt path/to/dependency.file \
            --depfile Cargo.toml \
            --force-analysis \
            --all-deps'
        }
      }

Exit Codes

The Phylum analysis stage will return a zero (0) exit code when it completes successfully and a non-zero code otherwise. The full and current list of exit codes is documented here.

Alternatives

There are times where it may not be possible to have full git history with a complete checkout. It may also be the case that a complete checkout is undesirable due to the extra time it takes. For these situation, or if using a "standard" or "standalone" pipeline configuration, the solution is to force analysis of all current dependencies so that no history is needed. This is done with the following flags:

      steps {
        // Force analysis for all current dependencies.
        sh 'phylum-ci -vv --force-analysis --all-deps'
      }

It is also possible to make direct use of the phylum Python package within CI. This may be necessary if the Docker image is unavailable or undesirable for some reason. To use the phylum package, install it and call the desired entry points from a script under your control. See the Installation and Usage sections of the README file for more detail.