Azure Pipelines Integration

Azure Pipelines Integration

Overview

Azure Pipelines supports several different source repositories. This integration works with repos
hosted on Azure Repos Git and GitHub.

Once configured for a repository, the Azure Pipelines integration will provide analysis of project dependencies from
manifests and lockfiles. This can happen in a branch pipeline run from a CI trigger or in a Pull Request (PR)
pipeline run from a PR trigger.

For PR triggered pipelines, analyzed dependencies will include any that are added/modified in the PR.

For CI triggered pipelines, the analyzed dependencies will be determined by comparing dependency files in the branch
to the default branch. All dependencies will be analyzed when the CI triggered pipeline is run on the default
branch.

The results will be provided in the pipeline logs and provided as a comment in a thread on the PR. The CI job will
return an error (i.e., fail the build) if any of the analyzed dependencies fail to meet the established policy.

There will be no comment if no dependencies were added or modified for a given PR.
If one or more dependencies are still processing (no results available), then the comment will make that clear and
the CI pipeline job will only fail if dependencies that have completed analysis results do not meet the active policy.

Prerequisites

The Azure Pipelines environment is primarily supported through the use of a Docker image.
The pre-requisites for using this image are:

  • Access to the phylumio/phylum-ci Docker image
  • Azure DevOps Services is used with an Azure Repos Git or GitHub repository type
    • Azure DevOps Server versions are not guaranteed to work at this time
    • Bitbucket Cloud hosted repositories are not supported at this time
  • An Azure token with API access
    • This is only required when the build repository is Azure Repos Git and PR triggers are enabled
    • Can be the default System.AccessToken provided automatically at the start of each pipeline build
    • Can be a personal access token (PAT) - see documentation
      • Needs at least the Pull Request Threads scope (read & write)
      • Consider using a service account for this token
  • A GitHub PAT with API access
    • This is only required when the build repository is GitHub and PR triggers are enabled
    • Can be a fine-grained PAT
    • Can be a classic PAT
      • Needs the repo scope or minimally the public_repo scope if private repositories are not used
      • See documentation
  • A Phylum token with API access
  • Access to the Phylum API endpoints
    • That usually means a connection to the internet, optionally via a proxy
    • Support for on-premises installs are not available at this time

Configure azure-pipelines.yml

Phylum analysis of dependencies can be added to existing pipelines or on it's own with this minimal configuration:

trigger:
  - main
pr:
  - main

jobs:
  - job: Phylum
    pool:
      vmImage: ubuntu-latest
    container: phylumio/phylum-ci:latest
    steps:
      - checkout: self
        fetchDepth: 0
        persistCredentials: true
      - script: phylum-ci
        displayName: Analyze dependencies with Phylum
        env:
          PHYLUM_API_KEY: $(PHYLUM_TOKEN)
          AZURE_TOKEN: $(AZURE_PAT)     # For Azure repos only
          GITHUB_TOKEN: $(GITHUB_PAT)   # For GitHub repos only

This single stage pipeline configuration contains a single container job named Phylum, triggered to run on pushes
or PRs targeting the main branch. It does not override any of the phylum-ci arguments, which are all either
optional or default to secure values.

Let's take a deeper dive into each part of the configuration:

Pipeline control

Choose when to run the pipeline. See the YAML schema trigger definition and pr definition
documentation for more detail.

# This is a CI trigger that will cause the
# pipeline to run on pushes to the `main` branch
trigger:
  - main

It is recommended to also enable PR validation for the target trigger branch(es). To do so for GitHub repos, use
the pr keyword. See the YAML schema pr definition documentation for more detail.

# This is a PR trigger that will cause the pipeline to run when
# a pull request is opened with `main` as the target branch.
# NOTE: This has no affect for Azure Repos Git based repositories
pr:
  - main

To enable PR validation for Azure Repos Git, navigate to the branch policies for the desired branch
(main in this example), and configure the Build validation policy for that branch.
For more information, see the documentation on PR triggers for Azure Repos Git hosted repositories,
PR triggers for GitHub, or more broadly events that trigger pipelines.

Job names

The job name can be named differently or included in an existing stage/job.

jobs:
  - job: Phylum  # Name this what you like

Pool selection

The pool is specified at the job level here because this is a container job. While Azure Pipelines
allows container jobs for windows-2019 and ubuntu-* base vmImage images, only ubuntu-* is supported by
Phylum at this time. Keeping that restriction in mind, the pool can be specified at the pipeline or stage level
instead. See the YAML schema pool definition documentation for more detail.

    pool:
      vmImage: ubuntu-latest

Docker image selection

The container is specified at the job level here because this is a container job where all steps
in the job are meant to run with the same image. The container can also be specified as a
resource at the pipeline level and then
referenced by name in individual steps of a job instead.
See the YAML schema jobs.job.container definition and resource definition
documentation for more detail.

Choose the Docker image tag to match your comfort level with image dependencies. latest is a "rolling" tag that
will point to the image created for the latest released phylum-ci Python package. A particular version tag
(e.g., 0.21.0-CLIv4.0.1) is created for each release of the phylum-ci Python package and should not change
once published.

However, to be certain that the image does not change...or be warned when it does because it won't be available
anymore...use the SHA256 digest of the tag. The digest can be found by looking at the phylumio/phylum-ci
tags on Docker Hub or with the command:

# The command-line JSON processor `jq` is used here for the sake of a one line example. It is not required.
❯ docker manifest inspect --verbose phylumio/phylum-ci:0.21.0-CLIv4.0.1 | jq .Descriptor.digest
"sha256:7ddeb98897cd7af9dacae2e1474e8574dcf74b2e2e41e47327519d12242601cc"

For instance, at the time of this writing, all of these tag references pointed to the same image:

    # NOTE: These are examples. Only one container line for `phylum-ci` is expected.

    # Be explicit about wanting the `latest` tag
    container: phylumio/phylum-ci:latest

    # Use a specific release version of the `phylum-ci` package
    container: phylumio/phylum-ci:0.21.0-CLIv4.0.1

    # Use a specific image with it's SHA256 digest
    container: phylumio/phylum-ci@sha256:7ddeb98897cd7af9dacae2e1474e8574dcf74b2e2e41e47327519d12242601cc

Only the last tag reference, by SHA256 digest, is guaranteed to not have the underlying image it points to change.

The default phylum-ci Docker image contains git and the installed phylum Python package. It also contains an
installed version of the Phylum CLI and all required tools needed for lockfile generation.
An advantage of using the default Docker image is that the complete environment is packaged and made available
with components that are known to work together.

One disadvantage to the default image is it's size. It can take a while to download and may provide more
tools than required for your specific use case. Special slim tags of the phylum-ci image are provided as
an alternative. These tags differ from the default image in that they do not contain the required tools needed
for lockfile generation (with the exception of the pip tool). The slim tags are
significantly smaller and allow for faster action run times. They are useful for those instances where no
manifest files are present and/or only lockfiles are used.

Here are examples of using the slim image tags:

    # NOTE: These are examples. Only one container line for `phylum-ci` is expected.

    # Use the most current release of *both* `phylum-ci` and the Phylum CLI
    container: phylumio/phylum-ci:slim

    # Use the `slim` image with a specific release version of `phylum-ci` and Phylum CLI
    container: phylumio/phylum-ci:0.36.0-CLIv5.7.1-slim

Repository checkout

The phylum-ci logic for determining changes in dependency files requires git history beyond what is available in a
shallow clone/checkout/fetch. To ensure the shallow fetch option is disabled for the pipeline, an explicit checkout
step is specified here, with fetchDepth set to 0. It is also possible to disable the shallow fetch option in the
pipeline settings UI. See the YAML schema steps.checkout definition documentation
for more detail.

In order to support CI triggers, certain git operations are needed to determine the default branch name and set the
remote HEAD ref for it since Azure Pipelines does not do so during repository checkout. These operations require git
credentials to be available after the initial fetch, which is done with the persistCredentials property. This property
is not required if CI triggers are disabled (e.g., via trigger: none).

      # Reference: https://learn.microsoft.com/azure/devops/pipelines/yaml-schema/steps-checkout
      - checkout: self
        fetchDepth: 0
        persistCredentials: true    # Needed only for CI triggers

Script arguments

The arguments to the script step are the way to exert control over the execution of the Phylum analysis.
The entry here will run as a script in the phylum-ci based container job.
See the YAML schema steps.script definition and container job documentation for
more detail.

The phylum-ci script entry point is expected to be called. It has a number of arguments that are all optional
and defaulted to secure values. To view the arguments, their description, and default values,
run the script with --help output as specified in the Usage section of the top-level README.md or
view the script options output for the latest release.

      # NOTE: These are examples. Only one script entry line for `phylum-ci` is expected.

      # Use the defaults for all the arguments.
      # The default behavior is to only analyze newly added dependencies
      # against the active policy set at the Phylum project level.
      - script: phylum-ci

      # Provide debug level output
      - script: phylum-ci -vv

      # Consider all dependencies in analysis results instead of just the newly added ones.
      # The default is to only analyze newly added dependencies, which can be useful for
      # existing code bases that may not meet established policy rules yet,
      # but don't want to make things worse. Specifying `--all-deps` can be useful for
      # casting the widest net for strict adherence to Quality Assurance (QA) standards.
      - script: phylum-ci --all-deps

      # Some lockfile types (e.g., Python/pip `requirements.txt`) are ambiguous in that
      # they can be named differently and may or may not contain strict dependencies.
      # In these cases it is best to specify an explicit path, either with the `--lockfile`
      # option or in a `.phylum_project` file. The easiest way to do that is with the
      # Phylum CLI, using the `phylum init` command (https://docs.phylum.io/docs/phylum_init)
      # and committing the generated `.phylum_project` file.
      - script: phylum-ci --lockfile requirements-prod.txt

      # Specify multiple explicit dependency file paths
      - script: phylum-ci --lockfile requirements-prod.txt Cargo.toml path/to/dependency.file

      # Force analysis, even when no dependency file has changed. This can be useful for
      # manifests, where the loosely specified dependencies may not change often but the
      # completely resolved set of strict dependencies does.
      - script: phylum-ci --force-analysis

      # Force analysis for all dependencies in a manifest file. This is especially useful
      # for *workspace* manifest files where there is no companion lockfile (e.g., libraries).
      - script: phylum-ci --force-analysis --all-deps --lockfile Cargo.toml

      # Ensure the latest Phylum CLI is installed.
      - script: phylum-ci --force-install

      # Install a specific version of the Phylum CLI.
      - script: phylum-ci --phylum-release 4.8.0 --force-install

      # Mix and match for your specific use case.
      - script: |
        phylum-ci \
          -vv \
          --lockfile requirements-dev.txt \
          --lockfile requirements-prod.txt path/to/dependency.file \
          --lockfile Cargo.toml \
          --force-analysis \
          --all-deps

Script Variables

The script step environment variables are used to ensure the phylum-ci tool is able to perform it's job.

A Phylum token with API access is required to perform analysis on project dependencies.
Contact Phylum or register to gain access.
See also phylum auth register command documentation and consider using a bot or group account
for this token.

Azure Repos Git Build Repositories

An Azure DevOps token with API access is required to use the API (e.g., to post notes/comments) when the build
repository is Azure Repos Git and PR triggers are enabled.
This can be the default System.AccessToken provided automatically at the start of each pipeline build for the
scoped build identity or a personal access token (PAT).

If using a PAT, it will need at least the Pull Request Threads scope (read & write).
The account used to create the PAT will be the one that appears to post the comments on the pull request.
Therefore, it might be worth using a bot or service account.
See the Azure DevOps documentation for using PATs to authenticate for more info.

If using the System.AccessToken, the scoped build identity it attaches to needs at least the
Contribute to pull requests permission. For example, to use the System.AccessToken on a project-scoped
identity, follow these steps:

  • Go to project settings
  • Select the Repos --> Repositories menu
  • Select the Security tab
  • Select the user {Project Name} Build Service ({Org Name})
    • NOTE: This user will only exist after the first time the pipeline has run
  • Ensure the Contribute to pull requests permission is set to Allow

See the Azure DevOps documentation for using the System.AccessToken and setting it's
job authorization scope.

GitHub Build Repositories

A GitHub PAT with API access is required to use the API (e.g., to post notes/comments) when the build
repository is GitHub and PR triggers are enabled.
This can be a fine-grained or classic PAT.

If using a fine-grained PAT, it will need repository access and permissions for read access to metadata and
read/write access to pull requests. See permissions required for fine-grained PATs for more info.

If using a classic PAT, it will need the repo scope or minimally the public_repo scope if private
repositories are not used. See documentation for scopes for more info.

Setting Values

Values for the PHYLUM_API_KEY and either AZURE_TOKEN or GITHUB_TOKEN environment variable (e.g., PHYLUM_TOKEN
and one of either AZURE_PAT or GITHUB_PAT in the example here) can come from the pipeline UI, a variable group,
or an Azure Key Vault. View the full documentation for how to set secret variables for more
information. Since these tokens are sensitive, care should be taken to protect them appropriately.

        env:
          # Contact Phylum (phylum.io/contact-us) or register (app.phylum.io/register)
          # to gain access. See also `phylum auth register`
          # (https://docs.phylum.io/docs/phylum_auth_register) command documentation.
          # Consider using a bot or group account for this token.
          # This value (`PHYLUM_TOKEN`) will need to be set as a secret variable:
          # https://learn.microsoft.com/azure/devops/pipelines/process/set-secret-variables
          PHYLUM_API_KEY: $(PHYLUM_TOKEN)

          # NOTE: These are examples. Only one `AZURE_TOKEN` entry line is expected, and only
          #       when the build repository is hosted in Azure Repos Git with PR triggers enabled.
          #
          # Use the `System.AccessToken` provided automatically at the start of each pipeline build.
          # This value does not have to be set as a secret variable since it is provided by default.
          AZURE_TOKEN: $(System.AccessToken)
          #
          # Use a personal access token (PAT).
          # This value (`AZURE_PAT`) will need to be set as a secret variable:
          # https://learn.microsoft.com/azure/devops/pipelines/process/set-secret-variables
          AZURE_TOKEN: $(AZURE_PAT)

          # NOTE: A `GITHUB_TOKEN` entry is only needed for GitHub hosted build repositories
          #       with PR triggers enabled.
          #
          # Use a personal access token (PAT).
          # This value (`GITHUB_PAT`) will need to be set as a secret variable:
          # https://learn.microsoft.com/azure/devops/pipelines/process/set-secret-variables
          GITHUB_TOKEN: $(GITHUB_PAT)

Alternatives

It is also possible to make direct use of the phylum Python package within CI.
This may be necessary if the Docker image is unavailable or undesirable for some reason.
To use the phylum package, install it and call the desired entry points from a script under your control.
See the Installation and Usage sections of the README file for more detail.