Custom CI

You can extend the continuous integration (CI) checks that run in your repository by adding custom CI jobs or custom Gradle tasks.

Add custom CI jobs [Beta]

In a Foundry code repository, the ci.yml file defines the default CI jobs that Foundry manages. To add your own jobs to checks, define them in a ci.additions.yml file.

With custom CI jobs, you can:

• Add jobs to checks that each run a list of shell commands.
• Set environment variables for all jobs, including the default jobs, or for a single custom job.
• Connect all jobs, including the default jobs, to external systems using the egress policies and secrets of Data Connection sources.

Create a custom CI job

To add a job, create a file named ci.additions.yml at the root of your repository. Repository upgrades never overwrite this file.

Before you run the following example, prepare and import a source with the API name MySource and a secret named password. The example defines an environment variable, declares the source, and adds a job that prints the value of GREETING and verifies that the password secret is available.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
env:
  GREETING: Hello from a custom CI job

sources:
  - MySource

jobs:
  print-greeting:
    timeout: 5 minutes
    steps:
      - echo "$GREETING"
      - test -n "$MySource_password"

The test -n "$MySource_password" step fails if the MySource_password environment variable is empty or unset. This confirms that the secret is available without printing its value to the logs.

Commit the file to run checks. In the Checks tab, each job appears separately with its own status and logs.

File reference

The ci.additions.yml file supports the following top-level keys. All keys are optional. If the file contains any other top-level key, checks fail.

Each job in jobs supports the following fields:

Rules

Checks fail if ci.additions.yml does not follow these rules:

• Names in env and jobs must not already be defined in ci.yml. You can add new names, but you cannot change the configuration that Foundry manages.
• Jobs must not set order.

Custom jobs run after all jobs defined in ci.yml, in the order in which they appear in ci.additions.yml. If ci.yml orders its jobs with order and one of those jobs fails, the custom jobs do not run.

Use sources in jobs

Jobs can connect to external systems through Data Connection sources. When you declare a source in ci.additions.yml, checks apply the source's egress policies to all jobs, including the default jobs, and provide the source's secrets as environment variables.

Prepare the source

Before a job can use a source, configure the following settings on the source in Data Connection:

1. Navigate to Connection settings > Code import configuration, toggle on Allow this source to be imported into code repositories, and make sure that the source has an API name. Jobs reference the source by this name.
2. Navigate to Connection settings > Export configuration and toggle on Enable exports to this source without markings validations.

Import the source into the repository

Import the source into the repository that contains ci.additions.yml. For example:

• In Python transforms repositories, use the External systems side panel. For more information, review Import a source into code.
• In functions repositories, use the Resource imports side panel.

Importing a source requires Editor permission on the source.

Declare the source

List the API names of the sources that jobs can use in the top-level sources key, as shown in the custom CI job example.

Each declared source must be imported into the repository and have an API name. Jobs cannot use sources that are imported into the repository but not declared in ci.additions.yml.

Access source secrets

Checks provide each secret of a declared source as an environment variable named <source API name>_<secret name>. For example, if a source with the API name MySource has a secret named password, jobs can read the secret from the MySource_password environment variable.

Available tools

Jobs run in a minimal Linux environment that includes the following tools:

• bash and common command-line utilities, such as grep, sed, find, xargs, tar, gzip, bzip2, and base64
• git
• python3 and the Python standard library
• A Java Development Kit (JDK), located at $JAVA_HOME

Tools such as curl and ssh are not available. To make HTTP requests, use Python, for example with the urllib.request module ↗.

Create custom Gradle tasks

You can create custom checks as Gradle ↗ tasks. Add these tasks to the appropriate inner build.gradle files in the language subfolders. The following example defines a Gradle task that prints Hello World when it runs. The doLast method creates a task action that runs when the task executes, rather than at configuration time.

Copied!
1
2
3
4
5
6
// Define custom task
task customTask {
    doLast {
        println "Hello World"
    }
}

To run a task during CI checks, a CI task must depend on it. Define this dependency in the same build.gradle file. In the following example, the check task runs after customTask in CI, and the Hello World message appears in the CI logs.

Copied!
1
2
// Add CI dependency on custom task
project.tasks.check.dependsOn customTask

To run a task at the end of the task list, use the following syntax instead.

Copied!
1
2
// Execute custom task after CI checks are published
project.tasks.publish.configure { finalizedBy customTask }

To add dependencies, use the following CI tasks: project.tasks.check, project.tasks.test, and project.tasks.publish. Do not use or rely on other CI tasks, such as internal tasks, because they are not guaranteed and may change.

For more information about Gradle build scripts, review the Gradle documentation ↗.