Composite Modules

Composite Modules allow you to create a Module from other existing Modules. This enables you to manage complex configurations by breaking them down into smaller, reusable components. This section will walk through how to define composite Modules.

Define submodules in a Module YAML file

To define submodules in a Module YAML file, you must include a submodules section that lists the Modules you want to use. Consider two Modules, example-module-1 and example-module-2, which you want to compose together:

example-module-1.yml:

Copied!
1
2
3
4
5
6
7
8
name: example-module-1
displayName: Example Module 1
description: Example Module 1
entities:
  - type: helmChart
    helmChart:
      productId: com.palantir.example:backend-service
      k8sNamespace: app-namespace

And example-module-2.yml:

Copied!
1
2
3
4
5
6
7
8
name: example-module-2
displayName: Example Module 2
description: Example Module 2
entities:
  - type: helmChart
    helmChart:
      productId: com.palantir.example:monitoring-service
      k8sNamespace: app-namespace

To combine these Modules into a single composite Module, you can define another Module that uses these Modules as its submodules. The minimal definition of such a Module must specify the exact name and revision of the submodules you want to include. To define these submodules:

example-composite-module.yml:

Copied!
1
2
3
4
5
6
7
8
name: example-composite-module
displayName: Example Composite Module
description: Example Composite Module
submodules:
  - name: "example-module-1"
    revision: 0
  - name: "example-module-2"
    revision: 0

This Module will include all the entities from the example-module-1 and example-module-2 Modules.

Submodule definition

This section lists all available configuration options for submodules. For each submodule, you can define the following configuration items:

name

The name of the Module to include as a submodule. This Module must already exist in Apollo; otherwise, an error will be thrown during Module creation.

revision

The version of the submodule to use. The revision is immutable for a given revision of the parent Module. To update the revision of a submodule, you must create a new revision of the parent Module.

key (optional)

When the same Module is used as a submodule multiple times, a unique key is required to be provided to differentiate between them. key can be any string. A good use case for using the same Module multiple times is, for example, if you want to use the same Module with different configurations, or on different Kubernetes namespaces.

You may want to include the same submodule multiple times in a parent Module, but with different configurations. You can use the key field to differentiate between the submodules. In the following example, example-module is included twice, with different values for the k8sNamespace variable:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
name: example-composite-module
displayName: Example Composite Module
description: Example Composite Module
submodules:
  - name: "example-module"
    revision: 0
    key: "one-example-module"
    variableOverrides:
      k8sNamespace: "test-namespace"
  - name: "example-module"
    revision: 0
    key: "second-example-module"
    variableOverrides:
      k8sNamespace: "prod-namespace"

variableOverrides (optional)

Overrides for any variable defined in the submodule. This allows you to customize the variables of a Module that you include as a submodule. An override can either:

• Hardcode a value for the variable.
• Re-map the submodule's variable to variables defined in the parent Module.

Note that you can only override variables defined by the submodule, not variables defined by the submodule's submodule.

If you do not override a variable defined in the submodule, then you must provide a value when installing the Module.

Overriding variables with hardcoded values

Suppose you have a Module which is configurable because it exposes one or more variables. When defining a composite Module, you might want to reference this as a submodule but limit the configurability by hard-coding some of the variables to values you know upfront. This reduces the number of decisions that installers will have to take.

Consider a Module called example-module:

example-module.yml:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
name: example-module
displayName: Example Module
description: Example Module
variables:
  k8sNamespace:
    type: string
    string:
      defaultValue: app-namespace
      description: Default k8s namespace for all entities
entities:
  - type: helmChart
    helmChart:
      productId: com.palantir.example:backend-service
      k8sNamespace: "{{ moduleVariable.k8sNamespace }}"

Let's say you want to use example-module as a submodule in a Module called example-composite-module. You can override the k8sNamespace variable in example-module, as shown below:

Copied!
1
2
3
4
5
6
name: example-composite-module
submodules:
  - name: "example-module"
    revision: 0
    variableOverrides:
      k8sNamespace: "namespace"

When installing example-composite-module Module, you will not need to manually enter the namespace for example-module's entities.

Overriding variables with references to other variables

Overriding variables with references to other variables can be useful when you have multiple submodules used in one composite Module, and you want multiple variables to have the same value. Without variableOverrides, you would have to set the same value for each variable in each submodule. By leveraging variableOverrides, you can define one variable in the parent Module, and have all the submodules use the same value.

Imagine we are using two submodules in example-composite-module Module. Each submodule exposes a k8sNamespace variable, and we want all the entities to exist in the same namespace. This is how you can use variableOverrides to achieve this:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
name: example-composite-module
...
variables:
  k8sNamespace:
    type: string
    string:
      defaultValue: namespace
      description: The Kubernetes namespace for the Entities to exist in
submodules:
  - name: "example-module-1"
    revision: 0
    variableOverrides:
      k8sNamespace: "{{ moduleVariable.k8sNamespace }}"
  - name: "example-module-2"
    revision: 0
    variableOverrides:
      k8sNamespace: "{{ moduleVariable.k8sNamespace }}"

k8sNamespace from example-module-1 Module and example-module-2 are re-mapped to the k8sNamespace variable defined in the example-composite-module Module.

Copied!
1
2
variableOverrides:
  frontend-url: "https://{{ moduleVariable.tenant-name }}.{{ moduleVariable.domain }}.com"

entityOverrides (optional)

Allows for overriding configurations and secrets of any entities defined in the submodule. This is useful for cases where you want to reference a submodule, but some of the config overrides or secrets in the submodule are not correct for your use case. You can use entityOverrides to apply overrides on top of entities from the submodule. You can add new config overrides that are not configured in the entity at all, or overwrite existing ones.

Here is an example of how to use entityOverrides:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
name: example-composite-module
...
submodules:
  ...
  # Here we override the entity defined in example-module-1:
  - name: "example-module-1"
    revision: 0
    entityOverrides:
      - type: helmChart
        helmChart:
          productId: com.palantir.example:control-plane
          override:
            configOverrides: |
              2.0.0:
                overrides:
                  num-threads: 100
            secretRequirements:
              secret-name:
                type: multikey
                multikey:
                  description: API key for communicating between services
                  keys:
                    - a-different-key

Here is a breakdown of the example:

• type (required): The type of the Entity to override. This value must match the type of the entity from the referenced submodule.
• productId (required): The ProductId of the Entity to override.
• entityName (optional): Used to differentiate between Entities of the same type. This is useful when you have multiple Entities of the same productId.
• override (required): Overrides for the Entity:

  • configOverrides (optional): Should be formatted as a valid YAML string. Provided overrides will be merged with existing configOverrides for the Entity. If the same key is defined in both places, the value in the entityOverrides will take precedence. Example:

    Consider you have this entity defined in a submodule:

    Copied!
    1
    2
    3
    4
    5
    6
    7
    8
    - type: helmChart
      helmChart:
        productId: com.palantir.example:control-plane
        configOverrides: |
          2.0.0:
            overrides:
              num-threads: 50
              thread-pool-size: 10

    And you apply the following entityOverrides:

    Copied!
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    entityOverrides:
      - type: helmChart
        helmChart:
          productId: com.palantir.example:control-plane
          override:
            configOverrides: |
              2.0.0:
                overrides:
                  num-threads: 100
                storage-size: 10Gi

    The resulting entity will have the following configOverrides:

    Copied!
    1
    2
    3
    4
    5
    6
    configOverrides: |
      2.0.0:
        overrides:
          num-threads: 100
          thread-pool-size: 10
        storage-size: 10Gi

    • thread-pool-size is not overridden, as it is not defined in the entityOverrides.
    • storage-size is added, as it is defined in the entityOverrides.
    • num-threads is overridden, as it is defined in both the Entity and the entityOverrides.

  • secretRequirements (optional): Provided secret requirements will be merged with existing secretRequirements for the Entity. The format should match the secretRequirements definition of an Entity. Example:

    Consider you have this Entity defined in a submodule:

    Copied!
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    - type: helmChart
      helmChart:
        productId: com.palantir.example:control-plane
        secretRequirements:
          secret-name:
            type: multikey
            multikey:
              description: API key for communicating between services
              keys:
                - api-key

    And you override secret requirements with the following entityOverrides:

    Copied!
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    entityOverrides:
      - type: helmChart
        helmChart:
          productId: com.palantir.example:control-plane
          override:
            secretRequirements:
              secret-name:
                type: multikey
                multikey:
                  description: API key for communicating between services
                  keys:
                    - a-different-key
              another-secret:
                type: multikey
                multikey:
                  description: Another secret
                  keys:
                    - another-key

    The resulting Entity will have the following secrets:

    Copied!
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    secretRequirements:
      # this secret was overridden
      secret-name:
        type: multikey
        multikey:
          description: API key for communicating between services
          keys:
            - a-different-key
      # this secret was added
      another-secret:
        type: multikey
        multikey:
          description: Another secret
          keys:
            - another-key

Copied!
1
2
3
4
5
6
7
8
9
entityOverrides:
  - type: helmChart
    helmChart:
      productId: com.palantir.example:control-plane
      override:
        configOverrides: |
          2.0.0:
            overrides:
              num-threads: "{{ moduleVariable.numThreads }}"