The Apollo Product Specification defines a way of describing the properties, requirements and expectations of an Apollo product. The specification provides a simple, concise way to configure a Product that abstracts universal application requirements away from the underlying infrastructure platform (such as Kubernetes).
The specification includes a set of required components which define the intrinsic properties of an application and a set of optional (opt-in) extensions that enable more capabilities of the Spoke Control Plane.
An Apollo Product Definition is the definition of a service or application which conforms to the Apollo Product Specification. An Apollo Product Definition is intended to be versioned alongside a release of an Apollo Product and should be immutable for that Apollo Product Release.
Apollo Product Definitions are organized into different files, each of which have a specific purpose and enumerated Specification traits:
The individual components of an Apollo Product Definition are packaged into a compressed tarball and sent to Apollo as part of the Apollo Product Release registration process. Once published, the Apollo Product Definition will be represented as an Apollo Product Release within the Apollo Hub.
A working example of an Apollo Product Definition can be found below.
Based on their intended functions, services have different traits which describe their behavior or requirements. Traits represent required Product attributes (e.g. required memory allocations, service ports, certs for TLS-based communication) and the implementation of these traits is deferred to the Spoke Control Plane.
The Apollo Product Specification defines a small set of required properties to be defined in all Apollo Product Definitions in the Product's manifest.yml, regardless of Product type. These properties allow Apollo to understand how to model a Product Release. See the manifest specification docs for further details.
Below, we will walk through examples of using traits for common requirements of services and how developers can declare those in an Apollo Product Definition.
Services need compute resources to function, so the Spoke Control Plane accepts several different compute dimensions from Apollo Product Definitions. These include CPU, memory, and node replication.
Service configuration enables developers to define the compute resources each replica of a service needs.
Developers may provide both "requests" (the minimum quantity of a resource a service needs available) and "limits" (the maximum quantity of a resource a service should use) under a top-level resources field in the configuration.yml file:
Copied!1 2 3 4 5 6 7 8# configuration.yml resources: requests: cpu: 8 memory: 10Gi limits: cpu: 16 memory: 10Gi
See the compute resources specification for expected formatting and supported resource types.
Developers can declare the desired number of replicas (pods) of their service in its configuration.yml file under the replication.desired field:
Copied!1 2 3# configuration.yml replication: desired: 3
Refer to the replication specification for the complete specification for configuring replication. The Spoke Control Plane will use the resulting replication value when constructing pod controllers for the service.
Some services may need to store data to fulfill their responsibilities.
This storage can be requested with volume declarations in an Apollo Product Definition’s manifest and configuration.yml files.
Developers can declare the names and types of volumes for their product using the volumes-v2 manifest extension:
Copied!1 2 3 4 5 6 7 8 9# manifest.yml extensions: volumes-v2: my-volume: volume-type: durable-volume: {} my-other-volume: volume-type: durable-volume: {}
For each entry within the volumes-v2 manifest extension, developers must create a corresponding entry within the top-level volumes field in their product's configuration.yml:
Copied!1 2 3 4 5 6 7 8# configuration.yml volumes: my-volume: durable-storage-configuration: desired-size: 10G my-other-volume: durable-storage-configuration: desired-size: 5T
See the volumes specification for a full specification of the volumes-v2 manifest extension and the expected format of a configuration.yml file's top-level volumes field.
To enable access to a Product's requested volumes, developers can use configuration templating provided by the Spoke Control Plane's substitution language:
Copied!1 2 3 4 5# configuration.yml conf: startup: my-data-directory: '{{ volumes.my-volume.PathOnDisk }}' my-other-data-directory: '{{ volumes.my-other-volume.PathOnDisk }}'
Refer to Referencing Requested Volumes for more information on referencing volumes in configuration.
Services may declare the endpoints from which they need to serve networking requests via the endpoints field in a product’s configuration.yml file:
Copied!1 2 3 4 5 6# configuration.yml endpoints: definitions: my-service-endpoint: desired-port: 8080 path: /my-service-endpoint
If a configuration defines more than one endpoint, then developers must add a field under endpoints indicating which endpoint should be used for status (liveness and readiness) checks:
Copied!1 2 3 4 5 6 7# configuration.yml endpoints: sls-status-endpoint: my-service-endpoint definitions: my-service-endpoint: desired-port: 8080 path: /my-service-endpoint
The Spoke Control Plane expects the status endpoint to serve requests to the /status/liveness and /status/readiness subpaths. Refer to liveness and readiness specification for more details.
The Spoke Control Plane will use the status endpoint to configure Kubernetes liveness and readiness probes for the service.
To enable accessing defined endpoints in configuration, the Spoke Control Plane's substitution language supports referencing endpoint definitions:
Copied!1 2 3 4 5 6 7 8 9endpoints: definitions: my-endpoint: desired-port: 8443 path: /my-endpoint-path conf: server: my-endpoint-port: '{{ endpoints.definitions.port }}' my-endpoint-path: '{{ endpoints.definitions.path }}'
Some services require that their replicas are individually and deterministically addressable for clients making requests over the network. This could be for leader election or consistent routing of client requests. A product that needs this behavior can include the require-stable-hostname extension in its Apollo Product Definition’s manifest.yml file:
Copied!1 2 3# manifest.yml extensions: require-stable-hostname: true
The specification for the require-stable-hostname manifest extension is documented as part of the endpoints specification.
In rare cases, a service’s replicas need to receive traffic before becoming ready. Products may request this behavior by including the allow-traffic-when-not-ready manifest extension:
Copied!1 2 3# manifest.yml extensions: allow-traffic-when-not-ready: true
The liveness and readiness specification also documents the allow-traffic-when-not-ready manifest extension.
Services often need to serve traffic from outside the cluster (for example, to serve requests from web browsers).
Once endpoints are defined in a service’s configuration.yml, developers can declare which of these endpoints should be exposed to external traffic.
Making endpoints externally accessible requires referencing them in an Apollo Product Definition’s manifest under the public-proxy-endpoints and public-tcp-endpoints extensions:
Copied!1 2 3 4 5 6# manifest.yml extensions: public-proxy-endpoints: - my-service-endpoint public-tcp-endpoints: - my-tcp-endpoint
Endpoints which receive HTTPS traffic can use public-proxy-endpoints. If an endpoint receives raw TCP traffic, it may appear in the public-tcp-endpoints extension.
Endpoint definitions whose names appear in the public-tcp-endpoints manifest extension must include an external-port field and must not include a path field.
Endpoint definitions whose names do not appear in the public-tcp-endpoints manifest extension must not include the external-port field.
Updating the previous endpoints configuration example to make it compatible with the example manifest above yields the following:
Copied!1 2 3 4 5 6 7 8 9# configuration.yml endpoints: definitions: my-service-endpoint: desired-port: 8080 path: /my-service-endpoint my-tcp-endpoint: desired-port: 4567 external-port: 22
The full specification for exposing endpoints to external traffic can be found in the endpoints specification.
The Apollo Product Spec is designed to make it easy for developers to build software that is portable, with deployment that doesn’t require manual configuration steps and upgrades that are safe and autonomous. To enable this, Apollo’s Configuration Management focuses on the following sections:
An Apollo Product Definition should include the default configuration of the product, such that the service could be deployed and start correctly without requiring any additional manual configuration. The default configuration is declared in the configuration.yml file of the Apollo Production Configuration, under the conf header:
Copied!1 2 3 4 5 6 7 8# configuration.yml conf: some-config: a: b c: d some-other-config: b: c c: d
Occasionally, a service may require configuration overrides to work alongside setups that can’t be predicted in default configuration. This may be for integrating with an external auth provider, changing garbage collection intervals to match atypical usage patterns, or increasing replication in environments under heavy load. The Spoke Control Plane supports defining configuration overrides to be merged on top of an Apollo Product Definition’s original configuration.yml contents:
Copied!1 2 3 4 5 6 7 8 9 10 11 12 13 14 15# defaults defined in the configuration.yml conf: a: b b: c # overrides defined for a particular instantiation of a service conf: b: overridden new-field: now-present # result of merge conf: a: b b: overridden new-field: now-present
Map merges are performed recursively, meaning that merging array fields or fields of different types will replace the original value with the override’s value. The top-level block for defining generated secrets may not be overridden; this is to protect against outages or data loss resulting from inadvertently overwriting generated secret values.
Configuration overrides should be the exception rather than the rule. See Configuration Substitution for how the Spoke Control Plane minimizes the need for configuration overrides.
The Product Spec provides a simple substitution language that can be used in configuration. Developers can use the substitution language to reference other parts of configuration, or the outputs of the different Spoke Control Plane capabilities offered.
The substitution language leverages double curly brackets ({{ and }}) in configuration to reference other values. The different sections of the Apollo Product Definition can be referenced based on the top level key. The configuration is evaluated and rendered into the final state at runtime by the Spoke Control Plane. As an example:
Copied!1 2 3 4 5 6 7 8 9 10# configuration.yml conf: some-config: a: b c: {{ conf.some-other-config.c }} some-other-config: b: c c: d managed_files: some-conf.yml: some-config
would result in the following rendered contents:
Copied!1 2 3#some-conf.yml: a: b c: d
In addition to being able to reference other configuration sections, the substitution language has a powerful set of helper functions.
Applications and services read configuration from disk in a variety of different file formats and from application specific file names. The Product Spec provides the means for declaring how the configuration declared as part of the Apollo Product Definition should be mapped and rendered to files of different file formats. This is defined in the configuration.yml file of the Apollo Product Definition, under the managed_files header:
Copied!1 2 3 4 5 6 7 8 9 10 11 12 13# configuration.yml conf: some-config: a: b c: d some-other-config: b: c c: d managed_files: some-conf.yml: some-config some-other-conf.json: type: json content: some-other-config
Developers define a file name and the the subset of the configuration that should be rendered into the file. By default, all configuration files will get mounted within the /opt/palantir/services/<product-name>/var/conf directory of the main container.
Managed file declarations include the following fields:
type [required]: indicates the desired format of the file. Supported values are:
content [required]: indicates which portion of the conf block should be included in the managed filereload-type [optional]: indicates whether the service can reload the configuration file without restarting. Accepted values are:
install [default]: a restart is required to reload the fileruntime: the file can be reloaded without a restartThe Control supports a shorthand managed file entry, as is done for some-conf.yml in the example above, where the file extension for the entry’s key (.yml) is used as the managed file’s type, and the entry’s value is used as the managed file’s content. For entries which use this shorthand, the reload type defaults to install. For the above example, the following configuration files would be rendered into the main service container:
Copied!1 2 3 4 5 6# some-conf.yml a: b c: d # some-other-conf.json {"b":"c","c":"d"}
Some services may not accept configuration file formats described above in Managing Configuration Files. For these services, the Spoke Control Plane supports a tmpl file type as well:
Copied!1 2 3 4 5 6 7 8# configuration.yml conf: my-xml-conf: a: b b: c managed_files: offroading.xml: type: tmpl
When using the tmpl file type, developers must additionally provide a configuration file template in their Apollo Product Definition within the deployment/templates directory:
Copied!1 2 3 4|__ deployment |__ configuration.yml |__ templates |__ offroading.xml.tmpl
Configuration template files are treated as Go templates and provided the result of configuration substitution as input via the {{ .conf }} template variable. this enables developers to satisfy custom configuration formats while still benefitting from the Spoke Control Plane’s runtime configuration support. An example of using Go templates to produce a valid XML file using the result of substitution is shown below:
Copied!1 2 3 4<conf> <a>{{ .conf.my-xml-conf.a }}</a> <b>{{ .conf.my-xml-conf.b }}</b> </conf>
Some services rely on credentials or other secret material for authentication and authorization with other services. In some instances, this secret material can be randomly generated. To ease the manual configuration burden for these use cases, Apollo supports a variety of options for automatically generating the required secret material by:
configuration.yml. This is useful in the case where a service just needs some secret value to establish trust with another service within the cluster.Specifying secrets that you’d like the Spoke Control Plane to generate on behalf of your service can be done using the top-level generated_secrets field in your product’s configuration.yml file:
Copied!1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20# configuration.yml generated_secrets: myAppSecret: secret-type: random_string length: 24 myAppRandomByteSecret: secret-type: random_bytes length: 16 myAppCustomCharsetSecret: secret-type: random_string length: 16 allow-chars: 'abcdefg' myAppRsaKeyPair: secret-type: rsa_pkcs8 length: 2048 myAppStigPassword: secret-type: stig_password ec_pkcs8: secret-type: ec_pkcs8 length: 2048
The Spoke Control Plane generates secret values based on the requested secret-type and length fields and stores the generated values in Kubernetes secret objects. The following secret types are supported:
random_string: A random string of the provided length (or a default length of 16) will be generated. If the allow-chars field is present, the generated secret value will only contain characters which appear in the intersection of the allow-chars set and the base64 charset. Similarly, if the disallow-chars field is present, the generated secret value will only contain characters which appear in the set difference of the provided disallow-chars set and the base64 charset.rsa_pkcs8: An RSA PKCS8 key pair of the provided length will be generated.random_bytes: A cryptographically secure random binary value will be generated. The resulting value will be returned base-64 encoded.ec_pkcs8: An elliptic curve public/private key pair will be generated. Accepted length values are 224, 256, 384, and 521 (the standard curves defined in FIPS 186-3).It is highly recommended that the Kubernetes cluster running the Spoke Control Plane is configured to enable secret encryption at rest.
For cases where services must be provided credentials with pre-defined values, the Hub supports setting user-defined secrets via admin-authorized endpoints.
Services can read in generated and user-defined secrets from configuration using the substitution language’s secret method:
Copied!1 2 3 4 5 6 7 8 9# configuration.yml conf: oauth-client: id: my-oauth-client-id secret: '{{secret("my-oauth-secret")}}' generated_secrets: my-oauth-secret: type: random_string length: 32
See the Referencing Secrets section for full details on how to reference secrets from substitution.
The Spoke Control Plane provides certificates and certificate authority (CA) certs to managed services via a volume mount in the primary service container. This mount encompasses the following files:
ca.cer: A concatenation of all the PEM-encoded certificates which the Deployment Platform is configured to provide as CAs for applications.key.pem: A private RSA key in PEM format.cert.cer: The public RSA key for the key.pem.truststore.jks: A Java truststore containing the same set of CAs that appear in ca.cer. This file is provided as a convenience for Java applications.keystore.jks: A Java keystore containing the same private key that appears in key.pem. This file is provided as a convenience for Java applications.See References Certificate Material for how to reference these certificate files in configuration.
To ease the manual burden of configuring information that services in a distributed system need to operate correctly, Apollo’s Spoke Control Plane supports a discovery subsystem. Through discovery, services can produce information that can be consumed by other services running in the same environment. This "discovered" information can then be referenced via the configuration subsystem to enable services to reference environment or installation specific configuration information which is only knowable at runtime, without the human toil of configuring it for each environment.
A service produces information associated with a named "role". These discovery entries can then be consumed via the same named "role". Discovery production and consumption is declared in an service’s configuration.yml file:
Copied!1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23# configuration.yml conf: tags: - foo discovery: produces: my-api: role: my-api port: 8443 path: /my-api scheme: https my-oauth-client: role: oauth-client client: client-id: bcryptedOAuthSecret: '{{secret("my-oauth-secret", part: bcrypt_hash)}}' consumes: neighboring-api: role: neighboring-api select: uris stack: production tags: [bar]
The Spoke Control Plane supports consuming discovery entries produced by other services, using the following filtering:
role: A discovery entry passes this filter if its role matches that of the role field in the consumes entry being evaluated.stack: The stack in which the discovery entry is produced. By default, a discovery entry is only produced in the stack where the service lives. Discovery can be produced to additional stacks by including a produce-to-additional-stacks field under the top-level discovery block, in which case it will apply to all entries within a service’s produced discovery. Note that produce-to-additional-stacks does not work within an individual discovery entry.tags: Services can have discovery tags appear on their produced discovery entries by defining a string list under the conf.tags key (for instance, the configuration above includes the foo tag). By default, consumed discovery filtering accepts any and all tags; if tags are included in a consumes entry (as is the case for the neighboring-api entry above), then only discovery entries with all the specified tags pass this filter.One of the most common use cases of discovery is URI information to enable network requests from one service to another. The Spoke Control Plane simplifies this use case by detecting certain discovery entries to "augment" with URI information. Any discovery entry with a port field which is a valid port number will be augmented in this manner. Optional inputs for this augmentation are:
scheme: The scheme of the URI; defaults to httpspath: The path of the URI; defaults to /The Spoke Control Plane ensures that its discovery augmentation aligns with the URIs as determined by the Kubernetes services it creates on behalf of the service. For example, if a service with the above discovery configuration block has a Kubernetes service with name foo and namespace application-namespace, then the service's produced my-api discovery entry will have its payload augmented to include uri and uris fields:
Copied!1 2 3 4 5 6 7# configuration.yml port: 8443 path: /my-api scheme: https uri: https:foo.application-namespace.svc.cluster.local:8443/my-api uris: https:foo.application-namespace.svc.cluster.local:8443/my-api
Consuming the above payload directly can be clunky; for instance, developers may care only about the uris field when constructing an HTTP client.
To improve discovery ergonomics, the Spoke Control Plane supports defining a select field to ‘pick’ a field when consuming discovery. See tying it together for a working example.
Discovering Services and Config covers how to consume discovery produced by other services, but services still need to inject discovered values into the correct portions of their configuration. To that end, the substitution language has a discovered method for referencing consumed discovery:
Copied!1 2 3 4 5 6 7 8 9# configuration.yml conf: my-neighbor-client: uris: '{{discovered.my-neighbor-api-uris}}' discovery: consumes: my-neighbor-api-uris: role: my-neighbor-api select: uris
The above example would result in conf.my-neighbor-client.uris being set to a list of URIs crafted from the consumed ny-neighbor-api discovery entries.
Self-discovery is also supported:
Copied!1 2 3 4 5 6 7 8# configuration.yml conf: my-app-uris: uris: '{{self_discovered.my-app-uris.uris}}' discovery: produces: my-app-uris: port: 8080
In this example, the port field makes the produced my-app-uris role a candidate for URI augmentation, so the conf.my-app-uris.uris field will have a non-null value.
A working example of an Apollo Product Definition is the easiest way to tie together all the above information. Let’s say we want to deploy a service which:
/foo.bar using mutual TLS, with bar checking a bcrypt hash of foo’s client secret to authenticate requests.config.yml.Our product’s manifest would include a couple extensions in this case:
Copied!1 2 3 4 5 6 7 8# manifest.yml extensions: public-proxy-endpoints: - foo-service-endpoint volumes-v2: foo-service-data: volume-type: durable-volume: {}
Our configuration.yml would look like:
Copied!1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44# configuration.yml replication: desired: 3 endpoints: definitions: foo-service-endpoint: desired-port: 8080 path: /foo volumes: foo-service-data: desired-size: 10G generated_secrets: bar-client-shared-secret: secret-type: random_string length: 32 discovery: produces: bar-client: role: bar-client client-bcrypted-shared-secret: '{{"bar-client-shared-secret", part: "bcrypt_hash"}}' consumes: bar-api-uris: # discovery role under which bar produces its uris role: bar-api select: uris conf: config: security: key-file: '{{ssl.pem_path}}' cert-file: '{{ssl.cert_path}}' ca-file: '{{ssl.ca_path}}' server: security: '{{conf.config.security}}' port: '{{endpoints.definitions.foo-service-endpoint.desired-port}}' base-path: '{{endpoints.definitions.foo-service-endpoint.path}}' bar-client: security: '{{conf.config.security}}' uris: '{{discovered.bar-api-uris}}' auth-secret: '{{secret("bar-client-shared-secret")}}'