Breaking changes between Object Storage V1 and Object Storage V2

The transition from Object Storage V1 (Phonograph) to Object Storage V2 (OSv2) is a major architectural shift that requires a number of breaking changes. The legacy Object Storage V1 (Phonograph) has a large API surface area, exposing significant amounts of low-level database functionality directly. Conversely, the new Object Storage V2 architecture syncs objects through the Object Data Funnel service into specialized object databases to gain scale, performance, flexibility, and security improvements. This architectural transition leads to two main types of breaking changes:

• Specialized object databases, being designed for specific use cases, may not offer the same functionality as Object Storage V1. This may require workflows to be modeled differently in OSv2 than in OSv1.
• Separating dimensions of concern that had been consolidated in Object Storage V1 requires refactoring of some existing queries that interact directly with OSv1 APIs.

These breaking changes may block some OSv1-backed object types from migrating to OSv2 without refactoring or remediation. The documentation on migrating to OSv2 contains more information on how these issues are surfaced to the user, as well as how to perform a migration from OSv1 to OSv2.

Permanent breaking changes

This section provides a list of breaking changes between Object Storage V1 (Phonograph) and Object Storage V2 (OSv2). These breaking changes should be addressed before trying to migrate any object type to OSv2. All of these changes between OSv1 and OSv2 are intended to improve the quality of data going into the Ontology, ensure more deterministic behavior, and increase legibility across the platform.

Actions and edits

• OSv2 only supports user edits via Actions. All existing direct queries on OSv1 edit APIs must be refactored to use Actions before migrating to OSv2.
• OSv2 renames "writeback datasets" as "materializations". In OSv1, writeback datasets are required; in OSv2, materializations are optional. See Materializations for more details on materialized datasets.

Edit history

• It is not possible to migrate object types from OSv1 to OSv2 and maintain user OSv1 edit history.
• Previous edits from OSv1 will persist, however edit history from OSv1 will not be available in OSv2.
• Track user edit history will need to be enabled for object types in OSv2 within the Ontology Manager and object views will need to be updated to add the Edit History widget to display user edit histories.
• Users may also configure Action Log object types to otherwise capture how Actions affect objects; also, see the Action Log Timeline widget to display a history of Actions across targeted objects.

Primary keys and uniqueness

• OSv2 enforces unique object primary keys for datasources. If there are duplicate primary keys in a datasource, indexing that datasource into the Ontology will fail and throw an error.
• OSv2 prevents certain data types from being used as primary keys in order to encourage Ontology modeling best practices. The following types cannot be used as primary keys:
  • Geohash
  • Geoshapes
  • Arrays
  • Time series properties
  • Real number types (decimal, double, float)

Data and property type restrictions

• OSv2 enforces data type coherence between datasource schema and object type schema on every sync. Incompatible data types for a property will cause the build to fail.
• OSv2 does not allow NaN or ±infinity as property values.
  • Empty strings are not allowed in OSv2; in OSv1, empty strings were silently converted to nulls.
  • Lat, Long should be a comma-separated string with no parentheses, for example -29.123, 150.982.
• OSv2 does not allow properties with nested arrays.
• OSv2 does not allow properties with array data types to have null values.
  • OSv1 currently skips null values for indexed array properties and returns the array as containing null values; thus, querying for null will not match an object with null values, which can create an unwanted discrepancy between object loads and searches.

Changelog and incremental updates

• For object types with a changelog datasource, OSv2 does not consider the “latest timestamp wins” semantic used by legacy changelog datasets.
  • OSv2 indexes all pipelines incrementally by default; all relevant changelog calculations are automatically performed in the background by Funnel, rendering the "changelog python decorator" obsolete.
• OSv2 has stricter validations on Geohash properties.

Monitoring

• OSv2 object types can only be monitored by Monitoring Views, which replaces the Object Storage V1 (Phonograph) sync status health check.

Object Set Service (OSS) changes

• OSS APIs do not have query string support; OSv1 does have query string support.
• OSS cardinality metrics are not supported on object sets that contain objects from both Object Storage V1 and Object Storage V2, such as multi-backend object sets.

Temporary breaking changes

This section lists out the current feature gap between Object Storage V1 (Phonograph) and Object Storage V2 (OSv2). These features are in development and the list will be updated as the breaking changes are resolved.

• OSv2 currently does not support connecting schedules where the path between the upstream and downstream targets would go through a materialization.

• OSv2 currently has an edge case where it returns less data in search responses than what the user is allowed to see:

  • The issue happens when all of the following conditions are met:
    • An object type is indexed into OSv2 and uses a restricted view datasource.
    • The restricted view datasource has a Marking applied to an array column.
    • The array column contains empty arrays on some rows.
  • According to the definition of Marking, users should be allowed to see the rows with empty arrays, but OSv2 does not return the corresponding rows in search responses.
  • One temporary solution is to add an element to every array so that the array column used in the Marking no longer contains empty arrays. For example, you can add a Marking to which every user has access.

• OSv2 currently does not support decimal as a property data type.

• OSv2 currently does not support Not conditions in granular permissioning policies of restricted view datasources.

• OSv2 currently does not support custom analyzers.

• OSv2 currently does not support projections on incrementally updating backing datasets.

• OSv2 currently does not support indexing object types backed by incremental View datasets with multiple transactions.

• Foundry Rules Archetypes currently does not support the full feature set of OSv2. For example, object types backed by multiple datasources and object types with multiple materializations are not supported. You can refer to the Foundry Rules documentation for more details.