Migrate from Object Storage V1 (Phonograph) to Object Storage V2
The architecture changes necessary for the improvements in Object Storage V2 (OSv2) require a migration of object types and many-to-many link types with join tables in Object Storage V1 (Phonograph) to Object Storage V2 (OSv2).
The migration is currently not mandatory but recommended. Foundry's Ontology Manager provides an integrated framework for zero-downtime migration of object types and many-to-many link types.
The Foundry Ontology does not require migrating all object types and many-to-many link types at once. It will continue its normal operation with some of the object types being in Object Storage V1 (Phonograph) and some in Object Storage V2 (OSv2). Foundry's Ontology Manager also supports migrating ontology types in bulk.
Considerations before running a migration
Migration behavior
- Any changes to Ontology definitions on object types that would result in a Funnel replacement pipeline will abort any ongoing migrations. To avoid this, ensure that the object type schema remains stable for the entire migration (including soaking period), and perform any schema changes before initiating the migration.
- Object Storage V1 (Phonograph) registrations are updated synchronously for any changes saved in Ontology Manager. However, Funnel records ontology definition changes asynchronously, resulting in a delay between saving an ontology change in Ontology Manager and that change being detected by Funnel. Because of this, migration events like
startorabortmay appear in Ontology Manager with a delay of several seconds after the ontology is saved. - Object Storage V1 (Phonograph) supports editing object and link types directly through Object Storage V1 (Phonograph) edit APIs. This interaction is deprecated and not compatible with OSv2. Before initiating the migration for an object type with user edits, ensure that your usage is compatible. Then, locate the object type in Ontology Manager and navigate to the Datasources tab. Toggle on
Only allow edits via actionsto unblock the migration of that object type. - User edits will be automatically disabled during the entire migration period, including soak period, starting from the moment of migration definition. New user edits cannot be posted during the migration process, but object reads will remain accessible without disruption.
- The edits history and attribution of object types is not preserved during the migration. The latest state of each object instance from Object Storage V1 (Phonograph) is migrated to Object Storage V2. If you did not use the Action Log object type to capture edit history and attribution, contact your Palantir representative for additional assistance.
Incompatible usage
If the Incompatible usage view is not accessible, contact your Palantir representative about enabling this feature.
Certain interactions in Object Storage V2 are considered breaking changes and they are not compatible with Object Storage V2. This incompatible usage is tracked and reported in Ontology Manager. Selecting the Incompatible usage alert will provide a visualization of any incompatible usage in the last 30 days to assist in remediation.
Non-blocking incompatible usage
Some incompatible usage, like direct API calls to Object Storage V1 (Phonograph) endpoints, will not block initiating a migration to OSv2 but will fail after the object type is migrated to OSv2.
You should investigate incompatible usage and determine whether action is needed, such as migrating direct Object Storage V1 (Phonograph) calls to Object Set Service for object queries/reads, Actions for object edits, and Ontology Metadata Service for object type metadata information. If these incompatible usage are no longer relevant or can break without consequence, you can initiate the migration without remediation.
Blocking incompatible usage
Other changes between OSv1 and OSv2, such as not enabling editing by actions only, will actively block the migration in Ontology Manager.
In this situation, you must remove any occurrences of the incompatible usage before starting the migration.
No incompatible usage
Object types without any incompatible usage or breaking changes will display a green tick icon in the banner to indicate that the object type is ready to migrate.
Migrating an entity
Starting a migration
To start the migration, navigate to the Datasources tab of your object type or many-to-many link type with a join table, and go to the Indexing Metadata section. If the object type is eligible for migration, you will be able to select the Object Storage V2 radio button that will open a wizard for selecting migration parameters. If this section is not present, you are most likely working in an ontology where Object Storage V2 is enforced for all object types and join table link types.
The Ontology Metadata Service tracks incompatible usage of object types backed by Object Storage V1 (Phonograph) and will trigger an alert when attempting to migrate an object type with incompatible usage. Also, you must still ensure that the object type schema does not contain any breaking changes.
Transition windows
The Transition windows options allow you to set preferred time windows for a safe migration; for instance, you may want to set a day of the week and time when an object type's use case has minimal activity. Keep in mind that after the migration process begins, it may take up to 30 minutes to initiate the first Funnel pipeline. If the migration cannot be completed within the first transition window, then Object Set Service will wait until the next window to begin reading data from OSv2.
If no transition window is set, the migration will start as soon the first Funnel pipeline succeeds.
A transition window will be computed after the first sync to OSv2 finishes successfully. This means that if a first sync completes in the middle of a configured transition window the migration is not going to consider this as an acceptable transition window and will try to take the next one.
Soak period
If you need to revert to Object Storage V1 (Phonograph) during the migration process, this is possible without downtime during the soak period. The migration framework will keep the Object Storage V1 (Phonograph) index up-to-date along with Object Storage V2 (OSv2) until the end of the soak period. The soak period can be set in increments of days, up to a maximum of 14 days, and will only begin after the migration passes a transition window.
During the soak period the Foundry Ontology backend will automatically route all queries to OSv2 and any request to OSv1 will be rejected. This time period can be used to validate that the workflows are working as expected after the migration. If you experience any issues during the soak period, you can abort the migration to immediately switch back to using OSv1 indices.
Setting the soak period to 0 days will delete the OSv1 index for that object type as soon as the OSv2 index is ready and the transition window is activated. Migrating back to OSv1 after the soak period ends will require re-indexing the data and can cause downtime.
Note that object types will be dual-indexed in OSv1 and OSv2 during the soak period; this will incur additional compute and storage usage. If you would like to avoid the additional resource usage, you can set the soak period to 0 days when configuring the migration.
Migrating a writeback dataset
In Object Storage V1 (Phonograph), writeback datasets are required to enable user edits on an object type or a many-to-many link type with a join table. Object Storage V2 replaces writeback datasets with materialized datasets. Object Storage V2 does not require materialized datasets to enable user edits. With optional materialized datasets in OSv2, you only need to create materializations if they are required for downstream usage.
Writeback datasets of object and link types can be migrated as materialization datasets in OSv2. The materialization dataset will keep the same columns, retaining compatibility with existing downstream pipelines. This toggle is only relevant if you are using the current writeback dataset in other applications. User edits will still be migrated to OSv2.
If you choose not to migrate the writeback dataset, the dataset itself will not get deleted. Instead, the dataset will be static and contain data from the last build time.
If the writeback dataset contains columns that are not mapped to any object type property, those columns will be dropped as part of the migration.
Aborting the migration
If performance regression is observed or bugs are encountered after switching to Object Storage V2, you can safely abort an ongoing migration during the soak period and revert back to OSv1. The migration framework will ensure that the OSv1 index is kept up-to-date with any new data throughout the migration process, and there will be no downtime.
If the object type being migrated has existing user edits, any new user edits will be disabled until the migration is completed.
Sync failures
Your object type may fail to sync during the migration process. If this is the case, you will see a PIPELINE_FAILED error in the Indexing Metadata section of your object types Datasources tab. If that happens, follow the instructions for debugging funnel batch pipelines to investigate and remediate the issue.
Bulk migrations
You can run migrations in bulk using the same transition window and soak time configurations by navigating to the Object types section in the left-hand navigation bar and selecting multiple object types at the same time. The process and setup wizard are the same as when migrating a single object type. We recommend gradual migrations that bulk-migrate less than 100 entities at a time to prevent unexpected complexity or errors.
This wizard will not appear in the interface for ontologies that only have OSv2 enabled.
