Announcements
REMINDER: You can now sign up for the Foundry Newsletter to receive a summary of new products, features, and improvements across the platform directly to your inbox. For more information on how to subscribe, see the Foundry Newsletter and Product Feedback channels announcement.
Lightweight transforms now supports external transforms, incremental transforms, and unmarking
Date published: 2024-03-14
We are continuing to expand the feature set for lightweight transforms and are excited to announce that three new features are now available for improving performance, reducing startup times and overhead, and restricting access to resources based on a user's eligibility.
Incremental transforms compatibility
Incremental transforms are now compatible with lightweight transforms. You can use this integration to combine the performance benefits of incremental transforms with the reduced startup times of lightweight transforms. To make your lightweight build incremental, you just need to add an @incremental decorator as shown below:
import pandas
@lightweight()
@incremental(require_incremental=True)
@transform(my_input=Input("my-input"), my_output=Output("my-output"))
def my_incremental_transform(my_input, my_output):
my_output.write_pandas(my_input.pandas(mode="added"))
To start creating incremental lightweight transforms, upgrade your repository to the latest version and install the latest version of foundry-transforms-lib-python. For more details, review the documentation.
Improved external transforms performance with lightweight mode
External transforms now support the @lightweight API, resulting in improved performance by reducing startup overhead. This enhancement is ideal for external transforms as they are typically not Spark dependent or CPU-intensive, leading to improved latency and throughput. Below is an example:
@lightweight
@use_external_systems(
egress=EgressPolicy('<policy RID>'),
creds=Credential('<credential RID>')
)
@transform(
output=Output('/path/to/output/dataset')
)
def compute(egress, creds, ...):
Support for unmarking
Markings and Organizations restrict access to resources based on a user's eligibility. When restricted content is removed or obfuscated while deriving a dependent resource, you may wish to remove the marking and/or Organization on that derived resource. Removing markings, also called unmarking, is now supported for lightweight transforms.
For more information on removing markings in lightweight transforms, review the documentation.
Additional highlights
App Building | Slate
Refined Tooltip Customization with HTML | Users can now employ HTML to craft styled and multi-line tooltips within charts, enhancing the ability to tailor and present data more effectively.
Enhanced Script Compliance in Slate | Slate has been enhanced to comply with strict mode ↗ standards, ensuring scripts are parsed and errors are handled with greater precision. The use of reserved keywords as variables is now prohibited, aligning with ES6 module conventions.
Enhanced Asynchronous Capabilities in Slate | Slate has been enhanced to accommodate asynchronous functions, paving the way for more intricate use cases such as timeouts between queries and the integration of libraries that utilize async functions. This advancement introduces a more streamlined, promise-based approach to handling asynchronous operations within Slate, while maintaining synchronous function execution.
App Building | Workshop
Introducing Workshop's Loop Layouts | Workshop's capabilities have been expanded with the introduction of Loop layouts, allowing users to iterate over sets of objects and present an embedded module for each one. This enhancement streamlines the handling of data collections and elevates the user's ability to design interactive modules. Further information is available in the loop layouts documentation.
Mobile Workshop Module Integration Enhanced | The mobile version of Workshop now boasts a new feature: the integration of the embedded module widget. This advancement streamlines the incorporation of mobile-compatible modules within other modules, fostering a more interactive and engaging user experience on mobile devices. Embedding is exclusively available for mobile modules, as detailed in the mobile mode limitation section of our documentation.
Upgraded Workshop Notepad Template Button Flexibility | Workshop has been updated to permit the conditional visibility and deactivation of the Notepad template option, based on Boolean variables, mirroring the capabilities of the button group widget. This update also introduces tooltips that utilize string variables, offering improved guidance for users.
Enhanced Audio Preview in Workshop | Workshop now showcases a waveform audio player for media set media references, offering users a more intuitive method to preview audio files.
Enhanced Tab Scrolling in Workshop | The new Enable overflow configuration option allows users to effortlessly navigate through overflowing tabs in Workshop's section headers. When activated, arrows will be displayed on both sides of the tabs, ensuring users can access all tabs even when they exceed the section header's width.
App Building | Ontology SDK
Developer Console appearance refresh | Developer Console has been revamped with a new sidebar, application header, and light theme, along with an enhanced home page.
Python Batch Actions: Return Edits Support | Users can now use the optional parameter batch_action_config to set return_edits to ALL and receive the edits in the response when using Python batch apply actions.
Analytics | Notepad
Introducing batched functions in Notepad | Functions on Objects sections in Notepad may now be configured with batched functions, which can significantly reduce the time taken to generate a document from a template.
Batched functions are functions that accept an object set as an argument and return results for each object in the object set. Users can configure batched functions for Functions on Objects sections that are nested inside generator sections.
Using batched functions can decrease template generation time as unbatched functions must execute once for each object in an object set, whereas a batched function executes only once for the entire object set.
To start using batched functions, users may insert a Functions on Objects section into a Notepad document, select the Batched option from the configuration panel and select a batched function. Learn how to write or configure batched functions.
Revamped Notepad Entry Screen | Notepad's entry screen has been upgraded with a more intuitive search function, enabling users to swiftly locate notes by title or content. This enhancement introduces a sortable resource list with quick-access categories like Recent, My Creations, and Bookmarked. Additionally, a help icon is now available for immediate reference to Notepad guidelines and AIP assistance, alongside simplified processes for generating new Notepad documents or templates.
Improved Markdown Formatting Capabilities in Notepad | Notepad has been upgraded to better interpret Function on Objects output as Markdown, enhancing the creation of dynamic and visually-appealing text formats in templates.
Analytics | Code Workspaces
Introducing Container Transforms in Code Workspaces | It is now possible to publish Jupyter® notebooks and R scripts as data transformation pipelines as container transforms. Container transforms are easily reproducible and can be scheduled to run automatically. This feature gives Code Workspaces first-class integration with Foundry and enables you to easily share the outputs of your data analysis workflows with other users. To learn more about using container transforms, read the public documentation for Jupyter® and RStudio®.
Analytics | Contour
Streamlined Analysis Tagging in Contour | Contour now offers a refined way to categorize and manage your analyses with the introduction of analysis tagging. To assign a tag, select the more actions menu, proceed to the tags submenu, and specify your chosen tag. For an organized view, analyses can be sorted by these tags within the analysis sidebar, simplifying navigation and management of intricate analysis projects.
Ontology | Ontology Management
Refined Ontology Manager Filtering Options | The Ontology Manager Cleanup Tool now boasts improved filtering features, enabling users to order object types by the quantity of associated flags. This enhancement facilitates a more detailed and efficient approach to organizing Ontology cleanup activities, allowing users to effectively prioritize tasks based on distinct criteria.
Enhanced API Name Creation in Ontology Manager | Ontology Manager now generates API names differently. In the past, when an object type display name included '[' and ']' characters, the auto-generated API name would commence with an underscore. With this improvement, the API name will no longer begin with an underscore, offering a more intuitive and user-friendly naming convention. This alteration only impacts the default generated name, which users are advised to customize further.
Customizable Action Type API Names in Ontology Manager | Directly from the Overview tab, Ontology Manager now supports the customization of API names for action types, offering enhanced flexibility.
Ontology | Vertex
Enhanced Annotation Features in Vertex | Vertex has been upgraded to include a new annotation mode, allowing users to mark up and highlight using a variety of custom shapes and colors. This update offers users a dynamic and customizable approach to annotating visual content, enhancing the interactive experience.
Security | Projects
Streamlined Content Management and Dataset Integration | Users now have the enhanced ability to integrate with datasets and manage content directly from URLs, streamlining the process even in the absence of home folders. This improvement ensures that operations such as importing a sheet via URL are conducted effortlessly, regardless of home folder availability.
Administration | Control Panel
Organization markings can be managed in Control Panel | Control Panel now allows Organization administrators to adjust Organization Marking permissions in a new tab within Organization permissions. This enhancement is a stride towards consolidating permissions handling, offering a more unified and straightforward experience. Following the successful implementation of Phase 2 of the migration, anticipated in Q1 2024 and announced in 2023, Control Panel will become the sole hub for Organization permissions configuration, rendering the previous method in Platform Settings obsolete.
Model Integration | AIP Logic
Refined Debugging Interface for AIP Logic | The debugging tool within AIP Logic has been upgraded with new features including toggleable block cards and a streamlined prompt visualization, alongside a dedicated function to reset tool calls. These improvements are designed to enhance the user's ability to efficiently navigate and resolve issues within their logic flows.
Introducing Object Sets Outputs in AIP Logic | The Use LLM block within AIP Logic now supports the return of an object set as an output, enhancing the ease of integration with Workshop. This improvement empowers users to run prompts that yield multiple objects, enhancing the analytical functions available in their workflows. For example, with this feature, prompts like "Are there more F1 drivers from Spain or Italy? Return me those drivers" will return more than one object as a result. Example is notional and uses open source data.
Introducing Single Completion LLM blocks in AIP Logic | Single Completion LLM blocks
You can now run a single completion from an LLM block with no special handling or additional instructions. This feature is useful for improving performance when the additional capabilities of Chain-of-thought (CoT) prompting are not needed by adding the least token overhead among all prompting strategies. Note that when using the single completion LLM block, tools and specify return types are not available.
Introducing the Transform Block in AIP Logic | The Transform block allows you to use expressions within AIP Logic. This is a great way to perform transformations on your data with AIP Logic such as convert a string to a date, parse JSON, perform mathematical expressions and more.
Array Output Capabilities Expanded in AIP Logic | The capability to generate array outputs has been introduced to AIP Logic, facilitating the delivery of multiple suggestions or values through a singular response. This advancement enhances the dynamic nature of interactions with LLM blocks, to address problems that demand a spectrum of possible solutions.
Model Integration | Modeling
Refined Media Set Integration for Model Deployments | Model deployment procedures have been upgraded to incorporate media sets as selectable inputs, specifically when the model's API contains media references. This enhancement introduces a media set selector that appears contextually for applicable models, simplifying the configuration of deployments that involve media data.
Data Integration | Pipeline Builder
Advanced Property Extraction in Pipeline Builder | Pipeline Builder now delivers an enhanced user experience for extracting properties from geoJSON data. Users have the option to extract all properties as a string or define a set of properties to extract using a struct.
Dataset targets support manual data input | Dataset targets in Pipeline Builder now allow manual data entry, akin to the existing feature for object type targets.
Fallback Branches Support in Pipeline Builder | You can now configure fallback branches in the Manage branches dialog for dataset input resolution and pipeline building.
Direct Access to Ontology Manager from Pipeline Builder | Users can now directly open Ontology type outputs in Ontology Manager from Pipeline Builder. When an Ontology type output is selected, a new action is available to the right of that output, allowing users to view it in Ontology Manager without having to switch applications manually.
Streamlined Column Renaming in Pipeline Builder | The "Edit multiple" functionality within Pipeline Builder has been enhanced to align with the "Normalize columns" transformation board, now using lower snake case for column names and retaining special characters. This refinement promotes a more cohesive and user-friendly renaming process, meeting the normalization expectations of our users.
Data Integration | Data Connection
Improved Direct Media Extraction Launch | The Data Connection interface now features an additional button on the media extract overview page, enabling users to directly initiate media extracts. This enhancement simplifies the extraction process by offering a straightforward method to execute media extracts without the need to leave the current page.
Optimized Media Extract Scheduling | Directly within the creation workflow, users now have the capability to set up schedules for media extracts.
Streamlined Media Sync Interface in Data Connection | A newly-introduced media sync overview interface in Data Connection offers users a unified dashboard similar to batch syncs. This new version establishes a foundation for upcoming enhancements, such as an integrated media sync initiation feature and the capability to schedule syncs directly within the platform.
Data Integration | Code Repositories
Enhanced Media Embedding in Markdown Files | Markdown files within Code Repositories now support direct embedding of media items using markdown syntax. For example: .
Easy Imports for TypeScript Repositories | Import object types, link types, REST APIs and models in a single, consolidated workflow using the redesigned Resource imports panel. Easily search and preview assets before pulling them into a repository to use in TypeScript functions.
