2 - How does Foundry make packages available?

📖 Task Introduction

Up to this point, we’ve been taking for granted that we can simply import Python libraries for use in our code-based data transformations. You imported the shutil ↗ package for use in passengers_raw.py, for example.

Let’s start with a review of how Foundry constitutes your Python repository environment these packages available followed by a brief exercise in viewing available packages. As you’ll see in the prescribed reading below, you can view a listing of all direct and transitively available Python packages using a cURL command against the Foundry Artifacts endpoint. The task will walk you through searching for packages by name in the Code Repositories UI.

📚 Recommended Reading (~10 min read)

The following readings are critical to understanding how packages are made available in your environment. Read them before continuing.

• Python environment creation
• Discovering Python libraries

🔨 Task Instructions

1. Open the passengers_logic repository in your Datasource Project: Passengers folder.
2. Create a new branch from Master called yourName/feature/tutorial_using_libraries (e.g., jmeier/feature/tutorial_using_libraries).
3. Open your passenger_flight_alerts_preprocessed.py transform file.
4. Let’s say you want to use the Theano ↗ package. On line 3, add the following statement: import theano.
5. If Code Assist is running, the syntax is underlined in red. Hovering over the underlined text reveals a ModuleNotFoundError, indicating that Theano is not installed in your repository.
6. On the far left of your repository, click the “Libraries” icon just below your branch name.
7. In the Libraries search field, type “Theano.”
8. Theano is available via a Conda library (ref: recommended readings above) and can therefore be imported into your repository. Click the “theano” result from your search and then on the blue Add Library button to the right.
9. If prompted with a pop-up window to confirm, click Add Library once again.
10. Return to your passenger_flight_alerts_preprocessed.py file and notice that the red underline is gone (though it now has a green one telling you the library is imported but not referenced — a much more manageable issue).
11. As noted in the introduction to the recommended Foundry documentation on Python packages, your repository’s meta.yml file is the metadata file responsible for building the Conda recipe ↗ for your environment.
12. Return to the Files panel on the left side of your application and open /transforms-python/conda_recipe/meta.yml. Notice that the theano package now appears under the run section near the bottom of the file.
13. Remove theano from your environment by opening the Libraries panel, selecting theano package, and clicking the icon next to the Library Added indicator.

Adding libraries provides you with more code options, but it also introduces additional overhead to the CI check process. Be judicious in your use of packages, and be aware that some of the popular packages such as pandas and numpy are already installed via the repository’s Python library.