Databricks recommends using pip to install libraries. They are only available on Databricks Runtime ML up to Databricks Runtime ML 8.4, and on Databricks Runtime for Genomics. Note that %conda magic commands are not available on Databricks Runtime. For more information on installing Python packages with conda, see the conda install documentation.
The following sections contain examples of how to use %conda commands to manage your environment. The %conda command is equivalent to the conda command and supports the same API with some restrictions noted below. If you do not specify a channel, all conda commands will fail with PackagesNotFoundError.
You must also update the usage of conda commands in init-scripts to specify a channel using “-c”. You must update all usage of %conda install and %sh conda install to specify a channel using “-c”. To install or update packages using the %conda command, you must specify a channel.ĭefault Anaconda channels will be removed from all Databricks Runtime versions on or after May 25th, 2021.
Your use of any Anaconda channels is governed by their terms of service.Īs a result of this change, Databricks has removed the default channel configuration for the Conda package manager in Databricks Runtime ML 8.0. See Anaconda Commercial Edition FAQ for more information.
Based on the new terms of service you may require a commercial license if you rely on Anaconda’s packaging and distribution. updated their terms of service for channels in September 2020. If you require python libraries that can only be installed using conda, you can use conda-based docker containers to pre-install the libraries you need.Īnaconda Inc. Databricks recommends using %pip for managing notebook-scoped libraries. %conda commands have been deprecated, and will no longer be supported after Databricks Runtime ML 8.4. To install libraries for all notebooks attached to a cluster, use workspace or cluster-installed libraries. The library utility is supported only on Databricks Runtime, not Databricks Runtime ML or Databricks Runtime for Genomics. This article describes how to use these magic commands.
Databricks recommends using this approach for new workloads. The %pip command is supported on Databricks Runtime 7.1 and above, and on Databricks Runtime 6.4 ML and above. Run the %pip magic command in a notebook.There are two methods for installing notebook-scoped libraries: You must reinstall notebook-scoped libraries at the beginning of each session, or whenever the notebook is detached from a cluster. Notebook-scoped libraries do not persist across sessions. Other notebooks attached to the same cluster are not affected. When you install a notebook-scoped library, only the current notebook and any jobs associated with that notebook have access to that library. Notebook-scoped libraries let you create, modify, save, reuse, and share custom Python environments that are specific to a notebook.