Databricks Runtime 7.2 for ML(不支持)

Databricks于2020年8月发布了这张图片。

Databricks Runtime 7.2 for Machine Learning为机器学习和数据科学提供了一个现成的环境Databricks Runtime 7.2(不支持).Databricks Runtime ML包含许多流行的机器学习库,包括TensorFlow、PyTorch和XGBoost。它还支持使用Horovod进行分布式深度学习训练。

有关更多信息,包括创建Databricks Runtime ML集群的说明,请参见Databricks运行时机器学习

新特性和主要变化

Databricks Runtime 7.2 ML是建立在Databricks Runtime 7.2之上的。有关Databricks Runtime 7.2(包括Apache Spark MLlib和SparkR)中的新特性的信息,请参见Databricks Runtime 7.2(不支持)发行说明。

Databricks Runtime ML Python环境的主要更改

本节描述与已安装的Databricks Runtime ML Python环境相比的主要变化Databricks Runtime 7.1 for ML(不支持).您还应该查看Databricks Runtime Python环境的主要更改Databricks Runtime 7.2(不支持).有关已安装的Python包及其版本的完整列表,请参见Python库

新增Python包

  • joblibspark: 0.2.0

Python包升级

  • 图4.8.1 -> 4.8.2

  • Mleap 0.16.0 -> 0.16.1

对Databricks Runtime ML系统环境进行了重大更改

新增系统包

  • xvfb

系统包升级

  • Ttyd 1.6.0 -> 1.6.1

改进

Databricks Runtime 7.2改进了对TensorBoard的支持。看到Databricks Runtime 7.2发布说明

系统环境

Databricks Runtime 7.2 ML的系统环境与Databricks Runtime 7.2不同:

以下部分列出了Databricks Runtime 7.2 ML中包含的不同于Databricks Runtime 7.2的库。

Python库

Databricks Runtime 7.2 ML使用Conda进行Python包管理,包括许多流行的ML包。下面介绍Databricks Runtime 7.2 ML的Conda环境。

CPU集群上的Python

的名字databricks-ml渠道-pytorch-违约依赖关系-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.6.24 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.6.20 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-cpuonly = 1.0 = 0-密码= 2.8 = py37h1ba5d50_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_1-pickleshare = 0.7.5 = py37_1001-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情节= 4.8.2 = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.1 = py3.7_cpu_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap = 3.0.2 = py_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.1 = py37_cpu-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普-astunparse = = 1.6.3-azure-core = = 1.7.0-azure-storage-blob = = 12.3.2-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.2-大猩猩= = 0.3.0-horovod = = 0.19.5-hyperopt = = 0.2.4.db2-joblibspark = = 0.2.0-keras-preprocessing = = 1.1.2-考拉= = 1.1.0-mleap = = 0.16.1-mlflow = = 1.9.1-msr = = 0.6.17-opt-einsum = = 3.3.0-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.7.0-tensorflow-cpu = = 2.2.0-tensorflow-estimator = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀/砖/ conda / env / databricks-ml

GPU集群下的Python

的名字databricks-ml-gpu渠道-pytorch-违约依赖关系-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.6.24 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.6.20 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-密码= 2.8 = py37h1ba5d50_0-cudatoolkit = 10.1.243 = h6bb024c_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_1-pickleshare = 0.7.5 = py37_1001-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情节= 4.8.2 = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.1 = py3.7_cuda10.1.243_cudnn7.6.3_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap = 3.0.2 = py_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.1 = py37_cu101-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普-astunparse = = 1.6.3-azure-core = = 1.7.0-azure-storage-blob = = 12.3.2-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.2-大猩猩= = 0.3.0-horovod = = 0.19.5-hyperopt = = 0.2.4.db2-joblibspark = = 0.2.0-keras-preprocessing = = 1.1.2-考拉= = 1.1.0-mleap = = 0.16.1-mlflow = = 1.9.1-msr = = 0.6.17-opt-einsum = = 3.3.0-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.7.0-tensorflow = = 2.2.0-tensorflow-estimator = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀/砖/ conda / env / databricks-ml-gpu

包含Python模块的Spark包

火花包

Python模块

版本

graphframes

graphframes

0.8.0-db2-spark3.0

R库

R库与R库在Databricks运行时7.2。

Java和Scala库(Scala 2.12集群)

除了Java和Scala库在Databricks Runtime 7.2, Databricks Runtime 7.2 ML包含以下jar:

组ID

工件ID

版本

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.combust.mleap

mleap-databricks-runtime_2.12

0.17.2-4882dc3

ml.dmlc

xgboost4j-spark_2.12

1.0.0

ml.dmlc

xgboost4j_2.12

1.0.0

org.mlflow

mlflow-client

1.9.1

org.scala-lang.modules

scala-java8-compat_2.12

0.8.0

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0