Gets the smallest node type for databricks_cluster that fits search criteria, like amount of RAM or number of cores. AWS or Azure. Internally data source fetches node types available per cloud, similar to executing databricks clusters list-node-types
, and filters it to return the smallest possible node with criteria.
data "databricks_node_type" "with_gpu" {
local_disk = true
min_cores = 16
gb_per_core = 1
min_gpus = 1
}
data "databricks_spark_version" "gpu_ml" {
gpu = true
ml = true
}
resource "databricks_cluster" "research" {
cluster_name = "Research Cluster"
spark_version = data.databricks_spark_version.gpu_ml.id
node_type_id = data.databricks_node_type.with_gpu.id
autotermination_minutes = 20
autoscale {
min_workers = 1
max_workers = 50
}
}
Data source allows you to pick groups by the following attributes
min_memory_gb
- (Optional) Minimum amount of memory per node in gigabytes. Defaults to _0_.gb_per_core
- (Optional) Number of gigabytes per core available on instance. Conflicts with min_memory_gb
. Defaults to _0_.min_cores
- (Optional) Minimum number of CPU cores available on instance. Defaults to _0_.min_gpus
- (Optional) Minimum number of GPU's attached to instance. Defaults to _0_.local_disk
- (Optional) Pick only nodes with local storage. Defaults to _false_.local_disk_min_size
- (Optional) Pick only nodes that have size local storage greater or equal to given value. Defaults to _0_.category
- (Optional, case insensitive string) Node category, which can be one of (depending on the cloud environment, could be checked with databricks clusters list-node-types -o json|jq '.node_types[]|.category'|sort |uniq
):
General Purpose
(all clouds)General Purpose (HDD)
(Azure)Compute Optimized
(all clouds)Memory Optimized
(all clouds)Memory Optimized (Remote HDD)
(Azure)Storage Optimized
(AWS, Azure)GPU Accelerated
(AWS, Azure)photon_worker_capable
- (Optional) Pick only nodes that can run Photon workers. Defaults to _false_.photon_driver_capable
- (Optional) Pick only nodes that can run Photon driver. Defaults to _false_.graviton
- (boolean, optional) if we should limit the search only to nodes with AWS Graviton CPUs. Default to _false_.fleet
- (boolean, optional) if we should limit the search only to AWS fleet instance types. Default to _false_.is_io_cache_enabled
- (Optional) . Pick only nodes that have IO Cache. Defaults to _false_.support_port_forwarding
- (Optional) Pick only nodes that support port forwarding. Defaults to _false_.Data source exposes the following attributes:
id
- node type, that can be used for databricks_job, databricks_cluster, or databricks_instance_pool.The following resources are used in the same context: