azurerm_data_factory_linked_service_azure_databricks

Manages a Linked Service (connection) between Azure Databricks and Azure Data Factory.

Example Usage with managed identity & new cluster

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "East US"
}

#Create a Linked Service using managed identity and new cluster config
resource "azurerm_data_factory" "example" {
  name                = "TestDtaFactory92783401247"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  identity {
    type = "SystemAssigned"
  }
}

#Create a databricks instance
resource "azurerm_databricks_workspace" "example" {
  name                = "databricks-test"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "standard"
}

resource "azurerm_data_factory_linked_service_azure_databricks" "msi_linked" {
  name            = "ADBLinkedServiceViaMSI"
  data_factory_id = azurerm_data_factory.example.id
  description     = "ADB Linked Service via MSI"
  adb_domain      = "https://${azurerm_databricks_workspace.example.workspace_url}"

  msi_work_space_resource_id = azurerm_databricks_workspace.example.id

  new_cluster_config {
    node_type             = "Standard_NC12"
    cluster_version       = "5.5.x-gpu-scala2.11"
    min_number_of_workers = 1
    max_number_of_workers = 5
    driver_node_type      = "Standard_NC12"
    log_destination       = "dbfs:/logs"

    custom_tags = {
      custom_tag1 = "sct_value_1"
      custom_tag2 = "sct_value_2"
    }

    spark_config = {
      config1 = "value1"
      config2 = "value2"
    }

    spark_environment_variables = {
      envVar1 = "value1"
      envVar2 = "value2"
    }

    init_scripts = ["init.sh", "init2.sh"]
  }
}

Example Usage with access token & existing cluster

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "East US"
}

#Link to an existing cluster via access token
resource "azurerm_data_factory" "example" {
  name                = "TestDtaFactory92783401247"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

#Create a databricks instance
resource "azurerm_databricks_workspace" "example" {
  name                = "databricks-test"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "standard"
}

resource "azurerm_data_factory_linked_service_azure_databricks" "at_linked" {
  name                = "ADBLinkedServiceViaAccessToken"
  data_factory_id     = azurerm_data_factory.example.id
  description         = "ADB Linked Service via Access Token"
  existing_cluster_id = "0308-201146-sly615"

  access_token = "SomeDatabricksAccessToken"
  adb_domain   = "https://${azurerm_databricks_workspace.example.workspace_url}"
}

Arguments Reference

The following arguments are supported:


You must specify exactly one of the following authentication blocks:


You must specify exactly one of the following modes for cluster integration:



A key_vault_password block supports the following:


A new_cluster_config block supports the following:


A instance_pool block supports the following:


Attributes Reference

In addition to the Arguments listed above - the following Attributes are exported:

Timeouts

The timeouts block allows you to specify timeouts for certain actions:

Import

Data Factory Linked Services can be imported using the resource id, e.g.

terraform import azurerm_data_factory_linked_service_azure_databricks.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example/providers/Microsoft.DataFactory/factories/example/linkedservices/example