aws-cdk-lib.aws_stepfunctions_tasks.SparkSubmitJobDriver

interface SparkSubmitJobDriver

LanguageType name
.NETAmazon.CDK.AWS.StepFunctions.Tasks.SparkSubmitJobDriver
Gogithub.com/aws/aws-cdk-go/awscdk/v2/awsstepfunctionstasks#SparkSubmitJobDriver
Javasoftware.amazon.awscdk.services.stepfunctions.tasks.SparkSubmitJobDriver
Pythonaws_cdk.aws_stepfunctions_tasks.SparkSubmitJobDriver
TypeScript (source)aws-cdk-lib » aws_stepfunctions_tasks » SparkSubmitJobDriver

The information about job driver for Spark submit.

Example

new tasks.EmrContainersStartJobRun(this, 'EMR Containers Start Job Run', {
  virtualCluster: tasks.VirtualClusterInput.fromVirtualClusterId('de92jdei2910fwedz'),
  releaseLabel: tasks.ReleaseLabel.EMR_6_2_0,
  jobName: 'EMR-Containers-Job',
  jobDriver: {
    sparkSubmitJobDriver: {
      entryPoint: sfn.TaskInput.fromText('local:///usr/lib/spark/examples/src/main/python/pi.py'),
    },
  },
  applicationConfig: [{
    classification: tasks.Classification.SPARK_DEFAULTS,
    properties: {
      'spark.executor.instances': '1',
      'spark.executor.memory': '512M',
    },
  }],
});

Properties

NameTypeDescription
entryPointTaskInputThe entry point of job application.
entryPointArguments?TaskInputThe arguments for a job application in a task input object containing an array of strings.
sparkSubmitParameters?stringThe Spark submit parameters that are used for job runs.

entryPoint

Type: TaskInput

The entry point of job application.

Length Constraints: Minimum length of 1. Maximum length of 256.


entryPointArguments?

Type: TaskInput (optional, default: No arguments defined)

The arguments for a job application in a task input object containing an array of strings.

Length Constraints: Minimum length of 1. Maximum length of 10280.


sparkSubmitParameters?

Type: string (optional, default: No spark submit parameters)

The Spark submit parameters that are used for job runs.

Length Constraints: Minimum length of 1. Maximum length of 102400.