Job is the Schema for the Jobs API. Manages a job resource within a Dataproc cluster.
Type
CRD
Group
dataproc.gcp.upbound.io
Version
v1beta1
apiVersion: dataproc.gcp.upbound.io/v1beta1
kind: Job
JobSpec defines the desired state of Job
No description provided.
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
No description provided.
No description provided.
HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
No description provided.
The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
Reference to a Cluster in dataproc to populate clusterName.
Policies for referencing.
Selector for a Cluster in dataproc to populate clusterName.
Policies for selection.
No description provided.
Presto client tags to attach to this query.
No description provided.
The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver.
HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
No description provided.
HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
Reference to a Cluster in dataproc to populate region.
Policies for referencing.
Selector for a Cluster in dataproc to populate region.
Policies for selection.
No description provided.
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver.
HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
No description provided.
No description provided.
HCFS URIs of jar files to be added to the Spark CLASSPATH.
No description provided.
The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ProviderConfigReference specifies how the provider that will be used to create, observe, update, and delete this managed resource should be configured.
Policies for referencing.
ProviderReference specifies the provider that will be used to create, observe, update, and delete this managed resource. Deprecated: Please use ProviderConfigReference, i.e. providerConfigRef
Policies for referencing.
PublishConnectionDetailsTo specifies the connection secret config which contains a name, metadata and a reference to secret store config to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource.
WriteConnectionSecretToReference specifies the namespace and name of a Secret to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource. This field is planned to be replaced in a future release in favor of PublishConnectionDetailsTo. Currently, both could be set independently and connection details would be published to both without affecting each other.
JobStatus defines the observed state of Job.
No description provided.
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
No description provided.
No description provided.
HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
No description provided.
The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
No description provided.
Presto client tags to attach to this query.
No description provided.
The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver.
HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
No description provided.
HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
No description provided.
No description provided.
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
The arguments to pass to the driver.
HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
No description provided.
No description provided.
HCFS URIs of jar files to be added to the Spark CLASSPATH.
No description provided.
The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
No description provided.
Conditions of the resource.
spark
apiVersion: dataproc.gcp.upbound.io/v1beta1
kind: Job
metadata:
annotations:
meta.upbound.io/example-id: dataproc/v1beta1/job
labels:
testing.upbound.io/example-name: spark
name: spark
spec:
forProvider:
forceDelete: true
placement:
- clusterNameSelector:
matchLabels:
testing.upbound.io/example-name: mycluster
regionSelector:
matchLabels:
testing.upbound.io/example-name: mycluster
sparkConfig:
- args:
- "1000"
jarFileUris:
- file:///usr/lib/spark/examples/jars/spark-examples.jar
loggingConfig:
- driverLogLevels:
root: INFO
mainClass: org.apache.spark.examples.SparkPi
properties:
spark.logConf: "true"
© 2022 Upbound, Inc.
Discover the building blocksfor your internal cloud platform.