Job - upbound/provider-gcp@v0.32.0

The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

fileUris

array

HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

loggingConfig

array

No description provided.

requiredobject

string

string

object

array

No description provided.

continueOnFailure

boolean

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.

properties

object

queryFileUri

string

queryList

array

The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

labels

object

pigConfig

array

No description provided.

continueOnFailure

boolean

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.

loggingConfig

array

No description provided.

requiredobject

object

string

array

The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

placement

array

No description provided.

clusterName

string

clusterNameRef

object

Reference to a Cluster in dataproc to populate clusterName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

clusterNameSelector

object

Selector for a Cluster in dataproc to populate clusterName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

resolution

string

resolve

string

prestoConfig

array

No description provided.

clientTags

array

Presto client tags to attach to this query.

continueOnFailure

boolean

loggingConfig

array

No description provided.

requiredobject

string

object

string

array

The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri

project

string

pysparkConfig

array

No description provided.

archiveUris

array

HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

args

array

The arguments to pass to the driver.

fileUris

array

HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

loggingConfig

array

No description provided.

requiredobject

requiredstring

object

array

HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

reference

array

No description provided.

jobId

string

region

string

regionRef

object

Reference to a Cluster in dataproc to populate region.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

regionSelector

object

Selector for a Cluster in dataproc to populate region.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

resolution

string

resolve

string

scheduling

array

No description provided.

maxFailuresPerHour

requirednumber

maxFailuresTotal

requirednumber

sparkConfig

array

No description provided.

archiveUris

array

HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

args

array

The arguments to pass to the driver.

fileUris

array

HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

loggingConfig

array

No description provided.

requiredobject

string

string

object

array

No description provided.

jarFileUris

array

HCFS URIs of jar files to be added to the Spark CLASSPATH.

loggingConfig

array

No description provided.

requiredobject

object

string

array

The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

managementPolicy

string

providerConfigRef

object

ProviderConfigReference specifies how the provider that will be used to create, observe, update, and delete this managed resource should be configured.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

providerRef

object

ProviderReference specifies the provider that will be used to create, observe, update, and delete this managed resource. Deprecated: Please use ProviderConfigReference, i.e. providerConfigRef

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

publishConnectionDetailsTo

object

PublishConnectionDetailsTo specifies the connection secret config which contains a name, metadata and a reference to secret store config to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource.

configRef

object

SecretStoreConfigRef specifies which secret store config should be used for this ConnectionSecret.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

metadata

object

Metadata is the metadata for connection secret.

object

object

string

requiredstring

writeConnectionSecretToRef

object

WriteConnectionSecretToReference specifies the namespace and name of a Secret to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource. This field is planned to be replaced in a future release in favor of PublishConnectionDetailsTo. Currently, both could be set independently and connection details would be published to both without affecting each other.

name

requiredstring

namespace

requiredstring

status

object

JobStatus defines the observed state of Job.

atProvider

object

No description provided.

driverControlsFilesUri

string

driverOutputResourceUri

string

forceDelete

boolean

hadoopConfig

array

No description provided.

archiveUris

array

HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

args

array

The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

fileUris

array

HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

loggingConfig

array

No description provided.

object

string

string

object

array

No description provided.

continueOnFailure

boolean

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.

properties

object

queryFileUri

string

queryList

array

The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

string

labels

object

pigConfig

array

No description provided.

continueOnFailure

boolean

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.

loggingConfig

array

No description provided.

object

object

string

array

The list of Hive queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

placement

array

No description provided.

clusterName

string

clusterUuid

string

prestoConfig

array

No description provided.

clientTags

array

Presto client tags to attach to this query.

continueOnFailure

boolean

loggingConfig

array

No description provided.

object

string

object

string

array

The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri

project

string

pysparkConfig

array

No description provided.

archiveUris

array

HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

args

array

The arguments to pass to the driver.

fileUris

array

HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

loggingConfig

array

No description provided.

object

string

object

array

HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

reference

array

No description provided.

jobId

string

region

string

scheduling

array

No description provided.

maxFailuresPerHour

number

maxFailuresTotal

number

sparkConfig

array

No description provided.

archiveUris

array

HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

args

array

The arguments to pass to the driver.

fileUris

array

HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.

jarFileUris

array

HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

loggingConfig

array

No description provided.

object

string

string

object

array

No description provided.

jarFileUris

array

HCFS URIs of jar files to be added to the Spark CLASSPATH.

loggingConfig

array

No description provided.

object

object

string

array

The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri

scriptVariables

object

status

array

No description provided.

string

string

string

string

array

Conditions of the resource.

requiredstring

string

requiredstring

requiredstring

requiredstring

spark

apiVersion: dataproc.gcp.upbound.io/v1beta1
kind: Job
metadata:
  annotations:
    meta.upbound.io/example-id: dataproc/v1beta1/job
  labels:
    testing.upbound.io/example-name: spark
  name: spark
spec:
  forProvider:
    forceDelete: true
    placement:
      - clusterNameSelector:
          matchLabels:
            testing.upbound.io/example-name: mycluster
    regionSelector:
      matchLabels:
        testing.upbound.io/example-name: mycluster
    sparkConfig:
      - args:
          - "1000"
        jarFileUris:
          - file:///usr/lib/spark/examples/jars/spark-examples.jar
        loggingConfig:
          - driverLogLevels:
              root: INFO
        mainClass: org.apache.spark.examples.SparkPi
        properties:
          spark.logConf: "true"

Marketplace

Discover the building blocks for your internal cloud platform.

SolutionsProviders Configurations

LearnDocumentation Schedule Demo

CompanyAbout Contact Us

MorePrivacy Policy Terms & Conditions

Marketplace

Discover the building blocksfor your internal cloud platform.