Crawler - crossplane-contrib/provider-aws@v0.33.0

A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.

configuration

string

crawlerSecurityConfiguration

string

crawlerSecurityConfigurationRef

object

CrawlerSecurityConfigurationRef is a reference to an SecurityConfiguration used to set the CrawlerSecurityConfiguration.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

crawlerSecurityConfigurationSelector

object

CrawlerSecurityConfigurationSelector selects references to SecurityConfiguration used to set the CrawlerSecurityConfiguration.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

object

DatabaseNameRef is a reference to an Database used to set the DatabaseName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

databaseNameSelector

object

DatabaseNamesSelector selects references to Database used to set the DatabaseName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

object

Specifies data lineage configuration settings for the crawler.

crawlerLineageSettings

string

recrawlPolicy

object

A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.

string

requiredstring

string

object

RoleRef is a reference to an IAMRole used to set the Role.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

roleSelector

object

RoleSelector selects references to IAMRole used to set the Role.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

object

The policy for the crawler's update and deletion behavior.

string

string

string

object

requiredobject

A list of collection of targets to crawl. Targets is a required field

catalogTargets

array

Specifies Glue Data Catalog targets.

databaseName

string

databaseNameRef

object

DatabaseNameRef is a reference to an Database used to set the DatabaseName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

databaseNameSelector

object

DatabaseNamesSelector selects references to Database used to set the DatabaseName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

resolution

string

resolve

string

tables

requiredarray

A list of the tables to be synchronized. Tables is a required field

dynamoDBTargets

array

Specifies Amazon DynamoDB targets.

string

boolean

number

array

Specifies JDBC targets.

connectionName

string

connectionNameRef

object

ConnectionNameRef is a reference to an Connection used to set the ConnectionName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

connectionNameSelector

object

ConnectionNamesSelector selects references to Connection used to set the ConnectionName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

resolution

string

resolve

string

exclusions

array

A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler (https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html).

path

string

mongoDBTargets

array

Specifies Amazon DocumentDB or MongoDB targets.

connectionName

string

connectionNameRef

object

ConnectionNameRef is a reference to an Connection used to set the ConnectionName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

connectionNameSelector

object

ConnectionNamesSelector selects references to Connection used to set the ConnectionName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

boolean

array

Specifies Amazon Simple Storage Service (Amazon S3) targets.

connectionName

string

connectionNameRef

object

ConnectionNameRef is a reference to an Connection used to set the ConnectionName.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

connectionNameSelector

object

ConnectionNamesSelector selects references to Connection used to set the ConnectionName.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

object

DlqEventQueueARNRef is a reference to an SQSEventQueue used to set the DlqEventQueueARN.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

dlqEventQueueArnSelector

object

DlqEventQueueARNSelector selects references to SQSEventQueue used to set the DlqEventQueueARN.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

string

string

string

object

EventQueueARNRef is a reference to an SQSEventQueue used to set the EventQueueARN.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

eventQueueArnSelector

object

EventQueueARNSelector selects references to SQSEventQueue used to set the EventQueueARN.

matchControllerRef

boolean

matchLabels

object

policy

object

Policies for selection.

resolution

string

resolve

string

exclusions

array

A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler (https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html).

path

string

sampleSize

integer

providerConfigRef

object

ProviderConfigReference specifies how the provider that will be used to create, observe, update, and delete this managed resource should be configured.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

providerRef

object

ProviderReference specifies the provider that will be used to create, observe, update, and delete this managed resource. Deprecated: Please use ProviderConfigReference, i.e. providerConfigRef

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

publishConnectionDetailsTo

object

PublishConnectionDetailsTo specifies the connection secret config which contains a name, metadata and a reference to secret store config to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource.

configRef

object

SecretStoreConfigRef specifies which secret store config should be used for this ConnectionSecret.

name

requiredstring

policy

object

Policies for referencing.

resolution

string

resolve

string

metadata

object

Metadata is the metadata for connection secret.

object

object

string

requiredstring

writeConnectionSecretToRef

object

WriteConnectionSecretToReference specifies the namespace and name of a Secret to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource. This field is planned to be replaced in a future release in favor of PublishConnectionDetailsTo. Currently, both could be set independently and connection details would be published to both without affecting each other.

name

requiredstring

namespace

requiredstring

status

object

CrawlerStatus defines the observed state of Crawler.

atProvider

object

CrawlerObservation defines the observed state of Crawler

crawlElapsedTime

integer

creationTime

string

lastCrawl

object

The status of the last crawl, and potentially error information if an error occurred.

string

string

string

string

string

string

string

string

string

integer

array

Conditions of the resource.

requiredstring

string

requiredstring

requiredstring

requiredstring

glue-crawler

apiVersion: glue.aws.crossplane.io/v1alpha1
kind: Crawler
metadata:
  name: glue-crawler
spec:
  forProvider:
    classifierRefs:
      - name: glue-classifier-csv
    crawlerSecurityConfigurationRef:
      name: glue-securityconfiguration
    databaseNameRef:
      name: glue-database
    region: us-east-1
    roleRef:
      name: glue-role
    schedule: CroN(0/5 * * * ? *)
    schemaChangePolicy:
      deleteBehavior: LOG
      updateBehavior: UPDATE_IN_DATABASE
    tags:
      glue: crawler
      spider: speedy
    targets:
      dynamoDBTargets:
        - path: dynamoDBpath
        - path: anotherPATH
          scanAll: false
          scanRate: 1.1
      jdbcTargets:
        - connectionNameRef:
            name: glue-connection-jdbc
          exclusions:
            - myFolder2/*
            - "*.{csv,avro}"
      mongoDBTargets:
        - connectionNameRef:
            name: glue-connection-mongodb
          path: dbName/collectionName
          scanAll: false
      s3Targets:
        - connectionNameRef:
            name: glue-connection-network
          dlqEventQueueArnRef:
            name: test-queue2
          eventQueueArnRef:
            name: test-queue
          exclusions:
            - myFolder2/*
            - "*.{csv,avro}"
          path: s3://bucket/prefix/object
          sampleSize: 123
  providerConfigRef:
    name: example

Marketplace

Discover the building blocks for your internal cloud platform.

Solutions

Learn

Company

More