airbyte.SourceS3

Coming soon!

Coming soon!

Coming soon!

Coming soon!

name string

The unique name of the resource.

args SourceS3Args

The arguments to resource properties.

opts CustomResourceOptions

Bag of options to control resource's behavior.

resource_name str

The unique name of the resource.

args SourceS3Args

The arguments to resource properties.

opts ResourceOptions

Bag of options to control resource's behavior.

ctx Context

Context object for the current deployment.

name string

The unique name of the resource.

args SourceS3Args

The arguments to resource properties.

opts ResourceOption

Bag of options to control resource's behavior.

name string

The unique name of the resource.

args SourceS3Args

The arguments to resource properties.

opts CustomResourceOptions

Bag of options to control resource's behavior.

name String

The unique name of the resource.

args SourceS3Args

The arguments to resource properties.

options CustomResourceOptions

Bag of options to control resource's behavior.

Configuration SourceS3Configuration

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

WorkspaceId string

DefinitionId string

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

Name string

Name of the source e.g. dev-mysql-instance.

SecretId string

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

Configuration SourceS3ConfigurationArgs

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

WorkspaceId string

DefinitionId string

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

Name string

Name of the source e.g. dev-mysql-instance.

SecretId string

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

configuration SourceS3Configuration

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

workspaceId String

definitionId String

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

name String

Name of the source e.g. dev-mysql-instance.

secretId String

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

configuration SourceS3Configuration

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

workspaceId string

definitionId string

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

name string

Name of the source e.g. dev-mysql-instance.

secretId string

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

configuration SourceS3ConfigurationArgs

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

workspace_id str

definition_id str

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

name str

Name of the source e.g. dev-mysql-instance.

secret_id str

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

configuration Property Map

NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy S3 v3 configs into v4 configs using the File-Based CDK.

workspaceId String

definitionId String

The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.

name String

Name of the source e.g. dev-mysql-instance.

secretId String

Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

CreatedAt double

Id string

The provider-assigned unique ID for this managed resource.

SourceId string

SourceType string

CreatedAt float64

Id string

The provider-assigned unique ID for this managed resource.

SourceId string

SourceType string

createdAt Double

id String

The provider-assigned unique ID for this managed resource.

sourceId String

sourceType String

createdAt number

id string

The provider-assigned unique ID for this managed resource.

sourceId string

sourceType string

created_at float

id str

The provider-assigned unique ID for this managed resource.

source_id str

source_type str

createdAt Number

id String

The provider-assigned unique ID for this managed resource.

sourceId String

sourceType String

name

The unique name of the resulting resource.

id

The unique provider ID of the resource to lookup.

state

Any extra arguments used during the lookup.

opts

A bag of options that control this resource's behavior.

resource_name

The unique name of the resulting resource.

id

The unique provider ID of the resource to lookup.

name

The unique name of the resulting resource.

id

The unique provider ID of the resource to lookup.

state

Any extra arguments used during the lookup.

opts

A bag of options that control this resource's behavior.

name

The unique name of the resulting resource.

id

The unique provider ID of the resource to lookup.

state

Any extra arguments used during the lookup.

opts

A bag of options that control this resource's behavior.

name

The unique name of the resulting resource.

id

The unique provider ID of the resource to lookup.

state

Any extra arguments used during the lookup.

opts

A bag of options that control this resource's behavior.

Bucket string

Name of the S3 bucket where the file(s) exist.

Streams List<SourceS3ConfigurationStream>

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

AwsAccessKeyId string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

AwsSecretAccessKey string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

DeliveryMethod SourceS3ConfigurationDeliveryMethod

Endpoint string

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

RegionName string

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

RoleArn string

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

StartDate string

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

Bucket string

Name of the S3 bucket where the file(s) exist.

Streams []SourceS3ConfigurationStream

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

AwsAccessKeyId string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

AwsSecretAccessKey string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

DeliveryMethod SourceS3ConfigurationDeliveryMethod

Endpoint string

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

RegionName string

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

RoleArn string

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

StartDate string

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

bucket String

Name of the S3 bucket where the file(s) exist.

streams List<SourceS3ConfigurationStream>

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

awsAccessKeyId String

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

awsSecretAccessKey String

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

deliveryMethod SourceS3ConfigurationDeliveryMethod

endpoint String

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

regionName String

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

roleArn String

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

startDate String

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

bucket string

Name of the S3 bucket where the file(s) exist.

streams SourceS3ConfigurationStream[]

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

awsAccessKeyId string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

awsSecretAccessKey string

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

deliveryMethod SourceS3ConfigurationDeliveryMethod

endpoint string

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

regionName string

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

roleArn string

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

startDate string

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

bucket str

Name of the S3 bucket where the file(s) exist.

streams Sequence[SourceS3ConfigurationStream]

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

aws_access_key_id str

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

aws_secret_access_key str

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

delivery_method SourceS3ConfigurationDeliveryMethod

endpoint str

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

region_name str

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

role_arn str

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

start_date str

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

bucket String

Name of the S3 bucket where the file(s) exist.

streams List<Property Map>

Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.

awsAccessKeyId String

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

awsSecretAccessKey String

In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.

deliveryMethod Property Map

endpoint String

Endpoint to an S3 compatible service. Leave empty to use AWS. Default: ""

regionName String

AWS region where the S3 bucket is located. If not provided, the region will be determined automatically.

roleArn String

Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. Set the External ID to the Airbyte workspace ID, which can be found in the URL of this page.

startDate String

UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

CopyRawFiles SourceS3ConfigurationDeliveryMethodCopyRawFiles

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

ReplicateRecords SourceS3ConfigurationDeliveryMethodReplicateRecords

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

CopyRawFiles SourceS3ConfigurationDeliveryMethodCopyRawFiles

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

ReplicateRecords SourceS3ConfigurationDeliveryMethodReplicateRecords

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

copyRawFiles SourceS3ConfigurationDeliveryMethodCopyRawFiles

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

replicateRecords SourceS3ConfigurationDeliveryMethodReplicateRecords

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

copyRawFiles SourceS3ConfigurationDeliveryMethodCopyRawFiles

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

replicateRecords SourceS3ConfigurationDeliveryMethodReplicateRecords

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

copy_raw_files SourceS3ConfigurationDeliveryMethodCopyRawFiles

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

replicate_records SourceS3ConfigurationDeliveryMethodReplicateRecords

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

copyRawFiles Property Map

Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files.

replicateRecords Property Map

Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination.

PreserveDirectoryStructure bool

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

PreserveDirectoryStructure bool

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

preserveDirectoryStructure Boolean

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

preserveDirectoryStructure boolean

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

preserve_directory_structure bool

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

preserveDirectoryStructure Boolean

If enabled, sends subdirectory folder structure along with source file names to the destination. Otherwise, files will be synced by their names only. This option is ignored when file-based replication is not enabled. Default: true

Format SourceS3ConfigurationStreamFormat

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

Name string

The name of the stream.

DaysToSyncIfHistoryIsFull double

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

Globs List<string>

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

InputSchema string

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

RecentNFilesToReadForSchemaDiscovery double

The number of resent files which will be used to discover the schema for this stream.

Schemaless bool

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

ValidationPolicy string

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

Format SourceS3ConfigurationStreamFormat

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

Name string

The name of the stream.

DaysToSyncIfHistoryIsFull float64

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

Globs []string

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

InputSchema string

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

RecentNFilesToReadForSchemaDiscovery float64

The number of resent files which will be used to discover the schema for this stream.

Schemaless bool

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

ValidationPolicy string

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

format SourceS3ConfigurationStreamFormat

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

name String

The name of the stream.

daysToSyncIfHistoryIsFull Double

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

globs List<String>

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

inputSchema String

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

recentNFilesToReadForSchemaDiscovery Double

The number of resent files which will be used to discover the schema for this stream.

schemaless Boolean

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

validationPolicy String

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

format SourceS3ConfigurationStreamFormat

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

name string

The name of the stream.

daysToSyncIfHistoryIsFull number

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

globs string[]

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

inputSchema string

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

recentNFilesToReadForSchemaDiscovery number

The number of resent files which will be used to discover the schema for this stream.

schemaless boolean

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

validationPolicy string

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

format SourceS3ConfigurationStreamFormat

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

name str

The name of the stream.

days_to_sync_if_history_is_full float

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

globs Sequence[str]

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

input_schema str

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

recent_n_files_to_read_for_schema_discovery float

The number of resent files which will be used to discover the schema for this stream.

schemaless bool

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

validation_policy str

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

format Property Map

The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.

name String

The name of the stream.

daysToSyncIfHistoryIsFull Number

When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3

globs List<String>

The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.

inputSchema String

The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.

recentNFilesToReadForSchemaDiscovery Number

The number of resent files which will be used to discover the schema for this stream.

schemaless Boolean

When enabled, syncs will not validate or structure records against the stream's schema. Default: false

validationPolicy String

The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

AvroFormat SourceS3ConfigurationStreamFormatAvroFormat

CsvFormat SourceS3ConfigurationStreamFormatCsvFormat

ExcelFormat SourceS3ConfigurationStreamFormatExcelFormat

JsonlFormat SourceS3ConfigurationStreamFormatJsonlFormat

ParquetFormat SourceS3ConfigurationStreamFormatParquetFormat

UnstructuredDocumentFormat SourceS3ConfigurationStreamFormatUnstructuredDocumentFormat

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

AvroFormat SourceS3ConfigurationStreamFormatAvroFormat

CsvFormat SourceS3ConfigurationStreamFormatCsvFormat

ExcelFormat SourceS3ConfigurationStreamFormatExcelFormat

JsonlFormat SourceS3ConfigurationStreamFormatJsonlFormat

ParquetFormat SourceS3ConfigurationStreamFormatParquetFormat

UnstructuredDocumentFormat SourceS3ConfigurationStreamFormatUnstructuredDocumentFormat

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

avroFormat SourceS3ConfigurationStreamFormatAvroFormat

csvFormat SourceS3ConfigurationStreamFormatCsvFormat

excelFormat SourceS3ConfigurationStreamFormatExcelFormat

jsonlFormat SourceS3ConfigurationStreamFormatJsonlFormat

parquetFormat SourceS3ConfigurationStreamFormatParquetFormat

unstructuredDocumentFormat SourceS3ConfigurationStreamFormatUnstructuredDocumentFormat

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

avroFormat SourceS3ConfigurationStreamFormatAvroFormat

csvFormat SourceS3ConfigurationStreamFormatCsvFormat

excelFormat SourceS3ConfigurationStreamFormatExcelFormat

jsonlFormat SourceS3ConfigurationStreamFormatJsonlFormat

parquetFormat SourceS3ConfigurationStreamFormatParquetFormat

unstructuredDocumentFormat SourceS3ConfigurationStreamFormatUnstructuredDocumentFormat

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

avro_format SourceS3ConfigurationStreamFormatAvroFormat

csv_format SourceS3ConfigurationStreamFormatCsvFormat

excel_format SourceS3ConfigurationStreamFormatExcelFormat

jsonl_format SourceS3ConfigurationStreamFormatJsonlFormat

parquet_format SourceS3ConfigurationStreamFormatParquetFormat

unstructured_document_format SourceS3ConfigurationStreamFormatUnstructuredDocumentFormat

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

avroFormat Property Map

csvFormat Property Map

excelFormat Property Map

jsonlFormat Property Map

parquetFormat Property Map

unstructuredDocumentFormat Property Map

Extract text from document formats (.pdf, .docx, .md, .pptx) and emit as one record per file.

DoubleAsString bool

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

DoubleAsString bool

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

doubleAsString Boolean

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

doubleAsString boolean

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

double_as_string bool

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

doubleAsString Boolean

Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

Delimiter string

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

DoubleQuote bool

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

Encoding string

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

EscapeChar string

The character used for escaping special characters. To disallow escaping, leave this field blank.

FalseValues List<string>

A set of case-sensitive strings that should be interpreted as false values.

HeaderDefinition SourceS3ConfigurationStreamFormatCsvFormatHeaderDefinition

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

IgnoreErrorsOnFieldsMismatch bool

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

NullValues List<string>

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

QuoteChar string

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

SkipRowsAfterHeader double

The number of rows to skip after the header row. Default: 0

SkipRowsBeforeHeader double

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

StringsCanBeNull bool

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

TrueValues List<string>

A set of case-sensitive strings that should be interpreted as true values.

Delimiter string

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

DoubleQuote bool

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

Encoding string

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

EscapeChar string

The character used for escaping special characters. To disallow escaping, leave this field blank.

FalseValues []string

A set of case-sensitive strings that should be interpreted as false values.

HeaderDefinition SourceS3ConfigurationStreamFormatCsvFormatHeaderDefinition

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

IgnoreErrorsOnFieldsMismatch bool

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

NullValues []string

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

QuoteChar string

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

SkipRowsAfterHeader float64

The number of rows to skip after the header row. Default: 0

SkipRowsBeforeHeader float64

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

StringsCanBeNull bool

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

TrueValues []string

A set of case-sensitive strings that should be interpreted as true values.

delimiter String

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

doubleQuote Boolean

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

encoding String

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

escapeChar String

The character used for escaping special characters. To disallow escaping, leave this field blank.

falseValues List<String>

A set of case-sensitive strings that should be interpreted as false values.

headerDefinition SourceS3ConfigurationStreamFormatCsvFormatHeaderDefinition

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

ignoreErrorsOnFieldsMismatch Boolean

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

nullValues List<String>

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

quoteChar String

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

skipRowsAfterHeader Double

The number of rows to skip after the header row. Default: 0

skipRowsBeforeHeader Double

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

stringsCanBeNull Boolean

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

trueValues List<String>

A set of case-sensitive strings that should be interpreted as true values.

delimiter string

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

doubleQuote boolean

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

encoding string

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

escapeChar string

The character used for escaping special characters. To disallow escaping, leave this field blank.

falseValues string[]

A set of case-sensitive strings that should be interpreted as false values.

headerDefinition SourceS3ConfigurationStreamFormatCsvFormatHeaderDefinition

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

ignoreErrorsOnFieldsMismatch boolean

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

nullValues string[]

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

quoteChar string

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

skipRowsAfterHeader number

The number of rows to skip after the header row. Default: 0

skipRowsBeforeHeader number

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

stringsCanBeNull boolean

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

trueValues string[]

A set of case-sensitive strings that should be interpreted as true values.

delimiter str

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

double_quote bool

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

encoding str

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

escape_char str

The character used for escaping special characters. To disallow escaping, leave this field blank.

false_values Sequence[str]

A set of case-sensitive strings that should be interpreted as false values.

header_definition SourceS3ConfigurationStreamFormatCsvFormatHeaderDefinition

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

ignore_errors_on_fields_mismatch bool

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

null_values Sequence[str]

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

quote_char str

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

skip_rows_after_header float

The number of rows to skip after the header row. Default: 0

skip_rows_before_header float

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

strings_can_be_null bool

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

true_values Sequence[str]

A set of case-sensitive strings that should be interpreted as true values.

delimiter String

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","

doubleQuote Boolean

Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true

encoding String

The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"

escapeChar String

The character used for escaping special characters. To disallow escaping, leave this field blank.

falseValues List<String>

A set of case-sensitive strings that should be interpreted as false values.

headerDefinition Property Map

How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.

ignoreErrorsOnFieldsMismatch Boolean

Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false

nullValues List<String>

A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.

quoteChar String

The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """

skipRowsAfterHeader Number

The number of rows to skip after the header row. Default: 0

skipRowsBeforeHeader Number

The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0

stringsCanBeNull Boolean

Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true

trueValues List<String>

A set of case-sensitive strings that should be interpreted as true values.