Skip to content

Commit

Permalink
DeltaCDFRelation
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Jul 2, 2024
1 parent 404cc16 commit fddc6c3
Showing 1 changed file with 13 additions and 1 deletion.
14 changes: 13 additions & 1 deletion docs/change-data-feed/DeltaCDFRelation.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

`DeltaCDFRelation` is created when:

* `CDCReaderImpl` is requested for a [CDF-aware BaseRelation](CDCReaderImpl.md#getCDCRelation)
* `CDCReaderImpl` is requested for a [CDF-aware BaseRelation](CDCReaderImpl.md#getCDCRelation) and [emptyCDFRelation](CDCReaderImpl.md#emptyCDFRelation)

## Building Distributed Scan { #buildScan }

Expand All @@ -32,3 +32,15 @@
`buildScan` does column pruning with the `requiredColumns` defined (using `Dataset.select` operator).

In the end, `buildScan` converts the `DataFrame` to `RDD[Row]` (using `DataFrame.rdd` operator).

## Schema

??? note "BaseRelation"

```scala
schema: StructType
```

`schema` is part of the `BaseRelation` ([Spark SQL]({{ book.spark_sql }}/BaseRelation/#schema)) abstraction.

`schema` [cdcReadSchema](CDCReaderImpl.md#cdcReadSchema) for the [schema](../Metadata.md#schema) of the delta table (based on the [Metadata](../Snapshot.md#metadata) of the [snapshotForBatchSchema](#snapshotForBatchSchema)).

0 comments on commit fddc6c3

Please sign in to comment.