Skip to content

Commit

Permalink
MergeOutputGeneration
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Jul 3, 2023
1 parent f4adafc commit ddcb1ef
Showing 1 changed file with 111 additions and 1 deletion.
112 changes: 111 additions & 1 deletion docs/commands/merge/MergeOutputGeneration.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,113 @@
# MergeOutputGeneration

`MergeOutputGeneration` is...FIXME
`MergeOutputGeneration` is an extension of the [MergeIntoCommandBase](MergeIntoCommandBase.md) abstraction with logic to transform the merge clauses into expressions that can be evaluated to obtain the output of the [merge operation](index.md).

## Implementations

* [ClassicMergeExecutor](ClassicMergeExecutor.md)
* [InsertOnlyMergeExecutor](InsertOnlyMergeExecutor.md)

## generatePrecomputedConditionsAndDF { #generatePrecomputedConditionsAndDF }

```scala
generatePrecomputedConditionsAndDF(
sourceDF: DataFrame,
clauses: Seq[DeltaMergeIntoClause]): (DataFrame, Seq[DeltaMergeIntoClause])
```

`generatePrecomputedConditionsAndDF`...FIXME

---

`generatePrecomputedConditionsAndDF` is used when:

* `ClassicMergeExecutor` is requested to [writeAllChanges](ClassicMergeExecutor.md#writeAllChanges)
* `InsertOnlyMergeExecutor` is requested to [generateInsertsOnlyOutputDF](InsertOnlyMergeExecutor.md#generateInsertsOnlyOutputDF)

## generateWriteAllChangesOutputCols { #generateWriteAllChangesOutputCols }

```scala
generateWriteAllChangesOutputCols(
targetOutputCols: Seq[Expression],
outputColNames: Seq[String],
noopCopyExprs: Seq[Expression],
clausesWithPrecompConditions: Seq[DeltaMergeIntoClause],
cdcEnabled: Boolean,
shouldCountDeletedRows: Boolean = true): IndexedSeq[Column]
```

`generateWriteAllChangesOutputCols`...FIXME

---

`generateWriteAllChangesOutputCols` is used when:

* `ClassicMergeExecutor` is requested to [writeAllChanges](ClassicMergeExecutor.md#writeAllChanges)

### generateAllActionExprs { #generateAllActionExprs }

```scala
generateAllActionExprs(
targetOutputCols: Seq[Expression],
clausesWithPrecompConditions: Seq[DeltaMergeIntoClause],
cdcEnabled: Boolean,
shouldCountDeletedRows: Boolean): Seq[ProcessedClause]
```

`generateAllActionExprs`...FIXME

### generateClauseOutputExprs { #generateClauseOutputExprs }

```scala
generateClauseOutputExprs(
numOutputCols: Integer,
clauses: Seq[ProcessedClause],
noopExprs: Seq[Expression]): Seq[Expression]
```

`generateClauseOutputExprs`...FIXME

## generateCdcAndOutputRows { #generateCdcAndOutputRows }

```scala
generateCdcAndOutputRows(
sourceDf: DataFrame,
outputCols: Seq[Column],
outputColNames: Seq[String],
noopCopyExprs: Seq[Expression],
deduplicateDeletes: DeduplicateCDFDeletes): DataFrame
```

`generateCdcAndOutputRows`...FIXME

---

`generateCdcAndOutputRows` is used when:

* `ClassicMergeExecutor` is requested to [writeAllChanges](ClassicMergeExecutor.md#writeAllChanges)

### packAndExplodeCDCOutput { #packAndExplodeCDCOutput }

```scala
packAndExplodeCDCOutput(
sourceDf: DataFrame,
cdcArray: Column,
cdcToMainDataArray: Column,
outputColNames: Seq[String],
dedupColumns: Seq[Column]): DataFrame
```

`packAndExplodeCDCOutput`...FIXME

### deduplicateCDFDeletes { #deduplicateCDFDeletes }

```scala
deduplicateCDFDeletes(
deduplicateDeletes: DeduplicateCDFDeletes,
df: DataFrame,
cdcArray: Column,
cdcToMainDataArray: Column,
outputColNames: Seq[String]): DataFrame
```

`deduplicateCDFDeletes`...FIXME

0 comments on commit ddcb1ef

Please sign in to comment.