Deletion Vectors in DELETE, MERGE and UPDATE commands

docs/commands/alter/.pages

@@ -1,4 +1,4 @@
-title: ALTER
+title: ALTER TABLE
 nav:
     - index.md
     - ...

docs/commands/alter/AlterTableDropFeature.md

@@ -0,0 +1,21 @@
+---
+title: DROP FEATURE
+---
+
+# AlterTableDropFeature
+
+`AlterTableDropFeature` is a `AlterTableCommand` ([Spark SQL]({{ book.spark_sql }}/logical-operators/AlterTableCommand)) unary logical operator that represents [ALTER TABLE DROP FEATURE](../../sql/index.md#ALTER-TABLE-DROP-FEATURE) SQL command in a logical query plan.
+
+`AlterTableDropFeature` supports a single feature removal (by the [feature name](#featureName)).
+
+## Creating Instance
+
+`AlterTableDropFeature` takes the following to be created:
+
+* <span id="table"> Table (`LogicalPlan`)
+* <span id="featureName"> Feature name
+* <span id="truncateHistory"> `truncateHistory` flag
+
+`AlterTableDropFeature` is created when:
+
+* `DeltaSqlAstBuilder` is requested to [parse ALTER TABLE DROP FEATURE SQL command](../../sql/DeltaSqlAstBuilder.md#visitAlterTableDropFeature)

docs/commands/alter/index.md

@@ -1,8 +1,8 @@
 ---
-title: ALTER
+title: ALTER TABLE
 ---
 
-# ALTER Commands
+# ALTER TABLE Commands
 
 Delta Lake supports altering delta tables using `ALTER TABLE` SQL commands with the following clauses:
 
@@ -11,6 +11,7 @@ Delta Lake supports altering delta tables using `ALTER TABLE` SQL commands with
 * [CHANGE COLUMN](AlterTableChangeColumnDeltaCommand.md)
 * [DROP COLUMNS](AlterTableDropColumnsDeltaCommand.md)
 * [DROP CONSTRAINT](AlterTableDropConstraintDeltaCommand.md)
+* [DROP FEATURE](AlterTableDropFeature.md)
 * [REPLACE COLUMNS](AlterTableReplaceColumnsDeltaCommand.md)
 * [SET TBLPROPERTIES](AlterTableSetPropertiesDeltaCommand.md)
 * [UNSET TBLPROPERTIES](AlterTableUnsetPropertiesDeltaCommand.md)

docs/commands/delete/DeleteCommand.md

@@ -116,7 +116,7 @@ shouldWritePersistentDeletionVectors(
 `shouldWritePersistentDeletionVectors` is enabled (`true`) when the following all hold:
 
 1. [spark.databricks.delta.delete.deletionVectors.persistent](../../configuration-properties/DeltaSQLConf.md#DELETE_USE_PERSISTENT_DELETION_VECTORS) configuration property is enabled (`true`)
-1. [Protocol and table configuration support deletion vector feature](../../deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable)
+1. [Protocol and table configuration support deletion vectors feature](../../deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable)
 
 ## <span id="apply"> Creating DeleteCommand
 

docs/commands/merge/MergeIntoCommandBase.md

@@ -512,3 +512,22 @@ isCdcEnabled(
 `isCdcEnabled` is used when:
 
 * `ClassicMergeExecutor` is requested to [findTouchedFiles](ClassicMergeExecutor.md#findTouchedFiles), [writeAllChanges](ClassicMergeExecutor.md#writeAllChanges)
+
+## shouldWritePersistentDeletionVectors { #shouldWritePersistentDeletionVectors }
+
+```scala
+shouldWritePersistentDeletionVectors(
+  spark: SparkSession,
+  txn: OptimisticTransaction): Boolean
+```
+
+`shouldWritePersistentDeletionVectors` is enabled (`true`) when the following all hold:
+
+1. [spark.databricks.delta.merge.deletionVectors.persistent](../../configuration-properties/index.md#merge.deletionVectors.persistent) configuration property is enabled (`true`)
+1. [Protocol and table configuration support deletion vectors feature](../../deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable)
+
+---
+
+`shouldWritePersistentDeletionVectors` is used when:
+
+* `MergeIntoCommand` is requested to [run a merge](MergeIntoCommand.md#runMerge)

docs/commands/update/UpdateCommand.md

@@ -4,6 +4,8 @@
 
 `UpdateCommand` is a `RunnableCommand` ([Spark SQL]({{ book.spark_sql }}/logical-operators/RunnableCommand/)) logical operator.
 
+`UpdateCommand` can use [Deletion Vectors](../../deletion-vectors/index.md) table feature to _soft-delete_ records when [executed](#run) (based on [shouldWritePersistentDeletionVectors](#shouldWritePersistentDeletionVectors)).
+
 ## Creating Instance
 
 `UpdateCommand` takes the following to be created:
@@ -52,6 +54,8 @@ performUpdate(
 
 `performUpdate`...FIXME
 
+With [persistent Deletion Vectors enabled](#shouldWritePersistentDeletionVectors), `performUpdate`...FIXME and [findTouchedFiles](../../deletion-vectors/DMLWithDeletionVectorsHelper.md#findTouchedFiles).
+
 ### rewriteFiles
 
 ```scala
@@ -74,3 +78,22 @@ buildUpdatedColumns(
 ```
 
 `buildUpdatedColumns`...FIXME
+
+## shouldWritePersistentDeletionVectors { #shouldWritePersistentDeletionVectors }
+
+```scala
+shouldWritePersistentDeletionVectors(
+  spark: SparkSession,
+  txn: OptimisticTransaction): Boolean
+```
+
+`shouldWritePersistentDeletionVectors` is enabled (`true`) when the following all hold:
+
+1. [spark.databricks.delta.update.deletionVectors.persistent](../../configuration-properties/index.md#update.deletionVectors.persistent) configuration property is enabled (`true`)
+1. [Protocol and table configuration support deletion vectors feature](../../deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable)
+
+---
+
+`shouldWritePersistentDeletionVectors` is used when:
+
+* `UpdateCommand` is [executed](#run) (and [performUpdate](UpdateCommand.md#performUpdate))

docs/configuration-properties/DeltaSQLConf.md

@@ -42,6 +42,10 @@
 
 [spark.databricks.delta.history.maxKeysPerList](index.md#history.maxKeysPerList)
 
+## merge.deletionVectors.persistent { #MERGE_USE_PERSISTENT_DELETION_VECTORS }
+
+[spark.databricks.delta.merge.deletionVectors.persistent](index.md#merge.deletionVectors.persistent)
+
 ## merge.materializeSource { #DELTA_COLLECT_STATS_USING_TABLE_SCHEMA }
 
 [spark.databricks.delta.merge.materializeSource](index.md#merge.materializeSource)
@@ -106,6 +110,10 @@
 
 [spark.databricks.delta.timeTravel.resolveOnIdentifier.enabled](index.md#timeTravel.resolveOnIdentifier.enabled)
 
+## update.deletionVectors.persistent { #UPDATE_USE_PERSISTENT_DELETION_VECTORS }
+
+[spark.databricks.delta.update.deletionVectors.persistent](index.md#update.deletionVectors.persistent)
+
 ## write.txnVersion.autoReset.enabled { #DELTA_IDEMPOTENT_DML_AUTO_RESET_ENABLED }
 
 [spark.databricks.delta.write.txnVersion.autoReset.enabled](index.md#write.txnVersion.autoReset.enabled)

docs/configuration-properties/index.md

@@ -177,7 +177,7 @@ If disabled, merge the two configurations with the same semantics as update and
 
 Default: `true`
 
-### <span id="delete.deletionVectors.persistent"><span id="DELETE_USE_PERSISTENT_DELETION_VECTORS"> delete.deletionVectors.persistent
+### <span id="DELETE_USE_PERSISTENT_DELETION_VECTORS"> delete.deletionVectors.persistent { #delete.deletionVectors.persistent }
 
 **spark.databricks.delta.delete.deletionVectors.persistent**
 
@@ -319,6 +319,14 @@ Used when:
 
 Default: `50`
 
+### <span id="MERGE_USE_PERSISTENT_DELETION_VECTORS"> merge.deletionVectors.persistent { #merge.deletionVectors.persistent }
+
+**spark.databricks.delta.merge.deletionVectors.persistent**
+
+**(internal)** Enables [persistent Deletion Vectors](../deletion-vectors/index.md) in [MERGE](../commands/merge/index.md) command
+
+Default: `true`
+
 ### <span id="MERGE_MATERIALIZE_SOURCE"> merge.materializeSource { #merge.materializeSource }
 
 **spark.databricks.delta.merge.materializeSource**
@@ -682,12 +690,26 @@ Used when:
 
 * `DeltaSourceBase` is requested for the [allowUnsafeStreamingReadOnColumnMappingSchemaChanges](../spark-connector/DeltaSourceBase.md#allowUnsafeStreamingReadOnColumnMappingSchemaChanges)
 
-### <span id="timeTravel.resolveOnIdentifier.enabled"><span id="RESOLVE_TIME_TRAVEL_ON_IDENTIFIER"> timeTravel.resolveOnIdentifier.enabled
+### <span id="RESOLVE_TIME_TRAVEL_ON_IDENTIFIER"> timeTravel.resolveOnIdentifier.enabled { #timeTravel.resolveOnIdentifier.enabled }
+
+**spark.databricks.delta.timeTravel.resolveOnIdentifier.enabled**
+
+**(internal)** Enables [time travel](../time-travel/index.md) patterns (as `@v123` and `@yyyyMMddHHmmssSSS`) in the path identifiers of delta tables
 
-**spark.databricks.delta.timeTravel.resolveOnIdentifier.enabled** (internal) controls whether to resolve patterns as `@v123` and `@yyyyMMddHHmmssSSS` in path identifiers as [time travel](../time-travel/index.md) nodes.
+Default: `true`
+
+### <span id="UPDATE_USE_PERSISTENT_DELETION_VECTORS"> update.deletionVectors.persistent { #update.deletionVectors.persistent }
+
+**spark.databricks.delta.update.deletionVectors.persistent**
+
+**(internal)** Enables [persistent Deletion Vectors](../deletion-vectors/index.md) in [UPDATE](../commands/update/index.md) command
 
 Default: `true`
 
+Used when:
+
+* [UpdateCommand](../commands/update/UpdateCommand.md) is executed (and [shouldWritePersistentDeletionVectors](../commands/update/UpdateCommand.md#shouldWritePersistentDeletionVectors))
+
 ### <span id="vacuum.parallelDelete.enabled"><span id="DELTA_VACUUM_PARALLEL_DELETE_ENABLED"> vacuum.parallelDelete.enabled
 
 **spark.databricks.delta.vacuum.parallelDelete.enabled** enables parallelizing the deletion of files during [vacuum](../commands/vacuum/index.md) command.

docs/deletion-vectors/DMLWithDeletionVectorsHelper.md

@@ -1,5 +1,7 @@
 # DMLWithDeletionVectorsHelper
 
+`DMLWithDeletionVectorsHelper` is a [DeltaCommand](../commands/DeltaCommand.md) with utilities for DML operations to work with [Deletion Vectors](index.md).
+
 ## createTargetDfForScanningForMatches { #createTargetDfForScanningForMatches }
 
 ```scala
@@ -57,7 +59,9 @@ findTouchedFiles(
   opName: String): Seq[TouchedFileWithDV]
 ```
 
-`findTouchedFiles`...FIXME
+`findTouchedFiles` requests the given [TahoeFileIndex](../TahoeFileIndex.md) (that is assumed a [TahoeBatchFileIndex](../TahoeBatchFileIndex.md)) for the [AddFiles](../TahoeBatchFileIndex.md#addFiles).
+
+In the end, `findTouchedFiles` [findFilesWithMatchingRows](#findFilesWithMatchingRows) with [candidate file map](../commands/DeltaCommand.md#generateCandidateFileMap) and [matched row index sets](DeletionVectorBitmapGenerator.md#buildRowIndexSetsForFilesMatchingCondition).
 
 ---
 

docs/deletion-vectors/DeletionVectorUtils.md

@@ -30,13 +30,13 @@ deletionVectorsReadable(
 ## deletionVectorsWritable { #deletionVectorsWritable }
 
 ```scala
+deletionVectorsWritable(
+  protocol: Protocol,
+  metadata: Metadata): Boolean
 deletionVectorsWritable(
   snapshot: SnapshotDescriptor,
   newProtocol: Option[Protocol] = None,
   newMetadata: Option[Metadata] = None): Boolean
-deletionVectorsWritable(
-  protocol: Protocol,
-  metadata: Metadata): Boolean
 ```
 
 `deletionVectorsWritable` is enabled (`true`) when the following all hold:
@@ -52,6 +52,7 @@ deletionVectorsWritable(
 * [DELETE](../commands/delete/index.md) command is executed (and requested to [shouldWritePersistentDeletionVectors](../commands/delete/DeleteCommand.md#shouldWritePersistentDeletionVectors))
 * [MERGE](../commands/merge/index.md) command is executed (and requested to [shouldWritePersistentDeletionVectors](../commands/merge/MergeIntoCommandBase.md#shouldWritePersistentDeletionVectors))
 * [UPDATE](../commands/update/index.md) command is executed (and requested to [shouldWritePersistentDeletionVectors](../commands/update/UpdateCommand.md#shouldWritePersistentDeletionVectors))
+* `CheckNoDeletionVector` is executed (for Iceberg-compatibility)
 
 ## isTableDVFree { #isTableDVFree }
 

docs/deletion-vectors/index.md

@@ -2,12 +2,19 @@
 
 **Deletion Vectors** is a [table feature](../table-features/index.md) to _soft-delete_ records (merely marking them as removed without rewriting the underlying parquet data files).
 
-Deletion Vectors is supported by conditional [DELETE](../commands/delete/index.md)s (when executed with a delete condition).
+Deletion Vectors is supported by the following commands (based on their corresponding "guard" configuration properties):
+
+Command | Configuration Property
+-|-
+ [DELETE](../commands/delete/index.md)<br>(when executed with a condition) | [spark.databricks.delta.delete.deletionVectors.persistent](../configuration-properties/index.md#delete.deletionVectors.persistent)
+ [MERGE](../commands/merge/index.md) | [spark.databricks.delta.merge.deletionVectors.persistent](../configuration-properties/index.md#merge.deletionVectors.persistent)
+ [UPDATE](../commands/update/index.md) | [spark.databricks.delta.update.deletionVectors.persistent](../configuration-properties/index.md#update.deletionVectors.persistent)
 
 Deletion Vectors can be enabled on a delta table using [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property.
 
 ```sql
-ALTER TABLE my_delta_table SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+ALTER TABLE my_delta_table
+SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
 ```
 
 Deletion Vectors is used on a delta table when all of the following hold:
@@ -26,7 +33,18 @@ Deletion Vectors is used on a delta table when all of the following hold:
 
 [spark.databricks.delta.delete.deletionVectors.persistent](../configuration-properties/index.md#delete.deletionVectors.persistent)
 
-## Iceberg-Compatibility Feature
+## UniForm Iceberg
+
+UniForm Iceberg (`IcebergCompatV2` and `IcebergCompatV1`) uses `CheckNoDeletionVector` check to assert that Deletion Vectors are disabled on a delta table.
+
+```text
+IcebergCompatV<version> requires Deletion Vectors to be disabled on the table.
+Please use the ALTER TABLE DROP FEATURE command to disable Deletion Vectors
+and to remove the existing Deletion Vectors from the table.
+```
+
+??? note "ALTER TABLE DROP FEATURE SQL command"
+    Use [ALTER TABLE DROP FEATURE](../sql/index.md#ALTER-TABLE-DROP-FEATURE) SQL command to drop a feature on a delta table.
 
 The Iceberg-compatibility feature ([REORG TABLE](../commands/reorg/index.md) with `ICEBERG_COMPAT_VERSION`) turns Deletion Vectors off.
 

docs/sql/DeltaSqlAstBuilder.md

@@ -8,6 +8,7 @@ SQL Statement | Logical Command
 --------------|----------
  <span id="visitAddTableConstraint"> [ALTER TABLE ADD CONSTRAINT](index.md#ALTER-TABLE-ADD-CONSTRAINT) | [AlterTableAddConstraint](../check-constraints/AlterTableAddConstraint.md)
  <span id="visitDropTableConstraint"> [ALTER TABLE DROP CONSTRAINT](index.md#ALTER-TABLE-DROP-CONSTRAINT) | [AlterTableDropConstraint](../check-constraints/AlterTableDropConstraint.md)
+ <span id="visitAlterTableDropFeature"> [ALTER TABLE DROP FEATURE](index.md#ALTER-TABLE-DROP-FEATURE) | [AlterTableDropFeature](../commands/alter/AlterTableDropFeature.md)
  [CONVERT TO DELTA](index.md#CONVERT-TO-DELTA) | [ConvertToDeltaCommand](../commands/convert/ConvertToDeltaCommand.md)
  <span id="visitDescribeDeltaDetail"> [DESCRIBE DETAIL](index.md#DESCRIBE-DETAIL) | [DescribeDeltaDetailCommand](../commands/describe-detail/DescribeDeltaDetailCommand.md)
  <span id="visitDescribeDeltaHistory"> [DESCRIBE HISTORY](index.md#describe-history) | [DescribeDeltaHistoryCommand](../commands/describe-history/DescribeDeltaHistoryCommand.md)

docs/sql/index.md

@@ -6,7 +6,7 @@ The SQL statements support table identifiers of the format `` delta.`path` `` (w
 
 The SQL statements can also refer to tables that are registered in a catalog (_metastore_).
 
-## <span id="ALTER-TABLE-ADD-CONSTRAINT"> ALTER TABLE ADD CONSTRAINT
+## ALTER TABLE ADD CONSTRAINT { #ALTER-TABLE-ADD-CONSTRAINT }
 
 ```text
 ALTER TABLE table
@@ -16,7 +16,7 @@ CHECK (expr+)
 
 Creates an [AlterTableAddConstraint](../check-constraints/AlterTableAddConstraint.md)
 
-## <span id="ALTER-TABLE-DROP-CONSTRAINT"> ALTER TABLE DROP CONSTRAINT
+## ALTER TABLE DROP CONSTRAINT { #ALTER-TABLE-DROP-CONSTRAINT }
 
 ```text
 ALTER TABLE table
@@ -25,6 +25,15 @@ DROP CONSTRAINT (IF EXISTS)? name
 
 Creates a [AlterTableDropConstraint](../check-constraints/AlterTableDropConstraint.md)
 
+## ALTER TABLE DROP FEATURE { #ALTER-TABLE-DROP-FEATURE }
+
+```text
+ALTER TABLE table
+DROP FEATURE featureName (TRUNCATE HISTORY)?
+```
+
+Creates a [AlterTableDropFeature](../commands/alter/AlterTableDropFeature.md) logical operator
+
 ## CLONE { #clone }
 
 ```antlr
@@ -53,7 +62,7 @@ temporalClause
 
 Creates a [CloneTableStatement](../commands/clone/CloneTableStatement.md)
 
-## <span id="CONVERT-TO-DELTA"> CONVERT TO DELTA
+## CONVERT TO DELTA { #CONVERT-TO-DELTA }
 
 ```text
 CONVERT TO DELTA table
@@ -63,7 +72,7 @@ CONVERT TO DELTA table
 
 Creates a [ConvertToDeltaCommand](../commands/convert/ConvertToDeltaCommand.md)
 
-## <span id="DESCRIBE-DETAIL"> DESCRIBE DETAIL
+## DESCRIBE DETAIL { #DESCRIBE-DETAIL }
 
 ```text
 (DESC | DESCRIBE) DETAIL (path | table)
@@ -80,15 +89,15 @@ Executes [DescribeDeltaDetailCommand](../commands/describe-detail/DescribeDeltaD
 
 Creates a [DescribeDeltaHistory](../commands/describe-history/DescribeDeltaHistory.md)
 
-## <span id="GENERATE"> GENERATE
+## GENERATE { #GENERATE }
 
 ```text
 GENERATE modeName FOR TABLE table
 ```
 
 Executes [DeltaGenerateCommand](../commands/generate/DeltaGenerateCommand.md)
 
-## <span id="OPTIMIZE"> OPTIMIZE
+## OPTIMIZE { #OPTIMIZE }
 
 ```text
 OPTIMIZE (path | table)
@@ -105,7 +114,7 @@ Executes [OptimizeTableCommand](../commands/optimize/OptimizeTableCommand.md) on
 
 Parsed by [DeltaSqlAstBuilder](DeltaSqlAstBuilder.md#visitOptimizeTable) that creates an [OptimizeTableCommand](../commands/optimize/OptimizeTableCommand.md)
 
-## <span id="RESTORE"> RESTORE
+## RESTORE { #RESTORE }
 
 ```text
 RESTORE TABLE? table
@@ -119,7 +128,7 @@ temporalClause
 
 Creates a [RestoreTableStatement](../commands/restore/RestoreTableStatement.md)
 
-## <span id="VACUUM"> VACUUM
+## VACUUM { #VACUUM }
 
 ```text
 VACUUM (path | table)

mkdocs.yml

@@ -267,7 +267,7 @@ nav:
   - Commands:
     - commands/index.md
     - commands/DeltaCommand.md
-    - ALTER:
+    - ALTER TABLE:
       - ... | flat | commands/alter/**.md
     - CLONE:
       - ... | flat | commands/clone/**.md