Skip to content

Commit

Permalink
support decimal arithmetic precision loss mode
Browse files Browse the repository at this point in the history
  • Loading branch information
jinchengchenghh committed Aug 28, 2024
1 parent afd0ce2 commit 182d32a
Show file tree
Hide file tree
Showing 13 changed files with 354 additions and 59 deletions.
21 changes: 21 additions & 0 deletions velox/docs/functions/spark/config.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
========================
SparkRegistration Configuration properties
========================

struct SparkRegistrationConfig
---------------------
.. list-table::
:widths: 20 10 10 70
:header-rows: 1

* - Property Name
- Type
- Default Value
- Description
* - allowPrecisionLoss
- bool
- true
- When true, establishing the result type of an arithmetic operation according to Hive behavior and SQL ANSI 2011 specification, i.e.
rounding the decimal part of the result if an exact representation is not
possible. Otherwise, NULL is returned when the actual result cannot be represented with the calculated decimal type. Now we support add,
subtract, multiply and divide operations.
27 changes: 25 additions & 2 deletions velox/docs/functions/spark/decimal.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,35 @@ Multiplication

Division
--------

If the function is registered with ``allowPrecisionLoss`` being true as default:
::

p = p1 - s1 + s2 + max(6, s1 + p2 + 1)
s = max(6, s1 + p2 + 1)

If the function is registered with ``allowPrecisionLoss`` being false:
::

intDig = min(38, p1 - s1 + s2);
decDig = min(38, max(6, s1 + p2 + 1));

If ``indDig + decDig`` is more than 38:
::

p = 38
s = decDig - (intDig + decDig - 38) / 2 - 1

Otherwise:
::

p = intDig + decDig
s = decDig

Decimal Precision and Scale Adjustment
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

For above arithmetic operators, when the precision of result exceeds 38,
caps p at 38 and reduces the scale, in order to prevent the truncation of
caps p at 38 and reduces the scale when allowing precision loss, in order to prevent the truncation of
the integer part of the decimals. Below formula illustrates how the result
precision and scale are adjusted.

Expand All @@ -43,6 +64,8 @@ precision and scale are adjusted.
precision = 38
scale = max(38 - (p - s), min(s, 6))

Caps p and s at 38 when not allowing precision loss.

Users experience runtime errors when the actual result cannot be represented
with the calculated decimal type.

Expand Down
1 change: 1 addition & 0 deletions velox/docs/spark_functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The semantics of Spark functions match Spark 3.5 with ANSI OFF.
functions/spark/bitwise
functions/spark/decimal
functions/spark/comparison
functions/spark/config
functions/spark/string
functions/spark/datetime
functions/spark/array
Expand Down
Loading

0 comments on commit 182d32a

Please sign in to comment.