-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Fix regression for negative-scale decimal128 in log #19315
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix regression for negative-scale decimal128 in log #19315
Conversation
datafusion/functions/src/math/log.rs
Outdated
| let value_f64 = value as f64; | ||
|
|
||
| // Compute log_base(value) - use natural log and convert | ||
| // log_base(x) = ln(x) / ln(base) | ||
| let log_value = value_f64.ln() / base.ln(); | ||
|
|
||
| // Add the adjustment: (-scale) * log_base(10) | ||
| // log_base(10) = ln(10) / ln(base) | ||
| let log_10_base = 10.0_f64.ln() / base.ln(); | ||
| let adjustment = (-scale as f64) * log_10_base; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do wonder if we're better off just casting to float and doing log there? There seem to be multiple log operations here, does it provide more accuracy?
Co-authored-by: Martin Grigorov <[email protected]>
|
Thanks @shifluxxc & @martin-g |
|
Thanks for the fix @shifluxxc , do you mind cherry-picking the PR to branch-52, then we could include it in v52.0.0 |
Which issue does this PR close?
Rationale for this change
Previously, the
logfunction would fail when operating on decimal values with negative scales.Negative scales in decimals represent values where the scale indicates padding zeros to the right (e.g.,
Decimal128(38, -2)with value100represents10000). This PR restores support for negative-scale decimals in thelogfunction by implementing the logarithmic property:log_base(value * 10^(-scale)) = log_base(value) + (-scale) * log_base(10).What changes are included in this PR?
Enhanced
log_decimal128function:log_base(value) + (-scale) * log_base(10)instead of trying to convert to unscaled valueScalarValue(which doesn't support negative scales yet)Added comprehensive tests:
log.rsfor negative-scale decimals with various bases (2, 3, 10)decimal.sltusing scientific notation (e.g.,1e4,8e1) to create decimals with negative scalesAre these changes tested?
Yes, this PR includes comprehensive tests:
Unit tests:
test_log_decimal128_negative_scale: Tests array inputs with negative scalestest_log_decimal128_negative_scale_base2: Tests with base 2 and negative scalestest_log_decimal128_negative_scale_scalar: Tests scalar inputs with negative scalesSQL logic tests:
log(1e4))log(10, 1e4))log(2.0, 8e1),log(2.0, 16e1))log(5e3))All tests pass successfully.
Are there any user-facing changes?
Yes, this is a user-facing change:
Before: The
logfunction would fail with an error when operating on decimal values with negative scales:After: The
logfunction now correctly handles decimal values with negative scales: