std_residual_ subtraction computation creates NaN when residuals are 0

[danielfong-act]

Are you on the latest chainladder version?

• Yes, this bug occurs on the latest version.

Describe the bug in words

The standardized residual triangle computed here is missing the last value of origin year 1981. I think this is a floating point error caused by residual computation subtracting two numbers that are very close. This can happen at either end of the latest valuation of the triangle. Edit: This doesn't always happen when subtracting two very close numbers, and I'm not sure what causes the subtraction issue exactly.

I noticed similar behavior in BarnettZehnwirth's computatino of standardized residuals, but didn't note down the steps to replicate it. Replacing regular subtraction operations with np.subtract seems to rectify this problem.

Subtraction is a pretty common operation and I'm not sure if/how commonly this creates a problem elsewhere. I suspect it does, so I'll amend this issue if someone finds something else.

How can the bug be reproduced?

From the development section of the user guide

import chainladder as cl
raa = cl.load_sample('raa')
model = cl.Development().fit(raa)
model.std_residuals_

setting a breakpoint within development.py where resid is computed makes the problem more obvious

What is the expected behavior?

That NaN should be 0.0 instead

[henrydingliu]

this is not a bug, but an intentional and necessary step to enable optimization for the sparse backend. just apply fillna to the raw residuals.

[danielfong-act]

Huh, neat. Still dunno why it doesn't happen every time a zero is computed. No matter; It should be easy to fix the residuals in those two places.

[henrydingliu]

@jbogaardt I can't tell if this is a bug with fillna

raa = cl.load_sample('raa')
print(raa-raa)
print((raa-raa).fillna(0))
print((raa-raa).fillna(1))

the fillna(0) is doing absolutely nothing

[danielfong-act]

It looks like an unintended side effects, at least. fillna, and triangle addition/subtraction in general, treats 0s and NaNs the same.

nantriangle=raa-raa
nantriangle.iloc[0,0,0,0]=0
nantriangle.iloc[0,0,0,1]=1
print(nantriangle.fillna(0))
print(nantriangle.fillna(2))
print(nantriangle+1)
print(nantriangle+0)

I get why this behavior is necessary for big triangles that require a sparse backend, but it wasn't intuitive to me.

Also, turns out the last origin and development residuals in PTF Residuals are small but nonzero, so that's why it's not happening there.

[henrydingliu]

@jbogaardt-wcf just saw your new handle

[jbogaardt-wcf]

@jbogaardt-wcf just saw your new handle

Ah, this is my work GH account. Sent from my work machine.