Hi All,
We have several Texas Electronics 525 tipping bucket rain gauges that are producing some strange hourly values. Despite the multiplier in the PulseCount instruction being 0.01, we are occasionally getting values which are not limited to the hundredths place, for example 0.06999999, 0.09999999, and 0.6199999.
The gauges are connected to either a CR3000 or a CR6 and are using the PulseCount as suggested by ShortCut.
CR6 - PulseCount(Rain_in,1,U3,1,0,0.01,0)
CR3000 - PulseCount(Rain_in,1,2,2,0,0.01,0)
The onset of these anomalies seems to have been when the data type was switched from FP2 to IEEE4. IEEE4 is definitely overkill, but given the arithmetic it still doesn’t seem like it should result in values with a resolution beyond the hundredths place.
Totalize(1,Rain_in,IEEE4,0)
So, my question is how can an integer of tips multiplied by 0.01 possibly result in a number like 0.0699999, and should it be considered valid data? I'm reluctant to simply change the data type to FP2 without knowing if we would just be just papering over some underlying problem that we do want to know about and resolve, if possible.
The numbers are valid. IEEE4 cannot by nature store certain exact values. It will be a value extremely close. As the total is counting up, it will hit one of those inconvenient numbers and that tiny precision error carries over in the total. Rounding in post processing would show the exact total you expect.
FP2 is a binary coded decimal format instead of floating point, and will store exact values in this case. However, once the value reaches 80, FP2 would lose the hundreths decimal place.