Hi,
I am looking for someone who has some patience to think along a bit with a problem that I am experiencing with a crashing CR1000x logger. Campbell support tried for a while but told me my right for support had run out after a while.. basically I'd like to know if others experience similar crashes.
I am running two CR1000x (latest OS version) loggers in parallel with two Eddy covariance systems at 10Hz with a Gill R3 sonic and a LI-7500(RS) IRGA, logging via RS232 (sonic)and SDM as well as analog input (IRGA). I collect these data, store them with tablefile() on a third-party, but high-quality (SDC) sd card and send every 30 minutes via FTP as well as retrieve the primary data table via pakbus/loggernet protocols. In addition I a number of status checks and switches and I am calculating raw covariances with a running mean, which I retrieve via a slow data table. The system was solar/battery powered and I switched to mains (battery buffered) recently. I use a third-party modem, which communicates well.
Although this all works pretty well usually, I do run into trouble a bit too often: incomplete FTP transfers, SDM channels seeming mixed up... I can accept that this may be related to a fairly overloaded programme, however the stated processing times in the status table are less than the execution intervals. I need to frequently restart the progrgamme (remotely) to solve these, which is just about acceptable.
However what is really worrying, as I am experiencing now as well as last summer, is crashing dataloggers, where the loggers (both) stop collecting data while I can still connect to them: data tables are frozen, there seems to be corrupted files on the card, but clock is still running and battery voltage updated (for example). When I try to (remotely) reload the programme, it either does not compile or loses contact. I need to go there (1.5 h drive), disconnect and reconnect, and sometimes even re-load the OS to get thnigs running again.
I am a bit out of options. Trying to run a simpler programme is still an option but it would be hard to judge if this is the solution as the crashes happen so infrequently. It seems the programme ' hangs' somewhere but I can't imagine nor trace where. Looking back, it SEEMS to me that a these crashes happened especially while battery voltage was at the high end (12.5-14 V, either high solar power or (now) while connected to mains adapter). For the rest, I see no apparent anomalies at the moment of crashing. Last time (yesterday) the crashes occcurred on both loggers, now both conneccted to mains each via a battery charger and battery) almost (exactly?) at the same time (they are 100 m apart in two pastures). Would that provide a clue? No lightning in sight, no extreme weather... power surge unlikely(?)
... so far, sorry for the longish account. I can give more info and/or share the programme (which is fairly messy, admittedly) with whoever would feel they can think along with me.
Thanks!
Bart Kruijt
We had some confusing crashes or lock-ups on the first CR1000X systems I was working on, and I never figured out what the problem might have been. I'm not really connected to those systems these days, but I think they've managed to keep them going, perhaps by occasional power cycles and reloads. Maybe there was some sort of buffer overflow or resource exhaustion. The loggers were line-powered with a connected battery, used various serial and other connections to instruments at different rates, several 'slow sequence' sections, connection via IP to a LoggerNet instance for pulling downloads, etc.. No real help to offer, just commiserating I guess. Good luck!
There are known lockups with some third party SD cards. Try running the system with a model of SD card thoroughly tested by Campbell Scientific.
Only these exact models of micro SD cards have passed stability tests across the full temperature range in testing at Campbell Scientific. Anything else risks lockups, or data loss.
2GB
ATP AF2GUDI-5ACXX
Swissbit SFSD2048N1BW1MT-I-ME-111-STD
Centon S4-IS-MSDC6-2G-002
Delkin S202MFBSS-C1047-B
8GB ATP AF8GUDI-WACXM
16GB Delkin S316MMZDJ-U1000-4
Hi, Thanks for that advice. I have to say, the crashes are not frequent - albeit annoying - so wouldn't a poor SD card lead to much more frequent crashes? Would it explain the (more frequent) incomplete or currupt files?
I am using Swissbit 8 Gb SLC cards, as far as I can see of similarly high specs: https://swissbit.com/data/S-455u/S-455u_fact_sheet.pdf
the factsheet for the Swissbit card you recommend is here: https://nl.mouser.com/datasheet/2/615/S-300u_fact_sheet-1633263.pdf
Except for the capacity (8 vs 2 Gb), I find it hard to tell the difference in quality - just for my understanding, what would be the essential difference that makes the cr1000x so sensitive to it? Just the capacity ?
Also: would this mean that a CR1000x without a card would not/never crash?
and @kirving, were you using one of these tested SD cards?
thanks again.
Apart from the issue of the appropriate SD card, I wonder, would the CR1000x be sensitive to medium-high frequency pulses on the 12v power input? I am using a solar charge controller which switches to Pulse Width Modulation to reduce charge power when the battery is fully charged. I am not sure, neet to check in the field, but possibly this PWM signal would also be present in the load output (power to datalogger). As far as I can see this PWM signal could have fequencies of a few hundred Hz.... Would this interfere with the datalogger's functioing, possibly leading to the lockups I am experiencing? Would it help to stabilise the power before the logger?
thanks again..
Bart
A datalogger recording high speed data does a lot of small writes to a card. It is very different than typically usage on a PC, which typically is big writes infrequently. Running at 10Hz, the datalogger does not have a lot of free time. A SD card taking a couple extra milliseconds to write can mess up everything.
Campbell Scientific even had to work directly with Delkin to get the 16GB card model produced with our tight timing needs.
If a solar charge regulator is at fault, you will generally see problems coincide with the sun.
thanks. This makes sense. Though I do hear several people around me say that they never had problems writing to sd cards, even with cheap ones..
I wonder - would it be safer to write files with TableFile tot a file on internal memory (USR:), then close the file after each averaging interval (eg 30 minutes) and copy (rename) to CRD: in one go, and possibly FTP from CRD at a later stage during execution? It would be an additional copy action, but not many small writes to the SD card.