Hello. I am attempting to read 3 serial weather data sources. A table is defined for each source to include the record and the record length. Each source automatically sends a record approximately every 5 seconds with each burst of data seemingly delayed by the amount of time required to transmit the previous burst. Data is 9600, N, 8, 1. I cannot control the data source in any way. There are numerous seemingly random bytes above ASCII 127, mainly near the beginning of each record. Near the start of each burst is an SOH (ASCII 1) and near the end is an ETX (ASCII 4). There are occasional random ETX in the first few dozen characters but the final ETX is always preceded by an ASCII 177 (B1 hex). Therefore I open the COM port with the binary option. The first source sends different parameters with every burst with record lengths varying from approximately 270 to 550 characters. The other 2 sources both send records that are approximately 65 characters long. The SerialInRecord call uses the &H01 as the start character, no length and &HB104 as the end characters. Obviously the data burst from each source is not necessarily complete when a scan reads the COM port. With the first (longest record), I sometimes get the entire record but sometimes I get a NAN although the recorded record length seems reasonable. The full records may be a fluke due to the current timimg. From the other 2 shorter record length sources, I most often get a NAN with a 0 record length (as expected) but where there should be a record, there is a zero length text but with an record length approprite for the expected record. The program compiles automatically in pipeline mode. From other forum posts, I have noted buffer size issues but adjustments have not helped (tried 1500 in the SerialOpen). As coded below, MaxProcTime = 15872, BuffDepth = 0 and MaxBuffDepth = 0. My code and snippets from logged data are below. Looking ahead, if this can be resolved, would parsing and handling each received record be best placed in a slowsequence in an attempt to avoid missed scans? Thank you in advance!
Public DataStream(3) As String * 600
Public NbytesReturned(3)
DataTable(FirstStream,True,360)
DataInterval(0,1,Sec,60)
Sample(1,DataStream(1),String)
Sample(1,NbytesReturned(1),FP2)
EndTable
DataTable(SecondStream,True,360)
DataInterval(0,1,Sec,60)
Sample(1,DataStream(2),String)
Sample(1,NbytesReturned(2),FP2)
EndTable
DataTable(ThirdStream,True,360)
DataInterval(0,1,Sec,60)
Sample (1,DataStream(3),String)
Sample(1,NbytesReturned(3),FP2)
EndTable
BeginProg
SerialOpen (COM1,9600,3,0,600) ' 9600,N,8,1,None the '3' means binary and handles ascii codes > 127
SerialOpen (COM2,9600,3,0,600) ' 9600,N,8,1,None the '3' means binary and handles ascii codes > 127
SerialOpen (COM3,9600,3,0,600) ' 9600,N,8,1,None the '3' means binary and handles ascii codes > 127
Scan(1,Sec,3,0)
SerialInRecord (Com1,DataStream(1),&H01,0,&HB104,NbytesReturned(1),01)
SerialInRecord (Com2,DataStream(2),&H01,0,&HB104,NbytesReturned(2),01)
SerialInRecord (Com3,DataStream(3),&H01,0,&HB104,NbytesReturned(3),01)
CallTable FirstStream
CallTable SecondStream
CallTable ThirdStream
NextScan
EndProg
Main site data snippet. The number at the end is numBytesReturned from SerialInRecord. Note the 2 null strings yet the reasonable line length numbers.
"2017-09-20 11:30:42",37,"dataMAINsiteüxëJhumidity 16 (%rh) extinction_coeff 0.186411 (1/km) barometric_pressure1 930.3 (hPa) barometric_pressure2 930.5 (hPa) barometric_pressure3 930.0 (hPa) wind_speed 11.6 (kts) wind_direction 231 (deg) ambient_light 5128.811 (cd/m^2) air_temperature 26.1 (degC) ",273
"2017-09-20 11:30:43",38,"NAN",0
"2017-09-20 11:30:44",39,"NAN",0
"2017-09-20 11:30:45",40,"NAN",0
"2017-09-20 11:30:46",41,"NAN",0
"2017-09-20 11:30:47",42,"",548
"2017-09-20 11:30:48",43,"NAN",0
"2017-09-20 11:30:49",44,"NAN",0
"2017-09-20 11:30:50",45,"NAN",0
"2017-09-20 11:30:51",46,"NAN",0
"2017-09-20 11:30:52",47,"",273
"2017-09-20 11:30:53",48,"NAN",0
"2017-09-20 11:30:54",49,"NAN",0
"2017-09-20 11:30:55",50,"NAN",0
"2017-09-20 11:30:56",51,"NAN",0
"2017-09-20 11:30:57",52,"¡dataMAINsiteN!ücwind_speed 11.0 (kts) wind_direction 229 (deg) barometric_pressure1 930.3 (hPa) barometric_pressure2 930.4 (hPa) barometric_pressure3 930.0 (hPa) cloud_height1 00000 (ft) cloud_depth1 00000 (ft) cloud_height2 00000 (ft) cloud_depth2 00000 (ft) cloud_height3 00000 (ft) cloud_depth3 00000 (ft) vertical_visibility 00000 (ft) ceilometer_range 24610 (ft) ambient_light 5128.811 (cd/m^2) extinction_coeff 0.186411 (1/km) ",434
Second and third site data snippet. Again, where data is expected, there is a null string but a reasonable line length. The one line with data is only a partial line with the front 20 or so characters missing.
"2017-09-20 11:30:05",0,"NAN",0
"2017-09-20 11:30:06",1,"NAN",0
"2017-09-20 11:30:07",2,"NAN",0
"2017-09-20 11:30:08",3,"NAN",0
"2017-09-20 11:30:09",4,"",66
"2017-09-20 11:30:10",5,"NAN",0
"2017-09-20 11:30:11",6,"NAN",0
"2017-09-20 11:30:12",7,"NAN",0
"2017-09-20 11:30:13",8,"NAN",0
"2017-09-20 11:30:14",9,"Ñwind_speed 16.5 (kts) wind_direction 226 (deg) ",48
"2017-09-20 11:30:15",10,"NAN",0
"2017-09-20 11:30:16",11,"NAN",0
"2017-09-20 11:30:17",12,"NAN",0
"2017-09-20 11:30:18",13,"NAN",0
"2017-09-20 11:30:19",14,"",66
If you have NBytesReturned, it did read data into your string variable. String processing instructions stop at the first null character. So, you are getting data into the string, but it doesn't display.
OK, so after reading multiple posts and getting very little info from the CR1000 manual and the serial.pdf file, I see I need to use SerialInBlock because of the unfortunate NULLs. Below is the new test code and sample data with all of the non-printable characters - it works but...
The 2 new questions I have now are:
1. Is this the most efficient way to remove the NULLs so that I have a string of the non-NULL data?
2. This works in a 1 sec scan with 1 stream and no parsing or processing of the data. I have 2 other streams to read and remove NULLs from (tested individually - they work too) but I need to keep the main scan at 1 second which will likely not be enough time to read, parse and process all 3. Should the read, parse, process be in a slowsequence or perhaps just the parse and process?
Thanks for the previous clue JDavis!
Dim datain As String * 1000, datalng(1000) As Long, dataout As String * 1000
Dim i As Long, buflen As Long, datalen As Long
DataTable(TestStream,True,-1)
DataInterval(0,1,Sec,60)
Sample(1,datalen,FP2)
Sample(1,dataout,String)
EndTable
' Main Program
BeginProg
SerialOpen (COM1,9600,3,0,1200) ' 9600,N,8,1,None '3' is binary, handles ascii codes > 127
' SerialOpen (COM2,9600,3,0,1200) ' 9600,N,8,1,None '3' is binary, handles ascii codes > 127
' SerialOpen (COM3,9600,3,0,1200) ' 9600,N,8,1,None '3' is binary, handles ascii codes > 127
'Main Scan
Scan(1,Sec,3,0)
dataout = ""
Move(datalng,1000,0,1)
buflen = SerialInBlock(COM1,datain,1000)
For i = 1 To buflen
datalng(i) = ASCII(datain(1,1,i))
If datalng(i) = 0 then dataout = dataout + CHR(datalng(i)) ' Populate target data string without NULL characters
Next
datalen = Len(dataout)
If Len(dataout) > 0 then CallTable TestStream
NextScan
EndProg
"2017-09-20 17:21:02",0,281,"±%dataMAINsite$¼÷ambient_light 3421.693 (cd/m^2) extinction_coeff 0.186411 (1/km) air_temperature 25.8 (degC) humidity 35 (%rh) wind_speed 19.3 (kts) wind_direction 233 (deg) barometric_pressure1 928.2 (hPa) barometric_pressure2 928.5 (hPa) barometric_pressure3 928.0 (hPa) ±"
"2017-09-20 17:21:07",1,327,"±%/dataMAINsite'£¾0ambient_light 3421.693 (cd/m^2) extinction_coeff 0.186411 (1/km) rain_accumulation_30 0.000 (mm) rain_accumulation 0.000 (mm) wind_speed 19.2 (kts) wind_direction 232 (deg) barometric_pressure1 928.3 (hPa) barometric_pressure2 928.5 (hPa) barometric_pressure3 928.0 (hPa) ice_probe_frequency 39987 (Hz) ±"
"2017-09-20 17:21:12",2,281,"±%dataMAINsite@gÂIextinction_coeff 0.186411 (1/km) ambient_light 3421.693 (cd/m^2) air_temperature 25.8 (degC) humidity 35 (%rh) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) wind_speed 23.0 (kts) wind_direction 235 (deg) ±"
"2017-09-20 17:21:17",3,294,"±%dataMAINsitei]Fvextinction_coeff 0.186411 (1/km) ambient_light 3411.846 (cd/m^2) precipitation_id NP (NWS) precipitation_rate 0.000 (mm/hr) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) wind_speed 23.7 (kts) wind_direction 230 (deg) ±"
"2017-09-20 17:21:22",4,281,"±%dataMAINsiteX)gìextinction_coeff 0.186411 (1/km) wind_speed 22.8 (kts) wind_direction 238 (deg) humidity 35 (%rh) air_temperature 25.7 (degC) ambient_light 3411.846 (cd/m^2) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 928.0 (hPa) ±"
"2017-09-20 17:21:27",5,442,"±%¡dataMAINsiteٯƱ±extinction_coeff 0.186411 (1/km) wind_speed 21.6 (kts) wind_direction 240 (deg) cloud_height1 00000 (ft) cloud_depth1 00000 (ft) cloud_height2 00000 (ft) cloud_depth2 00000 (ft) cloud_height3 00000 (ft) cloud_depth3 00000 (ft) vertical_visibility 00000 (ft) ceilometer_range 24610 (ft) ambient_light 3411.846 (cd/m^2) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) ±"
"2017-09-20 17:21:32",6,281,"±%dataMAINsite´à·¥humidity 35 (%rh) extinction_coeff 0.186411 (1/km) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) air_temperature 25.8 (degC) ambient_light 3411.846 (cd/m^2) wind_speed 19.6 (kts) wind_direction 231 (deg) ±"
"2017-09-20 17:21:37",7,296,"±%dataMAINsiteÆ`&©extinction_coeff 0.186411 (1/km) rain_accumulation_30 0.000 (mm) rain_accumulation 0.000 (mm) wind_speed 23.1 (kts) wind_direction 233 (deg) barometric_pressure1 928.2 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) ambient_light 3411.846 (cd/m^2) ±"
"2017-09-20 17:21:42",8,281,"±%dataMAINsiteRJ§Åextinction_coeff 0.186411 (1/km) ambient_light 3411.846 (cd/m^2) humidity 35 (%rh) air_temperature 25.8 (degC) wind_speed 22.2 (kts) wind_direction 238 (deg) barometric_pressure1 928.2 (hPa) barometric_pressure2 928.5 (hPa) barometric_pressure3 928.0 (hPa) ±"
"2017-09-20 17:21:47",9,553,"±%dataMAINsite駳extinction_coeff 0.186411 (1/km) ambient_light 3411.846 (cd/m^2) flash_15_10_N 0 flash_15_10_NE 0 flash_15_10_E 0 flash_15_10_SE 0 flash_15_10_S 0 flash_15_10_SW 0 flash_15_10_W 0 flash_15_10_NW 0 flash_15_30_N 0 flash_15_30_NE 0 flash_15_30_E 0 flash_15_30_SE 0 flash_15_30_S 0 flash_15_30_SW 0 flash_15_30_W 0 flash_15_30_NW 0 flash_15_overhead 0 flash_15_cloud 0 flash_15_total 0 wind_speed 22.3 (kts) wind_direction 239 (deg) barometric_pressure1 928.1 (hPa) barometric_pressure2 928.5 (hPa) barometric_pressure3 927.8 (hPa) ±"
"2017-09-20 17:21:52",10,281,"±%dataMAINsitepŽi!air_temperature 25.8 (degC) humidity 35 (%rh) wind_speed 27.0 (kts) wind_direction 237 (deg) extinction_coeff 0.186411 (1/km) barometric_pressure1 928.0 (hPa) barometric_pressure2 928.2 (hPa) barometric_pressure3 927.8 (hPa) ambient_light 3405.122 (cd/m^2) ±"
"2017-09-20 17:21:57",11,441,"±%¡dataMAINsite~IOqcloud_height1 00000 (ft) cloud_depth1 00000 (ft) cloud_height2 00000 (ft) cloud_depth2 00000 (ft) cloud_height3 00000 (ft) cloud_depth3 00000 (ft) vertical_visibility 00000 (ft) ceilometer_range 24610 (ft) ambient_light 3405.122 (cd/m^2) wind_speed 24.0 (kts) wind_direction 235 (deg) extinction_coeff 0.186411 (1/km) barometric_pressure1 928.2 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.9 (hPa) ±"
"2017-09-20 17:22:02",12,281,"±%dataMAINsiteã#¤kbarometric_pressure1 928.1 (hPa) barometric_pressure2 928.4 (hPa) barometric_pressure3 927.8 (hPa) air_temperature 25.8 (degC) wind_speed 24.1 (kts) wind_direction 238 (deg) extinction_coeff 0.186411 (1/km) humidity 34 (%rh) ambient_light 3405.122 (cd/m^2) ±"
If you are expecting messages every 5 seconds why are you trying to read them every 1 s ? Another suggestion might be to use SerialiNChk() to see if there is a message to read, if not then don't try to read one.
As noted in the original post, each burst of data seems delayed by the amount of time required to transmit the previous burst (time creep) and the source does not have time sync so I have to check every second. I had performed original ingest work with a PC and saw the data timing move slowly. However, the built-in functionality of the CR1000 will save a lot of programming (averages, max/min, etc) if it works. I had already planned to use SerialInCheck down the road - the posted code was quick and dirty for testing to see if data was still arriving after the read (partial reception at read time).