2023-05-20T00:21:18.294Z,1684542078.294 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2023-05-20T00:21:22.406Z,1684542082.406 [SBIT](IMPORTANT): git: 2023-05-16 2023-05-20T00:21:22.407Z,1684542082.407 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2023-05-20T00:23:07.048Z,1684542187.048 [SBIT](IMPORTANT): Beginning Startup BIT 2023-05-20T00:23:07.053Z,1684542187.053 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T00:23:18.061Z,1684542198.061 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005946 CHAN A1 (24V): 0.000095 CHAN A2 (12V): -0.000098 CHAN A3 (5V): -0.000289 CHAN B0 (3.3V): -0.000293 CHAN B1 (3.15aV): 0.000359 CHAN B2 (3.15bV): 0.000285 CHAN B3 (GND): -0.000158 OPEN: 0.007716 Full Scale: +/- 1 mA 2023-05-20T00:24:01.164Z,1684542241.164 [SBIT](IMPORTANT): SBIT PASSED 2023-05-20T00:24:01.164Z,1684542241.164 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2023-05-20T00:24:01.165Z,1684542241.165 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=12 count; 2023-05-20T00:24:01.165Z,1684542241.165 [SBIT](IMPORTANT): CBIT.abortDepth=300 meter; 2023-05-20T00:24:01.165Z,1684542241.165 [SBIT](IMPORTANT): CBIT.stopDepth=290 meter; 2023-05-20T00:24:01.165Z,1684542241.165 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool; 2023-05-20T00:24:01.165Z,1684542241.165 [SBIT](IMPORTANT): DAT.maxAckTimeouts=1 count; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DAT.sbdAddress=-1 enum; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DAT.verbosity=1 count; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DATMMP.loadAtStartup=0 bool; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DATMMP.simulateHardware=0 bool; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): Depth_Keller.offset=3.6 decibar; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): DropWeight.loadAtStartup=0 bool; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=3 minute; 2023-05-20T00:24:01.166Z,1684542241.166 [SBIT](IMPORTANT): ESPComponent.simulateHardware=0 bool; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none CBIT.ampHoursUsed; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none TrackAcousticContact.azimuth_to_contact_vehicle_frame; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none TrackAcousticContact.range_to_contact; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none VerticalHomogeneityIndexCalculator.vertical_salinity_homogeneity_index; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none VerticalHomogeneityIndexCalculator.vertical_temperature_homogeneity_index; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): Express none height_above_sea_floor; 2023-05-20T00:24:01.167Z,1684542241.167 [SBIT](IMPORTANT): StratificationFrontDetector.loadAtStartup=0 bool; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): StratificationFrontDetector.threshold_VTHI=0.1 celsius; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): StratificationFrontDetector.verbosity=1 count; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=75.046417 cubic_centimeter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalControl.massDefault=21.239223 millimeter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-28 millimeter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalControl.massPositionLimitFwd=32 millimeter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalControl.surfaceThreshold=1.5 meter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth1=3 meter; 2023-05-20T00:24:01.168Z,1684542241.168 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth2=5 meter; 2023-05-20T00:24:01.169Z,1684542241.169 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth3=10 meter; 2023-05-20T00:24:01.169Z,1684542241.169 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth4=15 meter; 2023-05-20T00:24:01.169Z,1684542241.169 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.loadAtStartup=0 bool; 2023-05-20T00:24:01.568Z,1684542241.568 [MissionManager](IMPORTANT): Started mission Startup 2023-05-20T00:25:04.001Z,1684542304.001 [MissionManager](IMPORTANT): Started mission Default 2023-05-20T00:33:27.880Z,1684542807.880 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.391161 min 2023-05-20T00:34:28.596Z,1684542868.596 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=19181, MT Status=1, MTMSN=1324 2023-05-20T00:34:29.336Z,1684542869.336 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2023-05-20T00:34:29.336Z,1684542869.336 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/keepstation.tx 2023-05-20T00:34:30.020Z,1684542870.020 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl 2023-05-20T00:34:35.562Z,1684542875.562 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2023-05-20T00:34:35.563Z,1684542875.563 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.7215 degree 2023-05-20T00:34:35.563Z,1684542875.563 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2023-05-20T00:34:35.564Z,1684542875.564 [CommandExec](IMPORTANT): got command run 2023-05-20T00:34:35.583Z,1684542875.583 [CommandExec](IMPORTANT): Running 2023-05-20T00:34:35.864Z,1684542875.864 [MissionManager](IMPORTANT): Started mission keepstation 2023-05-20T00:36:25.341Z,1684542985.341 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.721500,-122.186897 2023-05-20T00:49:31.496Z,1684543771.496 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=19189, MT Status=1, MTMSN=1325 2023-05-20T00:49:32.310Z,1684543772.310 [CommandExec](IMPORTANT): got command schedule 20230520T015000 "load Science/circle_sample.xml;set circle_sample.NumSamplers 55 count;set circle_sample.MissionTimeout 140 h;set circle_sample.AcousticTrackingTimeout 0.5 h;set circle_sample.NeedCommsMissionStart 0 bool" 4zotb 1 4.000000 2023-05-20T00:49:32.310Z,1684543772.310 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4zotb 2023-05-20T00:49:32.311Z,1684543772.311 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 4 with id='4zotb'): "load Science/circle_sample.xml;set circle_sample.NumSamplers 55 count;set circle_sample.MissionTimeout 140 h;set circle_sample.AcousticTrackingTimeout 0.5 h;set circle_sample.NeedCommsMissionStart 0 bool", AT TIME: 2023-05-20T01:50:00Z 2023-05-20T00:51:35.955Z,1684543895.955 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=19190, MT Status=1, MTMSN=1326 2023-05-20T00:51:36.714Z,1684543896.714 [CommandExec](IMPORTANT): got command schedule 20230520T015000 "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.DepthCircle 100 m" 4zotb 2 4.000000 2023-05-20T00:51:36.715Z,1684543896.715 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4zotb 2023-05-20T00:51:36.716Z,1684543896.716 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 4 with id='4zotb'): "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.DepthCircle 100 m", AT TIME: 2023-05-20T01:50:00Z 2023-05-20T00:52:03.487Z,1684543923.487 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=19191, MT Status=1, MTMSN=1327 2023-05-20T00:52:04.243Z,1684543924.243 [CommandExec](IMPORTANT): got command schedule 20230520T015000 "set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree;set circle_sample.CenterLongitude -122.1869 degree" 4zotb 3 4.000000 2023-05-20T00:52:04.243Z,1684543924.243 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4zotb 2023-05-20T00:52:04.245Z,1684543924.245 [CommandExec](IMPORTANT): Scheduled #3 (#3 of 4 with id='4zotb'): "set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree;set circle_sample.CenterLongitude -122.1869 degree", AT TIME: 2023-05-20T01:50:00Z 2023-05-20T00:52:26.199Z,1684543946.199 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=19192, MT Status=1, MTMSN=1328 2023-05-20T00:52:26.865Z,1684543946.865 [CommandExec](IMPORTANT): got command schedule 20230520T015000 "set circle_sample.TrackingUpdatePeriod 25 s;set circle_sample.NumberOfStartingFixesToIgnore 1 count;run" 4zotb 4 4.000000 2023-05-20T00:52:26.866Z,1684543946.866 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4zotb 2023-05-20T00:52:26.867Z,1684543946.867 [CommandExec](IMPORTANT): Scheduled #4 (#4 of 4 with id='4zotb'): "set circle_sample.TrackingUpdatePeriod 25 s;set circle_sample.NumberOfStartingFixesToIgnore 1 count;run", AT TIME: 2023-05-20T01:50:00Z 2023-05-20T01:30:50.856Z,1684546250.856 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.721500,-122.186897 2023-05-20T01:50:00.081Z,1684547400.081 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2023-05-20T01:50:09.103Z,1684547409.103 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml 2023-05-20T01:50:09.168Z,1684547409.168 [MissionManager](IMPORTANT): Started mission Default 2023-05-20T01:50:25.493Z,1684547425.493 [CommandExec](IMPORTANT): got command set circle_sample.NumSamplers 55 count 2023-05-20T01:50:25.494Z,1684547425.494 [CommandExec](IMPORTANT): got command set circle_sample.MissionTimeout 140 hour 2023-05-20T01:50:25.495Z,1684547425.495 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTrackingTimeout 0.5 hour 2023-05-20T01:50:25.495Z,1684547425.495 [CommandExec](IMPORTANT): got command set circle_sample.NeedCommsMissionStart 0 bool 2023-05-20T01:50:43.038Z,1684547443.038 [CommandExec](IMPORTANT): got command set circle_sample.UseOtherLRAUVSignals 0 bool 2023-05-20T01:50:43.040Z,1684547443.040 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTargetDepth 0 meter 2023-05-20T01:50:43.045Z,1684547443.045 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitFirstSample 0 minute 2023-05-20T01:50:43.045Z,1684547443.045 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitMustFire 2 hour 2023-05-20T01:50:43.046Z,1684547443.046 [CommandExec](IMPORTANT): got command set circle_sample.DepthCircle 100 meter 2023-05-20T01:50:59.856Z,1684547459.856 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMinDepth 2 meter 2023-05-20T01:50:59.857Z,1684547459.857 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMaxDepth 280 meter 2023-05-20T01:50:59.858Z,1684547459.858 [CommandExec](IMPORTANT): got command set circle_sample.MaxDepth 290 meter 2023-05-20T01:50:59.858Z,1684547459.858 [CommandExec](IMPORTANT): got command set circle_sample.CenterLatitude 36.7215 degree 2023-05-20T01:50:59.859Z,1684547459.859 [CommandExec](IMPORTANT): got command set circle_sample.CenterLongitude -122.1869 degree 2023-05-20T01:51:08.585Z,1684547468.585 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 25 second 2023-05-20T01:51:08.601Z,1684547468.601 [CommandExec](IMPORTANT): got command set circle_sample.NumberOfStartingFixesToIgnore 1 count 2023-05-20T01:51:08.602Z,1684547468.602 [CommandExec](IMPORTANT): got command run 2023-05-20T01:51:08.617Z,1684547468.617 [CommandExec](IMPORTANT): Running 2023-05-20T01:51:08.945Z,1684547468.945 [MissionManager](IMPORTANT): Started mission circle_sample 2023-05-20T01:51:09.028Z,1684547469.028 [circle_sample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2023-05-20T01:51:10.976Z,1684547470.976 [circle_sample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2023-05-20T01:51:12.249Z,1684547472.249 [circle_sample:CircleAndSample:A](IMPORTANT): Circle around location 36.721500 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 100.000000 m . 2023-05-20T01:51:14.236Z,1684547474.236 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T01:51:17.447Z,1684547477.447 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 0.094351 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2023-05-20T02:03:06.147Z,1684548186.147 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.008492 m , settling for 5.000000 min 2023-05-20T02:08:08.716Z,1684548488.716 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2023-05-20T02:08:18.857Z,1684548498.857 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T02:08:18.962Z,1684548498.962 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T02:09:09.743Z,1684548549.743 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T02:09:10.556Z,1684548550.556 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -1 2023-05-20T02:09:10.556Z,1684548550.556 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T02:09:10.976Z,1684548550.976 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.475456 m in 9.197840 degC water with 0.020880 ug/l chlorophyll fluorescence. 2023-05-20T02:09:11.786Z,1684548551.786 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T02:09:12.167Z,1684548552.167 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T02:09:13.783Z,1684548553.783 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T02:09:45.698Z,1684548585.698 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@56,aRNAl_bac) 2023-05-20T02:09:48.528Z,1684548588.528 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T02:09:48.529Z,1684548588.529 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T02:09:48.926Z,1684548588.926 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T02:12:31.343Z,1684548751.343 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T02:12:31.738Z,1684548751.738 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T02:56:58.295Z,1684551418.295 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T02:56:58.699Z,1684551418.699 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T02:57:00.715Z,1684551420.715 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T02:57:01.120Z,1684551421.120 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,aRNAl_bac) 2023-05-20T02:58:36.070Z,1684551516.070 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@56,aRNAl_bac) 2023-05-20T02:58:43.748Z,1684551523.748 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T02:58:44.152Z,1684551524.152 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2023-05-20T03:03:43.939Z,1684551823.939 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2023-05-20T03:03:46.404Z,1684551826.404 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2023-05-20T03:03:46.807Z,1684551826.807 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T03:03:47.185Z,1684551827.185 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T03:03:47.592Z,1684551827.592 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T03:03:48.011Z,1684551828.011 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.326408 m 2023-05-20T03:03:48.407Z,1684551828.407 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.223108 degC 2023-05-20T03:03:48.807Z,1684551828.807 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T03:03:49.628Z,1684551829.628 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2023-05-20T03:03:55.897Z,1684551835.897 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T03:04:06.229Z,1684551846.229 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T03:04:21.349Z,1684551861.349 [DAT](FAULT): failed to set remote address 2023-05-20T03:04:36.722Z,1684551876.722 [DAT](FAULT): failed to set remote address 2023-05-20T03:04:52.097Z,1684551892.097 [DAT](FAULT): failed to set remote address 2023-05-20T03:05:07.477Z,1684551907.477 [DAT](FAULT): failed to set remote address 2023-05-20T03:05:22.854Z,1684551922.854 [DAT](FAULT): failed to set remote address 2023-05-20T03:05:38.229Z,1684551938.229 [DAT](FAULT): failed to set remote address 2023-05-20T03:05:53.606Z,1684551953.606 [DAT](FAULT): failed to set remote address 2023-05-20T03:06:09.002Z,1684551969.002 [DAT](FAULT): failed to set remote address 2023-05-20T03:06:24.374Z,1684551984.374 [DAT](FAULT): failed to set remote address 2023-05-20T03:06:39.758Z,1684551999.758 [DAT](FAULT): failed to set remote address 2023-05-20T03:06:55.142Z,1684552015.142 [DAT](FAULT): failed to set remote address 2023-05-20T03:07:10.546Z,1684552030.546 [DAT](FAULT): failed to set remote address 2023-05-20T03:07:25.926Z,1684552045.926 [DAT](FAULT): failed to set remote address 2023-05-20T03:07:41.302Z,1684552061.302 [DAT](FAULT): failed to set remote address 2023-05-20T03:07:56.689Z,1684552076.689 [DAT](FAULT): failed to set remote address 2023-05-20T03:08:12.061Z,1684552092.061 [DAT](FAULT): failed to set remote address 2023-05-20T03:08:27.433Z,1684552107.433 [DAT](FAULT): failed to set remote address 2023-05-20T03:08:42.825Z,1684552122.825 [DAT](FAULT): failed to set remote address 2023-05-20T03:08:53.466Z,1684552133.466 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T03:08:58.202Z,1684552138.202 [DAT](FAULT): failed to set remote address 2023-05-20T03:09:13.582Z,1684552153.582 [DAT](FAULT): failed to set remote address 2023-05-20T03:09:28.953Z,1684552168.953 [DAT](FAULT): failed to set remote address 2023-05-20T03:09:44.326Z,1684552184.326 [DAT](FAULT): failed to set remote address 2023-05-20T03:09:59.714Z,1684552199.714 [DAT](FAULT): failed to set remote address 2023-05-20T03:10:15.097Z,1684552215.097 [DAT](FAULT): failed to set remote address 2023-05-20T03:10:30.473Z,1684552230.473 [DAT](FAULT): failed to set remote address 2023-05-20T03:10:45.842Z,1684552245.842 [DAT](FAULT): failed to set remote address 2023-05-20T03:11:01.214Z,1684552261.214 [DAT](FAULT): failed to set remote address 2023-05-20T03:11:16.590Z,1684552276.590 [DAT](FAULT): failed to set remote address 2023-05-20T03:11:31.982Z,1684552291.982 [DAT](FAULT): failed to set remote address 2023-05-20T03:11:47.385Z,1684552307.385 [DAT](FAULT): failed to set remote address 2023-05-20T03:12:02.758Z,1684552322.758 [DAT](FAULT): failed to set remote address 2023-05-20T03:12:18.130Z,1684552338.130 [DAT](FAULT): failed to set remote address 2023-05-20T03:12:33.506Z,1684552353.506 [DAT](FAULT): failed to set remote address 2023-05-20T03:12:48.901Z,1684552368.901 [DAT](FAULT): failed to set remote address 2023-05-20T03:13:04.282Z,1684552384.282 [DAT](FAULT): failed to set remote address 2023-05-20T03:13:19.663Z,1684552399.663 [DAT](FAULT): failed to set remote address 2023-05-20T03:13:35.053Z,1684552415.053 [DAT](FAULT): failed to set remote address 2023-05-20T03:13:50.426Z,1684552430.426 [DAT](FAULT): failed to set remote address 2023-05-20T03:14:05.811Z,1684552445.811 [DAT](FAULT): failed to set remote address 2023-05-20T03:14:21.189Z,1684552461.189 [DAT](FAULT): failed to set remote address 2023-05-20T03:14:36.594Z,1684552476.594 [DAT](FAULT): failed to set remote address 2023-05-20T03:14:51.967Z,1684552491.967 [DAT](FAULT): failed to set remote address 2023-05-20T03:15:07.338Z,1684552507.338 [DAT](FAULT): failed to set remote address 2023-05-20T03:15:31.034Z,1684552531.034 [DAT](FAULT): failed to set remote address 2023-05-20T03:15:56.238Z,1684552556.238 [DAT](FAULT): failed to set remote address 2023-05-20T03:16:21.189Z,1684552581.189 [DAT](FAULT): failed to set remote address 2023-05-20T03:16:46.145Z,1684552606.145 [DAT](FAULT): failed to set remote address 2023-05-20T03:17:11.353Z,1684552631.353 [DAT](FAULT): failed to set remote address 2023-05-20T03:17:36.317Z,1684552656.317 [DAT](FAULT): failed to set remote address 2023-05-20T03:18:01.521Z,1684552681.521 [DAT](FAULT): failed to set remote address 2023-05-20T03:18:26.470Z,1684552706.470 [DAT](FAULT): failed to set remote address 2023-05-20T03:18:51.445Z,1684552731.445 [DAT](FAULT): failed to set remote address 2023-05-20T03:19:16.662Z,1684552756.662 [DAT](FAULT): failed to set remote address 2023-05-20T03:19:41.638Z,1684552781.638 [DAT](FAULT): failed to set remote address 2023-05-20T03:20:06.602Z,1684552806.602 [DAT](FAULT): failed to set remote address 2023-05-20T03:20:31.807Z,1684552831.807 [DAT](FAULT): failed to set remote address 2023-05-20T03:20:56.754Z,1684552856.754 [DAT](FAULT): failed to set remote address 2023-05-20T03:21:21.714Z,1684552881.714 [DAT](FAULT): failed to set remote address 2023-05-20T03:22:11.879Z,1684552931.879 [DAT](FAULT): failed to set remote address 2023-05-20T03:22:37.342Z,1684552957.342 [DAT](FAULT): failed to set remote address 2023-05-20T03:23:02.306Z,1684552982.306 [DAT](FAULT): failed to set remote address 2023-05-20T03:23:27.525Z,1684553007.525 [DAT](FAULT): failed to set remote address 2023-05-20T03:23:52.481Z,1684553032.481 [DAT](FAULT): failed to set remote address 2023-05-20T03:24:17.466Z,1684553057.466 [DAT](FAULT): failed to set remote address 2023-05-20T03:24:42.697Z,1684553082.697 [DAT](FAULT): failed to set remote address 2023-05-20T03:25:07.653Z,1684553107.653 [DAT](FAULT): failed to set remote address 2023-05-20T03:25:32.609Z,1684553132.609 [DAT](FAULT): failed to set remote address 2023-05-20T03:25:57.817Z,1684553157.817 [DAT](FAULT): failed to set remote address 2023-05-20T03:26:22.777Z,1684553182.777 [DAT](FAULT): failed to set remote address 2023-05-20T03:26:47.730Z,1684553207.730 [DAT](FAULT): failed to set remote address 2023-05-20T03:27:12.945Z,1684553232.945 [DAT](FAULT): failed to set remote address 2023-05-20T03:27:37.898Z,1684553257.898 [DAT](FAULT): failed to set remote address 2023-05-20T03:28:02.879Z,1684553282.879 [DAT](FAULT): failed to set remote address 2023-05-20T03:28:28.094Z,1684553308.094 [DAT](FAULT): failed to set remote address 2023-05-20T03:28:53.057Z,1684553333.057 [DAT](FAULT): failed to set remote address 2023-05-20T03:29:18.022Z,1684553358.022 [DAT](FAULT): failed to set remote address 2023-05-20T03:29:43.246Z,1684553383.246 [DAT](FAULT): failed to set remote address 2023-05-20T03:30:08.214Z,1684553408.214 [DAT](FAULT): failed to set remote address 2023-05-20T03:30:33.185Z,1684553433.185 [DAT](FAULT): failed to set remote address 2023-05-20T03:30:58.414Z,1684553458.414 [DAT](FAULT): failed to set remote address 2023-05-20T03:51:19.977Z,1684554679.977 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028035 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2023-05-20T03:54:25.829Z,1684554865.829 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.724731 m , settling for 5.000000 min 2023-05-20T03:59:38.689Z,1684555178.689 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T03:59:38.802Z,1684555178.802 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T04:00:30.722Z,1684555230.722 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T04:00:31.574Z,1684555231.574 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -1 2023-05-20T04:00:31.574Z,1684555231.574 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T04:00:31.965Z,1684555231.965 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.310921 m in 9.134974 degC water with 0.022040 ug/l chlorophyll fluorescence. 2023-05-20T04:00:32.339Z,1684555232.339 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T04:00:32.737Z,1684555232.737 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T04:00:34.758Z,1684555234.758 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T04:01:06.678Z,1684555266.678 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@55,aRNAl_bac) 2023-05-20T04:01:09.511Z,1684555269.511 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T04:01:09.512Z,1684555269.512 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T04:01:09.927Z,1684555269.927 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T04:03:52.496Z,1684555432.496 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T04:03:52.879Z,1684555432.879 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T04:12:58.473Z,1684555978.473 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T04:50:29.475Z,1684558229.475 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T04:50:29.879Z,1684558229.879 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T04:50:31.903Z,1684558231.903 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T04:50:32.315Z,1684558232.315 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,aRNAl_bac) 2023-05-20T04:52:07.252Z,1684558327.252 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@55,aRNAl_bac) 2023-05-20T04:52:14.919Z,1684558334.919 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T04:52:15.325Z,1684558335.325 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2023-05-20T04:57:15.118Z,1684558635.118 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2023-05-20T04:57:17.573Z,1684558637.573 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2023-05-20T04:57:17.986Z,1684558637.986 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T04:57:18.370Z,1684558638.370 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T04:57:18.782Z,1684558638.782 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T04:57:19.197Z,1684558639.197 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.354118 m 2023-05-20T04:57:19.609Z,1684558639.609 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.249598 degC 2023-05-20T04:57:19.990Z,1684558639.990 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T04:57:20.801Z,1684558640.801 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2023-05-20T04:57:26.997Z,1684558646.997 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T04:57:32.290Z,1684558652.290 [DAT](FAULT): failed to set remote address 2023-05-20T04:57:37.330Z,1684558657.330 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T04:58:18.662Z,1684558698.662 [DAT](FAULT): failed to set remote address 2023-05-20T04:58:43.873Z,1684558723.873 [DAT](FAULT): failed to set remote address 2023-05-20T04:59:08.834Z,1684558748.834 [DAT](FAULT): failed to set remote address 2023-05-20T04:59:33.791Z,1684558773.791 [DAT](FAULT): failed to set remote address 2023-05-20T04:59:59.002Z,1684558799.002 [DAT](FAULT): failed to set remote address 2023-05-20T05:00:23.981Z,1684558823.981 [DAT](FAULT): failed to set remote address 2023-05-20T05:00:48.938Z,1684558848.938 [DAT](FAULT): failed to set remote address 2023-05-20T05:01:13.946Z,1684558873.946 [DAT](FAULT): failed to set remote address 2023-05-20T05:01:39.150Z,1684558899.150 [DAT](FAULT): failed to set remote address 2023-05-20T05:02:04.102Z,1684558924.102 [DAT](FAULT): failed to set remote address 2023-05-20T05:02:24.643Z,1684558944.643 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T05:02:29.305Z,1684558949.305 [DAT](FAULT): failed to set remote address 2023-05-20T05:02:54.282Z,1684558974.282 [DAT](FAULT): failed to set remote address 2023-05-20T05:03:19.246Z,1684558999.246 [DAT](FAULT): failed to set remote address 2023-05-20T05:03:44.202Z,1684559024.202 [DAT](FAULT): failed to set remote address 2023-05-20T05:04:09.402Z,1684559049.402 [DAT](FAULT): failed to set remote address 2023-05-20T05:04:34.354Z,1684559074.354 [DAT](FAULT): failed to set remote address 2023-05-20T05:04:59.565Z,1684559099.565 [DAT](FAULT): failed to set remote address 2023-05-20T05:05:25.030Z,1684559125.030 [DAT](FAULT): failed to set remote address 2023-05-20T05:05:49.982Z,1684559149.982 [DAT](FAULT): failed to set remote address 2023-05-20T05:06:14.956Z,1684559174.956 [DAT](FAULT): failed to set remote address 2023-05-20T05:06:40.130Z,1684559200.130 [DAT](FAULT): failed to set remote address 2023-05-20T05:07:05.101Z,1684559225.101 [DAT](FAULT): failed to set remote address 2023-05-20T05:07:30.062Z,1684559250.062 [DAT](FAULT): failed to set remote address 2023-05-20T05:07:55.269Z,1684559275.269 [DAT](FAULT): failed to set remote address 2023-05-20T05:08:20.254Z,1684559300.254 [DAT](FAULT): failed to set remote address 2023-05-20T05:08:45.222Z,1684559325.222 [DAT](FAULT): failed to set remote address 2023-05-20T05:09:10.438Z,1684559350.438 [DAT](FAULT): failed to set remote address 2023-05-20T05:09:35.408Z,1684559375.408 [DAT](FAULT): failed to set remote address 2023-05-20T05:10:00.353Z,1684559400.353 [DAT](FAULT): failed to set remote address 2023-05-20T05:10:25.566Z,1684559425.566 [DAT](FAULT): failed to set remote address 2023-05-20T05:10:50.521Z,1684559450.521 [DAT](FAULT): failed to set remote address 2023-05-20T05:11:15.489Z,1684559475.489 [DAT](FAULT): failed to set remote address 2023-05-20T05:11:40.721Z,1684559500.721 [DAT](FAULT): failed to set remote address 2023-05-20T05:12:05.653Z,1684559525.653 [DAT](FAULT): failed to set remote address 2023-05-20T05:12:30.869Z,1684559550.869 [DAT](FAULT): failed to set remote address 2023-05-20T05:12:55.817Z,1684559575.817 [DAT](FAULT): failed to set remote address 2023-05-20T05:13:20.795Z,1684559600.795 [DAT](FAULT): failed to set remote address 2023-05-20T05:13:45.993Z,1684559625.993 [DAT](FAULT): failed to set remote address 2023-05-20T05:14:10.976Z,1684559650.976 [DAT](FAULT): failed to set remote address 2023-05-20T05:14:35.906Z,1684559675.906 [DAT](FAULT): failed to set remote address 2023-05-20T05:15:01.137Z,1684559701.137 [DAT](FAULT): failed to set remote address 2023-05-20T05:15:26.097Z,1684559726.097 [DAT](FAULT): failed to set remote address 2023-05-20T05:15:51.061Z,1684559751.061 [DAT](FAULT): failed to set remote address 2023-05-20T05:16:16.273Z,1684559776.273 [DAT](FAULT): failed to set remote address 2023-05-20T05:16:41.238Z,1684559801.238 [DAT](FAULT): failed to set remote address 2023-05-20T05:17:06.437Z,1684559826.437 [DAT](FAULT): failed to set remote address 2023-05-20T05:17:31.414Z,1684559851.414 [DAT](FAULT): failed to set remote address 2023-05-20T05:17:56.361Z,1684559876.361 [DAT](FAULT): failed to set remote address 2023-05-20T05:18:21.326Z,1684559901.326 [DAT](FAULT): failed to set remote address 2023-05-20T05:18:46.561Z,1684559926.561 [DAT](FAULT): failed to set remote address 2023-05-20T05:19:11.517Z,1684559951.517 [DAT](FAULT): failed to set remote address 2023-05-20T05:19:36.478Z,1684559976.478 [DAT](FAULT): failed to set remote address 2023-05-20T05:20:01.686Z,1684560001.686 [DAT](FAULT): failed to set remote address 2023-05-20T05:20:26.683Z,1684560026.683 [DAT](FAULT): failed to set remote address 2023-05-20T05:20:51.634Z,1684560051.634 [DAT](FAULT): failed to set remote address 2023-05-20T05:21:16.846Z,1684560076.846 [DAT](FAULT): failed to set remote address 2023-05-20T05:21:41.825Z,1684560101.825 [DAT](FAULT): failed to set remote address 2023-05-20T05:22:06.786Z,1684560126.786 [DAT](FAULT): failed to set remote address 2023-05-20T05:22:32.036Z,1684560152.036 [DAT](FAULT): failed to set remote address 2023-05-20T05:22:56.974Z,1684560176.974 [DAT](FAULT): failed to set remote address 2023-05-20T05:23:21.950Z,1684560201.950 [DAT](FAULT): failed to set remote address 2023-05-20T05:23:47.158Z,1684560227.158 [DAT](FAULT): failed to set remote address 2023-05-20T05:24:12.118Z,1684560252.118 [DAT](FAULT): failed to set remote address 2023-05-20T05:24:37.075Z,1684560277.075 [DAT](FAULT): failed to set remote address 2023-05-20T05:51:22.714Z,1684561882.714 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031006 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2023-05-20T05:55:32.781Z,1684562132.781 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.738182 m , settling for 5.000000 min 2023-05-20T06:00:45.543Z,1684562445.543 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T06:00:45.689Z,1684562445.689 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T06:01:37.639Z,1684562497.639 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T06:01:38.451Z,1684562498.451 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1 2023-05-20T06:01:38.451Z,1684562498.451 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T06:01:38.878Z,1684562498.878 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.487633 m in 9.252436 degC water with 0.022040 ug/l chlorophyll fluorescence. 2023-05-20T06:01:39.251Z,1684562499.251 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T06:01:39.719Z,1684562499.719 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T06:01:41.683Z,1684562501.683 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T06:02:13.596Z,1684562533.596 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@54,aRNAl_bac) 2023-05-20T06:02:16.431Z,1684562536.431 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T06:02:16.432Z,1684562536.432 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T06:02:16.836Z,1684562536.836 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T06:04:58.946Z,1684562698.946 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T06:04:59.344Z,1684562699.344 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T06:23:18.414Z,1684563798.414 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T06:23:29.240Z,1684563809.240 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003323 CHAN A1 (24V): -0.000152 CHAN A2 (12V): 0.000211 CHAN A3 (5V): -0.000401 CHAN B0 (3.3V): 0.000222 CHAN B1 (3.15aV): -0.000076 CHAN B2 (3.15bV): -0.000315 CHAN B3 (GND): -0.000416 OPEN: 0.007842 Full Scale: +/- 1 mA 2023-05-20T06:48:44.760Z,1684565324.760 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T06:48:45.168Z,1684565325.168 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T06:48:47.189Z,1684565327.189 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T06:48:47.592Z,1684565327.592 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,aRNAl_bac) 2023-05-20T06:50:22.530Z,1684565422.530 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@54,aRNAl_bac) 2023-05-20T06:50:30.207Z,1684565430.207 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T06:50:30.607Z,1684565430.607 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2023-05-20T06:55:30.383Z,1684565730.383 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2023-05-20T06:55:32.844Z,1684565732.844 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2023-05-20T06:55:33.238Z,1684565733.238 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T06:55:33.631Z,1684565733.631 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T06:55:34.035Z,1684565734.035 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T06:55:34.451Z,1684565734.451 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.349876 m 2023-05-20T06:55:34.860Z,1684565734.860 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.274805 degC 2023-05-20T06:55:35.251Z,1684565735.251 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T06:55:36.067Z,1684565736.067 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2023-05-20T06:55:42.137Z,1684565742.137 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T06:55:52.481Z,1684565752.481 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T06:56:07.608Z,1684565767.608 [DAT](FAULT): failed to set remote address 2023-05-20T06:56:30.790Z,1684565790.790 [DAT](FAULT): failed to set remote address 2023-05-20T06:56:55.752Z,1684565815.752 [DAT](FAULT): failed to set remote address 2023-05-20T06:57:11.122Z,1684565831.122 [DAT](FAULT): failed to set remote address 2023-05-20T06:57:26.502Z,1684565846.502 [DAT](FAULT): failed to set remote address 2023-05-20T06:57:41.878Z,1684565861.878 [DAT](FAULT): failed to set remote address 2023-05-20T06:57:57.278Z,1684565877.278 [DAT](FAULT): failed to set remote address 2023-05-20T06:58:12.657Z,1684565892.657 [DAT](FAULT): failed to set remote address 2023-05-20T06:58:28.030Z,1684565908.030 [DAT](FAULT): failed to set remote address 2023-05-20T06:58:43.414Z,1684565923.414 [DAT](FAULT): failed to set remote address 2023-05-20T06:58:58.798Z,1684565938.798 [DAT](FAULT): failed to set remote address 2023-05-20T06:59:14.189Z,1684565954.189 [DAT](FAULT): failed to set remote address 2023-05-20T06:59:29.561Z,1684565969.561 [DAT](FAULT): failed to set remote address 2023-05-20T06:59:44.942Z,1684565984.942 [DAT](FAULT): failed to set remote address 2023-05-20T07:00:00.328Z,1684566000.328 [DAT](FAULT): failed to set remote address 2023-05-20T07:00:15.706Z,1684566015.706 [DAT](FAULT): failed to set remote address 2023-05-20T07:00:31.097Z,1684566031.097 [DAT](FAULT): failed to set remote address 2023-05-20T07:00:39.897Z,1684566039.897 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T07:00:46.485Z,1684566046.485 [DAT](FAULT): failed to set remote address 2023-05-20T07:01:01.857Z,1684566061.857 [DAT](FAULT): failed to set remote address 2023-05-20T07:01:17.229Z,1684566077.229 [DAT](FAULT): failed to set remote address 2023-05-20T07:01:32.601Z,1684566092.601 [DAT](FAULT): failed to set remote address 2023-05-20T07:01:47.985Z,1684566107.985 [DAT](FAULT): failed to set remote address 2023-05-20T07:02:03.357Z,1684566123.357 [DAT](FAULT): failed to set remote address 2023-05-20T07:02:18.729Z,1684566138.729 [DAT](FAULT): failed to set remote address 2023-05-20T07:02:34.101Z,1684566154.101 [DAT](FAULT): failed to set remote address 2023-05-20T07:02:49.473Z,1684566169.473 [DAT](FAULT): failed to set remote address 2023-05-20T07:03:04.846Z,1684566184.846 [DAT](FAULT): failed to set remote address 2023-05-20T07:03:20.221Z,1684566200.221 [DAT](FAULT): failed to set remote address 2023-05-20T07:03:35.618Z,1684566215.618 [DAT](FAULT): failed to set remote address 2023-05-20T07:03:51.013Z,1684566231.013 [DAT](FAULT): failed to set remote address 2023-05-20T07:04:06.410Z,1684566246.410 [DAT](FAULT): failed to set remote address 2023-05-20T07:04:21.802Z,1684566261.802 [DAT](FAULT): failed to set remote address 2023-05-20T07:04:37.177Z,1684566277.177 [DAT](FAULT): failed to set remote address 2023-05-20T07:04:52.562Z,1684566292.562 [DAT](FAULT): failed to set remote address 2023-05-20T07:05:07.945Z,1684566307.945 [DAT](FAULT): failed to set remote address 2023-05-20T07:05:23.326Z,1684566323.326 [DAT](FAULT): failed to set remote address 2023-05-20T07:05:38.701Z,1684566338.701 [DAT](FAULT): failed to set remote address 2023-05-20T07:05:54.078Z,1684566354.078 [DAT](FAULT): failed to set remote address 2023-05-20T07:06:09.458Z,1684566369.458 [DAT](FAULT): failed to set remote address 2023-05-20T07:06:24.842Z,1684566384.842 [DAT](FAULT): failed to set remote address 2023-05-20T07:06:40.214Z,1684566400.214 [DAT](FAULT): failed to set remote address 2023-05-20T07:06:55.590Z,1684566415.590 [DAT](FAULT): failed to set remote address 2023-05-20T07:07:10.962Z,1684566430.962 [DAT](FAULT): failed to set remote address 2023-05-20T07:07:26.345Z,1684566446.345 [DAT](FAULT): failed to set remote address 2023-05-20T07:07:41.718Z,1684566461.718 [DAT](FAULT): failed to set remote address 2023-05-20T07:07:57.093Z,1684566477.093 [DAT](FAULT): failed to set remote address 2023-05-20T07:08:12.474Z,1684566492.474 [DAT](FAULT): failed to set remote address 2023-05-20T07:08:27.854Z,1684566507.854 [DAT](FAULT): failed to set remote address 2023-05-20T07:08:43.243Z,1684566523.243 [DAT](FAULT): failed to set remote address 2023-05-20T07:08:58.618Z,1684566538.618 [DAT](FAULT): failed to set remote address 2023-05-20T07:09:13.990Z,1684566553.990 [DAT](FAULT): failed to set remote address 2023-05-20T07:09:29.366Z,1684566569.366 [DAT](FAULT): failed to set remote address 2023-05-20T07:09:44.741Z,1684566584.741 [DAT](FAULT): failed to set remote address 2023-05-20T07:10:00.130Z,1684566600.130 [DAT](FAULT): failed to set remote address 2023-05-20T07:10:15.502Z,1684566615.502 [DAT](FAULT): failed to set remote address 2023-05-20T07:10:30.882Z,1684566630.882 [DAT](FAULT): failed to set remote address 2023-05-20T07:10:46.273Z,1684566646.273 [DAT](FAULT): failed to set remote address 2023-05-20T07:11:01.657Z,1684566661.657 [DAT](FAULT): failed to set remote address 2023-05-20T07:11:17.038Z,1684566677.038 [DAT](FAULT): failed to set remote address 2023-05-20T07:11:32.429Z,1684566692.429 [DAT](FAULT): failed to set remote address 2023-05-20T07:11:57.653Z,1684566717.653 [DAT](FAULT): failed to set remote address 2023-05-20T07:12:22.609Z,1684566742.609 [DAT](FAULT): failed to set remote address 2023-05-20T07:12:47.838Z,1684566767.838 [DAT](FAULT): failed to set remote address 2023-05-20T07:13:12.785Z,1684566792.785 [DAT](FAULT): failed to set remote address 2023-05-20T07:13:37.755Z,1684566817.755 [DAT](FAULT): failed to set remote address 2023-05-20T07:14:02.957Z,1684566842.957 [DAT](FAULT): failed to set remote address 2023-05-20T07:14:27.914Z,1684566867.914 [DAT](FAULT): failed to set remote address 2023-05-20T07:14:53.134Z,1684566893.134 [DAT](FAULT): failed to set remote address 2023-05-20T07:15:18.096Z,1684566918.096 [DAT](FAULT): failed to set remote address 2023-05-20T07:15:43.050Z,1684566943.050 [DAT](FAULT): failed to set remote address 2023-05-20T07:16:08.265Z,1684566968.265 [DAT](FAULT): failed to set remote address 2023-05-20T07:16:33.217Z,1684566993.217 [DAT](FAULT): failed to set remote address 2023-05-20T07:16:58.166Z,1684567018.166 [DAT](FAULT): failed to set remote address 2023-05-20T07:17:23.374Z,1684567043.374 [DAT](FAULT): failed to set remote address 2023-05-20T07:17:48.334Z,1684567068.334 [DAT](FAULT): failed to set remote address 2023-05-20T07:18:13.542Z,1684567093.542 [DAT](FAULT): failed to set remote address 2023-05-20T07:18:38.502Z,1684567118.502 [DAT](FAULT): failed to set remote address 2023-05-20T07:19:03.458Z,1684567143.458 [DAT](FAULT): failed to set remote address 2023-05-20T07:19:28.430Z,1684567168.430 [DAT](FAULT): failed to set remote address 2023-05-20T07:19:53.631Z,1684567193.631 [DAT](FAULT): failed to set remote address 2023-05-20T07:20:18.594Z,1684567218.594 [DAT](FAULT): failed to set remote address 2023-05-20T07:21:08.777Z,1684567268.777 [DAT](FAULT): failed to set remote address 2023-05-20T07:21:33.737Z,1684567293.737 [DAT](FAULT): failed to set remote address 2023-05-20T07:21:58.942Z,1684567318.942 [DAT](FAULT): failed to set remote address 2023-05-20T07:22:23.945Z,1684567343.945 [DAT](FAULT): failed to set remote address 2023-05-20T07:22:48.902Z,1684567368.902 [DAT](FAULT): failed to set remote address 2023-05-20T07:23:14.110Z,1684567394.110 [DAT](FAULT): failed to set remote address 2023-05-20T07:23:39.125Z,1684567419.125 [DAT](FAULT): failed to set remote address 2023-05-20T07:51:25.476Z,1684569085.476 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.033447 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 4.000000 count 2023-05-20T07:52:12.354Z,1684569132.354 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.171486 m , settling for 5.000000 min 2023-05-20T07:57:25.125Z,1684569445.125 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T07:57:25.303Z,1684569445.303 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T07:58:16.330Z,1684569496.330 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T07:58:17.141Z,1684569497.141 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -1 2023-05-20T07:58:17.141Z,1684569497.141 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T07:58:17.591Z,1684569497.591 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.393913 m in 9.311213 degC water with 0.020880 ug/l chlorophyll fluorescence. 2023-05-20T07:58:17.946Z,1684569497.946 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T07:58:18.352Z,1684569498.352 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T07:58:20.376Z,1684569500.376 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T07:58:52.291Z,1684569532.291 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@53,aRNAl_bac) 2023-05-20T07:58:55.127Z,1684569535.127 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T07:58:55.128Z,1684569535.128 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T07:58:55.522Z,1684569535.522 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T08:01:37.927Z,1684569697.927 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T08:01:38.332Z,1684569698.332 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T08:48:39.303Z,1684572519.303 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T08:48:39.699Z,1684572519.699 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T08:48:41.719Z,1684572521.719 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T08:48:42.127Z,1684572522.127 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,aRNAl_bac) 2023-05-20T08:50:17.068Z,1684572617.068 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@53,aRNAl_bac) 2023-05-20T08:50:24.748Z,1684572624.748 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T08:50:25.147Z,1684572625.147 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2023-05-20T08:55:24.958Z,1684572924.958 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2023-05-20T08:55:27.406Z,1684572927.406 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 4.000000 count 2023-05-20T08:55:27.810Z,1684572927.810 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T08:55:28.199Z,1684572928.199 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T08:55:28.603Z,1684572928.603 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T08:55:29.023Z,1684572929.023 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.172356 m 2023-05-20T08:55:29.428Z,1684572929.428 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.273890 degC 2023-05-20T08:55:29.814Z,1684572929.814 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T08:55:30.629Z,1684572930.629 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 4.000000 count completed. 2023-05-20T08:55:36.685Z,1684572936.685 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T08:55:47.026Z,1684572947.026 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T08:56:22.072Z,1684572982.072 [DAT](FAULT): failed to set remote address 2023-05-20T08:57:12.255Z,1684573032.255 [DAT](FAULT): failed to set remote address 2023-05-20T08:58:02.412Z,1684573082.412 [DAT](FAULT): failed to set remote address 2023-05-20T08:58:52.334Z,1684573132.334 [DAT](FAULT): failed to set remote address 2023-05-20T08:59:42.555Z,1684573182.555 [DAT](FAULT): failed to set remote address 2023-05-20T09:00:07.957Z,1684573207.957 [DAT](FAULT): failed to set remote address 2023-05-20T09:00:32.909Z,1684573232.909 [DAT](FAULT): failed to set remote address 2023-05-20T09:00:34.483Z,1684573234.483 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T09:00:58.115Z,1684573258.115 [DAT](FAULT): failed to set remote address 2023-05-20T09:01:23.070Z,1684573283.070 [DAT](FAULT): failed to set remote address 2023-05-20T09:01:48.034Z,1684573308.034 [DAT](FAULT): failed to set remote address 2023-05-20T09:02:13.242Z,1684573333.242 [DAT](FAULT): failed to set remote address 2023-05-20T09:02:38.198Z,1684573358.198 [DAT](FAULT): failed to set remote address 2023-05-20T09:03:03.178Z,1684573383.178 [DAT](FAULT): failed to set remote address 2023-05-20T09:03:28.401Z,1684573408.401 [DAT](FAULT): failed to set remote address 2023-05-20T09:51:28.030Z,1684576288.030 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028068 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 5.000000 count 2023-05-20T09:56:17.254Z,1684576577.254 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.689842 m , settling for 5.000000 min 2023-05-20T10:01:30.041Z,1684576890.041 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T10:01:30.157Z,1684576890.157 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T10:02:21.718Z,1684576941.718 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T10:02:22.537Z,1684576942.537 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -1 2023-05-20T10:02:22.537Z,1684576942.537 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T10:02:22.976Z,1684576942.976 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.584015 m in 9.305506 degC water with 0.022040 ug/l chlorophyll fluorescence. 2023-05-20T10:02:23.332Z,1684576943.332 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T10:02:23.738Z,1684576943.738 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T10:02:25.767Z,1684576945.767 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T10:02:57.683Z,1684576977.683 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@52,aRNAl_bac) 2023-05-20T10:03:00.502Z,1684576980.502 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T10:03:00.502Z,1684576980.502 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T10:03:00.920Z,1684576980.920 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T10:05:43.727Z,1684577143.727 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T10:05:44.122Z,1684577144.122 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T10:34:33.481Z,1684578873.481 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T10:34:33.481Z,1684578873.481 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T10:54:23.599Z,1684580063.599 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T10:54:23.966Z,1684580063.966 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T10:54:25.988Z,1684580065.988 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T10:54:26.391Z,1684580066.391 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,aRNAl_bac) 2023-05-20T10:56:01.344Z,1684580161.344 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@52,aRNAl_bac) 2023-05-20T10:56:09.009Z,1684580169.009 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T10:56:09.424Z,1684580169.424 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2023-05-20T11:01:09.198Z,1684580469.198 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2023-05-20T11:01:11.664Z,1684580471.664 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 5.000000 count 2023-05-20T11:01:12.039Z,1684580472.039 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T11:01:12.485Z,1684580472.485 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T11:01:12.863Z,1684580472.863 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T11:01:13.275Z,1684580473.275 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 98.886803 m 2023-05-20T11:01:13.660Z,1684580473.660 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.312891 degC 2023-05-20T11:01:14.062Z,1684580474.062 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T11:01:14.874Z,1684580474.874 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 5.000000 count completed. 2023-05-20T11:01:21.141Z,1684580481.141 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T11:01:31.497Z,1684580491.497 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T11:01:46.617Z,1684580506.617 [DAT](FAULT): failed to set remote address 2023-05-20T11:02:01.997Z,1684580521.997 [DAT](FAULT): failed to set remote address 2023-05-20T11:02:17.370Z,1684580537.370 [DAT](FAULT): failed to set remote address 2023-05-20T11:02:32.741Z,1684580552.741 [DAT](FAULT): failed to set remote address 2023-05-20T11:02:48.114Z,1684580568.114 [DAT](FAULT): failed to set remote address 2023-05-20T11:03:03.494Z,1684580583.494 [DAT](FAULT): failed to set remote address 2023-05-20T11:03:18.890Z,1684580598.890 [DAT](FAULT): failed to set remote address 2023-05-20T11:03:34.262Z,1684580614.262 [DAT](FAULT): failed to set remote address 2023-05-20T11:03:50.641Z,1684580630.641 [DAT](FAULT): failed to set remote address 2023-05-20T11:04:06.013Z,1684580646.013 [DAT](FAULT): failed to set remote address 2023-05-20T11:04:21.386Z,1684580661.386 [DAT](FAULT): failed to set remote address 2023-05-20T11:04:36.773Z,1684580676.773 [DAT](FAULT): failed to set remote address 2023-05-20T11:04:52.162Z,1684580692.162 [DAT](FAULT): failed to set remote address 2023-05-20T11:05:07.534Z,1684580707.534 [DAT](FAULT): failed to set remote address 2023-05-20T11:05:22.905Z,1684580722.905 [DAT](FAULT): failed to set remote address 2023-05-20T11:05:38.304Z,1684580738.304 [DAT](FAULT): failed to set remote address 2023-05-20T11:05:55.934Z,1684580755.934 [DAT](FAULT): failed to set remote address 2023-05-20T11:06:18.762Z,1684580778.762 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T11:06:20.905Z,1684580780.905 [DAT](FAULT): failed to set remote address 2023-05-20T11:06:45.896Z,1684580805.896 [DAT](FAULT): failed to set remote address 2023-05-20T11:07:10.846Z,1684580830.846 [DAT](FAULT): failed to set remote address 2023-05-20T11:07:36.054Z,1684580856.054 [DAT](FAULT): failed to set remote address 2023-05-20T11:07:51.430Z,1684580871.430 [DAT](FAULT): failed to set remote address 2023-05-20T11:08:06.821Z,1684580886.821 [DAT](FAULT): failed to set remote address 2023-05-20T11:08:22.198Z,1684580902.198 [DAT](FAULT): failed to set remote address 2023-05-20T11:08:37.594Z,1684580917.594 [DAT](FAULT): failed to set remote address 2023-05-20T11:08:52.974Z,1684580932.974 [DAT](FAULT): failed to set remote address 2023-05-20T11:09:16.162Z,1684580956.162 [DAT](FAULT): failed to set remote address 2023-05-20T11:09:31.538Z,1684580971.538 [DAT](FAULT): failed to set remote address 2023-05-20T11:09:46.922Z,1684580986.922 [DAT](FAULT): failed to set remote address 2023-05-20T11:10:02.309Z,1684581002.309 [DAT](FAULT): failed to set remote address 2023-05-20T11:10:17.694Z,1684581017.694 [DAT](FAULT): failed to set remote address 2023-05-20T11:10:33.078Z,1684581033.078 [DAT](FAULT): failed to set remote address 2023-05-20T11:10:48.454Z,1684581048.454 [DAT](FAULT): failed to set remote address 2023-05-20T11:11:03.825Z,1684581063.825 [DAT](FAULT): failed to set remote address 2023-05-20T11:11:19.202Z,1684581079.202 [DAT](FAULT): failed to set remote address 2023-05-20T11:11:34.573Z,1684581094.573 [DAT](FAULT): failed to set remote address 2023-05-20T11:11:49.946Z,1684581109.946 [DAT](FAULT): failed to set remote address 2023-05-20T11:12:05.317Z,1684581125.317 [DAT](FAULT): failed to set remote address 2023-05-20T11:12:20.689Z,1684581140.689 [DAT](FAULT): failed to set remote address 2023-05-20T11:12:36.070Z,1684581156.070 [DAT](FAULT): failed to set remote address 2023-05-20T11:12:51.455Z,1684581171.455 [DAT](FAULT): failed to set remote address 2023-05-20T11:13:06.838Z,1684581186.838 [DAT](FAULT): failed to set remote address 2023-05-20T11:13:22.210Z,1684581202.210 [DAT](FAULT): failed to set remote address 2023-05-20T11:13:37.594Z,1684581217.594 [DAT](FAULT): failed to set remote address 2023-05-20T11:13:52.966Z,1684581232.966 [DAT](FAULT): failed to set remote address 2023-05-20T11:14:08.338Z,1684581248.338 [DAT](FAULT): failed to set remote address 2023-05-20T11:14:23.722Z,1684581263.722 [DAT](FAULT): failed to set remote address 2023-05-20T11:14:39.105Z,1684581279.105 [DAT](FAULT): failed to set remote address 2023-05-20T11:14:54.485Z,1684581294.485 [DAT](FAULT): failed to set remote address 2023-05-20T11:15:09.857Z,1684581309.857 [DAT](FAULT): failed to set remote address 2023-05-20T11:15:25.238Z,1684581325.238 [DAT](FAULT): failed to set remote address 2023-05-20T11:15:40.627Z,1684581340.627 [DAT](FAULT): failed to set remote address 2023-05-20T11:15:56.013Z,1684581356.013 [DAT](FAULT): failed to set remote address 2023-05-20T11:16:11.401Z,1684581371.401 [DAT](FAULT): failed to set remote address 2023-05-20T11:16:26.777Z,1684581386.777 [DAT](FAULT): failed to set remote address 2023-05-20T11:16:42.158Z,1684581402.158 [DAT](FAULT): failed to set remote address 2023-05-20T11:16:57.542Z,1684581417.542 [DAT](FAULT): failed to set remote address 2023-05-20T11:17:12.913Z,1684581432.913 [DAT](FAULT): failed to set remote address 2023-05-20T11:17:28.302Z,1684581448.302 [DAT](FAULT): failed to set remote address 2023-05-20T11:17:43.674Z,1684581463.674 [DAT](FAULT): failed to set remote address 2023-05-20T11:17:59.050Z,1684581479.050 [DAT](FAULT): failed to set remote address 2023-05-20T11:18:14.430Z,1684581494.430 [DAT](FAULT): failed to set remote address 2023-05-20T11:18:29.806Z,1684581509.806 [DAT](FAULT): failed to set remote address 2023-05-20T11:18:45.181Z,1684581525.181 [DAT](FAULT): failed to set remote address 2023-05-20T11:19:00.554Z,1684581540.554 [DAT](FAULT): failed to set remote address 2023-05-20T11:19:15.935Z,1684581555.935 [DAT](FAULT): failed to set remote address 2023-05-20T11:19:31.357Z,1684581571.357 [DAT](FAULT): failed to set remote address 2023-05-20T11:20:07.918Z,1684581607.918 [DAT](FAULT): failed to set remote address 2023-05-20T11:20:32.873Z,1684581632.873 [DAT](FAULT): failed to set remote address 2023-05-20T11:20:58.082Z,1684581658.082 [DAT](FAULT): failed to set remote address 2023-05-20T11:21:23.034Z,1684581683.034 [DAT](FAULT): failed to set remote address 2023-05-20T11:21:47.986Z,1684581707.986 [DAT](FAULT): failed to set remote address 2023-05-20T11:22:13.189Z,1684581733.189 [DAT](FAULT): failed to set remote address 2023-05-20T11:22:38.141Z,1684581758.141 [DAT](FAULT): failed to set remote address 2023-05-20T11:23:03.353Z,1684581783.353 [DAT](FAULT): failed to set remote address 2023-05-20T11:23:28.322Z,1684581808.322 [DAT](FAULT): failed to set remote address 2023-05-20T11:23:53.274Z,1684581833.274 [DAT](FAULT): failed to set remote address 2023-05-20T11:24:18.481Z,1684581858.481 [DAT](FAULT): failed to set remote address 2023-05-20T11:24:43.450Z,1684581883.450 [DAT](FAULT): failed to set remote address 2023-05-20T11:25:08.422Z,1684581908.422 [DAT](FAULT): failed to set remote address 2023-05-20T11:25:33.622Z,1684581933.622 [DAT](FAULT): failed to set remote address 2023-05-20T11:25:58.578Z,1684581958.578 [DAT](FAULT): failed to set remote address 2023-05-20T11:26:23.550Z,1684581983.550 [DAT](FAULT): failed to set remote address 2023-05-20T11:26:48.781Z,1684582008.781 [DAT](FAULT): failed to set remote address 2023-05-20T11:27:13.737Z,1684582033.737 [DAT](FAULT): failed to set remote address 2023-05-20T11:27:38.686Z,1684582058.686 [DAT](FAULT): failed to set remote address 2023-05-20T11:28:03.919Z,1684582083.919 [DAT](FAULT): failed to set remote address 2023-05-20T11:28:28.850Z,1684582108.850 [DAT](FAULT): failed to set remote address 2023-05-20T11:29:43.994Z,1684582183.994 [DAT](FAULT): failed to set remote address 2023-05-20T11:30:09.207Z,1684582209.207 [DAT](FAULT): failed to set remote address 2023-05-20T11:30:34.166Z,1684582234.166 [DAT](FAULT): failed to set remote address 2023-05-20T11:30:59.122Z,1684582259.122 [DAT](FAULT): failed to set remote address 2023-05-20T11:31:24.330Z,1684582284.330 [DAT](FAULT): failed to set remote address 2023-05-20T11:31:48.469Z,1684582308.469 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T11:31:49.282Z,1684582309.282 [DAT](FAULT): failed to set remote address 2023-05-20T11:32:14.229Z,1684582334.229 [DAT](FAULT): failed to set remote address 2023-05-20T11:32:29.644Z,1684582349.644 [DAT](FAULT): failed to set remote address 2023-05-20T11:32:44.994Z,1684582364.994 [DAT](FAULT): failed to set remote address 2023-05-20T11:33:00.367Z,1684582380.367 [DAT](FAULT): failed to set remote address 2023-05-20T11:33:15.742Z,1684582395.742 [DAT](FAULT): failed to set remote address 2023-05-20T11:33:31.118Z,1684582411.118 [DAT](FAULT): failed to set remote address 2023-05-20T11:33:46.498Z,1684582426.498 [DAT](FAULT): failed to set remote address 2023-05-20T11:34:01.869Z,1684582441.869 [DAT](FAULT): failed to set remote address 2023-05-20T11:34:17.242Z,1684582457.242 [DAT](FAULT): failed to set remote address 2023-05-20T11:34:32.676Z,1684582472.676 [DAT](FAULT): failed to set remote address 2023-05-20T11:51:30.457Z,1684583490.457 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027808 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 6.000000 count 2023-05-20T11:53:17.626Z,1684583597.626 [ElevatorServo](FAULT): Reading outside of valid range:0.000000 2023-05-20T11:53:18.265Z,1684583598.265 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-05-20T11:56:30.707Z,1684583790.707 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.011269 m , settling for 5.000000 min 2023-05-20T12:01:43.513Z,1684584103.513 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T12:01:43.645Z,1684584103.645 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T12:02:35.137Z,1684584155.137 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T12:02:35.949Z,1684584155.949 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -1 2023-05-20T12:02:35.950Z,1684584155.950 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T12:02:36.385Z,1684584156.385 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.156837 m in 9.283564 degC water with 0.022040 ug/l chlorophyll fluorescence. 2023-05-20T12:02:36.756Z,1684584156.756 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T12:02:37.193Z,1684584157.193 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T12:02:39.178Z,1684584159.178 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T12:03:11.102Z,1684584191.102 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac) 2023-05-20T12:03:13.932Z,1684584193.932 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T12:03:13.933Z,1684584193.933 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T12:03:14.331Z,1684584194.331 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T12:05:56.767Z,1684584356.767 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T12:05:57.167Z,1684584357.167 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T12:08:53.509Z,1684584533.509 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T12:23:29.286Z,1684585409.286 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T12:23:40.300Z,1684585420.300 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002283 CHAN A1 (24V): -0.000094 CHAN A2 (12V): 0.000455 CHAN A3 (5V): -0.000560 CHAN B0 (3.3V): -0.000369 CHAN B1 (3.15aV): -0.000190 CHAN B2 (3.15bV): 0.000071 CHAN B3 (GND): -0.000577 OPEN: 0.007565 Full Scale: +/- 1 mA 2023-05-20T12:51:37.239Z,1684587097.239 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T12:51:37.643Z,1684587097.643 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T12:51:39.647Z,1684587099.647 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T12:51:40.056Z,1684587100.056 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac) 2023-05-20T12:53:14.988Z,1684587194.988 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@51,aRNAl_bac) 2023-05-20T12:53:23.073Z,1684587203.073 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T12:53:23.475Z,1684587203.475 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2023-05-20T12:58:23.258Z,1684587503.258 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2023-05-20T12:58:25.705Z,1684587505.705 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 6.000000 count 2023-05-20T12:58:26.112Z,1684587506.112 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T12:58:26.551Z,1684587506.551 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T12:58:26.920Z,1684587506.920 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T12:58:27.324Z,1684587507.324 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.272606 m 2023-05-20T12:58:27.734Z,1684587507.734 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.282709 degC 2023-05-20T12:58:28.145Z,1684587508.145 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T12:58:28.932Z,1684587508.932 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 6.000000 count completed. 2023-05-20T12:58:35.221Z,1684587515.221 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T12:58:40.517Z,1684587520.517 [DAT](FAULT): failed to set remote address 2023-05-20T12:58:45.557Z,1684587525.557 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T12:59:33.198Z,1684587573.198 [DAT](FAULT): failed to set remote address 2023-05-20T12:59:58.401Z,1684587598.401 [DAT](FAULT): failed to set remote address 2023-05-20T13:00:23.378Z,1684587623.378 [DAT](FAULT): failed to set remote address 2023-05-20T13:00:48.602Z,1684587648.602 [DAT](FAULT): failed to set remote address 2023-05-20T13:01:13.565Z,1684587673.565 [DAT](FAULT): failed to set remote address 2023-05-20T13:01:38.521Z,1684587698.521 [DAT](FAULT): failed to set remote address 2023-05-20T13:02:03.741Z,1684587723.741 [DAT](FAULT): failed to set remote address 2023-05-20T13:02:28.949Z,1684587748.949 [DAT](FAULT): failed to set remote address 2023-05-20T13:02:54.166Z,1684587774.166 [DAT](FAULT): failed to set remote address 2023-05-20T13:03:32.800Z,1684587812.800 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T13:04:09.326Z,1684587849.326 [DAT](FAULT): failed to set remote address 2023-05-20T13:04:34.290Z,1684587874.290 [DAT](FAULT): failed to set remote address 2023-05-20T13:04:59.238Z,1684587899.238 [DAT](FAULT): failed to set remote address 2023-05-20T13:05:24.445Z,1684587924.445 [DAT](FAULT): failed to set remote address 2023-05-20T13:05:49.406Z,1684587949.406 [DAT](FAULT): failed to set remote address 2023-05-20T13:06:14.386Z,1684587974.386 [DAT](FAULT): failed to set remote address 2023-05-20T13:06:39.606Z,1684587999.606 [DAT](FAULT): failed to set remote address 2023-05-20T13:07:29.519Z,1684588049.519 [DAT](FAULT): failed to set remote address 2023-05-20T13:08:09.858Z,1684588089.858 [DAT](FAULT): failed to set remote address 2023-05-20T13:08:25.233Z,1684588105.233 [DAT](FAULT): failed to set remote address 2023-05-20T13:08:40.605Z,1684588120.605 [DAT](FAULT): failed to set remote address 2023-05-20T13:08:56.002Z,1684588136.002 [DAT](FAULT): failed to set remote address 2023-05-20T13:09:11.374Z,1684588151.374 [DAT](FAULT): failed to set remote address 2023-05-20T13:09:26.754Z,1684588166.754 [DAT](FAULT): failed to set remote address 2023-05-20T13:09:42.130Z,1684588182.130 [DAT](FAULT): failed to set remote address 2023-05-20T13:09:57.514Z,1684588197.514 [DAT](FAULT): failed to set remote address 2023-05-20T13:10:12.886Z,1684588212.886 [DAT](FAULT): failed to set remote address 2023-05-20T13:10:28.257Z,1684588228.257 [DAT](FAULT): failed to set remote address 2023-05-20T13:10:43.653Z,1684588243.653 [DAT](FAULT): failed to set remote address 2023-05-20T13:10:59.038Z,1684588259.038 [DAT](FAULT): failed to set remote address 2023-05-20T13:11:14.410Z,1684588274.410 [DAT](FAULT): failed to set remote address 2023-05-20T13:11:29.794Z,1684588289.794 [DAT](FAULT): failed to set remote address 2023-05-20T13:11:45.171Z,1684588305.171 [DAT](FAULT): failed to set remote address 2023-05-20T13:12:05.078Z,1684588325.078 [DAT](FAULT): failed to set remote address 2023-05-20T13:12:30.313Z,1684588350.313 [DAT](FAULT): failed to set remote address 2023-05-20T13:12:55.269Z,1684588375.269 [DAT](FAULT): failed to set remote address 2023-05-20T13:13:20.233Z,1684588400.233 [DAT](FAULT): failed to set remote address 2023-05-20T13:13:45.437Z,1684588425.437 [DAT](FAULT): failed to set remote address 2023-05-20T13:14:10.409Z,1684588450.409 [DAT](FAULT): failed to set remote address 2023-05-20T13:14:25.773Z,1684588465.773 [DAT](FAULT): failed to set remote address 2023-05-20T13:14:41.154Z,1684588481.154 [DAT](FAULT): failed to set remote address 2023-05-20T13:15:01.061Z,1684588501.061 [DAT](FAULT): failed to set remote address 2023-05-20T13:15:26.042Z,1684588526.042 [DAT](FAULT): failed to set remote address 2023-05-20T13:15:50.994Z,1684588550.994 [DAT](FAULT): failed to set remote address 2023-05-20T13:16:41.177Z,1684588601.177 [DAT](FAULT): failed to set remote address 2023-05-20T13:17:06.159Z,1684588626.159 [DAT](FAULT): failed to set remote address 2023-05-20T13:17:31.334Z,1684588651.334 [DAT](FAULT): failed to set remote address 2023-05-20T13:17:56.785Z,1684588676.785 [DAT](FAULT): failed to set remote address 2023-05-20T13:18:21.804Z,1684588701.804 [DAT](FAULT): failed to set remote address 2023-05-20T13:18:46.754Z,1684588726.754 [DAT](FAULT): failed to set remote address 2023-05-20T13:19:11.714Z,1684588751.714 [DAT](FAULT): failed to set remote address 2023-05-20T13:19:36.934Z,1684588776.934 [DAT](FAULT): failed to set remote address 2023-05-20T13:20:01.918Z,1684588801.918 [DAT](FAULT): failed to set remote address 2023-05-20T13:20:26.882Z,1684588826.882 [DAT](FAULT): failed to set remote address 2023-05-20T13:51:33.010Z,1684590693.010 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.029053 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 7.000000 count 2023-05-20T13:56:18.640Z,1684590978.640 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.817345 m , settling for 5.000000 min 2023-05-20T14:01:31.437Z,1684591291.437 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T14:01:31.570Z,1684591291.570 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T14:02:23.074Z,1684591343.074 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T14:02:23.888Z,1684591343.888 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -1 2023-05-20T14:02:23.888Z,1684591343.888 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T14:02:24.336Z,1684591344.336 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.256241 m in 9.315546 degC water with 0.022040 ug/l chlorophyll fluorescence. 2023-05-20T14:02:24.695Z,1684591344.695 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T14:02:25.117Z,1684591345.117 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T14:02:27.121Z,1684591347.121 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T14:02:59.031Z,1684591379.031 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@50,aRNAl_bac) 2023-05-20T14:03:01.055Z,1684591381.055 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T14:03:01.466Z,1684591381.466 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T14:03:01.863Z,1684591381.863 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T14:05:45.078Z,1684591545.078 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T14:05:45.483Z,1684591545.483 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T14:29:18.517Z,1684592958.517 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T14:29:18.517Z,1684592958.517 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T14:41:38.360Z,1684593698.360 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T14:50:43.169Z,1684594243.169 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T14:50:43.573Z,1684594243.573 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T14:50:45.601Z,1684594245.601 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T14:50:46.003Z,1684594246.003 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,aRNAl_bac) 2023-05-20T14:52:20.947Z,1684594340.947 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@50,aRNAl_bac) 2023-05-20T14:52:28.618Z,1684594348.618 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T14:52:29.024Z,1684594349.024 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2023-05-20T14:57:28.831Z,1684594648.831 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2023-05-20T14:57:31.295Z,1684594651.295 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 7.000000 count 2023-05-20T14:57:31.706Z,1684594651.706 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T14:57:32.084Z,1684594652.084 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T14:57:32.487Z,1684594652.487 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T14:57:32.903Z,1684594652.903 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.619545 m 2023-05-20T14:57:33.330Z,1684594653.330 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.322107 degC 2023-05-20T14:57:33.697Z,1684594653.697 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T14:57:34.612Z,1684594654.612 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 7.000000 count completed. 2023-05-20T14:57:40.680Z,1684594660.680 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T14:57:45.981Z,1684594665.981 [DAT](FAULT): failed to set remote address 2023-05-20T14:57:51.022Z,1684594671.022 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T14:58:36.406Z,1684594716.406 [DAT](FAULT): failed to set remote address 2023-05-20T14:59:01.613Z,1684594741.613 [DAT](FAULT): failed to set remote address 2023-05-20T14:59:26.566Z,1684594766.566 [DAT](FAULT): failed to set remote address 2023-05-20T14:59:51.542Z,1684594791.542 [DAT](FAULT): failed to set remote address 2023-05-20T15:00:16.762Z,1684594816.762 [DAT](FAULT): failed to set remote address 2023-05-20T15:00:41.722Z,1684594841.722 [DAT](FAULT): failed to set remote address 2023-05-20T15:01:31.898Z,1684594891.898 [DAT](FAULT): failed to set remote address 2023-05-20T15:01:57.361Z,1684594917.361 [DAT](FAULT): failed to set remote address 2023-05-20T15:02:22.314Z,1684594942.314 [DAT](FAULT): failed to set remote address 2023-05-20T15:02:38.340Z,1684594958.340 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T15:02:47.273Z,1684594967.273 [DAT](FAULT): failed to set remote address 2023-05-20T15:03:12.477Z,1684594992.477 [DAT](FAULT): failed to set remote address 2023-05-20T15:03:37.433Z,1684595017.433 [DAT](FAULT): failed to set remote address 2023-05-20T15:04:02.397Z,1684595042.397 [DAT](FAULT): failed to set remote address 2023-05-20T15:04:27.605Z,1684595067.605 [DAT](FAULT): failed to set remote address 2023-05-20T15:04:52.570Z,1684595092.570 [DAT](FAULT): failed to set remote address 2023-05-20T15:05:17.546Z,1684595117.546 [DAT](FAULT): failed to set remote address 2023-05-20T15:05:42.750Z,1684595142.750 [DAT](FAULT): failed to set remote address 2023-05-20T15:06:07.701Z,1684595167.701 [DAT](FAULT): failed to set remote address 2023-05-20T15:06:32.666Z,1684595192.666 [DAT](FAULT): failed to set remote address 2023-05-20T15:06:57.878Z,1684595217.878 [DAT](FAULT): failed to set remote address 2023-05-20T15:07:22.842Z,1684595242.842 [DAT](FAULT): failed to set remote address 2023-05-20T15:07:48.053Z,1684595268.053 [DAT](FAULT): failed to set remote address 2023-05-20T15:08:13.022Z,1684595293.022 [DAT](FAULT): failed to set remote address 2023-05-20T15:08:37.997Z,1684595317.997 [DAT](FAULT): failed to set remote address 2023-05-20T15:09:02.966Z,1684595342.966 [DAT](FAULT): failed to set remote address 2023-05-20T15:09:28.181Z,1684595368.181 [DAT](FAULT): failed to set remote address 2023-05-20T15:09:53.133Z,1684595393.133 [DAT](FAULT): failed to set remote address 2023-05-20T15:10:18.338Z,1684595418.338 [DAT](FAULT): failed to set remote address 2023-05-20T15:10:43.298Z,1684595443.298 [DAT](FAULT): failed to set remote address 2023-05-20T15:11:08.265Z,1684595468.265 [DAT](FAULT): failed to set remote address 2023-05-20T15:11:23.638Z,1684595483.638 [DAT](FAULT): failed to set remote address 2023-05-20T15:11:58.426Z,1684595518.426 [DAT](FAULT): failed to set remote address 2023-05-20T15:12:23.410Z,1684595543.410 [DAT](FAULT): failed to set remote address 2023-05-20T15:12:48.617Z,1684595568.617 [DAT](FAULT): failed to set remote address 2023-05-20T15:13:13.573Z,1684595593.573 [DAT](FAULT): failed to set remote address 2023-05-20T15:13:38.535Z,1684595618.535 [DAT](FAULT): failed to set remote address 2023-05-20T15:14:03.734Z,1684595643.734 [DAT](FAULT): failed to set remote address 2023-05-20T15:14:28.682Z,1684595668.682 [DAT](FAULT): failed to set remote address 2023-05-20T15:51:35.524Z,1684597895.524 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028149 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 8.000000 count 2023-05-20T15:55:48.498Z,1684598148.498 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.669662 m , settling for 5.000000 min 2023-05-20T16:01:01.257Z,1684598461.257 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T16:01:01.375Z,1684598461.375 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T16:01:52.494Z,1684598512.494 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T16:01:53.309Z,1684598513.309 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2023-05-20T16:01:53.310Z,1684598513.310 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T16:01:53.744Z,1684598513.744 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.897758 m in 9.146509 degC water with 0.024360 ug/l chlorophyll fluorescence. 2023-05-20T16:01:54.116Z,1684598514.116 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T16:01:54.529Z,1684598514.529 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T16:01:56.540Z,1684598516.540 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T16:02:28.471Z,1684598548.471 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@49,aRNAl_bac) 2023-05-20T16:02:31.299Z,1684598551.299 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T16:02:31.300Z,1684598551.300 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T16:02:31.707Z,1684598551.707 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T16:05:14.555Z,1684598714.555 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T16:05:14.950Z,1684598714.950 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T16:22:25.770Z,1684599745.770 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T16:34:20.528Z,1684600460.528 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T16:34:20.529Z,1684600460.529 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T16:38:20.513Z,1684600700.513 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T16:38:20.513Z,1684600700.513 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T16:46:18.505Z,1684601178.505 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T16:51:12.391Z,1684601472.391 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T16:51:12.798Z,1684601472.798 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T16:51:14.808Z,1684601474.808 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T16:51:15.215Z,1684601475.215 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,aRNAl_bac) 2023-05-20T16:52:50.556Z,1684601570.556 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@49,aRNAl_bac) 2023-05-20T16:52:58.231Z,1684601578.231 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T16:52:58.652Z,1684601578.652 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2023-05-20T16:54:29.522Z,1684601669.522 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 25 second 2023-05-20T16:54:34.697Z,1684601674.697 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:54:49.826Z,1684601689.826 [DAT](FAULT): failed to set remote address 2023-05-20T16:55:03.018Z,1684601703.018 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 25 second 2023-05-20T16:55:08.474Z,1684601708.474 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:55:18.825Z,1684601718.825 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:55:29.546Z,1684601729.546 [CommandExec](IMPORTANT): got command failComponent hardware DAT 2023-05-20T16:55:29.547Z,1684601729.547 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault 2023-05-20T16:56:14.741Z,1684601774.741 [DAT](FAULT): failed to set transmit power 2023-05-20T16:56:15.751Z,1684601775.751 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:57:58.429Z,1684601878.429 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2023-05-20T16:58:00.877Z,1684601880.877 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 8.000000 count 2023-05-20T16:58:01.288Z,1684601881.288 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T16:58:01.720Z,1684601881.720 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T16:58:02.091Z,1684601882.091 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T16:58:02.496Z,1684601882.496 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 98.962128 m 2023-05-20T16:58:02.914Z,1684601882.914 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.122095 degC 2023-05-20T16:58:03.298Z,1684601883.298 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T16:58:04.128Z,1684601884.128 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 8.000000 count completed. 2023-05-20T16:58:10.185Z,1684601890.185 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:58:15.486Z,1684601895.486 [DAT](FAULT): failed to set remote address 2023-05-20T16:58:20.521Z,1684601900.521 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:58:55.305Z,1684601935.305 [DAT](FAULT): failed to set remote address 2023-05-20T17:03:08.072Z,1684602188.072 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T17:17:27.173Z,1684603047.173 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 45 second 2023-05-20T17:17:32.777Z,1684603052.777 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:17:47.905Z,1684603067.905 [DAT](FAULT): failed to set remote address 2023-05-20T17:18:03.278Z,1684603083.278 [DAT](FAULT): failed to set remote address 2023-05-20T17:18:18.650Z,1684603098.650 [DAT](FAULT): failed to set remote address 2023-05-20T17:18:33.878Z,1684603113.878 [CommandExec](IMPORTANT): got command failComponent hardware DAT 2023-05-20T17:18:33.879Z,1684603113.879 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault 2023-05-20T17:19:08.838Z,1684603148.838 [DAT](FAULT): failed to enter command mode 2023-05-20T17:19:10.603Z,1684603150.603 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:20:35.855Z,1684603235.855 [DAT](FAULT): Buffer send receipt timeout failure. 2023-05-20T17:20:46.194Z,1684603246.194 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:21:01.061Z,1684603261.061 [DAT](FAULT): failed to enter online mode 2023-05-20T17:21:01.061Z,1684603261.061 [DAT](FAULT): Failure returning to online mode 2023-05-20T17:21:49.133Z,1684603309.133 [DAT](FAULT): failed to set DatVerbose 2023-05-20T17:21:50.395Z,1684603310.395 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:51:38.232Z,1684605098.232 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031209 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 9.000000 count 2023-05-20T17:52:10.152Z,1684605130.152 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.668457 m , settling for 5.000000 min 2023-05-20T17:57:22.941Z,1684605442.941 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T17:57:23.040Z,1684605443.040 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T17:58:14.215Z,1684605494.215 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T17:58:15.030Z,1684605495.030 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2023-05-20T17:58:15.031Z,1684605495.031 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T17:58:15.456Z,1684605495.456 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.873215 m in 9.134699 degC water with 0.026680 ug/l chlorophyll fluorescence. 2023-05-20T17:58:15.835Z,1684605495.835 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T17:58:16.233Z,1684605496.233 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T17:58:18.295Z,1684605498.295 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T17:58:50.219Z,1684605530.219 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@48,aRNAl_bac) 2023-05-20T17:58:52.236Z,1684605532.236 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T17:58:52.645Z,1684605532.645 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T17:58:53.047Z,1684605533.047 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T18:01:36.263Z,1684605696.263 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T18:01:36.664Z,1684605696.664 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T18:23:40.421Z,1684607020.421 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T18:23:51.768Z,1684607031.768 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002041 CHAN A1 (24V): -0.000043 CHAN A2 (12V): 0.000349 CHAN A3 (5V): -0.000463 CHAN B0 (3.3V): -0.000352 CHAN B1 (3.15aV): 0.000101 CHAN B2 (3.15bV): 0.000350 CHAN B3 (GND): -0.000242 OPEN: 0.007575 Full Scale: +/- 1 mA 2023-05-20T18:47:26.812Z,1684608446.812 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T18:47:27.215Z,1684608447.215 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T18:47:29.232Z,1684608449.232 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T18:47:29.634Z,1684608449.634 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,aRNAl_bac) 2023-05-20T18:49:04.978Z,1684608544.978 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@48,aRNAl_bac) 2023-05-20T18:49:12.654Z,1684608552.654 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T18:49:13.071Z,1684608553.071 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2023-05-20T18:54:12.834Z,1684608852.834 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2023-05-20T18:54:13.643Z,1684608853.643 [ESPComponent](FAULT): LCB 2 fault: LCB Watchdog Reset. Current Limiter Activated. 2023-05-20T18:54:15.295Z,1684608855.295 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 9.000000 count 2023-05-20T18:54:15.707Z,1684608855.707 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T18:54:16.090Z,1684608856.090 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T18:54:16.510Z,1684608856.510 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T18:54:16.923Z,1684608856.923 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.605003 m 2023-05-20T18:54:17.302Z,1684608857.302 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.188501 degC 2023-05-20T18:54:17.703Z,1684608857.703 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 2023-05-20T18:54:18.529Z,1684608858.529 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 9.000000 count completed. 2023-05-20T18:54:24.785Z,1684608864.785 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T18:54:35.118Z,1684608875.118 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T18:54:51.254Z,1684608891.254 [DAT](FAULT): failed to enter command mode 2023-05-20T18:59:22.336Z,1684609162.336 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T19:51:40.724Z,1684612300.724 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027686 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 10.000000 count 2023-05-20T19:53:10.811Z,1684612390.811 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.203323 m , settling for 5.000000 min 2023-05-20T19:58:23.593Z,1684612703.593 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T19:58:23.750Z,1684612703.750 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T19:59:15.641Z,1684612755.641 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T19:59:16.461Z,1684612756.461 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2023-05-20T19:59:16.462Z,1684612756.462 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T19:59:16.896Z,1684612756.896 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.188637 m in 9.162317 degC water with 0.027840 ug/l chlorophyll fluorescence. 2023-05-20T19:59:17.262Z,1684612757.262 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T19:59:17.663Z,1684612757.663 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T19:59:19.687Z,1684612759.687 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T19:59:51.598Z,1684612791.598 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@47,aRNAl_bac) 2023-05-20T19:59:53.618Z,1684612793.618 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T19:59:54.024Z,1684612794.024 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T19:59:54.426Z,1684612794.426 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T20:02:37.646Z,1684612957.646 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T20:02:38.046Z,1684612958.046 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T20:52:00.243Z,1684615920.243 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T20:52:00.646Z,1684615920.646 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T20:52:02.664Z,1684615922.664 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T20:52:03.068Z,1684615923.068 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@47,aRNAl_bac) 2023-05-20T20:53:38.423Z,1684616018.423 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@47,aRNAl_bac) 2023-05-20T20:53:46.091Z,1684616026.091 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T20:53:46.509Z,1684616026.509 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2023-05-20T20:58:46.285Z,1684616326.285 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2023-05-20T20:58:48.741Z,1684616328.741 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 10.000000 count 2023-05-20T20:58:49.153Z,1684616329.153 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T20:58:49.571Z,1684616329.571 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T20:58:49.936Z,1684616329.936 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T20:58:50.350Z,1684616330.350 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.176529 m 2023-05-20T20:58:50.768Z,1684616330.768 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.321802 degC 2023-05-20T20:58:51.146Z,1684616331.146 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-20T20:58:51.963Z,1684616331.963 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 10.000000 count completed. 2023-05-20T20:58:58.181Z,1684616338.181 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T20:59:08.517Z,1684616348.517 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T20:59:23.645Z,1684616363.645 [DAT](FAULT): failed to enter command mode 2023-05-20T21:03:55.809Z,1684616635.809 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T21:51:43.214Z,1684619503.214 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027075 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 11.000000 count 2023-05-20T21:54:00.114Z,1684619640.114 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.645668 m , settling for 5.000000 min 2023-05-20T21:59:12.889Z,1684619952.889 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T21:59:13.010Z,1684619953.010 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T22:00:04.926Z,1684620004.926 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T22:00:05.739Z,1684620005.739 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2023-05-20T22:00:05.739Z,1684620005.739 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T22:00:06.175Z,1684620006.175 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 101.412468 m in 9.226252 degC water with 0.027840 ug/l chlorophyll fluorescence. 2023-05-20T22:00:06.542Z,1684620006.542 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T22:00:06.947Z,1684620006.947 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T22:00:08.975Z,1684620008.975 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T22:00:40.925Z,1684620040.925 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@46,aRNAl_bac) 2023-05-20T22:00:43.346Z,1684620043.346 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T22:00:43.347Z,1684620043.347 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T22:00:43.748Z,1684620043.748 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T22:03:26.967Z,1684620206.967 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T22:03:27.371Z,1684620207.371 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T22:26:34.513Z,1684621594.513 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T22:26:34.513Z,1684621594.513 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T22:36:38.524Z,1684622198.524 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T22:36:38.525Z,1684622198.525 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T22:49:57.481Z,1684622997.481 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T22:49:57.895Z,1684622997.895 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T22:49:59.915Z,1684622999.915 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T22:50:00.314Z,1684623000.314 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@46,aRNAl_bac) 2023-05-20T22:51:35.658Z,1684623095.658 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@46,aRNAl_bac) 2023-05-20T22:51:43.340Z,1684623103.340 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T22:51:43.756Z,1684623103.756 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2023-05-20T22:56:43.630Z,1684623403.630 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2023-05-20T22:56:46.010Z,1684623406.010 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 11.000000 count 2023-05-20T22:56:46.408Z,1684623406.408 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T22:56:46.785Z,1684623406.785 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T22:56:47.186Z,1684623407.186 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T22:56:47.601Z,1684623407.601 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 101.809906 m 2023-05-20T22:56:48.015Z,1684623408.015 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.220697 degC 2023-05-20T22:56:48.408Z,1684623408.408 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 2023-05-20T22:56:49.209Z,1684623409.209 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 11.000000 count completed. 2023-05-20T22:56:55.538Z,1684623415.538 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T22:57:05.869Z,1684623425.869 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T22:57:41.189Z,1684623461.189 [DAT](FAULT): failed to enter command mode 2023-05-20T23:01:53.059Z,1684623713.059 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T23:51:45.892Z,1684626705.892 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032210 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 12.000000 count 2023-05-20T23:54:14.161Z,1684626854.161 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.749802 m , settling for 5.000000 min 2023-05-20T23:59:26.938Z,1684627166.938 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T23:59:27.049Z,1684627167.049 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-21T00:00:18.949Z,1684627218.949 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-21T00:00:19.764Z,1684627219.764 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -1 2023-05-21T00:00:19.766Z,1684627219.766 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-21T00:00:20.202Z,1684627220.202 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.658653 m in 9.134027 degC water with 0.026680 ug/l chlorophyll fluorescence. 2023-05-21T00:00:20.575Z,1684627220.575 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-21T00:00:20.975Z,1684627220.975 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-21T00:00:22.994Z,1684627222.994 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-21T00:00:54.921Z,1684627254.921 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@45,aRNAl_bac) 2023-05-21T00:00:56.930Z,1684627256.930 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2023-05-21T00:00:57.349Z,1684627257.349 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-21T00:00:57.739Z,1684627257.739 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-21T00:03:40.969Z,1684627420.969 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2023-05-21T00:03:41.371Z,1684627421.371 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-21T00:23:51.929Z,1684628631.929 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-21T00:24:02.876Z,1684628642.876 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002739 CHAN A1 (24V): -0.000032 CHAN A2 (12V): 0.000110 CHAN A3 (5V): -0.000461 CHAN B0 (3.3V): -0.000300 CHAN B1 (3.15aV): 0.000360 CHAN B2 (3.15bV): 0.000275 CHAN B3 (GND): -0.000018 OPEN: 0.007683 Full Scale: +/- 1 mA 2023-05-21T00:50:56.391Z,1684630256.391 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@1000.0) 2023-05-21T00:50:56.794Z,1684630256.794 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-21T00:50:58.816Z,1684630258.816 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-21T00:50:59.210Z,1684630259.210 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@45,aRNAl_bac) 2023-05-21T00:52:34.554Z,1684630354.554 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@45,aRNAl_bac) 2023-05-21T00:52:42.240Z,1684630362.240 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-21T00:52:42.636Z,1684630362.636 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2023-05-21T00:57:42.426Z,1684630662.426 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2023-05-21T00:57:44.923Z,1684630664.923 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 12.000000 count 2023-05-21T00:57:45.335Z,1684630665.335 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-21T00:57:45.708Z,1684630665.708 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-21T00:57:46.158Z,1684630666.158 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-21T00:57:46.537Z,1684630666.537 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.350685 m 2023-05-21T00:57:46.922Z,1684630666.922 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.167688 degC 2023-05-21T00:57:47.334Z,1684630667.334 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 2023-05-21T00:57:48.144Z,1684630668.144 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 12.000000 count completed. 2023-05-21T00:57:54.357Z,1684630674.357 [DAT](FAULT): Sending transmit timeout failure. 2023-05-21T00:58:04.689Z,1684630684.689 [DAT](FAULT): Sending transmit timeout failure. 2023-05-21T01:02:51.947Z,1684630971.947 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-21T01:51:09.527Z,1684633869.527 [circle_sample:CycleLogs:A:B](IMPORTANT): Restarting logs.