2023-05-19T19:35:32.260Z,1684524932.260 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2023-05-19T19:35:37.858Z,1684524937.858 [SBIT](IMPORTANT): git: 2023-05-17 2023-05-19T19:35:37.859Z,1684524937.859 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2023-05-19T19:36:09.908Z,1684524969.908 [DAT](FAULT): failed to enter command mode 2023-05-19T19:37:17.294Z,1684525037.294 [SBIT](IMPORTANT): Beginning Startup BIT 2023-05-19T19:37:17.298Z,1684525037.298 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-19T19:37:28.279Z,1684525048.279 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005231 CHAN A1 (24V): 0.003241 CHAN A2 (12V): -0.003834 CHAN A3 (5V): -0.002151 CHAN B0 (3.3V): -0.001043 CHAN B1 (3.15aV): -0.000322 CHAN B2 (3.15bV): -0.000994 CHAN B3 (GND): 0.000149 OPEN: 0.006424 Full Scale: +/- 1 mA 2023-05-19T19:38:11.431Z,1684525091.431 [SBIT](IMPORTANT): SBIT PASSED 2023-05-19T19:38:11.431Z,1684525091.431 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2023-05-19T19:38:11.432Z,1684525091.432 [SBIT](IMPORTANT): CBIT.abortDepth=300 meter; 2023-05-19T19:38:11.432Z,1684525091.432 [SBIT](IMPORTANT): CBIT.stopDepth=290 meter; 2023-05-19T19:38:11.432Z,1684525091.432 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): DAT.maxAckTimeouts=1 count; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): DAT.verbosity=1 count; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): DATMMP.loadAtStartup=0 bool; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): DATMMP.simulateHardware=0 bool; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): Depth_Keller.offset=-0.8 decibar; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=3 minute; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): ESPComponent.simulateHardware=0 bool; 2023-05-19T19:38:11.433Z,1684525091.433 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none CTD_Seabird.bin_median_sea_water_salinity; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none CTD_Seabird.bin_median_sea_water_temperature; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none TrackAcousticContact.azimuth_to_contact_vehicle_frame; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none TrackAcousticContact.range_to_contact; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none VerticalHomogeneityIndexCalculator.vertical_salinity_homogeneity_index; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none VerticalHomogeneityIndexCalculator.vertical_temperature_homogeneity_index; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff470nm; 2023-05-19T19:38:11.434Z,1684525091.434 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff650nm; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_median_mass_concentration_of_chlorophyll_in_sea_water; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): Express linearApproximation depth 5.000000 meter; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): Express linearApproximation latitude 0.005000 degree; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): Express linearApproximation longitude 0.005000 degree; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): StratificationFrontDetector.loadAtStartup=0 bool; 2023-05-19T19:38:11.435Z,1684525091.435 [SBIT](IMPORTANT): StratificationFrontDetector.threshold_VTHI=0.05 celsius; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): StratificationFrontDetector.verbosity=1 count; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=94.457272 cubic_centimeter; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): VerticalControl.massDefault=19.867281 millimeter; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth3=20 meter; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth4=30 meter; 2023-05-19T19:38:11.436Z,1684525091.436 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.loadAtStartup=0 bool; 2023-05-19T19:38:11.828Z,1684525091.828 [MissionManager](IMPORTANT): Started mission Startup 2023-05-19T19:40:12.757Z,1684525212.757 [MissionManager](IMPORTANT): Started mission Default 2023-05-19T19:51:31.199Z,1684525891.199 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.300450 min 2023-05-19T19:51:58.310Z,1684525918.310 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5310, MT Status=1, MTMSN=2690 2023-05-19T19:51:59.179Z,1684525919.179 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2023-05-19T19:51:59.188Z,1684525919.188 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/keepstation.tx 2023-05-19T19:51:59.886Z,1684525919.886 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl 2023-05-19T19:52:07.295Z,1684525927.295 [CommandExec](IMPORTANT): got command set keepstation.MissionTimeout 4 hour 2023-05-19T19:52:07.295Z,1684525927.295 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2023-05-19T19:52:07.296Z,1684525927.296 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.7215 degree 2023-05-19T19:52:07.297Z,1684525927.297 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2023-05-19T19:52:07.297Z,1684525927.297 [CommandExec](IMPORTANT): got command run 2023-05-19T19:52:07.299Z,1684525927.299 [CommandExec](IMPORTANT): Running 2023-05-19T19:52:07.442Z,1684525927.442 [MissionManager](IMPORTANT): Started mission keepstation 2023-05-19T19:53:27.381Z,1684526007.381 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5312, MT Status=1, MTMSN=2691 2023-05-19T19:53:27.971Z,1684526007.971 [CommandExec](IMPORTANT): got command schedule clear 2023-05-19T19:53:27.972Z,1684526007.972 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2023-05-19T19:53:27.972Z,1684526007.972 [CommandExec](IMPORTANT): got command schedule resume 2023-05-19T19:57:40.497Z,1684526260.497 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.721500,-122.186897 2023-05-19T20:36:34.654Z,1684528594.654 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=5325, MT Status=1, MTMSN=2692 2023-05-19T20:36:35.427Z,1684528595.427 [CommandExec](IMPORTANT): got command schedule 20230519T204500 "load Science/circle_sample.xml;set circle_sample.NumSamplers 2 count;set circle_sample.MissionTimeout 5 h;set circle_sample.AcousticTrackingTimeout 0.25 h;set circle_sample.NeedCommsMissionStart 0 bool" 4zcff 1 4.000000 2023-05-19T20:36:35.427Z,1684528595.427 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4zcff 2023-05-19T20:36:35.428Z,1684528595.428 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 4 with id='4zcff'): "load Science/circle_sample.xml;set circle_sample.NumSamplers 2 count;set circle_sample.MissionTimeout 5 h;set circle_sample.AcousticTrackingTimeout 0.25 h;set circle_sample.NeedCommsMissionStart 0 bool", AT TIME: 2023-05-19T20:45:00Z 2023-05-19T20:37:41.818Z,1684528661.818 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5326, MT Status=1, MTMSN=2693 2023-05-19T20:37:42.693Z,1684528662.693 [CommandExec](IMPORTANT): got command schedule 20230519T204500 "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.MaxWaitNotAchievingDepth 0.5 h" 4zcff 2 4.000000 2023-05-19T20:37:42.694Z,1684528662.694 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4zcff 2023-05-19T20:37:42.695Z,1684528662.695 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 4 with id='4zcff'): "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.MaxWaitNotAchievingDepth 0.5 h", AT TIME: 2023-05-19T20:45:00Z 2023-05-19T20:38:04.317Z,1684528684.317 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5327, MT Status=1, MTMSN=2694 2023-05-19T20:38:05.085Z,1684528685.085 [CommandExec](IMPORTANT): got command schedule 20230519T204500 "set circle_sample.DepthCircle 100 m;set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree" 4zcff 3 4.000000 2023-05-19T20:38:05.086Z,1684528685.086 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4zcff 2023-05-19T20:38:05.087Z,1684528685.087 [CommandExec](IMPORTANT): Scheduled #3 (#3 of 4 with id='4zcff'): "set circle_sample.DepthCircle 100 m;set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree", AT TIME: 2023-05-19T20:45:00Z 2023-05-19T20:38:19.865Z,1684528699.865 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5328, MT Status=1, MTMSN=2695 2023-05-19T20:38:20.505Z,1684528700.505 [CommandExec](IMPORTANT): got command schedule 20230519T204500 "set circle_sample.CenterLongitude -122.1869 degree;run" 4zcff 4 4.000000 2023-05-19T20:38:20.505Z,1684528700.505 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4zcff 2023-05-19T20:38:20.506Z,1684528700.506 [CommandExec](IMPORTANT): Scheduled #4 (#4 of 4 with id='4zcff'): "set circle_sample.CenterLongitude -122.1869 degree;run", AT TIME: 2023-05-19T20:45:00Z 2023-05-19T20:45:00.108Z,1684529100.108 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2023-05-19T20:45:09.654Z,1684529109.654 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml 2023-05-19T20:45:09.919Z,1684529109.919 [MissionManager](IMPORTANT): Started mission Default 2023-05-19T20:45:24.371Z,1684529124.371 [CommandExec](IMPORTANT): got command set circle_sample.NumSamplers 2 count 2023-05-19T20:45:24.372Z,1684529124.372 [CommandExec](IMPORTANT): got command set circle_sample.MissionTimeout 5 hour 2023-05-19T20:45:24.373Z,1684529124.373 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTrackingTimeout 0.25 hour 2023-05-19T20:45:24.374Z,1684529124.374 [CommandExec](IMPORTANT): got command set circle_sample.NeedCommsMissionStart 0 bool 2023-05-19T20:45:41.495Z,1684529141.495 [CommandExec](IMPORTANT): got command set circle_sample.UseOtherLRAUVSignals 0 bool 2023-05-19T20:45:41.496Z,1684529141.496 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTargetDepth 0 meter 2023-05-19T20:45:41.497Z,1684529141.497 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitFirstSample 0 minute 2023-05-19T20:45:41.497Z,1684529141.497 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitMustFire 2 hour 2023-05-19T20:45:41.498Z,1684529141.498 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitNotAchievingDepth 0.5 hour 2023-05-19T20:45:55.858Z,1684529155.858 [CommandExec](IMPORTANT): got command set circle_sample.DepthCircle 100 meter 2023-05-19T20:45:55.859Z,1684529155.859 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMinDepth 2 meter 2023-05-19T20:45:55.859Z,1684529155.859 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMaxDepth 280 meter 2023-05-19T20:45:55.860Z,1684529155.860 [CommandExec](IMPORTANT): got command set circle_sample.MaxDepth 290 meter 2023-05-19T20:45:55.861Z,1684529155.861 [CommandExec](IMPORTANT): got command set circle_sample.CenterLatitude 36.7215 degree 2023-05-19T20:45:58.929Z,1684529158.929 [CommandExec](IMPORTANT): got command set circle_sample.CenterLongitude -122.1869 degree 2023-05-19T20:45:58.929Z,1684529158.929 [CommandExec](IMPORTANT): got command run 2023-05-19T20:45:59.015Z,1684529159.015 [CommandExec](IMPORTANT): Running 2023-05-19T20:45:59.084Z,1684529159.084 [MissionManager](IMPORTANT): Started mission circle_sample 2023-05-19T20:45:59.162Z,1684529159.162 [circle_sample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.25 hours. 2023-05-19T20:46:02.844Z,1684529162.844 [circle_sample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2023-05-19T20:46:04.049Z,1684529164.049 [circle_sample:CircleAndSample:A](IMPORTANT): Circle around location 36.721500 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 100.000000 m . 2023-05-19T20:46:06.085Z,1684529166.085 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-19T20:46:09.314Z,1684529169.314 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 0.094636 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2023-05-19T20:57:14.506Z,1684529834.506 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.057777 m , settling for 5.000000 min 2023-05-19T21:02:17.053Z,1684530137.053 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2023-05-19T21:02:27.199Z,1684530147.199 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-19T21:02:27.481Z,1684530147.481 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-19T21:03:16.194Z,1684530196.194 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-19T21:03:17.008Z,1684530197.008 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -1 2023-05-19T21:03:17.008Z,1684530197.008 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-19T21:03:17.450Z,1684530197.450 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.410973 m in 9.206873 degC water with 0.083200 ug/l chlorophyll fluorescence. 2023-05-19T21:03:18.214Z,1684530198.214 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-19T21:03:18.622Z,1684530198.622 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-19T21:03:20.233Z,1684530200.233 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-19T21:03:52.150Z,1684530232.150 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@57,aRNAl_bac) 2023-05-19T21:03:54.978Z,1684530234.978 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2023-05-19T21:03:54.979Z,1684530234.979 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-19T21:03:55.381Z,1684530235.381 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-19T21:06:42.641Z,1684530402.641 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2023-05-19T21:06:43.045Z,1684530403.045 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-19T21:55:06.049Z,1684533306.049 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1000.0) 2023-05-19T21:55:06.455Z,1684533306.455 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-19T21:55:08.474Z,1684533308.474 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-19T21:55:08.881Z,1684533308.881 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@57,aRNAl_bac) 2023-05-19T21:56:43.824Z,1684533403.824 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@57,aRNAl_bac) 2023-05-19T21:56:51.499Z,1684533411.499 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-19T21:56:51.902Z,1684533411.902 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2023-05-19T22:01:51.670Z,1684533711.670 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2023-05-19T22:01:54.129Z,1684533714.129 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2023-05-19T22:01:54.554Z,1684533714.554 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-19T22:01:54.958Z,1684533714.958 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-19T22:01:55.339Z,1684533715.339 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-19T22:01:55.757Z,1684533715.757 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.322914 m 2023-05-19T22:01:56.154Z,1684533716.154 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.158808 degC 2023-05-19T22:01:56.582Z,1684533716.582 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2023-05-19T22:01:57.373Z,1684533717.373 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2023-05-19T22:02:18.616Z,1684533738.616 [DAT](FAULT): Buffer send receipt timeout failure. 2023-05-19T22:07:01.223Z,1684534021.223 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-19T22:46:11.991Z,1684536371.991 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030615 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2023-05-19T22:46:56.845Z,1684536416.845 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.763565 m , settling for 5.000000 min 2023-05-19T22:52:09.724Z,1684536729.724 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-19T22:52:09.825Z,1684536729.825 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-19T22:52:56.657Z,1684536776.657 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-19T22:52:57.473Z,1684536777.473 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -1 2023-05-19T22:52:57.474Z,1684536777.474 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-19T22:52:57.929Z,1684536777.929 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.969017 m in 9.199335 degC water with 0.065520 ug/l chlorophyll fluorescence. 2023-05-19T22:52:58.275Z,1684536778.275 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-19T22:52:58.677Z,1684536778.677 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-19T22:53:00.697Z,1684536780.697 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-19T22:53:32.642Z,1684536812.642 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@56,aRNAl_bac) 2023-05-19T22:53:35.469Z,1684536815.469 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2023-05-19T22:53:35.470Z,1684536815.470 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-19T22:53:35.873Z,1684536815.873 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-19T22:56:23.145Z,1684536983.145 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2023-05-19T22:56:23.550Z,1684536983.550 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-19T23:06:05.580Z,1684537565.580 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-19T23:06:05.581Z,1684537565.581 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-19T23:42:56.549Z,1684539776.549 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0) 2023-05-19T23:42:56.953Z,1684539776.953 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-19T23:42:58.974Z,1684539778.974 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-19T23:42:59.378Z,1684539779.378 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,aRNAl_bac) 2023-05-19T23:44:34.317Z,1684539874.317 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@56,aRNAl_bac) 2023-05-19T23:44:41.994Z,1684539881.994 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-19T23:44:42.398Z,1684539882.398 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2023-05-19T23:49:42.198Z,1684540182.198 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2023-05-19T23:49:44.700Z,1684540184.700 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2023-05-19T23:49:45.099Z,1684540185.099 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-19T23:49:45.480Z,1684540185.480 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-19T23:49:45.897Z,1684540185.897 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-19T23:49:46.313Z,1684540186.313 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.113914 m 2023-05-19T23:49:46.696Z,1684540186.696 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.166986 degC 2023-05-19T23:49:47.099Z,1684540187.099 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.031200 ug/l 2023-05-19T23:49:47.932Z,1684540187.932 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2023-05-19T23:49:49.811Z,1684540189.811 [MissionManager](IMPORTANT): Started mission Default 2023-05-19T23:49:54.224Z,1684540194.224 [DAT](FAULT): Sending transmit timeout failure. 2023-05-19T23:49:59.516Z,1684540199.516 [DAT](FAULT): failed to set remote address 2023-05-19T23:50:04.556Z,1684540204.556 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T00:05:40.550Z,1684541140.550 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=5348, MT Status=1, MTMSN=2696 2023-05-20T00:05:41.365Z,1684541141.365 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2023-05-20T00:05:41.367Z,1684541141.367 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/keepstation.tx 2023-05-20T00:05:42.017Z,1684541142.017 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl 2023-05-20T00:05:51.591Z,1684541151.591 [CommandExec](IMPORTANT): got command set keepstation.MissionTimeout 4 hour 2023-05-20T00:05:51.592Z,1684541151.592 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2023-05-20T00:05:51.593Z,1684541151.593 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.7215 degree 2023-05-20T00:05:51.593Z,1684541151.593 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2023-05-20T00:05:51.594Z,1684541151.594 [CommandExec](IMPORTANT): got command run 2023-05-20T00:05:51.595Z,1684541151.595 [CommandExec](IMPORTANT): Running 2023-05-20T00:05:51.870Z,1684541151.870 [MissionManager](IMPORTANT): Started mission keepstation 2023-05-20T00:36:02.037Z,1684542962.037 [keepstation:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 30.000000 min minutes 2023-05-20T00:36:17.001Z,1684542977.001 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.721500,-122.186897 2023-05-20T00:52:50.393Z,1684543970.393 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-05-20T00:53:59.937Z,1684544039.937 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5371, MT Status=1, MTMSN=2697 2023-05-20T00:54:00.798Z,1684544040.798 [CommandExec](IMPORTANT): got command schedule 20230520T005000 "load Science/circle_sample.xml;set circle_sample.NumSamplers 54 count;set circle_sample.MissionTimeout 140 h;set circle_sample.AcousticTrackingTimeout 0.5 h;set circle_sample.NeedCommsMissionStart 0 bool" 4zoq5 1 4.000000 2023-05-20T00:54:00.798Z,1684544040.798 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4zoq5 2023-05-20T00:54:00.799Z,1684544040.799 [CommandExec](IMPORTANT): Scheduled #5 (#1 of 4 with id='4zoq5'): "load Science/circle_sample.xml;set circle_sample.NumSamplers 54 count;set circle_sample.MissionTimeout 140 h;set circle_sample.AcousticTrackingTimeout 0.5 h;set circle_sample.NeedCommsMissionStart 0 bool", AT TIME: 2023-05-20T00:50:00Z 2023-05-20T00:54:22.707Z,1684544062.707 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5372, MT Status=1, MTMSN=2698 2023-05-20T00:54:23.626Z,1684544063.626 [CommandExec](IMPORTANT): got command schedule 20230520T005000 "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.MaxWaitNotAchievingDepth 0.5 h" 4zoq5 2 4.000000 2023-05-20T00:54:23.626Z,1684544063.626 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4zoq5 2023-05-20T00:54:23.627Z,1684544063.627 [CommandExec](IMPORTANT): Scheduled #6 (#2 of 4 with id='4zoq5'): "set circle_sample.UseOtherLRAUVSignals 0 bool;set circle_sample.AcousticTargetDepth 0 m;set circle_sample.MaxWaitFirstSample 0 min;set circle_sample.MaxWaitMustFire 2 h;set circle_sample.MaxWaitNotAchievingDepth 0.5 h", AT TIME: 2023-05-20T00:50:00Z 2023-05-20T00:55:00.618Z,1684544100.618 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5373, MT Status=1, MTMSN=2699 2023-05-20T00:55:01.471Z,1684544101.471 [CommandExec](IMPORTANT): got command schedule 20230520T005000 "set circle_sample.DepthCircle 100 m;set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree" 4zoq5 3 4.000000 2023-05-20T00:55:01.477Z,1684544101.477 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4zoq5 2023-05-20T00:55:01.478Z,1684544101.478 [CommandExec](IMPORTANT): Scheduled #7 (#3 of 4 with id='4zoq5'): "set circle_sample.DepthCircle 100 m;set circle_sample.CircleYoYoMinDepth 2 m;set circle_sample.CircleYoYoMaxDepth 280 m;set circle_sample.MaxDepth 290 m;set circle_sample.CenterLatitude 36.7215 degree", AT TIME: 2023-05-20T00:50:00Z 2023-05-20T00:55:12.361Z,1684544112.361 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=5374, MT Status=1, MTMSN=2700 2023-05-20T00:55:13.337Z,1684544113.337 [CommandExec](IMPORTANT): got command schedule 20230520T005000 "set circle_sample.CenterLongitude -122.1869 degree;set circle_sample.TrackingUpdatePeriod 30 s;set circle_sample.NumberOfStartingFixesToIgnore 1 count;run" 4zoq5 4 4.000000 2023-05-20T00:55:13.337Z,1684544113.337 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4zoq5 2023-05-20T00:55:13.339Z,1684544113.339 [CommandExec](IMPORTANT): Scheduled #8 (#4 of 4 with id='4zoq5'): "set circle_sample.CenterLongitude -122.1869 degree;set circle_sample.TrackingUpdatePeriod 30 s;set circle_sample.NumberOfStartingFixesToIgnore 1 count;run", AT TIME: 2023-05-20T00:50:00Z 2023-05-20T00:55:13.461Z,1684544113.461 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2023-05-20T00:55:19.869Z,1684544119.869 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml 2023-05-20T00:55:20.259Z,1684544120.259 [MissionManager](IMPORTANT): Started mission Default 2023-05-20T00:55:32.360Z,1684544132.360 [CommandExec](IMPORTANT): got command set circle_sample.NumSamplers 54 count 2023-05-20T00:55:32.360Z,1684544132.360 [CommandExec](IMPORTANT): got command set circle_sample.MissionTimeout 140 hour 2023-05-20T00:55:32.361Z,1684544132.361 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTrackingTimeout 0.5 hour 2023-05-20T00:55:32.362Z,1684544132.362 [CommandExec](IMPORTANT): got command set circle_sample.NeedCommsMissionStart 0 bool 2023-05-20T00:55:49.546Z,1684544149.546 [CommandExec](IMPORTANT): got command set circle_sample.UseOtherLRAUVSignals 0 bool 2023-05-20T00:55:49.547Z,1684544149.547 [CommandExec](IMPORTANT): got command set circle_sample.AcousticTargetDepth 0 meter 2023-05-20T00:55:49.610Z,1684544149.610 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitFirstSample 0 minute 2023-05-20T00:55:49.610Z,1684544149.610 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitMustFire 2 hour 2023-05-20T00:55:49.611Z,1684544149.611 [CommandExec](IMPORTANT): got command set circle_sample.MaxWaitNotAchievingDepth 0.5 hour 2023-05-20T00:56:03.624Z,1684544163.624 [CommandExec](IMPORTANT): got command set circle_sample.DepthCircle 100 meter 2023-05-20T00:56:03.625Z,1684544163.625 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMinDepth 2 meter 2023-05-20T00:56:03.625Z,1684544163.625 [CommandExec](IMPORTANT): got command set circle_sample.CircleYoYoMaxDepth 280 meter 2023-05-20T00:56:03.626Z,1684544163.626 [CommandExec](IMPORTANT): got command set circle_sample.MaxDepth 290 meter 2023-05-20T00:56:03.627Z,1684544163.627 [CommandExec](IMPORTANT): got command set circle_sample.CenterLatitude 36.7215 degree 2023-05-20T00:56:14.735Z,1684544174.735 [CommandExec](IMPORTANT): got command set circle_sample.CenterLongitude -122.1869 degree 2023-05-20T00:56:14.736Z,1684544174.736 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 30 second 2023-05-20T00:56:14.736Z,1684544174.736 [CommandExec](IMPORTANT): got command set circle_sample.NumberOfStartingFixesToIgnore 1 count 2023-05-20T00:56:14.737Z,1684544174.737 [CommandExec](IMPORTANT): got command run 2023-05-20T00:56:14.739Z,1684544174.739 [CommandExec](IMPORTANT): Running 2023-05-20T00:56:15.037Z,1684544175.037 [MissionManager](IMPORTANT): Started mission circle_sample 2023-05-20T00:56:15.118Z,1684544175.118 [circle_sample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2023-05-20T00:56:17.051Z,1684544177.051 [circle_sample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2023-05-20T00:56:18.507Z,1684544178.507 [circle_sample:CircleAndSample:A](IMPORTANT): Circle around location 36.721500 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 100.000000 m . 2023-05-20T00:56:22.059Z,1684544182.059 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T00:56:25.247Z,1684544185.247 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 0.122972 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2023-05-20T01:07:45.206Z,1684544865.206 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.089630 m , settling for 5.000000 min 2023-05-20T01:12:57.956Z,1684545177.956 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T01:12:58.092Z,1684545178.092 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T01:13:47.188Z,1684545227.188 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T01:13:48.001Z,1684545228.001 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1 2023-05-20T01:13:48.001Z,1684545228.001 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T01:13:48.463Z,1684545228.463 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.103981 m in 9.202631 degC water with 0.073840 ug/l chlorophyll fluorescence. 2023-05-20T01:13:48.806Z,1684545228.806 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T01:13:49.209Z,1684545229.209 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T01:13:51.229Z,1684545231.229 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T01:14:23.145Z,1684545263.145 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@55,aRNAl_bac) 2023-05-20T01:14:25.974Z,1684545265.974 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T01:14:25.975Z,1684545265.975 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T01:14:26.378Z,1684545266.378 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T01:17:13.654Z,1684545433.654 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T01:17:14.057Z,1684545434.057 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T01:30:22.025Z,1684546222.025 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T01:32:40.209Z,1684546360.209 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2023-05-20T01:37:28.296Z,1684546648.296 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T01:37:39.123Z,1684546659.123 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005519 CHAN A1 (24V): 0.002908 CHAN A2 (12V): -0.002910 CHAN A3 (5V): -0.002174 CHAN B0 (3.3V): -0.000627 CHAN B1 (3.15aV): -0.000834 CHAN B2 (3.15bV): -0.001090 CHAN B3 (GND): 0.000467 OPEN: 0.006657 Full Scale: +/- 1 mA 2023-05-20T02:00:52.462Z,1684548052.462 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T02:00:52.866Z,1684548052.866 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T02:00:54.886Z,1684548054.886 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T02:00:55.294Z,1684548055.294 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,aRNAl_bac) 2023-05-20T02:02:30.229Z,1684548150.229 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@55,aRNAl_bac) 2023-05-20T02:02:37.906Z,1684548157.906 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T02:02:38.310Z,1684548158.310 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2023-05-20T02:07:38.101Z,1684548458.101 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2023-05-20T02:07:40.583Z,1684548460.583 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2023-05-20T02:07:41.005Z,1684548461.005 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T02:07:41.398Z,1684548461.398 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T02:07:41.772Z,1684548461.772 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T02:07:42.182Z,1684548462.182 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 98.247902 m 2023-05-20T02:07:42.609Z,1684548462.609 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.232386 degC 2023-05-20T02:07:43.006Z,1684548463.006 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.062400 ug/l 2023-05-20T02:07:43.810Z,1684548463.810 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2023-05-20T02:07:49.868Z,1684548469.868 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T02:08:00.200Z,1684548480.200 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T02:08:15.349Z,1684548495.349 [DAT](FAULT): failed to set remote address 2023-05-20T02:08:30.721Z,1684548510.721 [DAT](FAULT): failed to set remote address 2023-05-20T02:08:46.110Z,1684548526.110 [DAT](FAULT): failed to set remote address 2023-05-20T02:09:01.499Z,1684548541.499 [DAT](FAULT): failed to set remote address 2023-05-20T02:09:16.877Z,1684548556.877 [DAT](FAULT): failed to set remote address 2023-05-20T02:09:32.257Z,1684548572.257 [DAT](FAULT): failed to set remote address 2023-05-20T02:09:47.629Z,1684548587.629 [DAT](FAULT): failed to set remote address 2023-05-20T02:10:03.001Z,1684548603.001 [DAT](FAULT): failed to set remote address 2023-05-20T02:10:18.375Z,1684548618.375 [DAT](FAULT): failed to set remote address 2023-05-20T02:10:59.461Z,1684548659.461 [DAT](FAULT): failed to set remote address 2023-05-20T02:12:00.041Z,1684548720.041 [DAT](FAULT): failed to set remote address 2023-05-20T02:12:30.289Z,1684548750.289 [DAT](FAULT): failed to set remote address 2023-05-20T02:12:45.673Z,1684548765.673 [DAT](FAULT): failed to set remote address 2023-05-20T02:12:47.646Z,1684548767.646 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T02:13:01.065Z,1684548781.065 [DAT](FAULT): failed to set remote address 2023-05-20T02:13:16.449Z,1684548796.449 [DAT](FAULT): failed to set remote address 2023-05-20T02:13:31.828Z,1684548811.828 [DAT](FAULT): failed to set remote address 2023-05-20T02:13:47.197Z,1684548827.197 [DAT](FAULT): failed to set remote address 2023-05-20T02:14:02.581Z,1684548842.581 [DAT](FAULT): failed to set remote address 2023-05-20T02:14:31.598Z,1684548871.598 [DAT](FAULT): failed to set remote address 2023-05-20T02:15:01.849Z,1684548901.849 [DAT](FAULT): failed to set remote address 2023-05-20T02:15:32.122Z,1684548932.122 [DAT](FAULT): failed to set remote address 2023-05-20T02:16:02.369Z,1684548962.369 [DAT](FAULT): failed to set remote address 2023-05-20T02:16:32.864Z,1684548992.864 [DAT](FAULT): failed to set remote address 2023-05-20T02:17:03.121Z,1684549023.121 [DAT](FAULT): failed to set remote address 2023-05-20T02:17:33.443Z,1684549053.443 [DAT](FAULT): failed to set remote address 2023-05-20T02:18:03.681Z,1684549083.681 [DAT](FAULT): failed to set remote address 2023-05-20T02:18:33.953Z,1684549113.953 [DAT](FAULT): failed to set remote address 2023-05-20T02:19:04.213Z,1684549144.213 [DAT](FAULT): failed to set remote address 2023-05-20T02:19:34.473Z,1684549174.473 [DAT](FAULT): failed to set remote address 2023-05-20T02:20:35.249Z,1684549235.249 [DAT](FAULT): failed to set remote address 2023-05-20T02:21:05.502Z,1684549265.502 [DAT](FAULT): failed to set remote address 2023-05-20T02:21:35.778Z,1684549295.778 [DAT](FAULT): failed to set remote address 2023-05-20T02:21:51.150Z,1684549311.150 [DAT](FAULT): failed to set remote address 2023-05-20T02:22:06.545Z,1684549326.545 [DAT](FAULT): failed to set remote address 2023-05-20T02:22:21.929Z,1684549341.929 [DAT](FAULT): failed to set remote address 2023-05-20T02:22:37.302Z,1684549357.302 [DAT](FAULT): failed to set remote address 2023-05-20T02:22:52.673Z,1684549372.673 [DAT](FAULT): failed to set remote address 2023-05-20T02:23:08.073Z,1684549388.073 [DAT](FAULT): failed to set remote address 2023-05-20T02:23:23.444Z,1684549403.444 [DAT](FAULT): failed to set remote address 2023-05-20T02:23:38.817Z,1684549418.817 [DAT](FAULT): failed to set remote address 2023-05-20T02:23:54.188Z,1684549434.188 [DAT](FAULT): failed to set remote address 2023-05-20T02:24:09.572Z,1684549449.572 [DAT](FAULT): failed to set remote address 2023-05-20T02:24:24.983Z,1684549464.983 [DAT](FAULT): failed to set remote address 2023-05-20T02:25:07.565Z,1684549507.565 [DAT](FAULT): failed to set remote address 2023-05-20T02:25:22.937Z,1684549522.937 [DAT](FAULT): failed to set remote address 2023-05-20T02:25:38.316Z,1684549538.316 [DAT](FAULT): failed to set remote address 2023-05-20T02:26:08.101Z,1684549568.101 [DAT](FAULT): failed to set remote address 2023-05-20T02:26:38.608Z,1684549598.608 [DAT](FAULT): failed to set remote address 2023-05-20T02:26:53.996Z,1684549613.996 [DAT](FAULT): failed to set remote address 2023-05-20T02:27:09.373Z,1684549629.373 [DAT](FAULT): failed to set remote address 2023-05-20T02:27:39.118Z,1684549659.118 [DAT](FAULT): failed to set remote address 2023-05-20T02:28:09.377Z,1684549689.377 [DAT](FAULT): failed to set remote address 2023-05-20T02:28:39.616Z,1684549719.616 [DAT](FAULT): failed to set remote address 2023-05-20T02:28:55.001Z,1684549735.001 [DAT](FAULT): failed to set remote address 2023-05-20T02:29:10.381Z,1684549750.381 [DAT](FAULT): failed to set remote address 2023-05-20T02:29:40.385Z,1684549780.385 [DAT](FAULT): failed to set remote address 2023-05-20T02:29:55.788Z,1684549795.788 [DAT](FAULT): failed to set remote address 2023-05-20T02:30:11.165Z,1684549811.165 [DAT](FAULT): failed to set remote address 2023-05-20T02:30:40.905Z,1684549840.905 [DAT](FAULT): failed to set remote address 2023-05-20T02:31:11.158Z,1684549871.158 [DAT](FAULT): failed to set remote address 2023-05-20T02:31:26.560Z,1684549886.560 [DAT](FAULT): failed to set remote address 2023-05-20T02:31:41.937Z,1684549901.937 [DAT](FAULT): failed to set remote address 2023-05-20T02:31:57.348Z,1684549917.348 [DAT](FAULT): failed to set remote address 2023-05-20T02:56:27.945Z,1684551387.945 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031266 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2023-05-20T03:00:57.019Z,1684551657.019 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.257874 m , settling for 5.000000 min 2023-05-20T03:06:09.784Z,1684551969.784 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T03:06:09.923Z,1684551969.923 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T03:06:59.893Z,1684552019.893 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T03:07:00.706Z,1684552020.706 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -1 2023-05-20T03:07:00.706Z,1684552020.706 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T03:07:01.148Z,1684552021.148 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.079979 m in 9.226709 degC water with 0.071760 ug/l chlorophyll fluorescence. 2023-05-20T03:07:01.509Z,1684552021.509 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T03:07:01.913Z,1684552021.913 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T03:07:03.933Z,1684552023.933 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T03:07:35.850Z,1684552055.850 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@54,aRNAl_bac) 2023-05-20T03:07:38.678Z,1684552058.678 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T03:07:38.678Z,1684552058.678 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T03:07:39.084Z,1684552059.084 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T03:10:26.454Z,1684552226.454 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T03:10:26.858Z,1684552226.858 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T04:00:28.573Z,1684555228.573 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T04:00:28.977Z,1684555228.977 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T04:00:31.001Z,1684555231.001 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T04:00:31.401Z,1684555231.401 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,aRNAl_bac) 2023-05-20T04:02:06.350Z,1684555326.350 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@54,aRNAl_bac) 2023-05-20T04:02:14.031Z,1684555334.031 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T04:02:14.433Z,1684555334.433 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2023-05-20T04:07:14.437Z,1684555634.437 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2023-05-20T04:07:16.915Z,1684555636.915 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2023-05-20T04:07:17.335Z,1684555637.335 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T04:07:17.716Z,1684555637.716 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T04:07:18.122Z,1684555638.122 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T04:07:18.547Z,1684555638.547 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.192413 m 2023-05-20T04:07:18.916Z,1684555638.916 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.221186 degC 2023-05-20T04:07:19.341Z,1684555639.341 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2023-05-20T04:07:20.127Z,1684555640.127 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2023-05-20T04:07:26.409Z,1684555646.409 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T04:07:31.700Z,1684555651.700 [DAT](FAULT): failed to set remote address 2023-05-20T04:07:36.741Z,1684555656.741 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T04:08:08.753Z,1684555688.753 [DAT](FAULT): failed to set remote address 2023-05-20T04:08:24.129Z,1684555704.129 [DAT](FAULT): failed to set remote address 2023-05-20T04:08:39.501Z,1684555719.501 [DAT](FAULT): failed to set remote address 2023-05-20T04:08:54.885Z,1684555734.885 [DAT](FAULT): failed to set remote address 2023-05-20T04:09:10.257Z,1684555750.257 [DAT](FAULT): failed to set remote address 2023-05-20T04:09:25.637Z,1684555765.637 [DAT](FAULT): failed to set remote address 2023-05-20T04:09:41.008Z,1684555781.008 [DAT](FAULT): failed to set remote address 2023-05-20T04:09:56.385Z,1684555796.385 [DAT](FAULT): failed to set remote address 2023-05-20T04:10:11.773Z,1684555811.773 [DAT](FAULT): failed to set remote address 2023-05-20T04:10:27.169Z,1684555827.169 [DAT](FAULT): failed to set remote address 2023-05-20T04:10:42.557Z,1684555842.557 [DAT](FAULT): failed to set remote address 2023-05-20T04:10:57.933Z,1684555857.933 [DAT](FAULT): failed to set remote address 2023-05-20T04:11:13.305Z,1684555873.305 [DAT](FAULT): failed to set remote address 2023-05-20T04:11:28.709Z,1684555888.709 [DAT](FAULT): failed to set remote address 2023-05-20T04:11:44.089Z,1684555904.089 [DAT](FAULT): failed to set remote address 2023-05-20T04:11:59.479Z,1684555919.479 [DAT](FAULT): failed to set remote address 2023-05-20T04:12:14.853Z,1684555934.853 [DAT](FAULT): failed to set remote address 2023-05-20T04:12:24.007Z,1684555944.007 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T04:12:30.228Z,1684555950.228 [DAT](FAULT): failed to set remote address 2023-05-20T04:12:45.601Z,1684555965.601 [DAT](FAULT): failed to set remote address 2023-05-20T04:13:00.977Z,1684555980.977 [DAT](FAULT): failed to set remote address 2023-05-20T04:13:16.348Z,1684555996.348 [DAT](FAULT): failed to set remote address 2023-05-20T04:13:31.740Z,1684556011.740 [DAT](FAULT): failed to set remote address 2023-05-20T04:13:47.113Z,1684556027.113 [DAT](FAULT): failed to set remote address 2023-05-20T04:14:02.489Z,1684556042.489 [DAT](FAULT): failed to set remote address 2023-05-20T04:14:17.864Z,1684556057.864 [DAT](FAULT): failed to set remote address 2023-05-20T04:14:33.249Z,1684556073.249 [DAT](FAULT): failed to set remote address 2023-05-20T04:14:48.629Z,1684556088.629 [DAT](FAULT): failed to set remote address 2023-05-20T04:15:04.001Z,1684556104.001 [DAT](FAULT): failed to set remote address 2023-05-20T04:15:19.377Z,1684556119.377 [DAT](FAULT): failed to set remote address 2023-05-20T04:15:34.766Z,1684556134.766 [DAT](FAULT): failed to set remote address 2023-05-20T04:15:50.144Z,1684556150.144 [DAT](FAULT): failed to set remote address 2023-05-20T04:16:05.520Z,1684556165.520 [DAT](FAULT): failed to set remote address 2023-05-20T04:16:20.912Z,1684556180.912 [DAT](FAULT): failed to set remote address 2023-05-20T04:16:43.853Z,1684556203.853 [DAT](FAULT): failed to set remote address 2023-05-20T04:17:14.117Z,1684556234.117 [DAT](FAULT): failed to set remote address 2023-05-20T04:17:44.385Z,1684556264.385 [DAT](FAULT): failed to set remote address 2023-05-20T04:18:14.881Z,1684556294.881 [DAT](FAULT): failed to set remote address 2023-05-20T04:18:30.253Z,1684556310.253 [DAT](FAULT): failed to set remote address 2023-05-20T04:18:45.647Z,1684556325.647 [DAT](FAULT): failed to set remote address 2023-05-20T04:19:15.377Z,1684556355.377 [DAT](FAULT): failed to set remote address 2023-05-20T04:19:45.637Z,1684556385.637 [DAT](FAULT): failed to set remote address 2023-05-20T04:20:15.885Z,1684556415.885 [DAT](FAULT): failed to set remote address 2023-05-20T04:20:31.258Z,1684556431.258 [DAT](FAULT): failed to set remote address 2023-05-20T04:20:46.652Z,1684556446.652 [DAT](FAULT): failed to set remote address 2023-05-20T04:21:16.656Z,1684556476.656 [DAT](FAULT): failed to set remote address 2023-05-20T04:21:46.925Z,1684556506.925 [DAT](FAULT): failed to set remote address 2023-05-20T04:22:17.214Z,1684556537.214 [DAT](FAULT): failed to set remote address 2023-05-20T04:22:47.469Z,1684556567.469 [DAT](FAULT): failed to set remote address 2023-05-20T04:23:17.736Z,1684556597.736 [DAT](FAULT): failed to set remote address 2023-05-20T04:23:33.113Z,1684556613.113 [DAT](FAULT): failed to set remote address 2023-05-20T04:23:48.515Z,1684556628.515 [DAT](FAULT): failed to set remote address 2023-05-20T04:24:18.488Z,1684556658.488 [DAT](FAULT): failed to set remote address 2023-05-20T04:24:48.729Z,1684556688.729 [DAT](FAULT): failed to set remote address 2023-05-20T04:25:18.977Z,1684556718.977 [DAT](FAULT): failed to set remote address 2023-05-20T04:25:49.229Z,1684556749.229 [DAT](FAULT): failed to set remote address 2023-05-20T04:26:04.614Z,1684556764.614 [DAT](FAULT): failed to set remote address 2023-05-20T04:35:41.556Z,1684557341.556 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T04:56:30.727Z,1684558590.727 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031551 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2023-05-20T05:00:22.632Z,1684558822.632 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.762291 m , settling for 5.000000 min 2023-05-20T05:05:35.409Z,1684559135.409 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T05:05:35.537Z,1684559135.537 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T05:06:24.626Z,1684559184.626 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T05:06:25.437Z,1684559185.437 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -1 2023-05-20T05:06:25.438Z,1684559185.438 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T05:06:25.883Z,1684559185.883 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 98.317505 m in 9.260309 degC water with 0.081120 ug/l chlorophyll fluorescence. 2023-05-20T05:06:26.241Z,1684559186.241 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T05:06:26.645Z,1684559186.645 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T05:06:28.666Z,1684559188.666 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T05:07:00.582Z,1684559220.582 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@53,aRNAl_bac) 2023-05-20T05:07:01.113Z,1684559221.113 [RudderServo](FAULT): Reading outside of valid range:0.000000 2023-05-20T05:07:01.824Z,1684559221.824 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2023-05-20T05:07:03.531Z,1684559223.531 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T05:07:03.532Z,1684559223.532 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T05:07:03.934Z,1684559223.934 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T05:09:51.211Z,1684559391.211 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T05:09:51.613Z,1684559391.613 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T05:28:54.300Z,1684560534.300 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T05:42:17.608Z,1684561337.608 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T05:42:17.608Z,1684561337.608 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T05:55:30.031Z,1684562130.031 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T05:55:30.432Z,1684562130.432 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T05:55:32.453Z,1684562132.453 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T05:55:32.856Z,1684562132.856 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,aRNAl_bac) 2023-05-20T05:57:07.795Z,1684562227.795 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@53,aRNAl_bac) 2023-05-20T05:57:15.472Z,1684562235.472 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T05:57:15.880Z,1684562235.880 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2023-05-20T06:02:15.655Z,1684562535.655 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2023-05-20T06:02:18.132Z,1684562538.132 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2023-05-20T06:02:18.530Z,1684562538.530 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T06:02:18.935Z,1684562538.935 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T06:02:19.337Z,1684562539.337 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T06:02:19.750Z,1684562539.750 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.152267 m 2023-05-20T06:02:20.165Z,1684562540.165 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.257105 degC 2023-05-20T06:02:20.555Z,1684562540.555 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.072800 ug/l 2023-05-20T06:02:21.380Z,1684562541.380 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2023-05-20T06:02:27.648Z,1684562547.648 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T06:02:32.941Z,1684562552.941 [DAT](FAULT): failed to set remote address 2023-05-20T06:02:37.981Z,1684562557.981 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T06:03:16.818Z,1684562596.818 [DAT](FAULT): failed to set remote address 2023-05-20T06:03:47.077Z,1684562627.077 [DAT](FAULT): failed to set remote address 2023-05-20T06:04:17.577Z,1684562657.577 [DAT](FAULT): failed to set remote address 2023-05-20T06:04:47.853Z,1684562687.853 [DAT](FAULT): failed to set remote address 2023-05-20T06:05:18.113Z,1684562718.113 [DAT](FAULT): failed to set remote address 2023-05-20T06:05:48.361Z,1684562748.361 [DAT](FAULT): failed to set remote address 2023-05-20T06:06:18.629Z,1684562778.629 [DAT](FAULT): failed to set remote address 2023-05-20T06:06:48.893Z,1684562808.893 [DAT](FAULT): failed to set remote address 2023-05-20T06:07:19.400Z,1684562839.400 [DAT](FAULT): failed to set remote address 2023-05-20T06:07:25.182Z,1684562845.182 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T06:07:49.657Z,1684562869.657 [DAT](FAULT): failed to set remote address 2023-05-20T06:08:50.197Z,1684562930.197 [DAT](FAULT): failed to set remote address 2023-05-20T06:09:20.474Z,1684562960.474 [DAT](FAULT): failed to set remote address 2023-05-20T06:09:50.757Z,1684562990.757 [DAT](FAULT): failed to set remote address 2023-05-20T06:10:21.005Z,1684563021.005 [DAT](FAULT): failed to set remote address 2023-05-20T06:10:36.381Z,1684563036.381 [DAT](FAULT): failed to set remote address 2023-05-20T06:10:51.757Z,1684563051.757 [DAT](FAULT): failed to set remote address 2023-05-20T06:11:07.141Z,1684563067.141 [DAT](FAULT): failed to set remote address 2023-05-20T06:11:22.513Z,1684563082.513 [DAT](FAULT): failed to set remote address 2023-05-20T06:11:37.889Z,1684563097.889 [DAT](FAULT): failed to set remote address 2023-05-20T06:11:53.261Z,1684563113.261 [DAT](FAULT): failed to set remote address 2023-05-20T06:12:08.652Z,1684563128.652 [DAT](FAULT): failed to set remote address 2023-05-20T06:12:24.025Z,1684563144.025 [DAT](FAULT): failed to set remote address 2023-05-20T06:12:39.409Z,1684563159.409 [DAT](FAULT): failed to set remote address 2023-05-20T06:12:54.780Z,1684563174.780 [DAT](FAULT): failed to set remote address 2023-05-20T06:13:10.157Z,1684563190.157 [DAT](FAULT): failed to set remote address 2023-05-20T06:13:25.534Z,1684563205.534 [DAT](FAULT): failed to set remote address 2023-05-20T06:13:40.905Z,1684563220.905 [DAT](FAULT): failed to set remote address 2023-05-20T06:13:56.276Z,1684563236.276 [DAT](FAULT): failed to set remote address 2023-05-20T06:14:11.652Z,1684563251.652 [DAT](FAULT): failed to set remote address 2023-05-20T06:14:53.761Z,1684563293.761 [DAT](FAULT): failed to set remote address 2023-05-20T06:15:09.136Z,1684563309.136 [DAT](FAULT): failed to set remote address 2023-05-20T06:15:24.509Z,1684563324.509 [DAT](FAULT): failed to set remote address 2023-05-20T06:15:39.925Z,1684563339.925 [DAT](FAULT): failed to set remote address 2023-05-20T06:15:55.304Z,1684563355.304 [DAT](FAULT): failed to set remote address 2023-05-20T06:16:24.561Z,1684563384.561 [DAT](FAULT): failed to set remote address 2023-05-20T06:16:55.065Z,1684563415.065 [DAT](FAULT): failed to set remote address 2023-05-20T06:17:25.316Z,1684563445.316 [DAT](FAULT): failed to set remote address 2023-05-20T06:17:55.572Z,1684563475.572 [DAT](FAULT): failed to set remote address 2023-05-20T06:18:25.820Z,1684563505.820 [DAT](FAULT): failed to set remote address 2023-05-20T06:18:56.065Z,1684563536.065 [DAT](FAULT): failed to set remote address 2023-05-20T06:19:26.557Z,1684563566.557 [DAT](FAULT): failed to set remote address 2023-05-20T06:19:56.824Z,1684563596.824 [DAT](FAULT): failed to set remote address 2023-05-20T06:20:27.075Z,1684563627.075 [DAT](FAULT): failed to set remote address 2023-05-20T06:20:57.333Z,1684563657.333 [DAT](FAULT): failed to set remote address 2023-05-20T06:21:27.605Z,1684563687.605 [DAT](FAULT): failed to set remote address 2023-05-20T06:21:58.104Z,1684563718.104 [DAT](FAULT): failed to set remote address 2023-05-20T06:22:28.349Z,1684563748.349 [DAT](FAULT): failed to set remote address 2023-05-20T06:22:58.601Z,1684563778.601 [DAT](FAULT): failed to set remote address 2023-05-20T06:23:28.853Z,1684563808.853 [DAT](FAULT): failed to set remote address 2023-05-20T06:23:59.133Z,1684563839.133 [DAT](FAULT): failed to set remote address 2023-05-20T06:24:29.394Z,1684563869.394 [DAT](FAULT): failed to set remote address 2023-05-20T06:24:59.889Z,1684563899.889 [DAT](FAULT): failed to set remote address 2023-05-20T06:25:30.145Z,1684563930.145 [DAT](FAULT): failed to set remote address 2023-05-20T06:25:45.520Z,1684563945.520 [DAT](FAULT): failed to set remote address 2023-05-20T06:26:00.897Z,1684563960.897 [DAT](FAULT): failed to set remote address 2023-05-20T06:26:16.276Z,1684563976.276 [DAT](FAULT): failed to set remote address 2023-05-20T06:26:31.665Z,1684563991.665 [DAT](FAULT): failed to set remote address 2023-05-20T06:26:47.065Z,1684564007.065 [DAT](FAULT): failed to set remote address 2023-05-20T06:56:33.430Z,1684565793.430 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031893 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 4.000000 count 2023-05-20T06:57:58.681Z,1684565878.681 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.149879 m , settling for 5.000000 min 2023-05-20T07:03:11.453Z,1684566191.453 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T07:03:11.581Z,1684566191.581 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T07:04:00.717Z,1684566240.717 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T07:04:01.528Z,1684566241.528 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -1 2023-05-20T07:04:01.529Z,1684566241.529 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T07:04:01.980Z,1684566241.980 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.107002 m in 9.291681 degC water with 0.078000 ug/l chlorophyll fluorescence. 2023-05-20T07:04:02.334Z,1684566242.334 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T07:04:02.738Z,1684566242.738 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T07:04:04.758Z,1684566244.758 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T07:04:36.718Z,1684566276.718 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@52,aRNAl_bac) 2023-05-20T07:04:39.141Z,1684566279.141 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T07:04:39.142Z,1684566279.142 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T07:04:39.549Z,1684566279.549 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T07:07:27.609Z,1684566447.609 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T07:07:28.013Z,1684566448.013 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T07:13:47.568Z,1684566827.568 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T07:37:39.408Z,1684568259.408 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T07:37:50.423Z,1684568270.423 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.006253 CHAN A1 (24V): 0.002242 CHAN A2 (12V): -0.004381 CHAN A3 (5V): -0.002545 CHAN B0 (3.3V): -0.000873 CHAN B1 (3.15aV): -0.000822 CHAN B2 (3.15bV): -0.000670 CHAN B3 (GND): 0.000140 OPEN: 0.006698 Full Scale: +/- 1 mA 2023-05-20T07:52:56.433Z,1684569176.433 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T07:52:56.838Z,1684569176.838 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T07:52:58.858Z,1684569178.858 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T07:52:59.262Z,1684569179.262 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,aRNAl_bac) 2023-05-20T07:54:34.209Z,1684569274.209 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@52,aRNAl_bac) 2023-05-20T07:54:41.890Z,1684569281.890 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T07:54:42.297Z,1684569282.297 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2023-05-20T07:57:34.076Z,1684569454.076 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-05-20T07:59:42.285Z,1684569582.285 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2023-05-20T07:59:44.750Z,1684569584.750 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 4.000000 count 2023-05-20T07:59:45.151Z,1684569585.151 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T07:59:45.577Z,1684569585.577 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T07:59:45.975Z,1684569585.975 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T07:59:46.365Z,1684569586.365 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.830956 m 2023-05-20T07:59:46.774Z,1684569586.774 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.296289 degC 2023-05-20T07:59:47.193Z,1684569587.193 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.104000 ug/l 2023-05-20T07:59:47.984Z,1684569587.984 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 4.000000 count completed. 2023-05-20T07:59:54.073Z,1684569594.073 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T07:59:59.369Z,1684569599.369 [DAT](FAULT): failed to set remote address 2023-05-20T08:00:04.409Z,1684569604.409 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T08:00:56.353Z,1684569656.353 [DAT](FAULT): failed to set remote address 2023-05-20T08:01:26.853Z,1684569686.853 [DAT](FAULT): failed to set remote address 2023-05-20T08:01:57.096Z,1684569717.096 [DAT](FAULT): failed to set remote address 2023-05-20T08:02:57.641Z,1684569777.641 [DAT](FAULT): failed to set remote address 2023-05-20T08:03:27.912Z,1684569807.912 [DAT](FAULT): failed to set remote address 2023-05-20T08:03:58.154Z,1684569838.154 [DAT](FAULT): failed to set remote address 2023-05-20T08:04:28.413Z,1684569868.413 [DAT](FAULT): failed to set remote address 2023-05-20T08:04:51.913Z,1684569891.913 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T08:04:58.929Z,1684569898.929 [DAT](FAULT): failed to set remote address 2023-05-20T08:05:59.425Z,1684569959.425 [DAT](FAULT): failed to set remote address 2023-05-20T08:06:29.722Z,1684569989.722 [DAT](FAULT): failed to set remote address 2023-05-20T08:06:59.973Z,1684570019.973 [DAT](FAULT): failed to set remote address 2023-05-20T08:07:30.473Z,1684570050.473 [DAT](FAULT): failed to set remote address 2023-05-20T08:08:00.730Z,1684570080.730 [DAT](FAULT): failed to set remote address 2023-05-20T08:08:30.993Z,1684570110.993 [DAT](FAULT): failed to set remote address 2023-05-20T08:09:01.249Z,1684570141.249 [DAT](FAULT): failed to set remote address 2023-05-20T08:09:16.629Z,1684570156.629 [DAT](FAULT): failed to set remote address 2023-05-20T08:09:32.001Z,1684570172.001 [DAT](FAULT): failed to set remote address 2023-05-20T08:09:47.373Z,1684570187.373 [DAT](FAULT): failed to set remote address 2023-05-20T08:10:02.780Z,1684570202.780 [DAT](FAULT): failed to set remote address 2023-05-20T08:10:18.153Z,1684570218.153 [DAT](FAULT): failed to set remote address 2023-05-20T08:10:33.528Z,1684570233.528 [DAT](FAULT): failed to set remote address 2023-05-20T08:10:48.901Z,1684570248.901 [DAT](FAULT): failed to set remote address 2023-05-20T08:11:04.272Z,1684570264.272 [DAT](FAULT): failed to set remote address 2023-05-20T08:11:19.656Z,1684570279.656 [DAT](FAULT): failed to set remote address 2023-05-20T08:11:35.037Z,1684570295.037 [DAT](FAULT): failed to set remote address 2023-05-20T08:11:50.425Z,1684570310.425 [DAT](FAULT): failed to set remote address 2023-05-20T08:12:05.809Z,1684570325.809 [DAT](FAULT): failed to set remote address 2023-05-20T08:12:33.532Z,1684570353.532 [DAT](FAULT): failed to set remote address 2023-05-20T08:12:48.912Z,1684570368.912 [DAT](FAULT): failed to set remote address 2023-05-20T08:13:04.297Z,1684570384.297 [DAT](FAULT): failed to set remote address 2023-05-20T08:13:19.670Z,1684570399.670 [DAT](FAULT): failed to set remote address 2023-05-20T08:13:35.045Z,1684570415.045 [DAT](FAULT): failed to set remote address 2023-05-20T08:13:50.441Z,1684570430.441 [DAT](FAULT): failed to set remote address 2023-05-20T08:14:05.813Z,1684570445.813 [DAT](FAULT): failed to set remote address 2023-05-20T08:14:21.209Z,1684570461.209 [DAT](FAULT): failed to set remote address 2023-05-20T08:14:36.593Z,1684570476.593 [DAT](FAULT): failed to set remote address 2023-05-20T08:14:51.985Z,1684570491.985 [DAT](FAULT): failed to set remote address 2023-05-20T08:15:07.368Z,1684570507.368 [DAT](FAULT): failed to set remote address 2023-05-20T08:15:22.744Z,1684570522.744 [DAT](FAULT): failed to set remote address 2023-05-20T08:16:05.601Z,1684570565.601 [DAT](FAULT): failed to set remote address 2023-05-20T08:16:35.885Z,1684570595.885 [DAT](FAULT): failed to set remote address 2023-05-20T08:16:51.285Z,1684570611.285 [DAT](FAULT): failed to set remote address 2023-05-20T08:17:06.660Z,1684570626.660 [DAT](FAULT): failed to set remote address 2023-05-20T08:17:36.409Z,1684570656.409 [DAT](FAULT): failed to set remote address 2023-05-20T08:17:51.784Z,1684570671.784 [DAT](FAULT): failed to set remote address 2023-05-20T08:47:21.544Z,1684572441.544 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T08:47:21.545Z,1684572441.545 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T08:56:36.121Z,1684572996.121 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031169 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 5.000000 count 2023-05-20T09:02:57.583Z,1684573377.583 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.595490 m , settling for 5.000000 min 2023-05-20T09:08:10.352Z,1684573690.352 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T09:08:10.494Z,1684573690.494 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T09:08:59.605Z,1684573739.605 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T09:09:00.418Z,1684573740.418 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -1 2023-05-20T09:09:00.418Z,1684573740.418 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T09:09:00.866Z,1684573740.866 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.693985 m in 9.279566 degC water with 0.076960 ug/l chlorophyll fluorescence. 2023-05-20T09:09:01.221Z,1684573741.221 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T09:09:01.626Z,1684573741.626 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T09:09:03.645Z,1684573743.645 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T09:09:35.567Z,1684573775.567 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac) 2023-05-20T09:09:38.394Z,1684573778.394 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T09:09:38.395Z,1684573778.395 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T09:09:38.798Z,1684573778.798 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T09:12:26.093Z,1684573946.093 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T09:12:26.498Z,1684573946.498 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T09:17:15.572Z,1684574235.572 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T09:17:15.572Z,1684574235.572 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T09:29:30.568Z,1684574970.568 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T09:29:30.568Z,1684574970.568 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T09:57:35.534Z,1684576655.534 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T09:57:35.938Z,1684576655.938 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T09:57:37.958Z,1684576657.958 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T09:57:38.361Z,1684576658.361 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac) 2023-05-20T09:59:13.309Z,1684576753.309 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@51,aRNAl_bac) 2023-05-20T09:59:20.984Z,1684576760.984 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T09:59:21.386Z,1684576761.386 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2023-05-20T10:04:21.172Z,1684577061.172 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2023-05-20T10:04:23.633Z,1684577063.633 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 5.000000 count 2023-05-20T10:04:24.038Z,1684577064.038 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T10:04:24.447Z,1684577064.447 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T10:04:24.865Z,1684577064.865 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T10:04:25.249Z,1684577065.249 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.444611 m 2023-05-20T10:04:25.654Z,1684577065.654 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.307306 degC 2023-05-20T10:04:26.061Z,1684577066.061 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.083200 ug/l 2023-05-20T10:04:26.858Z,1684577066.858 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 5.000000 count completed. 2023-05-20T10:04:33.144Z,1684577073.144 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T10:04:38.444Z,1684577078.444 [DAT](FAULT): failed to set remote address 2023-05-20T10:04:43.484Z,1684577083.484 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T10:04:58.612Z,1684577098.612 [DAT](FAULT): failed to set remote address 2023-05-20T10:05:14.019Z,1684577114.019 [DAT](FAULT): failed to set remote address 2023-05-20T10:05:29.409Z,1684577129.409 [DAT](FAULT): failed to set remote address 2023-05-20T10:05:44.788Z,1684577144.788 [DAT](FAULT): failed to set remote address 2023-05-20T10:06:00.161Z,1684577160.161 [DAT](FAULT): failed to set remote address 2023-05-20T10:06:15.541Z,1684577175.541 [DAT](FAULT): failed to set remote address 2023-05-20T10:06:30.916Z,1684577190.916 [DAT](FAULT): failed to set remote address 2023-05-20T10:06:46.293Z,1684577206.293 [DAT](FAULT): failed to set remote address 2023-05-20T10:07:01.685Z,1684577221.685 [DAT](FAULT): failed to set remote address 2023-05-20T10:07:17.065Z,1684577237.065 [DAT](FAULT): failed to set remote address 2023-05-20T10:07:32.444Z,1684577252.444 [DAT](FAULT): failed to set remote address 2023-05-20T10:07:47.816Z,1684577267.816 [DAT](FAULT): failed to set remote address 2023-05-20T10:08:12.035Z,1684577292.035 [DAT](FAULT): failed to set remote address 2023-05-20T10:08:42.257Z,1684577322.257 [DAT](FAULT): failed to set remote address 2023-05-20T10:09:12.532Z,1684577352.532 [DAT](FAULT): failed to set remote address 2023-05-20T10:09:30.692Z,1684577370.692 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T10:09:43.056Z,1684577383.056 [DAT](FAULT): failed to set remote address 2023-05-20T10:10:13.316Z,1684577413.316 [DAT](FAULT): failed to set remote address 2023-05-20T10:10:43.593Z,1684577443.593 [DAT](FAULT): failed to set remote address 2023-05-20T10:11:13.860Z,1684577473.860 [DAT](FAULT): failed to set remote address 2023-05-20T10:11:44.633Z,1684577504.633 [DAT](FAULT): failed to set remote address 2023-05-20T10:12:14.885Z,1684577534.885 [DAT](FAULT): failed to set remote address 2023-05-20T10:12:45.136Z,1684577565.136 [DAT](FAULT): failed to set remote address 2023-05-20T10:13:15.393Z,1684577595.393 [DAT](FAULT): failed to set remote address 2023-05-20T10:13:45.660Z,1684577625.660 [DAT](FAULT): failed to set remote address 2023-05-20T10:14:16.154Z,1684577656.154 [DAT](FAULT): failed to set remote address 2023-05-20T10:15:16.656Z,1684577716.656 [DAT](FAULT): failed to set remote address 2023-05-20T10:15:46.941Z,1684577746.941 [DAT](FAULT): failed to set remote address 2023-05-20T10:16:17.188Z,1684577777.188 [DAT](FAULT): failed to set remote address 2023-05-20T10:16:47.440Z,1684577807.440 [DAT](FAULT): failed to set remote address 2023-05-20T10:17:17.956Z,1684577837.956 [DAT](FAULT): failed to set remote address 2023-05-20T10:17:48.217Z,1684577868.217 [DAT](FAULT): failed to set remote address 2023-05-20T10:18:18.465Z,1684577898.465 [DAT](FAULT): failed to set remote address 2023-05-20T10:18:48.744Z,1684577928.744 [DAT](FAULT): failed to set remote address 2023-05-20T10:19:19.029Z,1684577959.029 [DAT](FAULT): failed to set remote address 2023-05-20T10:19:49.293Z,1684577989.293 [DAT](FAULT): failed to set remote address 2023-05-20T10:20:19.309Z,1684578019.309 [DAT](FAULT): failed to set remote address 2023-05-20T10:20:49.578Z,1684578049.578 [DAT](FAULT): failed to set remote address 2023-05-20T10:21:19.845Z,1684578079.845 [DAT](FAULT): failed to set remote address 2023-05-20T10:21:50.089Z,1684578110.089 [DAT](FAULT): failed to set remote address 2023-05-20T10:22:20.601Z,1684578140.601 [DAT](FAULT): failed to set remote address 2023-05-20T10:22:50.605Z,1684578170.605 [DAT](FAULT): failed to set remote address 2023-05-20T10:23:05.992Z,1684578185.992 [DAT](FAULT): failed to set remote address 2023-05-20T10:23:21.396Z,1684578201.396 [DAT](FAULT): failed to set remote address 2023-05-20T10:23:36.769Z,1684578216.769 [DAT](FAULT): failed to set remote address 2023-05-20T10:23:52.150Z,1684578232.150 [DAT](FAULT): failed to set remote address 2023-05-20T10:24:07.528Z,1684578247.528 [DAT](FAULT): failed to set remote address 2023-05-20T10:24:22.905Z,1684578262.905 [DAT](FAULT): failed to set remote address 2023-05-20T10:24:38.276Z,1684578278.276 [DAT](FAULT): failed to set remote address 2023-05-20T10:24:53.652Z,1684578293.652 [DAT](FAULT): failed to set remote address 2023-05-20T10:25:09.033Z,1684578309.033 [DAT](FAULT): failed to set remote address 2023-05-20T10:25:24.425Z,1684578324.425 [DAT](FAULT): failed to set remote address 2023-05-20T10:25:39.809Z,1684578339.809 [DAT](FAULT): failed to set remote address 2023-05-20T10:25:55.219Z,1684578355.219 [DAT](FAULT): failed to set remote address 2023-05-20T10:26:22.692Z,1684578382.692 [DAT](FAULT): failed to set remote address 2023-05-20T10:26:38.073Z,1684578398.073 [DAT](FAULT): failed to set remote address 2023-05-20T10:26:53.449Z,1684578413.449 [DAT](FAULT): failed to set remote address 2023-05-20T10:27:23.245Z,1684578443.245 [DAT](FAULT): failed to set remote address 2023-05-20T10:56:38.614Z,1684580198.614 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027767 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 6.000000 count 2023-05-20T10:59:24.658Z,1684580364.658 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.103264 m , settling for 5.000000 min 2023-05-20T11:04:37.416Z,1684580677.416 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T11:04:37.532Z,1684580677.532 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T11:05:26.622Z,1684580726.622 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T11:05:27.432Z,1684580727.432 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2023-05-20T11:05:27.433Z,1684580727.433 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T11:05:27.882Z,1684580727.882 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.164452 m in 9.294153 degC water with 0.067600 ug/l chlorophyll fluorescence. 2023-05-20T11:05:28.238Z,1684580728.238 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T11:05:28.641Z,1684580728.641 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T11:05:30.662Z,1684580730.662 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T11:05:35.080Z,1684580735.080 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-05-20T11:06:02.707Z,1684580762.707 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@50,aRNAl_bac) 2023-05-20T11:06:05.529Z,1684580765.529 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T11:06:05.530Z,1684580765.530 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T11:06:05.937Z,1684580765.937 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T11:09:09.349Z,1684580949.349 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T11:09:09.755Z,1684580949.755 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T11:44:16.571Z,1684583056.571 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T11:44:16.572Z,1684583056.572 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T11:54:25.587Z,1684583665.587 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T11:54:25.992Z,1684583665.992 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T11:54:28.010Z,1684583668.010 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T11:54:28.414Z,1684583668.414 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,aRNAl_bac) 2023-05-20T11:56:03.769Z,1684583763.769 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@50,aRNAl_bac) 2023-05-20T11:56:11.446Z,1684583771.446 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T11:56:11.859Z,1684583771.859 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2023-05-20T12:01:11.701Z,1684584071.701 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2023-05-20T12:01:14.180Z,1684584074.180 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 6.000000 count 2023-05-20T12:01:14.563Z,1684584074.563 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T12:01:14.969Z,1684584074.969 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T12:01:15.383Z,1684584075.383 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T12:01:15.801Z,1684584075.801 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.129318 m 2023-05-20T12:01:16.179Z,1684584076.179 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.290003 degC 2023-05-20T12:01:16.716Z,1684584076.716 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.093600 ug/l 2023-05-20T12:01:17.403Z,1684584077.403 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 6.000000 count completed. 2023-05-20T12:01:23.697Z,1684584083.697 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T12:01:34.029Z,1684584094.029 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T12:01:49.904Z,1684584109.904 [DAT](FAULT): failed to set remote address 2023-05-20T12:02:20.153Z,1684584140.153 [DAT](FAULT): failed to set remote address 2023-05-20T12:02:50.429Z,1684584170.429 [DAT](FAULT): failed to set remote address 2023-05-20T12:03:20.441Z,1684584200.441 [DAT](FAULT): failed to set remote address 2023-05-20T12:03:50.717Z,1684584230.717 [DAT](FAULT): failed to set remote address 2023-05-20T12:04:20.961Z,1684584260.961 [DAT](FAULT): failed to set remote address 2023-05-20T12:04:51.469Z,1684584291.469 [DAT](FAULT): failed to set remote address 2023-05-20T12:05:21.740Z,1684584321.740 [DAT](FAULT): failed to set remote address 2023-05-20T12:05:52.008Z,1684584352.008 [DAT](FAULT): failed to set remote address 2023-05-20T12:06:21.378Z,1684584381.378 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T12:06:22.268Z,1684584382.268 [DAT](FAULT): failed to set remote address 2023-05-20T12:06:52.532Z,1684584412.532 [DAT](FAULT): failed to set remote address 2023-05-20T12:07:22.820Z,1684584442.820 [DAT](FAULT): failed to set remote address 2023-05-20T12:07:53.089Z,1684584473.089 [DAT](FAULT): failed to set remote address 2023-05-20T12:08:23.604Z,1684584503.604 [DAT](FAULT): failed to set remote address 2023-05-20T12:08:53.868Z,1684584533.868 [DAT](FAULT): failed to set remote address 2023-05-20T12:09:24.157Z,1684584564.157 [DAT](FAULT): failed to set remote address 2023-05-20T12:09:54.406Z,1684584594.406 [DAT](FAULT): failed to set remote address 2023-05-20T12:10:24.652Z,1684584624.652 [DAT](FAULT): failed to set remote address 2023-05-20T12:10:54.905Z,1684584654.905 [DAT](FAULT): failed to set remote address 2023-05-20T12:11:25.413Z,1684584685.413 [DAT](FAULT): failed to set remote address 2023-05-20T12:11:55.673Z,1684584715.673 [DAT](FAULT): failed to set remote address 2023-05-20T12:12:25.941Z,1684584745.941 [DAT](FAULT): failed to set remote address 2023-05-20T12:12:56.209Z,1684584776.209 [DAT](FAULT): failed to set remote address 2023-05-20T12:13:26.455Z,1684584806.455 [DAT](FAULT): failed to set remote address 2023-05-20T12:13:56.737Z,1684584836.737 [DAT](FAULT): failed to set remote address 2023-05-20T12:14:26.985Z,1684584866.985 [DAT](FAULT): failed to set remote address 2023-05-20T12:15:27.753Z,1684584927.753 [DAT](FAULT): failed to set remote address 2023-05-20T12:15:57.996Z,1684584957.996 [DAT](FAULT): failed to set remote address 2023-05-20T12:16:28.237Z,1684584988.237 [DAT](FAULT): failed to set remote address 2023-05-20T12:16:58.493Z,1684585018.493 [DAT](FAULT): failed to set remote address 2023-05-20T12:17:28.992Z,1684585048.992 [DAT](FAULT): failed to set remote address 2023-05-20T12:17:59.241Z,1684585079.241 [DAT](FAULT): failed to set remote address 2023-05-20T12:18:29.505Z,1684585109.505 [DAT](FAULT): failed to set remote address 2023-05-20T12:18:59.780Z,1684585139.780 [DAT](FAULT): failed to set remote address 2023-05-20T12:19:30.057Z,1684585170.057 [DAT](FAULT): failed to set remote address 2023-05-20T12:20:00.553Z,1684585200.553 [DAT](FAULT): failed to set remote address 2023-05-20T12:20:30.814Z,1684585230.814 [DAT](FAULT): failed to set remote address 2023-05-20T12:21:01.060Z,1684585261.060 [DAT](FAULT): failed to set remote address 2023-05-20T12:21:31.342Z,1684585291.342 [DAT](FAULT): failed to set remote address 2023-05-20T12:22:01.601Z,1684585321.601 [DAT](FAULT): failed to set remote address 2023-05-20T12:22:32.109Z,1684585352.109 [DAT](FAULT): failed to set remote address 2023-05-20T12:22:47.484Z,1684585367.484 [DAT](FAULT): failed to set remote address 2023-05-20T12:23:02.868Z,1684585382.868 [DAT](FAULT): failed to set remote address 2023-05-20T12:23:32.616Z,1684585412.616 [DAT](FAULT): failed to set remote address 2023-05-20T12:56:41.298Z,1684587401.298 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030599 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 7.000000 count 2023-05-20T12:57:42.300Z,1684587462.300 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 100.146370 m , settling for 5.000000 min 2023-05-20T13:02:55.069Z,1684587775.069 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T13:02:55.205Z,1684587775.205 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T13:03:41.148Z,1684587821.148 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T13:03:41.963Z,1684587821.963 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2023-05-20T13:03:41.965Z,1684587821.965 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T13:03:42.402Z,1684587822.402 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.955276 m in 9.304285 degC water with 0.070720 ug/l chlorophyll fluorescence. 2023-05-20T13:03:42.765Z,1684587822.765 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T13:03:43.169Z,1684587823.169 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T13:03:45.191Z,1684587825.191 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T13:04:17.173Z,1684587857.173 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@49,aRNAl_bac) 2023-05-20T13:04:20.001Z,1684587860.001 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T13:04:20.002Z,1684587860.002 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T13:04:20.408Z,1684587860.408 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T13:07:08.089Z,1684588028.089 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T13:07:08.489Z,1684588028.489 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T13:24:15.569Z,1684589055.569 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T13:24:15.570Z,1684589055.570 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T13:33:13.596Z,1684589593.596 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T13:37:50.792Z,1684589870.792 [CBIT](IMPORTANT): Beginning ground fault scan 2023-05-20T13:38:01.623Z,1684589881.623 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.011066 CHAN A1 (24V): 0.021821 CHAN A2 (12V): 0.005566 CHAN A3 (5V): 0.003894 CHAN B0 (3.3V): 0.003687 CHAN B1 (3.15aV): 0.003635 CHAN B2 (3.15bV): 0.003591 CHAN B3 (GND): -0.000963 OPEN: 0.007054 Full Scale: +/- 1 mA 2023-05-20T13:53:58.989Z,1684590838.989 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T13:53:59.397Z,1684590839.397 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T13:54:01.419Z,1684590841.419 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T13:54:01.817Z,1684590841.817 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,aRNAl_bac) 2023-05-20T13:55:36.758Z,1684590936.758 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@49,aRNAl_bac) 2023-05-20T13:55:44.434Z,1684590944.434 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T13:55:44.838Z,1684590944.838 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2023-05-20T14:00:44.617Z,1684591244.617 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2023-05-20T14:00:47.080Z,1684591247.080 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 7.000000 count 2023-05-20T14:00:47.495Z,1684591247.495 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T14:00:47.900Z,1684591247.900 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T14:00:48.299Z,1684591248.299 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T14:00:48.699Z,1684591248.699 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.576271 m 2023-05-20T14:00:49.114Z,1684591249.114 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.318415 degC 2023-05-20T14:00:49.532Z,1684591249.532 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.062400 ug/l 2023-05-20T14:00:50.317Z,1684591250.317 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 7.000000 count completed. 2023-05-20T14:00:56.548Z,1684591256.548 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T14:01:06.864Z,1684591266.864 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T14:01:21.989Z,1684591281.989 [DAT](FAULT): failed to set remote address 2023-05-20T14:01:37.392Z,1684591297.392 [DAT](FAULT): failed to set remote address 2023-05-20T14:01:52.769Z,1684591312.769 [DAT](FAULT): failed to set remote address 2023-05-20T14:02:08.141Z,1684591328.141 [DAT](FAULT): failed to set remote address 2023-05-20T14:02:23.542Z,1684591343.542 [DAT](FAULT): failed to set remote address 2023-05-20T14:02:38.921Z,1684591358.921 [DAT](FAULT): failed to set remote address 2023-05-20T14:02:54.297Z,1684591374.297 [DAT](FAULT): failed to set remote address 2023-05-20T14:03:09.669Z,1684591389.669 [DAT](FAULT): failed to set remote address 2023-05-20T14:03:25.060Z,1684591405.060 [DAT](FAULT): failed to set remote address 2023-05-20T14:03:40.440Z,1684591420.440 [DAT](FAULT): failed to set remote address 2023-05-20T14:03:55.828Z,1684591435.828 [DAT](FAULT): failed to set remote address 2023-05-20T14:04:33.645Z,1684591473.645 [DAT](FAULT): failed to set remote address 2023-05-20T14:05:03.894Z,1684591503.894 [DAT](FAULT): failed to set remote address 2023-05-20T14:05:34.140Z,1684591534.140 [DAT](FAULT): failed to set remote address 2023-05-20T14:05:54.144Z,1684591554.144 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T14:06:04.404Z,1684591564.404 [DAT](FAULT): failed to set remote address 2023-05-20T14:06:34.464Z,1684591594.464 [DAT](FAULT): failed to set remote address 2023-05-20T14:07:04.696Z,1684591624.696 [DAT](FAULT): failed to set remote address 2023-05-20T14:07:34.965Z,1684591654.965 [DAT](FAULT): failed to set remote address 2023-05-20T14:08:05.217Z,1684591685.217 [DAT](FAULT): failed to set remote address 2023-05-20T14:08:35.469Z,1684591715.469 [DAT](FAULT): failed to set remote address 2023-05-20T14:09:05.754Z,1684591745.754 [DAT](FAULT): failed to set remote address 2023-05-20T14:09:36.261Z,1684591776.261 [DAT](FAULT): failed to set remote address 2023-05-20T14:10:06.501Z,1684591806.501 [DAT](FAULT): failed to set remote address 2023-05-20T14:10:36.793Z,1684591836.793 [DAT](FAULT): failed to set remote address 2023-05-20T14:11:07.091Z,1684591867.091 [DAT](FAULT): failed to set remote address 2023-05-20T14:11:37.314Z,1684591897.314 [DAT](FAULT): failed to set remote address 2023-05-20T14:12:07.570Z,1684591927.570 [DAT](FAULT): failed to set remote address 2023-05-20T14:12:38.073Z,1684591958.073 [DAT](FAULT): failed to set remote address 2023-05-20T14:13:08.321Z,1684591988.321 [DAT](FAULT): failed to set remote address 2023-05-20T14:13:38.601Z,1684592018.601 [DAT](FAULT): failed to set remote address 2023-05-20T14:14:08.901Z,1684592048.901 [DAT](FAULT): failed to set remote address 2023-05-20T14:14:39.177Z,1684592079.177 [DAT](FAULT): failed to set remote address 2023-05-20T14:15:09.429Z,1684592109.429 [DAT](FAULT): failed to set remote address 2023-05-20T14:15:39.681Z,1684592139.681 [DAT](FAULT): failed to set remote address 2023-05-20T14:16:09.941Z,1684592169.941 [DAT](FAULT): failed to set remote address 2023-05-20T14:16:40.445Z,1684592200.445 [DAT](FAULT): failed to set remote address 2023-05-20T14:17:10.696Z,1684592230.696 [DAT](FAULT): failed to set remote address 2023-05-20T14:17:26.077Z,1684592246.077 [DAT](FAULT): failed to set remote address 2023-05-20T14:17:41.461Z,1684592261.461 [DAT](FAULT): failed to set remote address 2023-05-20T14:18:11.222Z,1684592291.222 [DAT](FAULT): failed to set remote address 2023-05-20T14:18:26.593Z,1684592306.593 [DAT](FAULT): failed to set remote address 2023-05-20T14:18:41.977Z,1684592321.977 [DAT](FAULT): failed to set remote address 2023-05-20T14:19:11.981Z,1684592351.981 [DAT](FAULT): failed to set remote address 2023-05-20T14:19:42.241Z,1684592382.241 [DAT](FAULT): failed to set remote address 2023-05-20T14:56:43.764Z,1684594603.764 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028215 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 8.000000 count 2023-05-20T15:02:49.815Z,1684594969.815 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.687546 m , settling for 5.000000 min 2023-05-20T15:08:02.568Z,1684595282.568 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T15:08:02.713Z,1684595282.713 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T15:08:49.809Z,1684595329.809 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T15:08:50.622Z,1684595330.622 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2023-05-20T15:08:50.623Z,1684595330.623 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T15:08:51.065Z,1684595331.065 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.818970 m in 9.300837 degC water with 0.067600 ug/l chlorophyll fluorescence. 2023-05-20T15:08:51.425Z,1684595331.425 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T15:08:51.829Z,1684595331.829 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T15:08:53.849Z,1684595333.849 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T15:09:25.766Z,1684595365.766 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@48,aRNAl_bac) 2023-05-20T15:09:27.785Z,1684595367.785 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T15:09:28.190Z,1684595368.190 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T15:09:28.596Z,1684595368.596 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T15:12:16.322Z,1684595536.322 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T15:12:16.726Z,1684595536.726 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T16:00:13.137Z,1684598413.137 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1000.0) 2023-05-20T16:00:13.543Z,1684598413.543 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1000.0) 2023-05-20T16:00:15.564Z,1684598415.564 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T16:00:15.966Z,1684598415.966 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,aRNAl_bac) 2023-05-20T16:01:50.913Z,1684598510.913 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@48,aRNAl_bac) 2023-05-20T16:01:58.631Z,1684598518.631 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T16:01:59.034Z,1684598519.034 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2023-05-20T16:03:36.712Z,1684598616.712 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-05-20T16:03:36.712Z,1684598616.712 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-05-20T16:06:58.825Z,1684598818.825 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2023-05-20T16:07:01.306Z,1684598821.306 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 8.000000 count 2023-05-20T16:07:01.707Z,1684598821.707 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2023-05-20T16:07:02.085Z,1684598822.085 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2023-05-20T16:07:02.497Z,1684598822.497 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2023-05-20T16:07:02.913Z,1684598822.913 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.798820 m 2023-05-20T16:07:03.337Z,1684598823.337 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.216303 degC 2023-05-20T16:07:03.707Z,1684598823.707 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.083200 ug/l 2023-05-20T16:07:04.543Z,1684598824.543 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 8.000000 count completed. 2023-05-20T16:07:10.717Z,1684598830.717 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:07:16.011Z,1684598836.011 [DAT](FAULT): failed to set remote address 2023-05-20T16:07:21.056Z,1684598841.056 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:08:15.017Z,1684598895.017 [DAT](FAULT): failed to set remote address 2023-05-20T16:08:45.282Z,1684598925.282 [DAT](FAULT): failed to set remote address 2023-05-20T16:09:15.537Z,1684598955.537 [DAT](FAULT): failed to set remote address 2023-05-20T16:09:45.793Z,1684598985.793 [DAT](FAULT): failed to set remote address 2023-05-20T16:10:16.037Z,1684599016.037 [DAT](FAULT): failed to set remote address 2023-05-20T16:10:46.555Z,1684599046.555 [DAT](FAULT): failed to set remote address 2023-05-20T16:11:16.788Z,1684599076.788 [DAT](FAULT): failed to set remote address 2023-05-20T16:12:08.355Z,1684599128.355 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min . 2023-05-20T16:12:17.292Z,1684599137.292 [DAT](FAULT): failed to set remote address 2023-05-20T16:12:32.660Z,1684599152.660 [DAT](FAULT): failed to set remote address 2023-05-20T16:12:48.041Z,1684599168.041 [DAT](FAULT): failed to set remote address 2023-05-20T16:13:18.117Z,1684599198.117 [DAT](FAULT): failed to set remote address 2023-05-20T16:13:33.493Z,1684599213.493 [DAT](FAULT): failed to set remote address 2023-05-20T16:13:48.877Z,1684599228.877 [DAT](FAULT): failed to set remote address 2023-05-20T16:14:18.648Z,1684599258.648 [DAT](FAULT): failed to set remote address 2023-05-20T16:14:48.893Z,1684599288.893 [DAT](FAULT): failed to set remote address 2023-05-20T16:15:04.268Z,1684599304.268 [DAT](FAULT): failed to set remote address 2023-05-20T16:15:19.660Z,1684599319.660 [DAT](FAULT): failed to set remote address 2023-05-20T16:15:49.437Z,1684599349.437 [DAT](FAULT): failed to set remote address 2023-05-20T16:16:19.732Z,1684599379.732 [DAT](FAULT): failed to set remote address 2023-05-20T16:16:50.241Z,1684599410.241 [DAT](FAULT): failed to set remote address 2023-05-20T16:17:20.509Z,1684599440.509 [DAT](FAULT): failed to set remote address 2023-05-20T16:17:35.897Z,1684599455.897 [DAT](FAULT): failed to set remote address 2023-05-20T16:17:51.268Z,1684599471.268 [DAT](FAULT): failed to set remote address 2023-05-20T16:18:21.012Z,1684599501.012 [DAT](FAULT): failed to set remote address 2023-05-20T16:18:51.274Z,1684599531.274 [DAT](FAULT): failed to set remote address 2023-05-20T16:19:21.773Z,1684599561.773 [DAT](FAULT): failed to set remote address 2023-05-20T16:19:52.029Z,1684599592.029 [DAT](FAULT): failed to set remote address 2023-05-20T16:20:07.413Z,1684599607.413 [DAT](FAULT): failed to set remote address 2023-05-20T16:20:22.793Z,1684599622.793 [DAT](FAULT): failed to set remote address 2023-05-20T16:20:38.169Z,1684599638.169 [DAT](FAULT): failed to set remote address 2023-05-20T16:20:53.541Z,1684599653.541 [DAT](FAULT): failed to set remote address 2023-05-20T16:21:08.914Z,1684599668.914 [DAT](FAULT): failed to set remote address 2023-05-20T16:21:24.293Z,1684599684.293 [DAT](FAULT): failed to set remote address 2023-05-20T16:21:53.297Z,1684599713.297 [DAT](FAULT): failed to set remote address 2023-05-20T16:22:08.689Z,1684599728.689 [DAT](FAULT): failed to set remote address 2023-05-20T16:22:24.065Z,1684599744.065 [DAT](FAULT): failed to set remote address 2023-05-20T16:22:53.824Z,1684599773.824 [DAT](FAULT): failed to set remote address 2023-05-20T16:23:24.077Z,1684599804.077 [DAT](FAULT): failed to set remote address 2023-05-20T16:23:54.341Z,1684599834.341 [DAT](FAULT): failed to set remote address 2023-05-20T16:27:02.267Z,1684600022.267 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-05-20T16:56:46.345Z,1684601806.345 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028971 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 9.000000 count 2023-05-20T16:58:44.969Z,1684601924.969 [CommandExec](IMPORTANT): got command 2023-05-20T16:58:44.969Z,1684601924.969 [CommandExec](FAULT): Incomplete syntax. Try: help 2023-05-20T16:58:54.996Z,1684601934.996 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:59:10.126Z,1684601950.126 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 25 second 2023-05-20T16:59:15.428Z,1684601955.428 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T16:59:25.534Z,1684601965.534 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.788994 m , settling for 5.000000 min 2023-05-20T16:59:30.657Z,1684601970.657 [CommandExec](IMPORTANT): got command failComponent hardware DAT 2023-05-20T16:59:30.664Z,1684601970.664 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault 2023-05-20T16:59:52.027Z,1684601992.027 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:02:04.261Z,1684602124.261 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 45 second 2023-05-20T17:04:38.316Z,1684602278.316 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-05-20T17:04:38.430Z,1684602278.430 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-05-20T17:05:28.337Z,1684602328.337 [ESPComponent](IMPORTANT): ESP has connected as client 2023-05-20T17:05:29.148Z,1684602329.148 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2023-05-20T17:05:29.148Z,1684602329.148 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-05-20T17:05:29.618Z,1684602329.618 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.377945 m in 9.076136 degC water with 0.076960 ug/l chlorophyll fluorescence. 2023-05-20T17:05:29.953Z,1684602329.953 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2023-05-20T17:05:30.357Z,1684602330.357 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-05-20T17:05:32.377Z,1684602332.377 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2023-05-20T17:06:04.294Z,1684602364.294 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@47,aRNAl_bac) 2023-05-20T17:06:07.126Z,1684602367.126 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2023-05-20T17:06:07.127Z,1684602367.127 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-05-20T17:06:07.525Z,1684602367.525 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-05-20T17:08:55.207Z,1684602535.207 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2023-05-20T17:08:55.611Z,1684602535.611 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-05-20T17:31:00.283Z,1684603860.283 [CommandExec](IMPORTANT): got command failComponent hardware DAT 2023-05-20T17:31:00.284Z,1684603860.284 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault 2023-05-20T17:31:35.493Z,1684603895.493 [DAT](FAULT): failed to enter command mode 2023-05-20T17:31:37.258Z,1684603897.258 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:32:37.865Z,1684603957.865 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 35 second 2023-05-20T17:33:41.168Z,1684604021.168 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 35 second 2023-05-20T17:33:47.638Z,1684604027.638 [DAT](FAULT): Buffer send receipt timeout failure. 2023-05-20T17:33:57.970Z,1684604037.970 [DAT](FAULT): Sending transmit timeout failure. 2023-05-20T17:43:59.979Z,1684604639.979 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 35 second 2023-05-20T17:44:49.924Z,1684604689.924 [CommandExec](IMPORTANT): got command set circle_sample.TrackingUpdatePeriod 35 second 2023-05-20T17:44:56.108Z,1684604696.108 [DAT](FAULT): Buffer send receipt timeout failure. 2023-05-20T17:48:29.658Z,1684604909.658 [CBIT](FAULT): WATER ALARM AUX. 2023-05-20T17:48:29.983Z,1684604909.983 [CBIT](FAULT): WATER ALARM AUX. 2023-05-20T17:48:30.379Z,1684604910.379 [CBIT](FAULT): WATER ALARM AUX. 2023-05-20T17:48:30.771Z,1684604910.771 [CBIT](FAULT): WATER ALARM AUX. 2023-05-20T17:48:30.783Z,1684604910.783 [CBIT](CRITICAL): WATER DETECTED IN PRESSURE HULL. BURNWIRE ACTIVATED 2023-05-20T17:48:30.785Z,1684604910.785 [CBIT](IMPORTANT): Powered off ESP with command: ! echo 0 > /dev/loadA6 2023-05-20T17:48:30.825Z,1684604910.825 [CommandExec](IMPORTANT): got command ! echo 0 > /dev/loadA6 2023-05-20T17:48:31.627Z,1684604911.627 [CommandExec](FAULT): Scheduling is paused 2023-05-20T17:48:31.639Z,1684604911.639 [ESPComponent](IMPORTANT): [sample #11] Stop sampling requested. 2023-05-20T17:48:31.683Z,1684604911.683 [MissionManager](IMPORTANT): Started mission Default 2023-05-20T17:48:32.041Z,1684604912.041 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.pauseFiltering 2023-05-20T17:48:38.101Z,1684604918.101 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@882.9) 2023-05-20T17:48:38.506Z,1684604918.506 [ESPComponent](IMPORTANT): In S_PAUSED received result: 882.9 2023-05-20T17:48:38.909Z,1684604918.909 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@882.9) 2023-05-20T17:48:40.931Z,1684604920.931 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-05-20T17:48:41.334Z,1684604921.334 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@47,aRNAl_bac) 2023-05-20T17:49:14.075Z,1684604954.075 [DropWeight](CRITICAL): DROP WEIGHT MISSING. 2023-05-20T17:49:14.232Z,1684604954.232 [CBIT](CRITICAL): Hardware Fault in component: DropWeight 2023-05-20T17:50:16.275Z,1684605016.275 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@47,aRNAl_bac) 2023-05-20T17:50:23.953Z,1684605023.953 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-05-20T17:50:24.356Z,1684605024.356 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2023-05-20T17:55:24.129Z,1684605324.129 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2023-05-20T17:56:45.689Z,1684605405.689 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-05-20T18:04:15.829Z,1684605855.829 [DropWeight](CRITICAL): DROP WEIGHT MISSING. 2023-05-20T18:04:15.877Z,1684605855.877 [CBIT](CRITICAL): Hardware Fault in component: DropWeight 2023-05-20T18:08:26.224Z,1684606106.224 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5380, MT Status=1, MTMSN=2701 2023-05-20T18:08:26.900Z,1684606106.900 [CommandExec](IMPORTANT): got command schedule clear 2023-05-20T18:08:26.900Z,1684606106.900 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2023-05-20T18:08:26.901Z,1684606106.901 [CommandExec](IMPORTANT): got command schedule resume 2023-05-20T18:08:26.901Z,1684606106.901 [CommandExec](IMPORTANT): Scheduling is resumed 2023-05-20T18:17:49.173Z,1684606669.173 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=5381, MT Status=1, MTMSN=2702 2023-05-20T18:17:50.054Z,1684606670.054 [CommandExec](IMPORTANT): got command burn off 2023-05-20T18:17:50.054Z,1684606670.054 [CommandExec](IMPORTANT): Deactivating dropweight wire 2023-05-20T18:17:50.071Z,1684606670.071 [CommandExec](IMPORTANT): got command configSet DropWeight.loadAtStartup 0 bool persist 2023-05-20T18:17:50.071Z,1684606670.071 [CommandExec](IMPORTANT): configSet DropWeight.loadAtStartup requires a restart to take effect. 2023-05-20T18:17:50.078Z,1684606670.078 [CommandExec](IMPORTANT): got command restart application