2024-09-21T06:10:00.455Z,1726899000.455 [CommandExec](IMPORTANT): got command restart logs 2024-09-21T06:20:19.734Z,1726899619.734 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028573 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 7.000000 count 2024-09-21T06:21:06.213Z,1726899666.213 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.653458 m , settling for 1.000000 min 2024-09-21T06:22:18.985Z,1726899738.985 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T06:22:19.101Z,1726899739.101 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T06:23:11.124Z,1726899791.124 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T06:23:11.934Z,1726899791.934 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -1 2024-09-21T06:23:11.934Z,1726899791.934 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T06:23:12.398Z,1726899792.398 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.319839 m in 9.345453 degC water with 0.056160 ug/l chlorophyll fluorescence. 2024-09-21T06:23:12.738Z,1726899792.738 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T06:23:13.143Z,1726899793.143 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T06:23:15.163Z,1726899795.163 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T06:23:46.680Z,1726899826.680 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@44,aRNAl_bac) 2024-09-21T06:23:49.103Z,1726899829.103 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T06:23:49.104Z,1726899829.104 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T06:23:49.507Z,1726899829.507 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T06:28:08.073Z,1726900088.073 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@0.0) 2024-09-21T06:28:15.749Z,1726900095.749 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T06:28:16.152Z,1726900096.152 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2024-09-21T06:33:15.932Z,1726900395.932 [ESPComponent](CRITICAL): [sample #12] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 261.3962 2024-09-21T06:33:16.055Z,1726900396.055 [CommandExec](FAULT): Scheduling is paused for 1 commands 2024-09-21T06:33:17.049Z,1726900397.049 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T06:38:35.461Z,1726900715.461 [ElevatorServo](FAULT): Overload Error 2024-09-21T06:48:42.438Z,1726901322.438 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17671, MT Status=1, MTMSN=4160 2024-09-21T06:48:43.015Z,1726901323.015 [CommandExec](IMPORTANT): got command schedule resume 2024-09-21T06:48:43.015Z,1726901323.015 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-21T06:51:54.888Z,1726901514.888 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17681, MT Status=1, MTMSN=4161 2024-09-21T06:51:55.634Z,1726901515.634 [CommandExec](IMPORTANT): got command schedule 20240921T080000 "load Science/circle_sample.xml;set CircleSample.NumSamplers 40 count;set CircleSample.MissionTimeout 90 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4hn1h 1 4.000000 2024-09-21T06:51:55.635Z,1726901515.635 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4hn1h 2024-09-21T06:51:55.636Z,1726901515.636 [CommandExec](IMPORTANT): Scheduled #19 (#1 of 4 with id='4hn1h'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 40 count;set CircleSample.MissionTimeout 90 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", AT TIME: 2024-09-21T08:00:00Z 2024-09-21T06:52:50.763Z,1726901570.763 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17682, MT Status=1, MTMSN=4162 2024-09-21T06:52:51.543Z,1726901571.543 [CommandExec](IMPORTANT): got command schedule 20240921T080000 "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m" 4hn1h 2 4.000000 2024-09-21T06:52:51.543Z,1726901571.543 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4hn1h 2024-09-21T06:52:51.544Z,1726901571.544 [CommandExec](IMPORTANT): Scheduled #20 (#2 of 4 with id='4hn1h'): "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m", AT TIME: 2024-09-21T08:00:00Z 2024-09-21T06:54:25.027Z,1726901665.027 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17683, MT Status=1, MTMSN=4163 2024-09-21T06:54:25.815Z,1726901665.815 [CommandExec](IMPORTANT): got command schedule 20240921T080000 "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree" 4hn1h 3 4.000000 2024-09-21T06:54:25.815Z,1726901665.815 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4hn1h 2024-09-21T06:54:25.825Z,1726901665.825 [CommandExec](IMPORTANT): Scheduled #21 (#3 of 4 with id='4hn1h'): "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree", AT TIME: 2024-09-21T08:00:00Z 2024-09-21T06:55:56.582Z,1726901756.582 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17684, MT Status=1, MTMSN=4164 2024-09-21T06:55:57.346Z,1726901757.346 [CommandExec](IMPORTANT): got command schedule 20240921T080000 "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4hn1h 4 4.000000 2024-09-21T06:55:57.347Z,1726901757.347 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4hn1h 2024-09-21T06:55:57.348Z,1726901757.348 [CommandExec](IMPORTANT): Scheduled #22 (#4 of 4 with id='4hn1h'): "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-21T08:00:00Z 2024-09-21T06:56:10.760Z,1726901770.760 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17685, MT Status=1, MTMSN=4165 2024-09-21T06:56:11.522Z,1726901771.522 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-21T06:56:13.138Z,1726901773.138 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-21T06:56:19.918Z,1726901779.918 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2024-09-21T06:56:19.919Z,1726901779.919 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-21T06:56:19.919Z,1726901779.919 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-21T06:56:19.920Z,1726901779.920 [CommandExec](IMPORTANT): got command run 2024-09-21T06:56:19.922Z,1726901779.922 [CommandExec](IMPORTANT): Running 2024-09-21T06:56:20.088Z,1726901780.088 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-21T07:04:41.518Z,1726902281.518 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-21T07:13:06.116Z,1726902786.116 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.712467,-122.186897 2024-09-21T07:30:48.974Z,1726903848.974 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2024-09-21T08:00:00.094Z,1726905600.094 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-21T08:00:07.171Z,1726905607.171 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-21T08:00:07.373Z,1726905607.373 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T08:00:20.611Z,1726905620.611 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 40 count 2024-09-21T08:00:20.612Z,1726905620.612 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 90 hour 2024-09-21T08:00:20.613Z,1726905620.613 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-21T08:00:20.613Z,1726905620.613 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-21T08:00:36.371Z,1726905636.371 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-21T08:00:36.372Z,1726905636.372 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-21T08:00:36.373Z,1726905636.373 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-21T08:00:36.373Z,1726905636.373 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-21T08:00:36.374Z,1726905636.374 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-21T08:00:51.035Z,1726905651.035 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-21T08:00:51.036Z,1726905651.036 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-21T08:00:51.062Z,1726905651.062 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-21T08:00:51.063Z,1726905651.063 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-21T08:00:51.065Z,1726905651.065 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-21T08:01:08.241Z,1726905668.241 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-21T08:01:08.248Z,1726905668.248 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-21T08:01:08.248Z,1726905668.248 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 65 second 2024-09-21T08:01:08.249Z,1726905668.249 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-21T08:01:08.250Z,1726905668.250 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-21T08:01:08.250Z,1726905668.250 [CommandExec](IMPORTANT): got command run 2024-09-21T08:01:08.252Z,1726905668.252 [CommandExec](IMPORTANT): Running 2024-09-21T08:01:08.376Z,1726905668.376 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-21T08:01:08.425Z,1726905668.425 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-21T08:01:10.396Z,1726905670.396 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-21T08:01:11.622Z,1726905671.622 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-21T08:01:13.651Z,1726905673.651 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T08:11:12.993Z,1726906272.993 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.029563 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-21T08:12:37.443Z,1726906357.443 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.052414 m , settling for 1.000000 min 2024-09-21T08:13:50.221Z,1726906430.221 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T08:13:50.402Z,1726906430.402 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T08:14:41.942Z,1726906481.942 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T08:14:42.754Z,1726906482.754 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -1 2024-09-21T08:14:42.754Z,1726906482.754 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T08:14:43.218Z,1726906483.218 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.383820 m in 9.510675 degC water with 0.064480 ug/l chlorophyll fluorescence. 2024-09-21T08:14:43.559Z,1726906483.559 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T08:14:43.963Z,1726906483.963 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T08:14:45.983Z,1726906485.983 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T08:15:17.543Z,1726906517.543 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@40,aRNAl_bac) 2024-09-21T08:15:20.371Z,1726906520.371 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T08:15:20.372Z,1726906520.372 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T08:15:20.775Z,1726906520.775 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T08:18:16.119Z,1726906696.119 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2024-09-21T08:18:16.522Z,1726906696.522 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-21T09:03:26.861Z,1726909406.861 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-21T09:03:37.803Z,1726909417.803 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.064033 CHAN A1 (24V): 0.064896 CHAN A2 (12V): 0.023972 CHAN A3 (5V): 0.004668 CHAN B0 (3.3V): 0.003018 CHAN B1 (3.15aV): 0.003268 CHAN B2 (3.15bV): 0.002958 CHAN B3 (GND): -0.001528 OPEN: 0.007701 Full Scale: +/- 1 mA 2024-09-21T09:08:37.807Z,1726909717.807 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1000.0) 2024-09-21T09:08:38.211Z,1726909718.211 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-21T09:08:40.238Z,1726909720.238 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-21T09:08:40.635Z,1726909720.635 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,aRNAl_bac) 2024-09-21T09:10:15.574Z,1726909815.574 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@40,aRNAl_bac) 2024-09-21T09:10:23.252Z,1726909823.252 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T09:10:23.655Z,1726909823.655 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2024-09-21T09:15:23.435Z,1726910123.435 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2024-09-21T09:15:25.911Z,1726910125.911 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2024-09-21T09:15:26.326Z,1726910126.326 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-21T09:15:26.700Z,1726910126.700 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-21T09:15:27.114Z,1726910127.114 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-21T09:15:27.524Z,1726910127.524 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 151.092346 m 2024-09-21T09:15:27.920Z,1726910127.920 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.382013 degC 2024-09-21T09:15:28.318Z,1726910128.318 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.072800 ug/l 2024-09-21T09:15:29.139Z,1726910129.139 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-09-21T09:15:33.598Z,1726910133.598 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T09:16:07.819Z,1726910167.819 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-21T09:22:12.813Z,1726910532.813 [RudderServo](FAULT): Reading outside of valid range:0.000000 2024-09-21T09:22:13.509Z,1726910533.509 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-09-21T09:24:16.373Z,1726910656.373 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-09-21T10:04:20.202Z,1726913060.202 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-09-21T10:04:20.202Z,1726913060.202 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-09-21T10:11:15.519Z,1726913475.519 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028866 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-09-21T10:11:47.038Z,1726913507.038 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.807602 m , settling for 1.000000 min 2024-09-21T10:12:59.813Z,1726913579.813 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T10:12:59.941Z,1726913579.941 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T10:13:47.799Z,1726913627.799 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T10:13:48.609Z,1726913628.609 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -1 2024-09-21T10:13:48.610Z,1726913628.610 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T10:13:49.056Z,1726913629.056 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.817017 m in 9.441095 degC water with 0.066560 ug/l chlorophyll fluorescence. 2024-09-21T10:13:49.415Z,1726913629.415 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T10:13:49.818Z,1726913629.818 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T10:13:51.855Z,1726913631.855 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T10:14:23.372Z,1726913663.372 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@39,aRNAl_bac) 2024-09-21T10:14:26.199Z,1726913666.199 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T10:14:26.200Z,1726913666.200 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T10:14:26.605Z,1726913666.605 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T10:17:23.555Z,1726913843.555 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2024-09-21T10:17:23.959Z,1726913843.959 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-21T11:05:20.056Z,1726916720.056 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@1000.0) 2024-09-21T11:05:20.459Z,1726916720.459 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-21T11:05:22.479Z,1726916722.479 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-21T11:05:22.884Z,1726916722.884 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,aRNAl_bac) 2024-09-21T11:06:57.823Z,1726916817.823 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@39,aRNAl_bac) 2024-09-21T11:07:05.505Z,1726916825.505 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T11:07:05.905Z,1726916825.905 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED 2024-09-21T11:12:05.675Z,1726917125.675 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally. 2024-09-21T11:12:08.244Z,1726917128.244 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2024-09-21T11:12:08.600Z,1726917128.600 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-21T11:12:09.026Z,1726917129.026 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-21T11:12:09.410Z,1726917129.410 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-21T11:12:09.813Z,1726917129.813 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.134964 m 2024-09-21T11:12:10.238Z,1726917130.238 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.470301 degC 2024-09-21T11:12:10.635Z,1726917130.635 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2024-09-21T11:12:11.441Z,1726917131.441 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-09-21T11:12:15.884Z,1726917135.884 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T11:39:56.185Z,1726918796.185 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-09-21T11:39:56.185Z,1726918796.185 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-09-21T12:11:17.967Z,1726920677.967 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027303 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2024-09-21T12:11:21.278Z,1726920681.278 [ElevatorServo](FAULT): Overload Error 2024-09-21T12:13:45.435Z,1726920825.435 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.686951 m , settling for 1.000000 min 2024-09-21T12:14:58.221Z,1726920898.221 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T12:14:58.374Z,1726920898.374 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T12:15:50.279Z,1726920950.279 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T12:15:51.092Z,1726920951.092 [ESPComponent](IMPORTANT): [sample #15] Setting cartridge argument to -1 2024-09-21T12:15:51.099Z,1726920951.099 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T12:15:51.538Z,1726920951.538 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.880646 m in 9.489588 degC water with 0.066560 ug/l chlorophyll fluorescence. 2024-09-21T12:15:51.894Z,1726920951.894 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T12:15:52.299Z,1726920952.299 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T12:15:54.318Z,1726920954.318 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T12:16:25.826Z,1726920985.826 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PRIMING (@38,aRNAl_bac) 2024-09-21T12:16:27.847Z,1726920987.847 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T12:16:28.252Z,1726920988.252 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T12:16:28.657Z,1726920988.657 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T12:19:25.606Z,1726921165.606 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_FILTERING (@0.0) 2024-09-21T12:19:26.011Z,1726921166.011 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-21T13:09:10.559Z,1726924150.559 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PAUSED (@1000.0) 2024-09-21T13:09:10.958Z,1726924150.958 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-21T13:09:12.980Z,1726924152.980 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-21T13:09:13.383Z,1726924153.383 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,aRNAl_bac) 2024-09-21T13:10:48.323Z,1726924248.323 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPING (@38,aRNAl_bac) 2024-09-21T13:10:55.999Z,1726924255.999 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T13:10:56.403Z,1726924256.403 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPED 2024-09-21T13:15:56.178Z,1726924556.178 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence completed normally. 2024-09-21T13:15:58.662Z,1726924558.662 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2024-09-21T13:15:59.060Z,1726924559.060 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-21T13:15:59.462Z,1726924559.462 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-21T13:15:59.879Z,1726924559.879 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-21T13:16:00.266Z,1726924560.266 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.770309 m 2024-09-21T13:16:00.662Z,1726924560.662 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.478998 degC 2024-09-21T13:16:01.062Z,1726924561.062 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2024-09-21T13:16:01.880Z,1726924561.880 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2024-09-21T13:16:06.338Z,1726924566.338 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T13:26:47.129Z,1726925207.129 [ElevatorServo](FAULT): Overload Error 2024-09-21T14:11:20.490Z,1726927880.490 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028516 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 4.000000 count 2024-09-21T14:11:57.718Z,1726927917.718 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.101654 m , settling for 1.000000 min 2024-09-21T14:13:10.458Z,1726927990.458 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T14:13:10.573Z,1726927990.573 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T14:14:03.350Z,1726928043.350 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T14:14:04.161Z,1726928044.161 [ESPComponent](IMPORTANT): [sample #16] Setting cartridge argument to -1 2024-09-21T14:14:04.162Z,1726928044.162 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T14:14:04.638Z,1726928044.638 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.012405 m in 9.529200 degC water with 0.059280 ug/l chlorophyll fluorescence. 2024-09-21T14:14:04.968Z,1726928044.968 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T14:14:05.371Z,1726928045.371 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T14:14:07.391Z,1726928047.391 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T14:14:38.903Z,1726928078.903 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PRIMING (@37,aRNAl_bac) 2024-09-21T14:14:40.923Z,1726928080.923 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T14:14:41.328Z,1726928081.328 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T14:14:41.731Z,1726928081.731 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T14:17:38.283Z,1726928258.283 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_FILTERING (@0.0) 2024-09-21T14:17:38.687Z,1726928258.687 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-21T15:03:38.029Z,1726931018.029 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-21T15:03:49.311Z,1726931029.311 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.025168 CHAN A1 (24V): 0.031721 CHAN A2 (12V): 0.011380 CHAN A3 (5V): 0.001948 CHAN B0 (3.3V): 0.001340 CHAN B1 (3.15aV): 0.001461 CHAN B2 (3.15bV): 0.001178 CHAN B3 (GND): -0.001365 OPEN: 0.006310 Full Scale: +/- 1 mA 2024-09-21T15:05:16.891Z,1726931116.891 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PAUSED (@1000.0) 2024-09-21T15:05:17.291Z,1726931117.291 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-21T15:05:19.311Z,1726931119.311 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-21T15:05:19.715Z,1726931119.715 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,aRNAl_bac) 2024-09-21T15:06:54.668Z,1726931214.668 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPING (@37,aRNAl_bac) 2024-09-21T15:07:02.344Z,1726931222.344 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T15:07:02.838Z,1726931222.838 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPED 2024-09-21T15:12:02.623Z,1726931522.623 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence completed normally. 2024-09-21T15:12:05.126Z,1726931525.126 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 4.000000 count 2024-09-21T15:12:05.514Z,1726931525.514 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-21T15:12:05.916Z,1726931525.916 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-21T15:12:06.318Z,1726931526.318 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-21T15:12:06.702Z,1726931526.702 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.449539 m 2024-09-21T15:12:07.123Z,1726931527.123 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.555902 degC 2024-09-21T15:12:07.534Z,1726931527.534 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2024-09-21T15:12:08.318Z,1726931528.318 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 4.000000 count completed. 2024-09-21T15:12:12.839Z,1726931532.839 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T15:12:56.518Z,1726931576.518 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-21T15:23:23.025Z,1726932203.025 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-09-21T15:23:23.026Z,1726932203.026 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-09-21T16:11:23.081Z,1726935083.081 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 1.097897 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 5.000000 count 2024-09-21T16:11:26.762Z,1726935086.762 [ElevatorServo](FAULT): Overload Error 2024-09-21T16:11:27.584Z,1726935087.584 [ElevatorServo](FAULT): LCB fault: Invalid Command. 2024-09-21T16:13:55.479Z,1726935235.479 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.814987 m , settling for 1.000000 min 2024-09-21T16:15:08.253Z,1726935308.253 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T16:15:08.378Z,1726935308.378 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T16:16:00.302Z,1726935360.302 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T16:16:01.116Z,1726935361.116 [ESPComponent](IMPORTANT): [sample #17] Setting cartridge argument to -1 2024-09-21T16:16:01.116Z,1726935361.116 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T16:16:01.589Z,1726935361.589 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.575500 m in 9.509089 degC water with 0.074880 ug/l chlorophyll fluorescence. 2024-09-21T16:16:01.919Z,1726935361.919 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T16:16:02.327Z,1726935362.327 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T16:16:04.343Z,1726935364.343 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T16:16:35.855Z,1726935395.855 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PRIMING (@36,aRNAl_bac) 2024-09-21T16:16:37.874Z,1726935397.874 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T16:16:38.282Z,1726935398.282 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T16:16:38.706Z,1726935398.706 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T16:19:54.224Z,1726935594.224 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPING (@0.0) 2024-09-21T16:20:01.904Z,1726935601.904 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T16:20:02.357Z,1726935602.357 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPED 2024-09-21T16:25:02.298Z,1726935902.298 [ESPComponent](CRITICAL): [sample #17] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 198.3688 2024-09-21T16:25:02.503Z,1726935902.503 [CommandExec](FAULT): Scheduling is paused for 1 commands 2024-09-21T16:25:03.458Z,1726935903.458 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T16:30:15.093Z,1726936215.093 [ElevatorServo](FAULT): Overload Error 2024-09-21T16:47:14.863Z,1726937234.863 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17735, MT Status=1, MTMSN=4166 2024-09-21T16:47:15.431Z,1726937235.431 [CommandExec](IMPORTANT): got command ! onESPclient preston -- 'spare:5' 2024-09-21T16:47:15.879Z,1726937235.879 [CommandExec](IMPORTANT): /mnt/mmc/started/2024-09-21UTC164715_onESPclient 2024-09-21T17:04:50.549Z,1726938290.549 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 39.778015 min 2024-09-21T17:13:37.216Z,1726938817.216 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 48.555741 min 2024-09-21T17:14:21.238Z,1726938861.238 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17766, MT Status=1, MTMSN=4167 2024-09-21T17:14:21.792Z,1726938861.792 [CommandExec](IMPORTANT): got command schedule resume 2024-09-21T17:14:21.792Z,1726938861.792 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-21T17:21:18.385Z,1726939278.385 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 56.241951 min 2024-09-21T17:21:54.382Z,1726939314.382 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17776, MT Status=1, MTMSN=4171 2024-09-21T17:21:55.163Z,1726939315.163 [CommandExec](IMPORTANT): got command schedule 20240921T180000 "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4ig18 4 4.000000 2024-09-21T17:21:55.163Z,1726939315.163 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4ig18 2024-09-21T17:21:55.165Z,1726939315.165 [CommandExec](IMPORTANT): Scheduled #23 (#4 of 4 with id='4ig18'): "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-21T18:00:00Z 2024-09-21T17:28:52.073Z,1726939732.073 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 63.803088 min 2024-09-21T17:29:19.113Z,1726939759.113 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17783, MT Status=1, MTMSN=4172 2024-09-21T17:29:19.799Z,1726939759.799 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-21T17:29:21.466Z,1726939761.466 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-21T17:29:27.892Z,1726939767.892 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2024-09-21T17:29:27.893Z,1726939767.893 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-21T17:29:27.894Z,1726939767.894 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-21T17:29:27.894Z,1726939767.894 [CommandExec](IMPORTANT): got command run 2024-09-21T17:29:27.896Z,1726939767.896 [CommandExec](IMPORTANT): Running 2024-09-21T17:29:27.996Z,1726939767.996 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-21T17:35:52.523Z,1726940152.523 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-21T17:42:17.112Z,1726940537.112 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.712467,-122.186897 2024-09-21T18:10:33.640Z,1726942233.640 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17797, MT Status=1, MTMSN=4173 2024-09-21T18:10:34.158Z,1726942234.158 [CommandExec](IMPORTANT): got command stop 2024-09-21T18:10:34.158Z,1726942234.158 [CommandExec](IMPORTANT): Scheduling is paused for 2 commands 2024-09-21T18:10:34.867Z,1726942234.867 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T18:18:45.999Z,1726942725.999 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17798, MT Status=1, MTMSN=4174 2024-09-21T18:18:46.507Z,1726942726.507 [CommandExec](IMPORTANT): got command schedule resume 2024-09-21T18:18:46.507Z,1726942726.507 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-21T18:25:38.765Z,1726943138.765 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 15.058305 min 2024-09-21T18:27:31.078Z,1726943251.078 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17804, MT Status=1, MTMSN=4175 2024-09-21T18:27:31.786Z,1726943251.786 [CommandExec](IMPORTANT): got command schedule asap "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4iiy5 1 4.000000 2024-09-21T18:27:31.787Z,1726943251.787 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4iiy5 2024-09-21T18:27:31.788Z,1726943251.788 [CommandExec](IMPORTANT): Scheduled #24 (#1 of 4 with id='4iiy5'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", ASAP 2024-09-21T18:27:50.911Z,1726943270.911 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17805, MT Status=1, MTMSN=4176 2024-09-21T18:27:51.689Z,1726943271.689 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m" 4iiy5 2 4.000000 2024-09-21T18:27:51.689Z,1726943271.689 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4iiy5 2024-09-21T18:27:51.690Z,1726943271.690 [CommandExec](IMPORTANT): Scheduled #25 (#2 of 4 with id='4iiy5'): "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m", ASAP 2024-09-21T18:28:14.036Z,1726943294.036 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17806, MT Status=1, MTMSN=4177 2024-09-21T18:28:14.826Z,1726943294.826 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree" 4iiy5 3 4.000000 2024-09-21T18:28:14.826Z,1726943294.826 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4iiy5 2024-09-21T18:28:14.827Z,1726943294.827 [CommandExec](IMPORTANT): Scheduled #26 (#3 of 4 with id='4iiy5'): "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree", ASAP 2024-09-21T18:28:32.699Z,1726943312.699 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17807, MT Status=1, MTMSN=4178 2024-09-21T18:28:33.481Z,1726943313.481 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4iiy5 4 4.000000 2024-09-21T18:28:33.481Z,1726943313.481 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4iiy5 2024-09-21T18:28:33.482Z,1726943313.482 [CommandExec](IMPORTANT): Scheduled #27 (#4 of 4 with id='4iiy5'): "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", ASAP 2024-09-21T18:28:33.635Z,1726943313.635 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-21T18:28:40.786Z,1726943320.786 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-21T18:28:52.675Z,1726943332.675 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 37 count 2024-09-21T18:28:52.676Z,1726943332.676 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 80 hour 2024-09-21T18:28:52.676Z,1726943332.676 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-21T18:28:52.678Z,1726943332.678 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-21T18:29:06.916Z,1726943346.916 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-21T18:29:06.916Z,1726943346.916 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-21T18:29:06.918Z,1726943346.918 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-21T18:29:06.919Z,1726943346.919 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-21T18:29:06.919Z,1726943346.919 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-21T18:29:20.724Z,1726943360.724 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-21T18:29:20.725Z,1726943360.725 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-21T18:29:20.726Z,1726943360.726 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-21T18:29:20.726Z,1726943360.726 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-21T18:29:20.727Z,1726943360.727 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-21T18:29:36.781Z,1726943376.781 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-21T18:29:36.781Z,1726943376.781 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-21T18:29:36.782Z,1726943376.782 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 65 second 2024-09-21T18:29:36.783Z,1726943376.783 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-21T18:29:36.783Z,1726943376.783 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-21T18:29:36.784Z,1726943376.784 [CommandExec](IMPORTANT): got command run 2024-09-21T18:29:36.786Z,1726943376.786 [CommandExec](IMPORTANT): Running 2024-09-21T18:29:37.100Z,1726943377.100 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-21T18:29:37.132Z,1726943377.132 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-21T18:29:39.116Z,1726943379.116 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-21T18:29:40.351Z,1726943380.351 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-21T18:29:42.381Z,1726943382.381 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T18:39:41.632Z,1726943981.632 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.028280 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-21T18:43:21.429Z,1726944201.429 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.107132 m , settling for 1.000000 min 2024-09-21T18:44:34.194Z,1726944274.194 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T18:44:34.331Z,1726944274.331 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T18:45:25.870Z,1726944325.870 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T18:45:26.682Z,1726944326.682 [ESPComponent](IMPORTANT): [sample #18] Setting cartridge argument to -1 2024-09-21T18:45:26.683Z,1726944326.683 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T18:45:27.139Z,1726944327.139 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.145264 m in 9.548731 degC water with 0.065520 ug/l chlorophyll fluorescence. 2024-09-21T18:45:27.486Z,1726944327.486 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T18:45:27.891Z,1726944327.891 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T18:45:29.911Z,1726944329.911 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T18:46:01.423Z,1726944361.423 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PRIMING (@35,aRNAl_bac) 2024-09-21T18:46:04.251Z,1726944364.251 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T18:46:04.252Z,1726944364.252 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T18:46:04.660Z,1726944364.660 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T18:49:20.599Z,1726944560.599 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPING (@0.0) 2024-09-21T18:49:28.275Z,1726944568.275 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T18:49:28.679Z,1726944568.679 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPED 2024-09-21T18:54:28.459Z,1726944868.459 [ESPComponent](CRITICAL): [sample #18] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 199.1760 2024-09-21T18:54:28.586Z,1726944868.586 [CommandExec](FAULT): Scheduling is paused for 2 commands 2024-09-21T18:54:29.578Z,1726944869.578 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T18:59:43.542Z,1726945183.542 [ElevatorServo](FAULT): Overload Error 2024-09-21T19:04:56.616Z,1726945496.616 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-09-21T19:13:59.800Z,1726946039.800 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17821, MT Status=1, MTMSN=4179 2024-09-21T19:14:00.360Z,1726946040.360 [CommandExec](IMPORTANT): got command schedule clear 2024-09-21T19:14:00.360Z,1726946040.360 [CommandExec](IMPORTANT): Cleared 2 scheduled commands. 2024-09-21T19:14:00.360Z,1726946040.360 [CommandExec](IMPORTANT): got command schedule resume 2024-09-21T19:14:00.360Z,1726946040.360 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-21T19:15:15.916Z,1726946115.916 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17825, MT Status=1, MTMSN=4180 2024-09-21T19:15:16.678Z,1726946116.678 [CommandExec](IMPORTANT): got command schedule 20240921T200000 "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4ilgm 1 4.000000 2024-09-21T19:15:16.679Z,1726946116.679 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4ilgm 2024-09-21T19:15:16.680Z,1726946116.680 [CommandExec](IMPORTANT): Scheduled #28 (#1 of 4 with id='4ilgm'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", AT TIME: 2024-09-21T20:00:00Z 2024-09-21T19:15:46.762Z,1726946146.762 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17826, MT Status=1, MTMSN=4181 2024-09-21T19:15:47.512Z,1726946147.512 [CommandExec](IMPORTANT): got command schedule 20240921T200000 "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m" 4ilgm 2 4.000000 2024-09-21T19:15:47.512Z,1726946147.512 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4ilgm 2024-09-21T19:15:47.514Z,1726946147.514 [CommandExec](IMPORTANT): Scheduled #29 (#2 of 4 with id='4ilgm'): "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m", AT TIME: 2024-09-21T20:00:00Z 2024-09-21T19:16:13.068Z,1726946173.068 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17828, MT Status=1, MTMSN=4182 2024-09-21T19:16:13.854Z,1726946173.854 [CommandExec](IMPORTANT): got command schedule 20240921T200000 "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree" 4ilgm 3 4.000000 2024-09-21T19:16:13.855Z,1726946173.855 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4ilgm 2024-09-21T19:16:13.856Z,1726946173.856 [CommandExec](IMPORTANT): Scheduled #30 (#3 of 4 with id='4ilgm'): "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree", AT TIME: 2024-09-21T20:00:00Z 2024-09-21T19:16:47.134Z,1726946207.134 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17830, MT Status=1, MTMSN=4183 2024-09-21T19:16:47.911Z,1726946207.911 [CommandExec](IMPORTANT): got command schedule 20240921T200000 "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4ilgm 4 4.000000 2024-09-21T19:16:47.911Z,1726946207.911 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4ilgm 2024-09-21T19:16:47.912Z,1726946207.912 [CommandExec](IMPORTANT): Scheduled #31 (#4 of 4 with id='4ilgm'): "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-21T20:00:00Z 2024-09-21T19:24:43.675Z,1726946683.675 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.228448 min 2024-09-21T19:24:56.247Z,1726946696.247 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-09-21T19:32:42.535Z,1726947162.535 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 38.209444 min 2024-09-21T19:34:21.631Z,1726947261.631 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17841, MT Status=1, MTMSN=4184 2024-09-21T19:34:22.371Z,1726947262.371 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-21T19:34:24.061Z,1726947264.061 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-21T19:34:30.771Z,1726947270.771 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 5 minute 2024-09-21T19:34:30.772Z,1726947270.772 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-21T19:34:30.772Z,1726947270.772 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-21T19:34:30.773Z,1726947270.773 [CommandExec](IMPORTANT): got command run 2024-09-21T19:34:30.775Z,1726947270.775 [CommandExec](IMPORTANT): Running 2024-09-21T19:34:31.047Z,1726947271.047 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-21T19:36:48.771Z,1726947408.771 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-21T19:39:13.379Z,1726947553.379 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.712467,-122.186897 2024-09-21T19:43:06.879Z,1726947786.879 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17848, MT Status=1, MTMSN=4185 2024-09-21T19:43:07.496Z,1726947787.496 [CommandExec](IMPORTANT): got command ! onESPclient preston -- 'spare:5' 2024-09-21T19:43:07.870Z,1726947787.870 [CommandExec](IMPORTANT): /mnt/mmc/started/2024-09-21UTC194307_onESPclient 2024-09-21T20:00:00.048Z,1726948800.048 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-21T20:00:07.388Z,1726948807.388 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-21T20:00:07.773Z,1726948807.773 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T20:00:19.728Z,1726948819.728 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 37 count 2024-09-21T20:00:19.729Z,1726948819.729 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 80 hour 2024-09-21T20:00:19.730Z,1726948819.730 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-21T20:00:19.730Z,1726948819.730 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-21T20:00:34.471Z,1726948834.471 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-21T20:00:34.471Z,1726948834.471 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-21T20:00:34.472Z,1726948834.472 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-21T20:00:34.472Z,1726948834.472 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-21T20:00:34.502Z,1726948834.502 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-21T20:00:48.527Z,1726948848.527 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-21T20:00:48.528Z,1726948848.528 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-21T20:00:48.529Z,1726948848.529 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-21T20:00:48.530Z,1726948848.530 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-21T20:00:48.530Z,1726948848.530 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-21T20:01:04.904Z,1726948864.904 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-21T20:01:04.905Z,1726948864.905 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-21T20:01:04.906Z,1726948864.906 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 65 second 2024-09-21T20:01:04.906Z,1726948864.906 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-21T20:01:04.907Z,1726948864.907 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-21T20:01:04.908Z,1726948864.908 [CommandExec](IMPORTANT): got command run 2024-09-21T20:01:04.910Z,1726948864.910 [CommandExec](IMPORTANT): Running 2024-09-21T20:01:05.163Z,1726948865.163 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-21T20:01:05.193Z,1726948865.193 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-21T20:01:07.171Z,1726948867.171 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-21T20:01:08.427Z,1726948868.427 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-21T20:01:10.464Z,1726948870.464 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T20:05:43.269Z,1726949143.269 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-09-21T20:05:43.269Z,1726949143.269 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-09-21T20:11:09.606Z,1726949469.606 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.027031 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-21T20:11:36.270Z,1726949496.270 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.812073 m , settling for 1.000000 min 2024-09-21T20:12:49.049Z,1726949569.049 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T20:12:49.189Z,1726949569.189 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T20:13:41.102Z,1726949621.102 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T20:13:41.909Z,1726949621.909 [ESPComponent](IMPORTANT): [sample #19] Setting cartridge argument to -1 2024-09-21T20:13:41.909Z,1726949621.909 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T20:13:42.367Z,1726949622.367 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.613434 m in 9.553614 degC water with 0.060320 ug/l chlorophyll fluorescence. 2024-09-21T20:13:42.716Z,1726949622.716 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T20:13:43.119Z,1726949623.119 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T20:13:45.138Z,1726949625.138 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T20:14:16.655Z,1726949656.655 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PRIMING (@34,aRNAl_bac) 2024-09-21T20:14:19.479Z,1726949659.479 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T20:14:19.480Z,1726949659.480 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T20:14:19.883Z,1726949659.883 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T20:17:13.603Z,1726949833.603 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_FILTERING (@0.0) 2024-09-21T20:17:14.007Z,1726949834.007 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-21T21:03:49.545Z,1726952629.545 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-21T21:04:00.465Z,1726952640.465 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.021712 CHAN A1 (24V): 0.027741 CHAN A2 (12V): 0.009779 CHAN A3 (5V): 0.000154 CHAN B0 (3.3V): 0.002577 CHAN B1 (3.15aV): 0.000520 CHAN B2 (3.15bV): 0.000526 CHAN B3 (GND): -0.000780 OPEN: 0.007050 Full Scale: +/- 1 mA 2024-09-21T21:05:45.367Z,1726952745.367 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PAUSED (@1000.0) 2024-09-21T21:05:45.767Z,1726952745.767 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-21T21:05:47.787Z,1726952747.787 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-21T21:05:48.191Z,1726952748.191 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PROCESSED (@34,aRNAl_bac) 2024-09-21T21:07:23.131Z,1726952843.131 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPING (@34,aRNAl_bac) 2024-09-21T21:07:30.811Z,1726952850.811 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T21:07:31.215Z,1726952851.215 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPED 2024-09-21T21:12:31.084Z,1726953151.084 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence completed normally. 2024-09-21T21:12:33.586Z,1726953153.586 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2024-09-21T21:12:33.958Z,1726953153.958 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-21T21:12:34.362Z,1726953154.362 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-21T21:12:34.772Z,1726953154.772 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-21T21:12:35.172Z,1726953155.172 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.918777 m 2024-09-21T21:12:35.578Z,1726953155.578 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.572412 degC 2024-09-21T21:12:35.995Z,1726953155.995 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.072800 ug/l 2024-09-21T21:12:36.788Z,1726953156.788 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-09-21T21:12:41.243Z,1726953161.243 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-21T21:39:20.790Z,1726954760.790 [ElevatorServo](FAULT): Overload Error 2024-09-21T22:02:14.966Z,1726956134.966 [ThrusterHE](FAULT): Could not get status:serial timeout 2024-09-21T22:11:12.239Z,1726956672.239 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030688 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-09-21T22:11:15.925Z,1726956675.925 [ElevatorServo](FAULT): Overload Error 2024-09-21T22:14:14.073Z,1726956854.073 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.700729 m , settling for 1.000000 min 2024-09-21T22:15:26.865Z,1726956926.865 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-21T22:15:26.988Z,1726956926.988 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-21T22:16:18.934Z,1726956978.934 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-21T22:16:19.745Z,1726956979.745 [ESPComponent](IMPORTANT): [sample #20] Setting cartridge argument to -1 2024-09-21T22:16:19.746Z,1726956979.746 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-21T22:16:20.251Z,1726956980.251 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.644669 m in 9.573969 degC water with 0.069680 ug/l chlorophyll fluorescence. 2024-09-21T22:16:20.550Z,1726956980.550 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-21T22:16:20.955Z,1726956980.955 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-21T22:16:22.975Z,1726956982.975 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-21T22:16:54.486Z,1726957014.486 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PRIMING (@33,aRNAl_bac) 2024-09-21T22:16:56.910Z,1726957016.910 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PRIMING (@100.0) 2024-09-21T22:16:56.911Z,1726957016.911 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-21T22:16:57.315Z,1726957017.315 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-21T22:20:12.127Z,1726957212.127 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPING (@0.0) 2024-09-21T22:20:19.804Z,1726957219.804 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-21T22:20:20.207Z,1726957220.207 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPED 2024-09-21T22:25:20.027Z,1726957520.027 [ESPComponent](CRITICAL): [sample #20] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 197.6339 2024-09-21T22:25:20.203Z,1726957520.203 [CommandExec](FAULT): Scheduling is paused 2024-09-21T22:25:21.128Z,1726957521.128 [MissionManager](IMPORTANT): Started mission Default 2024-09-21T22:30:48.755Z,1726957848.755 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-09-21T22:41:52.362Z,1726958512.362 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17871, MT Status=1, MTMSN=4186 2024-09-21T22:41:52.863Z,1726958512.863 [CommandExec](IMPORTANT): got command schedule resume 2024-09-21T22:41:52.863Z,1726958512.863 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-21T22:47:45.570Z,1726958865.570 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17877, MT Status=1, MTMSN=4187 2024-09-21T22:47:46.310Z,1726958866.310 [CommandExec](IMPORTANT): got command schedule 20240922T000000 "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4iv74 1 4.000000 2024-09-21T22:47:46.310Z,1726958866.310 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4iv74 2024-09-21T22:47:46.332Z,1726958866.332 [CommandExec](IMPORTANT): Scheduled #32 (#1 of 4 with id='4iv74'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 37 count;set CircleSample.MissionTimeout 80 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", AT TIME: 2024-09-22T00:00:00Z 2024-09-21T22:48:14.246Z,1726958894.246 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17878, MT Status=1, MTMSN=4188 2024-09-21T22:48:14.990Z,1726958894.990 [CommandExec](IMPORTANT): got command schedule 20240922T000000 "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m" 4iv74 2 4.000000 2024-09-21T22:48:14.990Z,1726958894.990 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4iv74 2024-09-21T22:48:14.992Z,1726958894.992 [CommandExec](IMPORTANT): Scheduled #33 (#2 of 4 with id='4iv74'): "set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min;set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m", AT TIME: 2024-09-22T00:00:00Z 2024-09-21T22:50:02.135Z,1726959002.135 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17879, MT Status=1, MTMSN=4189 2024-09-21T22:50:02.950Z,1726959002.950 [CommandExec](IMPORTANT): got command schedule 20240922T000000 "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree" 4iv74 3 4.000000 2024-09-21T22:50:02.951Z,1726959002.951 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4iv74 2024-09-21T22:50:02.952Z,1726959002.952 [CommandExec](IMPORTANT): Scheduled #34 (#3 of 4 with id='4iv74'): "set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m;set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree", AT TIME: 2024-09-22T00:00:00Z 2024-09-21T22:50:24.355Z,1726959024.355 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=17880, MT Status=1, MTMSN=4190 2024-09-21T22:50:25.138Z,1726959025.138 [CommandExec](IMPORTANT): got command schedule 20240922T000000 "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4iv74 4 4.000000 2024-09-21T22:50:25.138Z,1726959025.138 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4iv74 2024-09-21T22:50:25.140Z,1726959025.140 [CommandExec](IMPORTANT): Scheduled #35 (#4 of 4 with id='4iv74'): "set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 65 s;set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-22T00:00:00Z 2024-09-21T22:56:42.404Z,1726959402.404 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.347705 min 2024-09-21T22:58:16.663Z,1726959496.663 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17888, MT Status=1, MTMSN=4191 2024-09-21T22:58:17.240Z,1726959497.240 [CommandExec](IMPORTANT): got command ! onESPclient preston -- slot spare:10 2024-09-21T22:58:17.610Z,1726959497.610 [CommandExec](IMPORTANT): /mnt/mmc/started/2024-09-21UTC225817_onESPclient 2024-09-21T23:04:22.034Z,1726959862.034 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 39.009595 min 2024-09-21T23:05:46.228Z,1726959946.228 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=17894, MT Status=1, MTMSN=4192 2024-09-21T23:05:46.968Z,1726959946.968 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-21T23:05:48.801Z,1726959948.801 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-21T23:05:55.134Z,1726959955.134 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2024-09-21T23:05:55.135Z,1726959955.135 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-21T23:05:55.136Z,1726959955.136 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-21T23:05:55.136Z,1726959955.136 [CommandExec](IMPORTANT): got command run 2024-09-21T23:05:55.138Z,1726959955.138 [CommandExec](IMPORTANT): Running 2024-09-21T23:05:55.175Z,1726959955.175 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-21T23:08:36.815Z,1726960116.815 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-21T23:31:55.234Z,1726961515.234 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.712467,-122.186897 2024-09-22T00:00:00.181Z,1726963200.181 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-22T00:00:07.299Z,1726963207.299 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-22T00:00:07.428Z,1726963207.428 [MissionManager](IMPORTANT): Started mission Default 2024-09-22T00:00:18.992Z,1726963218.992 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 37 count 2024-09-22T00:00:18.993Z,1726963218.993 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 80 hour 2024-09-22T00:00:18.994Z,1726963218.994 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-22T00:00:18.994Z,1726963218.994 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-22T00:00:33.242Z,1726963233.242 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-22T00:00:33.243Z,1726963233.243 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-22T00:00:33.243Z,1726963233.243 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-22T00:00:33.244Z,1726963233.244 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-22T00:00:33.244Z,1726963233.244 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-22T00:00:46.643Z,1726963246.643 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-22T00:00:46.644Z,1726963246.644 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-22T00:00:46.652Z,1726963246.652 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-22T00:00:46.657Z,1726963246.657 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-22T00:00:46.658Z,1726963246.658 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-22T00:01:02.662Z,1726963262.662 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-22T00:01:02.663Z,1726963262.663 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-22T00:01:02.663Z,1726963262.663 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 65 second 2024-09-22T00:01:02.664Z,1726963262.664 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-22T00:01:02.664Z,1726963262.664 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-22T00:01:02.665Z,1726963262.665 [CommandExec](IMPORTANT): got command run 2024-09-22T00:01:02.667Z,1726963262.667 [CommandExec](IMPORTANT): Running 2024-09-22T00:01:02.988Z,1726963262.988 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-22T00:01:03.050Z,1726963263.050 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-22T00:01:05.007Z,1726963265.007 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-22T00:01:06.256Z,1726963266.256 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-22T00:01:08.314Z,1726963268.314 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T00:11:07.513Z,1726963867.513 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.028459 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-22T00:15:30.912Z,1726964130.912 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.023743 m , settling for 1.000000 min 2024-09-22T00:16:43.697Z,1726964203.697 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T00:16:43.821Z,1726964203.821 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T00:17:34.121Z,1726964254.121 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T00:17:34.934Z,1726964254.934 [ESPComponent](IMPORTANT): [sample #21] Setting cartridge argument to -1 2024-09-22T00:17:34.936Z,1726964254.936 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T00:17:35.370Z,1726964255.370 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.018387 m in 9.521570 degC water with 0.058240 ug/l chlorophyll fluorescence. 2024-09-22T00:17:35.738Z,1726964255.738 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T00:17:36.142Z,1726964256.142 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T00:17:38.159Z,1726964258.159 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T00:18:09.671Z,1726964289.671 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PRIMING (@32,aRNAl_bac) 2024-09-22T00:18:12.499Z,1726964292.499 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T00:18:12.500Z,1726964292.500 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T00:18:12.903Z,1726964292.903 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T00:21:06.787Z,1726964466.787 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T00:21:07.197Z,1726964467.197 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T01:11:07.072Z,1726967467.072 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T01:11:07.475Z,1726967467.475 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T01:11:09.496Z,1726967469.496 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T01:11:09.899Z,1726967469.899 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PROCESSED (@32,aRNAl_bac) 2024-09-22T01:12:44.840Z,1726967564.840 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPING (@32,aRNAl_bac) 2024-09-22T01:12:52.515Z,1726967572.515 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T01:12:52.919Z,1726967572.919 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPED 2024-09-22T01:17:52.779Z,1726967872.779 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence completed normally. 2024-09-22T01:17:55.313Z,1726967875.313 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2024-09-22T01:17:55.710Z,1726967875.710 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T01:17:56.069Z,1726967876.069 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T01:17:56.486Z,1726967876.486 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T01:17:56.859Z,1726967876.859 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 151.665421 m 2024-09-22T01:17:57.270Z,1726967877.270 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.491388 degC 2024-09-22T01:17:57.692Z,1726967877.692 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.062400 ug/l 2024-09-22T01:17:58.484Z,1726967878.484 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-09-22T01:18:02.928Z,1726967882.928 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T01:18:15.993Z,1726967895.993 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T01:19:19.521Z,1726967959.521 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T01:28:28.037Z,1726968508.037 [ElevatorServo](FAULT): Overload Error 2024-09-22T02:03:13.966Z,1726970593.966 [ElevatorServo](FAULT): Overload Error 2024-09-22T02:11:10.298Z,1726971070.298 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032153 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-09-22T02:12:02.006Z,1726971122.006 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.192215 m , settling for 1.000000 min 2024-09-22T02:13:14.766Z,1726971194.766 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T02:13:14.914Z,1726971194.914 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T02:14:02.835Z,1726971242.835 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T02:14:03.657Z,1726971243.657 [ESPComponent](IMPORTANT): [sample #22] Setting cartridge argument to -1 2024-09-22T02:14:03.657Z,1726971243.657 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T02:14:04.085Z,1726971244.085 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.194916 m in 9.624323 degC water with 0.079040 ug/l chlorophyll fluorescence. 2024-09-22T02:14:04.451Z,1726971244.451 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T02:14:04.855Z,1726971244.855 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T02:14:06.874Z,1726971246.874 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T02:14:38.407Z,1726971278.407 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PRIMING (@31,aRNAl_bac) 2024-09-22T02:14:40.834Z,1726971280.834 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T02:14:40.835Z,1726971280.835 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T02:14:41.236Z,1726971281.236 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T02:18:03.643Z,1726971483.643 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T02:18:04.047Z,1726971484.047 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T03:04:00.509Z,1726974240.509 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-22T03:04:11.544Z,1726974251.544 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.019644 CHAN A1 (24V): 0.022680 CHAN A2 (12V): 0.008949 CHAN A3 (5V): 0.000791 CHAN B0 (3.3V): 0.000555 CHAN B1 (3.15aV): 0.000557 CHAN B2 (3.15bV): 0.000695 CHAN B3 (GND): -0.000839 OPEN: 0.007531 Full Scale: +/- 1 mA 2024-09-22T03:04:45.227Z,1726974285.227 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T03:04:45.631Z,1726974285.631 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T03:04:47.651Z,1726974287.651 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T03:04:48.054Z,1726974288.054 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PROCESSED (@30,aRNAl_bac) 2024-09-22T03:06:22.994Z,1726974382.994 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPING (@30,aRNAl_bac) 2024-09-22T03:06:30.672Z,1726974390.672 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T03:06:31.075Z,1726974391.075 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPED 2024-09-22T03:11:30.866Z,1726974690.866 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence completed normally. 2024-09-22T03:11:33.340Z,1726974693.340 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2024-09-22T03:11:33.756Z,1726974693.756 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T03:11:34.163Z,1726974694.163 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T03:11:34.550Z,1726974694.550 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T03:11:34.954Z,1726974694.954 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.701508 m 2024-09-22T03:11:35.351Z,1726974695.351 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.721491 degC 2024-09-22T03:11:35.759Z,1726974695.759 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2024-09-22T03:11:36.585Z,1726974696.585 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-09-22T03:11:41.030Z,1726974701.030 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T03:12:03.502Z,1726974723.502 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T03:12:14.338Z,1726974734.338 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T03:40:57.097Z,1726976457.097 [ElevatorServo](FAULT): Overload Error 2024-09-22T04:11:12.722Z,1726978272.722 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028035 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2024-09-22T04:11:16.397Z,1726978276.397 [ElevatorServo](FAULT): Overload Error 2024-09-22T04:14:06.070Z,1726978446.070 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.794266 m , settling for 1.000000 min 2024-09-22T04:15:18.834Z,1726978518.834 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T04:15:19.009Z,1726978519.009 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T04:16:10.967Z,1726978570.967 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T04:16:11.775Z,1726978571.775 [ESPComponent](IMPORTANT): [sample #23] Setting cartridge argument to -1 2024-09-22T04:16:11.775Z,1726978571.775 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T04:16:12.221Z,1726978572.221 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.624313 m in 9.538202 degC water with 0.068640 ug/l chlorophyll fluorescence. 2024-09-22T04:16:12.579Z,1726978572.579 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T04:16:12.986Z,1726978572.986 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T04:16:15.006Z,1726978575.006 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T04:16:47.331Z,1726978607.331 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PRIMING (@29,aRNAl_bac) 2024-09-22T04:16:50.161Z,1726978610.161 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T04:16:50.162Z,1726978610.162 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T04:16:50.563Z,1726978610.563 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T04:20:55.387Z,1726978855.387 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T04:20:55.792Z,1726978855.792 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T05:11:57.186Z,1726981917.186 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T05:11:57.591Z,1726981917.591 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T05:11:59.612Z,1726981919.612 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T05:12:00.014Z,1726981920.014 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PROCESSED (@27,aRNAl_bac) 2024-09-22T05:13:34.955Z,1726982014.955 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPING (@27,aRNAl_bac) 2024-09-22T05:13:42.632Z,1726982022.632 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T05:13:43.036Z,1726982023.036 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPED 2024-09-22T05:18:42.811Z,1726982322.811 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence completed normally. 2024-09-22T05:18:45.322Z,1726982325.322 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2024-09-22T05:18:45.691Z,1726982325.691 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T05:18:46.077Z,1726982326.077 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T05:18:46.483Z,1726982326.483 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T05:18:46.899Z,1726982326.899 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.365494 m 2024-09-22T05:18:47.308Z,1726982327.308 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.495386 degC 2024-09-22T05:18:47.696Z,1726982327.696 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.072800 ug/l 2024-09-22T05:18:48.541Z,1726982328.541 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2024-09-22T05:18:52.983Z,1726982332.983 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T05:18:55.278Z,1726982335.278 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T05:27:30.922Z,1726982850.922 [ElevatorServo](FAULT): Overload Error 2024-09-22T06:11:15.400Z,1726985475.400 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030623 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 4.000000 count 2024-09-22T06:13:00.431Z,1726985580.431 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.691864 m , settling for 1.000000 min 2024-09-22T06:14:13.221Z,1726985653.221 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T06:14:13.346Z,1726985653.346 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T06:15:06.915Z,1726985706.915 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T06:15:07.726Z,1726985707.726 [ESPComponent](IMPORTANT): [sample #24] Setting cartridge argument to -1 2024-09-22T06:15:07.726Z,1726985707.726 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T06:15:08.214Z,1726985708.214 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.207092 m in 9.440027 degC water with 0.066560 ug/l chlorophyll fluorescence. 2024-09-22T06:15:08.530Z,1726985708.530 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T06:15:08.936Z,1726985708.936 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T06:15:10.955Z,1726985710.955 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T06:15:42.467Z,1726985742.467 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PRIMING (@26,aRNAl_bac) 2024-09-22T06:15:44.487Z,1726985744.487 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T06:15:44.892Z,1726985744.892 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T06:15:45.296Z,1726985745.296 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T06:18:41.848Z,1726985921.848 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T06:18:42.251Z,1726985922.251 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T07:25:07.719Z,1726989907.719 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T07:25:08.125Z,1726989908.125 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T07:25:10.146Z,1726989910.146 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T07:25:10.552Z,1726989910.552 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PROCESSED (@25,aRNAl_bac) 2024-09-22T07:26:45.486Z,1726990005.486 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPING (@25,aRNAl_bac) 2024-09-22T07:26:53.164Z,1726990013.164 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T07:26:53.568Z,1726990013.568 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPED 2024-09-22T07:31:53.351Z,1726990313.351 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence completed normally. 2024-09-22T07:31:55.828Z,1726990315.828 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 4.000000 count 2024-09-22T07:31:56.255Z,1726990316.255 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T07:31:56.622Z,1726990316.622 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T07:31:57.031Z,1726990317.031 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T07:31:57.452Z,1726990317.452 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.112396 m 2024-09-22T07:31:57.828Z,1726990317.828 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.350794 degC 2024-09-22T07:31:58.250Z,1726990318.250 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.083200 ug/l 2024-09-22T07:31:59.043Z,1726990319.043 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 4.000000 count completed. 2024-09-22T07:32:03.523Z,1726990323.523 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T08:00:42.282Z,1726992042.282 [ElevatorServo](FAULT): Overload Error 2024-09-22T08:11:18.173Z,1726992678.173 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032983 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 5.000000 count 2024-09-22T08:14:48.260Z,1726992888.260 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.234772 m , settling for 1.000000 min 2024-09-22T08:16:01.049Z,1726992961.049 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T08:16:01.254Z,1726992961.254 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T08:16:53.203Z,1726993013.203 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T08:16:54.019Z,1726993014.019 [ESPComponent](IMPORTANT): [sample #25] Setting cartridge argument to -1 2024-09-22T08:16:54.020Z,1726993014.020 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T08:16:54.460Z,1726993014.460 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.744614 m in 9.403986 degC water with 0.068640 ug/l chlorophyll fluorescence. 2024-09-22T08:16:54.819Z,1726993014.819 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T08:16:55.223Z,1726993015.223 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T08:16:57.243Z,1726993017.243 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T08:17:28.757Z,1726993048.757 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PRIMING (@24,aRNAl_bac) 2024-09-22T08:17:31.583Z,1726993051.583 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T08:17:31.584Z,1726993051.584 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T08:17:31.987Z,1726993051.987 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T08:20:28.559Z,1726993228.559 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T08:20:28.963Z,1726993228.963 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T09:04:11.565Z,1726995851.565 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-22T09:04:22.487Z,1726995862.487 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.021066 CHAN A1 (24V): 0.006875 CHAN A2 (12V): 0.005437 CHAN A3 (5V): -0.000297 CHAN B0 (3.3V): 0.000426 CHAN B1 (3.15aV): 0.000358 CHAN B2 (3.15bV): -0.000067 CHAN B3 (GND): -0.001804 OPEN: 0.006610 Full Scale: +/- 1 mA 2024-09-22T09:09:42.723Z,1726996182.723 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T09:09:43.128Z,1726996183.128 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T09:09:45.155Z,1726996185.155 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T09:09:45.551Z,1726996185.551 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PROCESSED (@24,aRNAl_bac) 2024-09-22T09:11:20.891Z,1726996280.891 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPING (@24,aRNAl_bac) 2024-09-22T09:11:28.567Z,1726996288.567 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T09:11:28.971Z,1726996288.971 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPED 2024-09-22T09:16:28.746Z,1726996588.746 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence completed normally. 2024-09-22T09:16:31.312Z,1726996591.312 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 5.000000 count 2024-09-22T09:16:31.614Z,1726996591.614 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T09:16:32.035Z,1726996592.035 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T09:16:32.450Z,1726996592.450 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T09:16:32.822Z,1726996592.822 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.314941 m 2024-09-22T09:16:33.236Z,1726996593.236 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.453913 degC 2024-09-22T09:16:33.647Z,1726996593.647 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.052000 ug/l 2024-09-22T09:16:34.442Z,1726996594.442 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 5.000000 count completed. 2024-09-22T09:16:38.910Z,1726996598.910 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T09:46:24.922Z,1726998384.922 [ElevatorServo](FAULT): Overload Error 2024-09-22T10:11:20.919Z,1726999880.919 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032365 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 6.000000 count 2024-09-22T10:12:04.591Z,1726999924.591 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.118469 m , settling for 1.000000 min 2024-09-22T10:13:17.370Z,1726999997.370 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T10:13:17.499Z,1726999997.499 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T10:14:10.258Z,1727000050.258 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T10:14:11.070Z,1727000051.070 [ESPComponent](IMPORTANT): [sample #26] Setting cartridge argument to -1 2024-09-22T10:14:11.070Z,1727000051.070 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T10:14:11.520Z,1727000051.520 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.822968 m in 9.478510 degC water with 0.067600 ug/l chlorophyll fluorescence. 2024-09-22T10:14:11.876Z,1727000051.876 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T10:14:12.279Z,1727000052.279 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T10:14:14.299Z,1727000054.299 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T10:14:45.811Z,1727000085.811 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PRIMING (@23,aRNAl_bac) 2024-09-22T10:14:48.234Z,1727000088.234 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T10:14:48.236Z,1727000088.236 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T10:14:48.642Z,1727000088.642 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T10:17:45.187Z,1727000265.187 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T10:17:45.591Z,1727000265.591 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T11:03:17.765Z,1727002997.765 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-09-22T11:04:04.083Z,1727003044.083 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T11:04:04.486Z,1727003044.486 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T11:04:06.509Z,1727003046.509 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T11:04:06.911Z,1727003046.911 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PROCESSED (@23,aRNAl_bac) 2024-09-22T11:05:42.255Z,1727003142.255 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPING (@23,aRNAl_bac) 2024-09-22T11:05:49.932Z,1727003149.932 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T11:05:50.336Z,1727003150.336 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPED 2024-09-22T11:10:50.115Z,1727003450.115 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence completed normally. 2024-09-22T11:10:50.923Z,1727003450.923 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated. 2024-09-22T11:10:52.599Z,1727003452.599 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 6.000000 count 2024-09-22T11:10:53.031Z,1727003453.031 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T11:10:53.400Z,1727003453.400 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T11:10:53.784Z,1727003453.784 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T11:10:54.199Z,1727003454.199 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.015686 m 2024-09-22T11:10:54.602Z,1727003454.602 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.654596 degC 2024-09-22T11:10:55.037Z,1727003455.037 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.041600 ug/l 2024-09-22T11:10:55.810Z,1727003455.810 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 6.000000 count completed. 2024-09-22T11:11:00.282Z,1727003460.282 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T11:11:01.542Z,1727003461.542 [ElevatorServo](FAULT): Overload Error 2024-09-22T11:11:24.162Z,1727003484.162 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T11:43:11.278Z,1727005391.278 [ElevatorServo](FAULT): Overload Error 2024-09-22T12:01:03.424Z,1727006463.424 [CircleSample:CycleLogs:A:B](IMPORTANT): Restarting logs.