2022-05-20T14:16:17.733Z,1653056177.733 [CommandExec](IMPORTANT): got command restart logs 2022-05-20T14:18:21.194Z,1653056301.194 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T141817 2022-05-20T14:18:28.757Z,1653056308.757 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 45 min;set sci2.Lat1 34.3244 degree;set sci2.Lon1 -119.8820 degree;set sci2.Speed 1.2 m/s;set sci2.YoYoMaxDepth 20 m;set sci2.MaxDepth 40 m" 1od1t 1 2.000000 2022-05-20T14:18:28.757Z,1653056308.757 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=1od1t 2022-05-20T14:18:28.758Z,1653056308.758 [CommandExec](IMPORTANT): Scheduled #21 (#1 of 2 with id='1od1t'): "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 45 min;set sci2.Lat1 34.3244 degree;set sci2.Lon1 -119.8820 degree;set sci2.Speed 1.2 m/s;set sci2.YoYoMaxDepth 20 m;set sci2.MaxDepth 40 m", ASAP 2022-05-20T14:18:29.686Z,1653056309.686 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T141826 2022-05-20T14:18:38.807Z,1653056318.807 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MinOffshore 1 km;run" 1od1t 2 2.000000 2022-05-20T14:18:38.807Z,1653056318.807 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=1od1t 2022-05-20T14:18:38.808Z,1653056318.808 [CommandExec](IMPORTANT): Scheduled #22 (#2 of 2 with id='1od1t'): "set sci2.MinOffshore 1 km;run", ASAP 2022-05-20T14:18:38.923Z,1653056318.923 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-20T14:18:42.273Z,1653056322.273 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-20T14:18:55.327Z,1653056335.327 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 12 hour 2022-05-20T14:18:55.334Z,1653056335.334 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 45 minute 2022-05-20T14:18:55.335Z,1653056335.335 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.3244 degree 2022-05-20T14:18:55.336Z,1653056335.336 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.8820 degree 2022-05-20T14:18:55.337Z,1653056335.337 [CommandExec](IMPORTANT): got command set sci2.Speed 1.2 meter_per_second 2022-05-20T14:18:55.337Z,1653056335.337 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 20 meter 2022-05-20T14:18:55.338Z,1653056335.338 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 40 meter 2022-05-20T14:18:56.839Z,1653056336.839 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 1 kilometer 2022-05-20T14:18:56.840Z,1653056336.840 [CommandExec](IMPORTANT): got command run 2022-05-20T14:18:56.842Z,1653056336.842 [CommandExec](IMPORTANT): Running 2022-05-20T14:18:57.036Z,1653056337.036 [MissionManager](IMPORTANT): Started mission sci2 2022-05-20T14:24:02.705Z,1653056642.705 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-20T15:09:06.560Z,1653059346.560 [ThrusterHE](FAULT): Could not get status:serial timeout 2022-05-20T15:09:06.656Z,1653059346.656 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-05-20T15:09:06.657Z,1653059346.657 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-05-20T15:14:25.232Z,1653059665.232 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-20T15:28:26.136Z,1653060506.136 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-05-20T15:44:26.452Z,1653061466.452 [ElevatorServo](FAULT): Overload Error 2022-05-20T15:44:32.604Z,1653061472.604 [ThrusterHE](FAULT): Could not get status:serial timeout 2022-05-20T15:44:32.700Z,1653061472.700 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-05-20T15:44:32.700Z,1653061472.700 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-05-20T15:57:14.853Z,1653062234.853 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 34.324402,-119.882004 2022-05-20T15:59:11.622Z,1653062351.622 [MissionManager](IMPORTANT): Started mission Default 2022-05-20T15:59:14.196Z,1653062354.196 [ThrusterHE](FAULT): Could not get status:serial timeout 2022-05-20T15:59:14.292Z,1653062354.292 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-05-20T15:59:14.292Z,1653062354.292 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-05-20T16:05:12.007Z,1653062712.007 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-20T16:06:58.013Z,1653062818.013 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T160654 2022-05-20T16:07:05.565Z,1653062825.565 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h" 1oi3e 1 4.000000 2022-05-20T16:07:05.565Z,1653062825.565 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=1oi3e 2022-05-20T16:07:05.566Z,1653062825.566 [CommandExec](IMPORTANT): Scheduled #23 (#1 of 4 with id='1oi3e'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h", ASAP 2022-05-20T16:07:06.562Z,1653062826.562 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T160703 2022-05-20T16:07:14.140Z,1653062834.140 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 1 min;set IsothermDepthSampling.HeadingLeavingSide1 210 degree" 1oi3e 2 4.000000 2022-05-20T16:07:14.141Z,1653062834.141 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=1oi3e 2022-05-20T16:07:14.142Z,1653062834.142 [CommandExec](IMPORTANT): Scheduled #24 (#2 of 4 with id='1oi3e'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 1 min;set IsothermDepthSampling.HeadingLeavingSide1 210 degree", ASAP 2022-05-20T16:07:15.078Z,1653062835.078 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T160711 2022-05-20T16:07:22.904Z,1653062842.904 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.HeadingLeavingSide2 270 degree;set IsothermDepthSampling.HeadingLeavingSide3 30 degree;set IsothermDepthSampling.HeadingLeavingSide4 120 degree;set IsothermDepthSampling.MaxDepth 30 m" 1oi3e 3 4.000000 2022-05-20T16:07:22.905Z,1653062842.905 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=1oi3e 2022-05-20T16:07:22.906Z,1653062842.906 [CommandExec](IMPORTANT): Scheduled #25 (#3 of 4 with id='1oi3e'): "set IsothermDepthSampling.HeadingLeavingSide2 270 degree;set IsothermDepthSampling.HeadingLeavingSide3 30 degree;set IsothermDepthSampling.HeadingLeavingSide4 120 degree;set IsothermDepthSampling.MaxDepth 30 m", ASAP 2022-05-20T16:07:23.858Z,1653062843.858 [DataOverHttps](IMPORTANT): SBD MTMSN=20220520T160720 2022-05-20T16:07:31.440Z,1653062851.440 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10.0 min;run" 1oi3e 4 4.000000 2022-05-20T16:07:31.441Z,1653062851.441 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=1oi3e 2022-05-20T16:07:31.442Z,1653062851.442 [CommandExec](IMPORTANT): Scheduled #26 (#4 of 4 with id='1oi3e'): "set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10.0 min;run", ASAP 2022-05-20T16:07:31.589Z,1653062851.589 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2022-05-20T16:08:04.916Z,1653062884.916 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2022-05-20T16:08:31.590Z,1653062911.590 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2022-05-20T16:08:31.591Z,1653062911.591 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2022-05-20T16:08:31.592Z,1653062911.592 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 hour 2022-05-20T16:08:59.825Z,1653062939.825 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2022-05-20T16:08:59.825Z,1653062939.825 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2022-05-20T16:08:59.826Z,1653062939.826 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.InitialDiveDuration 1 minute 2022-05-20T16:08:59.827Z,1653062939.827 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide1 210 degree 2022-05-20T16:09:26.198Z,1653062966.198 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide2 270 degree 2022-05-20T16:09:26.199Z,1653062966.199 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide3 30 degree 2022-05-20T16:09:26.199Z,1653062966.199 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide4 120 degree 2022-05-20T16:09:26.200Z,1653062966.200 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxDepth 30 meter 2022-05-20T16:10:05.657Z,1653063005.657 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MinOffshore 4 kilometer 2022-05-20T16:10:05.657Z,1653063005.657 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 2 meter 2022-05-20T16:10:05.658Z,1653063005.658 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2022-05-20T16:10:05.659Z,1653063005.659 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10.0 minute 2022-05-20T16:10:05.659Z,1653063005.659 [CommandExec](IMPORTANT): got command run 2022-05-20T16:10:05.742Z,1653063005.742 [CommandExec](IMPORTANT): Running 2022-05-20T16:10:05.766Z,1653063005.766 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2022-05-20T16:10:05.914Z,1653063005.914 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 21.00 hours. 2022-05-20T16:15:11.902Z,1653063311.902 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-20T16:15:16.394Z,1653063316.394 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.377996 m 2022-05-20T16:16:26.062Z,1653063386.062 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2022-05-20T16:19:41.634Z,1653063581.634 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.068583 m 2022-05-20T16:21:57.037Z,1653063717.037 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.798048 m , 3.380000 ug/l , 7.989211 m , 12.198965 degC , 15.315043 ug/l 2022-05-20T16:21:57.407Z,1653063717.407 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 7.989211 m . 2022-05-20T16:21:57.826Z,1653063717.826 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2022-05-20T16:24:19.254Z,1653063859.254 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 7.990987 m , settling for 10.000000 min 2022-05-20T16:34:21.546Z,1653064461.546 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2022-05-20T16:34:31.684Z,1653064471.684 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-20T16:34:31.985Z,1653064471.985 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-20T16:35:27.136Z,1653064527.136 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-20T16:35:28.066Z,1653064528.066 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -10 2022-05-20T16:35:28.067Z,1653064528.067 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-20T16:35:28.489Z,1653064528.489 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 8.233812 m in 12.329828 degC water with 13.845520 ug/l chlorophyll fluorescence. 2022-05-20T16:35:29.159Z,1653064529.159 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-20T16:35:29.561Z,1653064529.561 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-20T16:35:31.178Z,1653064531.178 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-20T16:36:11.785Z,1653064571.785 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@49,daSPR_phyto) 2022-05-20T16:36:14.210Z,1653064574.210 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2022-05-20T16:36:14.211Z,1653064574.211 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-20T16:36:14.613Z,1653064574.613 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-20T16:38:29.962Z,1653064709.962 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2022-05-20T16:38:30.366Z,1653064710.366 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-20T17:31:20.287Z,1653067880.287 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@770.2) 2022-05-20T17:31:20.691Z,1653067880.691 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@770.2) 2022-05-20T17:31:22.712Z,1653067882.712 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-20T17:31:23.127Z,1653067883.127 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,daSPR_phyto) 2022-05-20T18:28:23.049Z,1653071303.049 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@49,daSPR_phyto) 2022-05-20T18:28:33.958Z,1653071313.958 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-20T18:28:34.440Z,1653071314.440 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2022-05-20T18:33:34.213Z,1653071614.213 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2022-05-20T18:33:35.088Z,1653071615.088 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2022-05-20T18:33:38.097Z,1653071618.097 [MissionManager](IMPORTANT): Started mission Default