2024-04-27T15:32:11.755Z,1714231931.755 [CommandExec](IMPORTANT): got command restart logs 2024-04-27T15:36:17.016Z,1714232177.016 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T153617 2024-04-27T15:36:28.351Z,1714232188.351 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 2 h;set sci2_sampling.PeakDetectChl 1 bool;set sci2_sampling.TriggerThresholdChl 0.1 ug/l;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree" 23a0d 1 4.000000 2024-04-27T15:36:28.352Z,1714232188.352 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=23a0d 2024-04-27T15:36:28.361Z,1714232188.361 [CommandExec](IMPORTANT): Scheduled #26 (#1 of 4 with id='23a0d'): "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 2 h;set sci2_sampling.PeakDetectChl 1 bool;set sci2_sampling.TriggerThresholdChl 0.1 ug/l;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree", ASAP 2024-04-27T15:36:29.344Z,1714232189.344 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T153629 2024-04-27T15:36:36.886Z,1714232196.886 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.NumSamplersLocation1 1 count;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;set sci2_sampling.Lat2 36.80 degree;set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count" 23a0d 2 4.000000 2024-04-27T15:36:36.886Z,1714232196.886 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=23a0d 2024-04-27T15:36:36.887Z,1714232196.887 [CommandExec](IMPORTANT): Scheduled #27 (#2 of 4 with id='23a0d'): "set sci2_sampling.NumSamplersLocation1 1 count;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;set sci2_sampling.Lat2 36.80 degree;set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count", ASAP 2024-04-27T15:36:37.856Z,1714232197.856 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T153638 2024-04-27T15:36:45.458Z,1714232205.458 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree;set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count" 23a0d 3 4.000000 2024-04-27T15:36:45.458Z,1714232205.458 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=23a0d 2024-04-27T15:36:45.459Z,1714232205.459 [CommandExec](IMPORTANT): Scheduled #28 (#3 of 4 with id='23a0d'): "set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree;set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count", ASAP 2024-04-27T15:36:46.359Z,1714232206.359 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T153646 2024-04-27T15:36:53.984Z,1714232213.984 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run" 23a0d 4 4.000000 2024-04-27T15:36:53.985Z,1714232213.985 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=23a0d 2024-04-27T15:36:53.986Z,1714232213.986 [CommandExec](IMPORTANT): Scheduled #29 (#4 of 4 with id='23a0d'): "set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run", ASAP 2024-04-27T15:36:54.148Z,1714232214.148 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_sampling.xml 2024-04-27T15:37:06.071Z,1714232226.071 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_sampling.xml id=sci2_sampling 2024-04-27T15:37:25.013Z,1714232245.013 [CommandExec](IMPORTANT): got command set sci2_sampling.MissionTimeout 2 hour 2024-04-27T15:37:25.014Z,1714232245.014 [CommandExec](IMPORTANT): got command set sci2_sampling.PeakDetectChl 1 bool 2024-04-27T15:37:25.015Z,1714232245.015 [CommandExec](IMPORTANT): got command set sci2_sampling.TriggerThresholdChl 0.1 microgram_per_liter 2024-04-27T15:37:25.015Z,1714232245.015 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat1 36.80 degree 2024-04-27T15:37:25.016Z,1714232245.016 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon1 -121.848 degree 2024-04-27T15:37:57.796Z,1714232277.796 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation1 1 count 2024-04-27T15:37:57.797Z,1714232277.797 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location1 1 minute 2024-04-27T15:37:57.798Z,1714232277.798 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat2 36.80 degree 2024-04-27T15:37:57.799Z,1714232277.799 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon2 -121.848 degree 2024-04-27T15:37:57.800Z,1714232277.800 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation2 1 count 2024-04-27T15:38:23.598Z,1714232303.598 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location2 1 minute 2024-04-27T15:38:23.599Z,1714232303.599 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat3 36.80 degree 2024-04-27T15:38:23.599Z,1714232303.599 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon3 -121.848 degree 2024-04-27T15:38:23.600Z,1714232303.600 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation3 1 count 2024-04-27T15:38:35.303Z,1714232315.303 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location3 1 minute 2024-04-27T15:38:35.303Z,1714232315.303 [CommandExec](IMPORTANT): got command run 2024-04-27T15:38:35.305Z,1714232315.305 [CommandExec](IMPORTANT): Running 2024-04-27T15:38:35.457Z,1714232315.457 [MissionManager](IMPORTANT): Started mission sci2_sampling 2024-04-27T15:38:35.485Z,1714232315.485 [sci2_sampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2024-04-27T15:42:12.944Z,1714232532.944 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T15:47:01.590Z,1714232821.590 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T15:48:32.069Z,1714232912.069 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.224850 m 2024-04-27T15:49:32.674Z,1714232972.674 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T15:49:34.809Z,1714232974.809 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T15:51:20.949Z,1714233080.949 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.102358 m 2024-04-27T15:54:21.567Z,1714233261.567 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.893638 m , 1.067200 ug/l , 5.179821 m , 12.837741 degC , 1.889637 ug/l 2024-04-27T15:54:21.977Z,1714233261.977 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 5.179821 m . 2024-04-27T15:54:22.351Z,1714233262.351 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T15:57:34.655Z,1714233454.655 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.187168 m , settling for 1.000000 min 2024-04-27T15:58:37.264Z,1714233517.264 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2024-04-27T15:58:47.398Z,1714233527.398 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T15:58:47.680Z,1714233527.680 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T15:59:45.280Z,1714233585.280 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T15:59:46.183Z,1714233586.183 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2024-04-27T15:59:46.184Z,1714233586.184 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T15:59:46.532Z,1714233586.532 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.098572 m in 12.854395 degC water with 2.010280 ug/l chlorophyll fluorescence. 2024-04-27T15:59:47.304Z,1714233587.304 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T15:59:47.709Z,1714233587.709 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T15:59:49.345Z,1714233589.345 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T16:00:24.126Z,1714233624.126 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@59,archive_bac) 2024-04-27T16:00:26.077Z,1714233626.077 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T16:00:26.489Z,1714233626.489 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T16:00:26.898Z,1714233626.898 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T16:01:08.903Z,1714233668.903 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T16:01:09.296Z,1714233669.296 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T16:01:38.788Z,1714233698.788 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T16:01:39.193Z,1714233699.193 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T16:01:41.214Z,1714233701.214 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T16:01:41.618Z,1714233701.618 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@59,archive_bac) 2024-04-27T16:04:31.712Z,1714233871.712 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@59,archive_bac) 2024-04-27T16:04:43.026Z,1714233883.026 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T16:04:43.597Z,1714233883.597 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2024-04-27T16:09:43.209Z,1714234183.209 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2024-04-27T16:09:44.047Z,1714234184.047 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T16:09:47.660Z,1714234187.660 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T16:09:57.686Z,1714234197.686 [DAT](FAULT): Buffer send receipt timeout failure. 2024-04-27T16:15:14.153Z,1714234514.153 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T16:15:56.935Z,1714234556.935 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T16:16:32.535Z,1714234592.535 [DAT](FAULT): failed to enter online mode 2024-04-27T16:16:32.535Z,1714234592.535 [DAT](FAULT): Failure returning to online mode 2024-04-27T16:20:57.099Z,1714234857.099 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T16:23:15.276Z,1714234995.276 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T16:24:43.792Z,1714235083.792 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.226578 m 2024-04-27T16:25:44.371Z,1714235144.371 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T16:25:46.409Z,1714235146.409 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T16:27:16.092Z,1714235236.092 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.109272 m 2024-04-27T16:28:19.478Z,1714235299.478 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 6, 8, 32, 33, 34, 35, 37, 38, 39, 62. 2024-04-27T16:29:52.867Z,1714235392.867 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.909629 m , 1.740000 ug/l , 9.518502 m , 12.864408 degC , 2.032902 ug/l 2024-04-27T16:29:53.232Z,1714235393.232 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 9.518502 m . 2024-04-27T16:29:53.664Z,1714235393.664 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T16:32:01.709Z,1714235521.709 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 9.476914 m , settling for 1.000000 min 2024-04-27T16:33:14.514Z,1714235594.514 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T16:33:14.638Z,1714235594.638 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T16:34:11.399Z,1714235651.399 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T16:34:12.230Z,1714235652.230 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6 2024-04-27T16:34:12.230Z,1714235652.230 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T16:34:12.672Z,1714235652.672 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 9.136799 m in 12.865656 degC water with 2.432520 ug/l chlorophyll fluorescence. 2024-04-27T16:34:13.025Z,1714235653.025 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T16:34:13.422Z,1714235653.422 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T16:34:15.449Z,1714235655.449 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T16:34:50.188Z,1714235690.188 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@58,archive_bac) 2024-04-27T16:34:53.016Z,1714235693.016 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T16:34:53.017Z,1714235693.017 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T16:34:53.425Z,1714235693.425 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T16:35:35.030Z,1714235735.030 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T16:35:35.455Z,1714235735.455 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T16:36:04.525Z,1714235764.525 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T16:36:04.932Z,1714235764.932 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T16:36:06.946Z,1714235766.946 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T16:36:07.367Z,1714235767.367 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@58,archive_bac) 2024-04-27T16:38:57.440Z,1714235937.440 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@58,archive_bac) 2024-04-27T16:39:08.351Z,1714235948.351 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T16:39:08.757Z,1714235948.757 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2024-04-27T16:44:08.526Z,1714236248.526 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2024-04-27T16:44:09.376Z,1714236249.376 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T16:44:12.990Z,1714236252.990 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T16:49:54.779Z,1714236594.779 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T16:50:48.733Z,1714236648.733 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T16:54:23.252Z,1714236863.252 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T16:55:14.166Z,1714236914.166 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.157428 m 2024-04-27T16:55:48.869Z,1714236948.869 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T16:56:14.764Z,1714236974.764 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T16:56:16.791Z,1714236976.791 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T16:58:03.431Z,1714237083.431 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.065624 m 2024-04-27T17:00:40.185Z,1714237240.185 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.930804 m , 1.519600 ug/l , 10.881414 m , 12.842310 degC , 1.936624 ug/l 2024-04-27T17:00:40.574Z,1714237240.574 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 10.881414 m . 2024-04-27T17:00:40.985Z,1714237240.985 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T17:02:37.360Z,1714237357.360 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 10.859416 m , settling for 1.000000 min 2024-04-27T17:03:50.150Z,1714237430.150 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T17:03:50.282Z,1714237430.282 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T17:04:47.409Z,1714237487.409 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T17:04:48.227Z,1714237488.227 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2024-04-27T17:04:48.228Z,1714237488.228 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T17:04:48.703Z,1714237488.703 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 10.717666 m in 12.841974 degC water with 2.214440 ug/l chlorophyll fluorescence. 2024-04-27T17:04:49.043Z,1714237489.043 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T17:04:49.430Z,1714237489.430 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T17:04:51.456Z,1714237491.456 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T17:05:26.188Z,1714237526.188 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@57,archive_bac) 2024-04-27T17:05:29.027Z,1714237529.027 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T17:05:29.028Z,1714237529.028 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T17:05:29.422Z,1714237529.422 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T17:06:11.033Z,1714237571.033 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T17:06:11.436Z,1714237571.436 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T17:06:40.550Z,1714237600.550 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T17:06:40.933Z,1714237600.933 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T17:06:42.964Z,1714237602.964 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T17:06:43.352Z,1714237603.352 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@57,archive_bac) 2024-04-27T17:09:33.454Z,1714237773.454 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@57,archive_bac) 2024-04-27T17:09:44.753Z,1714237784.753 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T17:09:45.161Z,1714237785.161 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2024-04-27T17:14:44.927Z,1714238084.927 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2024-04-27T17:14:45.761Z,1714238085.761 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T17:14:49.411Z,1714238089.411 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T17:20:52.534Z,1714238452.534 [MissionManager](IMPORTANT): Started mission Default 2024-04-27T17:27:22.418Z,1714238842.418 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.491472 min 2024-04-27T17:27:29.025Z,1714238849.025 [DataOverHttps](IMPORTANT): SBD MOMSN=19602017, MTMSN=20240427T172729 2024-04-27T17:27:39.033Z,1714238859.033 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-04-27T17:27:40.425Z,1714238860.425 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T172740 2024-04-27T17:27:41.111Z,1714238861.111 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-04-27T17:28:04.004Z,1714238884.004 [CommandExec](IMPORTANT): got command set keepstation.MissionTimeout 4 hour 2024-04-27T17:28:04.005Z,1714238884.005 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 45 minute 2024-04-27T17:28:04.009Z,1714238884.009 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.797 degree 2024-04-27T17:28:04.010Z,1714238884.010 [CommandExec](IMPORTANT): got command set keepstation.Longitude -121.847 degree 2024-04-27T17:28:04.011Z,1714238884.011 [CommandExec](IMPORTANT): got command run 2024-04-27T17:28:04.012Z,1714238884.012 [CommandExec](IMPORTANT): Running 2024-04-27T17:28:04.046Z,1714238884.046 [CommandExec](IMPORTANT): got command stop 2024-04-27T17:28:04.046Z,1714238884.046 [CommandExec](IMPORTANT): Scheduling is paused 2024-04-27T17:28:04.123Z,1714238884.123 [MissionManager](IMPORTANT): Started mission keepstation 2024-04-27T17:28:04.506Z,1714238884.506 [MissionManager](IMPORTANT): Started mission Default 2024-04-27T17:33:18.692Z,1714239198.692 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T173319 2024-04-27T17:33:26.298Z,1714239206.298 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 3 h;set sci2_sampling.PeakDetectChl 0 bool;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.NumSamplersLocation1 1 count" 23ffi 1 5.000000 2024-04-27T17:33:26.298Z,1714239206.298 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=23ffi 2024-04-27T17:33:26.299Z,1714239206.299 [CommandExec](IMPORTANT): Scheduled #30 (#1 of 5 with id='23ffi'): "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 3 h;set sci2_sampling.PeakDetectChl 0 bool;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.NumSamplersLocation1 1 count", ASAP 2024-04-27T17:33:27.264Z,1714239207.264 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T173327 2024-04-27T17:33:34.938Z,1714239214.938 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.SampleAtPeakChlLocation1 0 bool;set sci2_sampling.DepDiffFromPeakChlLocation1 NaN m;set sci2_sampling.DepLocation1 40 m;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;set sci2_sampling.Lat2 36.80 degree" 23ffi 2 5.000000 2024-04-27T17:33:34.938Z,1714239214.938 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=23ffi 2024-04-27T17:33:34.939Z,1714239214.939 [CommandExec](IMPORTANT): Scheduled #31 (#2 of 5 with id='23ffi'): "set sci2_sampling.SampleAtPeakChlLocation1 0 bool;set sci2_sampling.DepDiffFromPeakChlLocation1 NaN m;set sci2_sampling.DepLocation1 40 m;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;set sci2_sampling.Lat2 36.80 degree", ASAP 2024-04-27T17:33:35.884Z,1714239215.884 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T173336 2024-04-27T17:33:43.514Z,1714239223.514 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count;set sci2_sampling.SampleAtPeakChlLocation2 0 bool;set sci2_sampling.DepDiffFromPeakChlLocation2 NaN m;set sci2_sampling.DepLocation2 40 m" 23ffi 3 5.000000 2024-04-27T17:33:43.514Z,1714239223.514 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=23ffi 2024-04-27T17:33:43.515Z,1714239223.515 [CommandExec](IMPORTANT): Scheduled #32 (#3 of 5 with id='23ffi'): "set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count;set sci2_sampling.SampleAtPeakChlLocation2 0 bool;set sci2_sampling.DepDiffFromPeakChlLocation2 NaN m;set sci2_sampling.DepLocation2 40 m", ASAP 2024-04-27T17:33:44.432Z,1714239224.432 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T173344 2024-04-27T17:33:52.046Z,1714239232.046 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree;set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count;set sci2_sampling.SampleAtPeakChlLocation3 0 bool" 23ffi 4 5.000000 2024-04-27T17:33:52.046Z,1714239232.046 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=23ffi 2024-04-27T17:33:52.047Z,1714239232.047 [CommandExec](IMPORTANT): Scheduled #33 (#4 of 5 with id='23ffi'): "set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree;set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count;set sci2_sampling.SampleAtPeakChlLocation3 0 bool", ASAP 2024-04-27T17:33:53.040Z,1714239233.040 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T173353 2024-04-27T17:34:02.469Z,1714239242.469 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.DepDiffFromPeakChlLocation3 NaN m;set sci2_sampling.DepLocation3 40 m;set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run" 23ffi 5 5.000000 2024-04-27T17:34:02.470Z,1714239242.470 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=23ffi 2024-04-27T17:34:02.471Z,1714239242.471 [CommandExec](IMPORTANT): Scheduled #34 (#5 of 5 with id='23ffi'): "set sci2_sampling.DepDiffFromPeakChlLocation3 NaN m;set sci2_sampling.DepLocation3 40 m;set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run", ASAP 2024-04-27T17:34:02.655Z,1714239242.655 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_sampling.xml 2024-04-27T17:34:23.216Z,1714239263.216 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_sampling.xml id=sci2_sampling 2024-04-27T17:34:50.914Z,1714239290.914 [CommandExec](IMPORTANT): got command set sci2_sampling.MissionTimeout 3 hour 2024-04-27T17:34:50.915Z,1714239290.915 [CommandExec](IMPORTANT): got command set sci2_sampling.PeakDetectChl 0 bool 2024-04-27T17:34:50.916Z,1714239290.916 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat1 36.80 degree 2024-04-27T17:34:50.918Z,1714239290.918 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon1 -121.848 degree 2024-04-27T17:34:50.919Z,1714239290.919 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation1 1 count 2024-04-27T17:35:04.707Z,1714239304.707 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.996809 min 2024-04-27T17:35:22.797Z,1714239322.797 [CommandExec](IMPORTANT): got command set sci2_sampling.SampleAtPeakChlLocation1 0 bool 2024-04-27T17:35:22.802Z,1714239322.802 [CommandExec](IMPORTANT): got command set sci2_sampling.DepDiffFromPeakChlLocation1 NaN meter 2024-04-27T17:35:22.802Z,1714239322.802 [CommandExec](IMPORTANT): got command set sci2_sampling.DepLocation1 40 meter 2024-04-27T17:35:22.803Z,1714239322.803 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location1 1 minute 2024-04-27T17:35:22.804Z,1714239322.804 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat2 36.80 degree 2024-04-27T17:36:04.905Z,1714239364.905 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon2 -121.848 degree 2024-04-27T17:36:04.913Z,1714239364.913 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation2 1 count 2024-04-27T17:36:04.914Z,1714239364.914 [CommandExec](IMPORTANT): got command set sci2_sampling.SampleAtPeakChlLocation2 0 bool 2024-04-27T17:36:04.915Z,1714239364.915 [CommandExec](IMPORTANT): got command set sci2_sampling.DepDiffFromPeakChlLocation2 NaN meter 2024-04-27T17:36:04.916Z,1714239364.916 [CommandExec](IMPORTANT): got command set sci2_sampling.DepLocation2 40 meter 2024-04-27T17:36:48.694Z,1714239408.694 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location2 1 minute 2024-04-27T17:36:48.694Z,1714239408.694 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat3 36.80 degree 2024-04-27T17:36:48.695Z,1714239408.695 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon3 -121.848 degree 2024-04-27T17:36:48.696Z,1714239408.696 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation3 1 count 2024-04-27T17:36:48.697Z,1714239408.697 [CommandExec](IMPORTANT): got command set sci2_sampling.SampleAtPeakChlLocation3 0 bool 2024-04-27T17:37:09.369Z,1714239429.369 [CommandExec](IMPORTANT): got command set sci2_sampling.DepDiffFromPeakChlLocation3 NaN meter 2024-04-27T17:37:09.370Z,1714239429.370 [CommandExec](IMPORTANT): got command set sci2_sampling.DepLocation3 40 meter 2024-04-27T17:37:09.370Z,1714239429.370 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location3 1 minute 2024-04-27T17:37:09.371Z,1714239429.371 [CommandExec](IMPORTANT): got command run 2024-04-27T17:37:09.400Z,1714239429.400 [CommandExec](IMPORTANT): Running 2024-04-27T17:37:09.655Z,1714239429.655 [MissionManager](IMPORTANT): Started mission sci2_sampling 2024-04-27T17:37:09.685Z,1714239429.685 [sci2_sampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2024-04-27T17:39:04.034Z,1714239544.034 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T17:42:59.980Z,1714239779.980 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T17:44:03.427Z,1714239843.427 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T17:44:04.999Z,1714239844.999 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T17:44:05.481Z,1714239845.481 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , nan m , nan degC , 40.000000 m , 1.000000 min . 2024-04-27T17:44:09.065Z,1714239849.065 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 40.000000 m . 2024-04-27T17:44:09.548Z,1714239849.548 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T17:50:49.476Z,1714240249.476 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 40.025898 m , settling for 1.000000 min 2024-04-27T17:52:02.233Z,1714240322.233 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T17:52:02.364Z,1714240322.364 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T17:53:00.343Z,1714240380.343 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T17:53:01.142Z,1714240381.142 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2024-04-27T17:53:01.143Z,1714240381.143 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T17:53:01.599Z,1714240381.599 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 39.958923 m in 10.330866 degC water with 0.422240 ug/l chlorophyll fluorescence. 2024-04-27T17:53:01.953Z,1714240381.953 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T17:53:02.356Z,1714240382.356 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T17:53:04.379Z,1714240384.379 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T17:53:39.184Z,1714240419.184 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@56,archive_bac) 2024-04-27T17:53:41.938Z,1714240421.938 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T17:53:41.939Z,1714240421.939 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T17:53:42.342Z,1714240422.342 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T17:54:23.977Z,1714240463.977 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T17:54:24.365Z,1714240464.365 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T17:54:53.453Z,1714240493.453 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T17:54:53.860Z,1714240493.860 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T17:54:55.869Z,1714240495.869 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T17:54:56.272Z,1714240496.272 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,archive_bac) 2024-04-27T17:57:46.765Z,1714240666.765 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@56,archive_bac) 2024-04-27T17:57:57.674Z,1714240677.674 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T17:57:58.081Z,1714240678.081 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2024-04-27T18:02:57.860Z,1714240977.860 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2024-04-27T18:02:58.723Z,1714240978.723 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T18:03:02.311Z,1714240982.311 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T18:10:24.684Z,1714241424.684 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T18:11:08.901Z,1714241468.901 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T18:12:52.862Z,1714241572.862 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. Invalid Command. 2024-04-27T18:16:09.967Z,1714241769.967 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T18:16:30.588Z,1714241790.588 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T18:17:28.787Z,1714241848.787 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T18:17:30.885Z,1714241850.885 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , nan m , nan degC , 40.000000 m , 1.000000 min . 2024-04-27T18:17:34.422Z,1714241854.422 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 40.000000 m . 2024-04-27T18:17:34.847Z,1714241854.847 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T18:21:34.800Z,1714242094.800 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 7, 8, 32, 33, 34, 35, 37, 38, 39, 62. 2024-04-27T18:24:18.460Z,1714242258.460 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 40.017254 m , settling for 1.000000 min 2024-04-27T18:25:31.242Z,1714242331.242 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T18:25:31.372Z,1714242331.372 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T18:26:28.916Z,1714242388.916 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T18:26:29.733Z,1714242389.733 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6 2024-04-27T18:26:29.734Z,1714242389.734 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T18:26:30.170Z,1714242390.170 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 39.540211 m in 10.452387 degC water with 0.470960 ug/l chlorophyll fluorescence. 2024-04-27T18:26:30.525Z,1714242390.525 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T18:26:30.930Z,1714242390.930 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T18:26:32.956Z,1714242392.956 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T18:27:07.703Z,1714242427.703 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@55,archive_bac) 2024-04-27T18:27:10.524Z,1714242430.524 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T18:27:10.525Z,1714242430.525 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T18:27:10.924Z,1714242430.924 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T18:27:52.544Z,1714242472.544 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T18:27:52.949Z,1714242472.949 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T18:28:22.030Z,1714242502.030 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T18:28:22.432Z,1714242502.432 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T18:28:24.460Z,1714242504.460 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T18:28:24.878Z,1714242504.878 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,archive_bac) 2024-04-27T18:31:15.345Z,1714242675.345 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@55,archive_bac) 2024-04-27T18:31:25.856Z,1714242685.856 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T18:31:26.280Z,1714242686.280 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2024-04-27T18:36:26.028Z,1714242986.028 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2024-04-27T18:36:26.935Z,1714242986.935 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T18:36:30.506Z,1714242990.506 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T18:37:28.870Z,1714243048.870 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-04-27T18:37:28.870Z,1714243048.870 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-04-27T18:43:53.680Z,1714243433.680 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T18:44:43.109Z,1714243483.109 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T18:46:07.273Z,1714243567.273 [Radio_Surface](FAULT): LCB fault: Invalid Command. 2024-04-27T18:49:01.678Z,1714243741.678 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T18:49:44.060Z,1714243784.060 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T18:50:18.243Z,1714243818.243 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T18:50:20.148Z,1714243820.148 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , nan m , nan degC , 40.000000 m , 1.000000 min . 2024-04-27T18:50:23.676Z,1714243823.676 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 40.000000 m . 2024-04-27T18:50:24.147Z,1714243824.147 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T18:57:03.292Z,1714244223.292 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 40.007751 m , settling for 1.000000 min 2024-04-27T18:58:16.050Z,1714244296.050 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T18:58:16.396Z,1714244296.396 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T18:59:13.560Z,1714244353.560 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T18:59:14.380Z,1714244354.380 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6 2024-04-27T18:59:14.381Z,1714244354.381 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T18:59:14.821Z,1714244354.821 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 40.500347 m in 10.210413 degC water with 0.374680 ug/l chlorophyll fluorescence. 2024-04-27T18:59:15.173Z,1714244355.173 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T18:59:15.575Z,1714244355.575 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T18:59:17.596Z,1714244357.596 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T18:59:52.346Z,1714244392.346 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@54,archive_bac) 2024-04-27T18:59:55.173Z,1714244395.173 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T18:59:55.174Z,1714244395.174 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T18:59:55.581Z,1714244395.581 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T19:00:37.310Z,1714244437.310 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T19:00:37.599Z,1714244437.599 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T19:01:06.697Z,1714244466.697 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T19:01:07.084Z,1714244467.084 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T19:01:09.110Z,1714244469.110 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T19:01:09.531Z,1714244469.531 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,archive_bac) 2024-04-27T19:02:36.788Z,1714244556.788 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 7, 8, 32, 33, 34, 35, 37, 38, 39, 62. 2024-04-27T19:04:00.008Z,1714244640.008 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@54,archive_bac) 2024-04-27T19:04:10.542Z,1714244650.542 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T19:04:10.989Z,1714244650.989 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2024-04-27T19:09:10.692Z,1714244950.692 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2024-04-27T19:09:11.542Z,1714244951.542 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T19:09:15.236Z,1714244955.236 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T19:16:05.929Z,1714245365.929 [MissionManager](IMPORTANT): Started mission Default 2024-04-27T19:21:59.891Z,1714245719.891 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.893173 min 2024-04-27T19:23:08.831Z,1714245788.831 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T192309 2024-04-27T19:23:20.009Z,1714245800.009 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 4 h;set sci2_sampling.ProfileStationAtWaypoint 1 bool;set sci2_sampling.MaxDurationProfileStation 10 min;set sci2_sampling.PeakDetectChl 1 bool" 23kem 1 4.000000 2024-04-27T19:23:20.010Z,1714245800.010 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=23kem 2024-04-27T19:23:20.011Z,1714245800.011 [CommandExec](IMPORTANT): Scheduled #35 (#1 of 4 with id='23kem'): "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 4 h;set sci2_sampling.ProfileStationAtWaypoint 1 bool;set sci2_sampling.MaxDurationProfileStation 10 min;set sci2_sampling.PeakDetectChl 1 bool", ASAP 2024-04-27T19:23:20.229Z,1714245800.229 [DataOverHttps](IMPORTANT): SBD MOMSN=19602418, MTMSN=20240427T192321 2024-04-27T19:23:28.914Z,1714245808.914 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.TriggerThresholdChl 0.1 ug/l;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.NumSamplersLocation1 1 count;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min" 23kem 2 4.000000 2024-04-27T19:23:28.915Z,1714245808.915 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=23kem 2024-04-27T19:23:28.916Z,1714245808.916 [CommandExec](IMPORTANT): Scheduled #36 (#2 of 4 with id='23kem'): "set sci2_sampling.TriggerThresholdChl 0.1 ug/l;set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.NumSamplersLocation1 1 count;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min", ASAP 2024-04-27T19:23:30.041Z,1714245810.041 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T192330 2024-04-27T19:23:37.497Z,1714245817.497 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.Lat2 36.80 degree;set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count;set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree" 23kem 3 4.000000 2024-04-27T19:23:37.497Z,1714245817.497 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=23kem 2024-04-27T19:23:37.498Z,1714245817.498 [CommandExec](IMPORTANT): Scheduled #37 (#3 of 4 with id='23kem'): "set sci2_sampling.Lat2 36.80 degree;set sci2_sampling.Lon2 -121.848 degree;set sci2_sampling.NumSamplersLocation2 1 count;set sci2_sampling.WaitTimeDepUndulation1Location2 1 min;set sci2_sampling.Lat3 36.80 degree", ASAP 2024-04-27T19:23:38.689Z,1714245818.689 [DataOverHttps](IMPORTANT): SBD MOMSN=19602423, MTMSN=20240427T192339 2024-04-27T19:23:48.946Z,1714245828.946 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count;set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run" 23kem 4 4.000000 2024-04-27T19:23:48.946Z,1714245828.946 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=23kem 2024-04-27T19:23:48.947Z,1714245828.947 [CommandExec](IMPORTANT): Scheduled #38 (#4 of 4 with id='23kem'): "set sci2_sampling.Lon3 -121.848 degree;set sci2_sampling.NumSamplersLocation3 1 count;set sci2_sampling.WaitTimeDepUndulation1Location3 1 min;run", ASAP 2024-04-27T19:23:49.144Z,1714245829.144 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_sampling.xml 2024-04-27T19:24:11.675Z,1714245851.675 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_sampling.xml id=sci2_sampling 2024-04-27T19:24:54.854Z,1714245894.854 [CommandExec](IMPORTANT): got command set sci2_sampling.MissionTimeout 4 hour 2024-04-27T19:24:54.855Z,1714245894.855 [CommandExec](IMPORTANT): got command set sci2_sampling.ProfileStationAtWaypoint 1 bool 2024-04-27T19:24:54.856Z,1714245894.856 [CommandExec](IMPORTANT): got command set sci2_sampling.MaxDurationProfileStation 10 minute 2024-04-27T19:24:54.862Z,1714245894.862 [CommandExec](IMPORTANT): got command set sci2_sampling.PeakDetectChl 1 bool 2024-04-27T19:25:28.872Z,1714245928.872 [CommandExec](IMPORTANT): got command set sci2_sampling.TriggerThresholdChl 0.1 microgram_per_liter 2024-04-27T19:25:28.872Z,1714245928.872 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat1 36.80 degree 2024-04-27T19:25:28.873Z,1714245928.873 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon1 -121.848 degree 2024-04-27T19:25:28.874Z,1714245928.874 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation1 1 count 2024-04-27T19:25:28.876Z,1714245928.876 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location1 1 minute 2024-04-27T19:26:08.001Z,1714245968.001 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat2 36.80 degree 2024-04-27T19:26:08.002Z,1714245968.002 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon2 -121.848 degree 2024-04-27T19:26:08.003Z,1714245968.003 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation2 1 count 2024-04-27T19:26:08.003Z,1714245968.003 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location2 1 minute 2024-04-27T19:26:08.004Z,1714245968.004 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat3 36.80 degree 2024-04-27T19:26:41.892Z,1714246001.892 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon3 -121.848 degree 2024-04-27T19:26:41.893Z,1714246001.893 [CommandExec](IMPORTANT): got command set sci2_sampling.NumSamplersLocation3 1 count 2024-04-27T19:26:41.894Z,1714246001.894 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location3 1 minute 2024-04-27T19:26:41.894Z,1714246001.894 [CommandExec](IMPORTANT): got command run 2024-04-27T19:26:41.902Z,1714246001.902 [CommandExec](IMPORTANT): Running 2024-04-27T19:26:42.016Z,1714246002.016 [MissionManager](IMPORTANT): Started mission sci2_sampling 2024-04-27T19:26:42.088Z,1714246002.088 [sci2_sampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2024-04-27T19:31:18.210Z,1714246278.210 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T19:34:11.099Z,1714246451.099 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T19:51:11.970Z,1714247471.970 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T19:51:12.002Z,1714247472.002 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T19:51:12.359Z,1714247472.359 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.186385 m 2024-04-27T19:52:13.084Z,1714247533.084 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T19:52:15.098Z,1714247535.098 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T19:54:19.526Z,1714247659.526 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.049635 m 2024-04-27T19:56:13.027Z,1714247773.027 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T19:56:45.843Z,1714247805.843 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.923026 m , 1.125200 ug/l , 6.731772 m , 12.849371 degC , 2.158181 ug/l 2024-04-27T19:56:46.258Z,1714247806.258 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 6.731772 m . 2024-04-27T19:56:46.577Z,1714247806.577 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T19:59:38.693Z,1714247978.693 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.701512 m , settling for 1.000000 min 2024-04-27T20:00:51.482Z,1714248051.482 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T20:00:51.758Z,1714248051.758 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T20:01:48.970Z,1714248108.970 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T20:01:49.732Z,1714248109.732 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2024-04-27T20:01:49.732Z,1714248109.732 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T20:01:50.157Z,1714248110.157 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.401151 m in 12.875666 degC water with 2.475440 ug/l chlorophyll fluorescence. 2024-04-27T20:01:50.540Z,1714248110.540 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T20:01:50.956Z,1714248110.956 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T20:01:52.974Z,1714248112.974 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T20:02:27.709Z,1714248147.709 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@52,archive_bac) 2024-04-27T20:02:30.528Z,1714248150.528 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T20:02:30.529Z,1714248150.529 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T20:02:30.935Z,1714248150.935 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T20:03:12.565Z,1714248192.565 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T20:03:12.953Z,1714248192.953 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T20:03:42.053Z,1714248222.053 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T20:03:42.463Z,1714248222.463 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T20:03:44.470Z,1714248224.470 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T20:03:44.872Z,1714248224.872 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,archive_bac) 2024-04-27T20:06:35.359Z,1714248395.359 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@52,archive_bac) 2024-04-27T20:06:45.862Z,1714248405.862 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T20:06:46.276Z,1714248406.276 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2024-04-27T20:11:46.040Z,1714248706.040 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2024-04-27T20:11:46.879Z,1714248706.879 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T20:11:50.502Z,1714248710.502 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T20:17:26.616Z,1714249046.616 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T20:18:16.971Z,1714249096.971 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T20:23:17.920Z,1714249397.920 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T20:27:02.566Z,1714249622.566 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T20:40:03.288Z,1714250403.288 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T20:40:03.302Z,1714250403.302 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T20:44:35.713Z,1714250675.713 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T20:45:22.826Z,1714250722.826 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.197190 m 2024-04-27T20:46:23.568Z,1714250783.568 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T20:46:25.608Z,1714250785.608 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T20:49:51.496Z,1714250991.496 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.081617 m 2024-04-27T20:50:22.571Z,1714251022.571 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T20:52:44.849Z,1714251164.849 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.873329 m , 0.928000 ug/l , 7.617844 m , 12.795050 degC , 1.645450 ug/l 2024-04-27T20:52:45.214Z,1714251165.214 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 7.617844 m . 2024-04-27T20:52:45.611Z,1714251165.611 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T20:53:23.238Z,1714251203.238 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 7.538632 m , settling for 1.000000 min 2024-04-27T20:54:36.002Z,1714251276.002 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T20:54:36.126Z,1714251276.126 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T20:55:33.267Z,1714251333.267 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T20:55:34.200Z,1714251334.200 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6 2024-04-27T20:55:34.201Z,1714251334.201 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T20:55:34.521Z,1714251334.521 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 7.389533 m in 12.786982 degC water with 1.685480 ug/l chlorophyll fluorescence. 2024-04-27T20:55:34.889Z,1714251334.889 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T20:55:35.288Z,1714251335.288 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T20:55:37.316Z,1714251337.316 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T20:56:12.070Z,1714251372.070 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@51,archive_bac) 2024-04-27T20:56:14.882Z,1714251374.882 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T20:56:14.883Z,1714251374.883 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T20:56:15.285Z,1714251375.285 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T20:56:56.896Z,1714251416.896 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T20:56:57.308Z,1714251417.308 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T20:57:26.793Z,1714251446.793 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T20:57:27.201Z,1714251447.201 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T20:57:29.225Z,1714251449.225 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T20:57:29.630Z,1714251449.630 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,archive_bac) 2024-04-27T21:00:19.717Z,1714251619.717 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@51,archive_bac) 2024-04-27T21:00:30.209Z,1714251630.209 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T21:00:30.617Z,1714251630.617 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2024-04-27T21:04:17.717Z,1714251857.717 [CBIT](IMPORTANT): Beginning ground fault scan 2024-04-27T21:04:29.037Z,1714251869.037 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001920 CHAN A1 (24V): -0.001767 CHAN A2 (12V): -0.000451 CHAN A3 (5V): -0.001063 CHAN B0 (3.3V): 0.000343 CHAN B1 (3.15aV): 0.000415 CHAN B2 (3.15bV): 0.000537 CHAN B3 (GND): -0.000699 OPEN: 0.007898 Full Scale: +/- 1 mA 2024-04-27T21:05:30.396Z,1714251930.396 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2024-04-27T21:05:31.222Z,1714251931.222 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T21:05:34.856Z,1714251934.856 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T21:10:04.089Z,1714252204.089 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T21:10:04.094Z,1714252204.094 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T21:11:10.786Z,1714252270.786 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T21:11:52.681Z,1714252312.681 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T21:16:52.869Z,1714252612.869 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T21:27:13.887Z,1714253233.887 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T21:28:05.173Z,1714253285.173 [sci2_sampling:ProfileStationWpt:E.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T21:40:35.427Z,1714254035.427 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T21:40:35.433Z,1714254035.433 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T21:44:27.867Z,1714254267.867 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.211884 m 2024-04-27T21:45:27.992Z,1714254327.992 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T21:45:30.013Z,1714254330.013 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T21:48:35.151Z,1714254515.151 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.086802 m 2024-04-27T21:49:28.453Z,1714254568.453 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T21:51:07.526Z,1714254667.526 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.876784 m , 0.568400 ug/l , 7.120449 m , 12.747602 degC , 1.475512 ug/l 2024-04-27T21:51:07.889Z,1714254667.889 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 7.120449 m . 2024-04-27T21:51:08.271Z,1714254668.271 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T21:51:55.969Z,1714254715.969 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 7.070157 m , settling for 1.000000 min 2024-04-27T21:53:08.738Z,1714254788.738 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T21:53:08.879Z,1714254788.879 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T21:54:03.332Z,1714254843.332 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T21:54:04.127Z,1714254844.127 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6 2024-04-27T21:54:04.128Z,1714254844.128 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T21:54:04.541Z,1714254844.541 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.988475 m in 12.636286 degC water with 1.408240 ug/l chlorophyll fluorescence. 2024-04-27T21:54:04.911Z,1714254844.911 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T21:54:05.312Z,1714254845.312 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T21:54:07.334Z,1714254847.334 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T21:54:42.069Z,1714254882.069 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@50,archive_bac) 2024-04-27T21:54:44.088Z,1714254884.088 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T21:54:44.495Z,1714254884.495 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T21:54:44.916Z,1714254884.916 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T21:55:26.932Z,1714254926.932 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T21:55:27.348Z,1714254927.348 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T21:55:56.810Z,1714254956.810 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T21:55:57.240Z,1714254957.240 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T21:55:59.232Z,1714254959.232 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T21:55:59.641Z,1714254959.641 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,archive_bac) 2024-04-27T21:58:49.744Z,1714255129.744 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@50,archive_bac) 2024-04-27T21:59:00.630Z,1714255140.630 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T21:59:01.041Z,1714255141.041 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2024-04-27T22:04:00.805Z,1714255440.805 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2024-04-27T22:04:01.632Z,1714255441.632 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T22:04:05.272Z,1714255445.272 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T22:09:40.689Z,1714255780.689 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T22:10:39.155Z,1714255839.155 [MissionManager](IMPORTANT): Started mission Default 2024-04-27T22:12:38.736Z,1714255958.736 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T221239 2024-04-27T22:12:46.432Z,1714255966.432 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 2 h;set sci2_sampling.ProfileStationAtWaypoint 0 bool;set sci2_sampling.PeakDetectChl 1 bool;set sci2_sampling.TriggerThresholdChl 0.1 ug/l" 23scg 1 2.000000 2024-04-27T22:12:46.433Z,1714255966.433 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=23scg 2024-04-27T22:12:46.434Z,1714255966.434 [CommandExec](IMPORTANT): Scheduled #39 (#1 of 2 with id='23scg'): "load Science/sci2_sampling.xml;set sci2_sampling.MissionTimeout 2 h;set sci2_sampling.ProfileStationAtWaypoint 0 bool;set sci2_sampling.PeakDetectChl 1 bool;set sci2_sampling.TriggerThresholdChl 0.1 ug/l", ASAP 2024-04-27T22:12:47.351Z,1714255967.351 [DataOverHttps](IMPORTANT): SBD MTMSN=20240427T221248 2024-04-27T22:12:54.985Z,1714255974.985 [CommandExec](IMPORTANT): got command schedule asap "set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;run" 23scg 2 2.000000 2024-04-27T22:12:54.985Z,1714255974.985 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=23scg 2024-04-27T22:12:54.986Z,1714255974.986 [CommandExec](IMPORTANT): Scheduled #40 (#2 of 2 with id='23scg'): "set sci2_sampling.Lat1 36.80 degree;set sci2_sampling.Lon1 -121.848 degree;set sci2_sampling.WaitTimeDepUndulation1Location1 1 min;run", ASAP 2024-04-27T22:12:55.130Z,1714255975.130 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_sampling.xml 2024-04-27T22:13:07.006Z,1714255987.006 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_sampling.xml id=sci2_sampling 2024-04-27T22:13:26.100Z,1714256006.100 [CommandExec](IMPORTANT): got command set sci2_sampling.MissionTimeout 2 hour 2024-04-27T22:13:26.100Z,1714256006.100 [CommandExec](IMPORTANT): got command set sci2_sampling.ProfileStationAtWaypoint 0 bool 2024-04-27T22:13:26.102Z,1714256006.102 [CommandExec](IMPORTANT): got command set sci2_sampling.PeakDetectChl 1 bool 2024-04-27T22:13:26.102Z,1714256006.102 [CommandExec](IMPORTANT): got command set sci2_sampling.TriggerThresholdChl 0.1 microgram_per_liter 2024-04-27T22:13:42.074Z,1714256022.074 [CommandExec](IMPORTANT): got command set sci2_sampling.Lat1 36.80 degree 2024-04-27T22:13:42.075Z,1714256022.075 [CommandExec](IMPORTANT): got command set sci2_sampling.Lon1 -121.848 degree 2024-04-27T22:13:42.076Z,1714256022.076 [CommandExec](IMPORTANT): got command set sci2_sampling.WaitTimeDepUndulation1Location1 1 minute 2024-04-27T22:13:42.080Z,1714256022.080 [CommandExec](IMPORTANT): got command run 2024-04-27T22:13:42.128Z,1714256022.128 [CommandExec](IMPORTANT): Running 2024-04-27T22:13:42.147Z,1714256022.147 [MissionManager](IMPORTANT): Started mission sci2_sampling 2024-04-27T22:13:42.230Z,1714256022.230 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T22:13:42.279Z,1714256022.279 [sci2_sampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2024-04-27T22:13:42.281Z,1714256022.281 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T22:15:37.656Z,1714256137.656 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T22:16:27.213Z,1714256187.213 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-121.848000 2024-04-27T22:21:27.674Z,1714256487.674 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-04-27T22:34:10.049Z,1714257250.049 [sci2_sampling:TransitToWaypoint:Transit:F.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-121.848000 2024-04-27T22:34:38.739Z,1714257278.739 [sci2_sampling:ProfileStationWpt:E.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T22:34:39.962Z,1714257279.962 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.608200 m 2024-04-27T22:35:40.566Z,1714257340.566 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 1.000000 count 2024-04-27T22:35:42.588Z,1714257342.588 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 1.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T22:37:27.621Z,1714257447.621 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 20.098036 m 2024-04-27T22:40:13.836Z,1714257613.836 [sci2_sampling:SampleAtPeakChlDepOrTemp:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:K](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 14.943769 m , 0.452400 ug/l , 6.535556 m , 12.781772 degC , 1.390841 ug/l 2024-04-27T22:40:14.076Z,1714257614.076 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 6.535556 m . 2024-04-27T22:40:14.487Z,1714257614.487 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T22:40:54.929Z,1714257654.929 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.439614 m , settling for 1.000000 min 2024-04-27T22:42:07.674Z,1714257727.674 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T22:42:07.797Z,1714257727.797 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T22:43:05.744Z,1714257785.744 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T22:43:06.557Z,1714257786.557 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -6 2024-04-27T22:43:06.558Z,1714257786.558 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T22:43:06.994Z,1714257786.994 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.268041 m in 12.760767 degC water with 1.390840 ug/l chlorophyll fluorescence. 2024-04-27T22:43:07.360Z,1714257787.360 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T22:43:07.766Z,1714257787.766 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T22:43:09.794Z,1714257789.794 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T22:43:44.534Z,1714257824.534 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@49,archive_bac) 2024-04-27T22:43:47.356Z,1714257827.356 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T22:43:47.357Z,1714257827.357 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T22:43:47.787Z,1714257827.787 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T22:44:29.380Z,1714257869.380 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T22:44:29.800Z,1714257869.800 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T22:44:59.295Z,1714257899.295 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T22:44:59.686Z,1714257899.686 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T22:45:01.712Z,1714257901.712 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T22:45:02.104Z,1714257902.104 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,archive_bac) 2024-04-27T22:47:52.197Z,1714258072.197 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@49,archive_bac) 2024-04-27T22:48:02.693Z,1714258082.693 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T22:48:03.162Z,1714258083.162 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2024-04-27T22:53:02.892Z,1714258382.892 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2024-04-27T22:53:03.707Z,1714258383.707 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2024-04-27T22:53:10.987Z,1714258390.987 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 2.000000 count 2024-04-27T22:53:13.020Z,1714258393.020 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 2.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T22:53:16.707Z,1714258396.707 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 6.535556 m . 2024-04-27T22:53:17.044Z,1714258397.044 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T22:53:53.002Z,1714258433.002 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.472891 m , settling for 1.000000 min 2024-04-27T22:55:05.794Z,1714258505.794 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T22:55:05.963Z,1714258505.963 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T22:56:03.636Z,1714258563.636 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T22:56:04.439Z,1714258564.439 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -6 2024-04-27T22:56:04.439Z,1714258564.439 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T22:56:04.886Z,1714258564.886 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.966002 m in 12.794214 degC water with 1.583400 ug/l chlorophyll fluorescence. 2024-04-27T22:56:05.266Z,1714258565.266 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T22:56:05.660Z,1714258565.660 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T22:56:07.749Z,1714258567.749 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T22:56:42.413Z,1714258602.413 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@48,archive_bac) 2024-04-27T22:56:45.286Z,1714258605.286 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T22:56:45.287Z,1714258605.287 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T22:56:45.702Z,1714258605.702 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T22:57:27.261Z,1714258647.261 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T22:57:27.668Z,1714258647.668 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T22:57:57.165Z,1714258677.165 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T22:57:57.568Z,1714258677.568 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T22:57:59.582Z,1714258679.582 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T22:57:59.981Z,1714258679.981 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,archive_bac) 2024-04-27T23:00:50.078Z,1714258850.078 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@48,archive_bac) 2024-04-27T23:01:01.800Z,1714258861.800 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T23:01:02.266Z,1714258862.266 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2024-04-27T23:06:02.077Z,1714259162.077 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2024-04-27T23:06:02.926Z,1714259162.926 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed. 2024-04-27T23:06:10.151Z,1714259170.151 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:A](IMPORTANT): CntSamples = 3.000000 count 2024-04-27T23:06:12.194Z,1714259172.194 [sci2_sampling:SampleAtPeakChlDepOrTemp:SetSampleOptions:F](IMPORTANT): CntSamples = 3.000000 count . Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 1.000000 min . 2024-04-27T23:06:15.799Z,1714259175.799 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Target depth = 6.535556 m . 2024-04-27T23:06:16.325Z,1714259176.325 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2024-04-27T23:06:58.251Z,1714259218.251 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.547225 m , settling for 1.000000 min 2024-04-27T23:08:11.054Z,1714259291.054 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-04-27T23:08:11.186Z,1714259291.186 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-04-27T23:09:09.188Z,1714259349.188 [ESPComponent](IMPORTANT): ESP has connected as client 2024-04-27T23:09:10.034Z,1714259350.034 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -6 2024-04-27T23:09:10.034Z,1714259350.034 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-04-27T23:09:10.452Z,1714259350.452 [sci2_sampling:SampleAtPeakChlDepOrTemp:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.556733 m in 12.752161 degC water with 1.675040 ug/l chlorophyll fluorescence. 2024-04-27T23:09:10.812Z,1714259350.812 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2024-04-27T23:09:11.209Z,1714259351.209 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-04-27T23:09:13.238Z,1714259353.238 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2024-04-27T23:09:47.985Z,1714259387.985 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@46,archive_bac) 2024-04-27T23:09:50.805Z,1714259390.805 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@5.0) 2024-04-27T23:09:50.806Z,1714259390.806 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=4.640000 issuedCmd_=Cmd.startFiltering 2024-04-27T23:09:51.232Z,1714259391.232 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0) 2024-04-27T23:10:33.671Z,1714259433.671 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2024-04-27T23:10:34.044Z,1714259434.044 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-04-27T23:11:03.210Z,1714259463.210 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@10.0) 2024-04-27T23:11:03.566Z,1714259463.566 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@10.0) 2024-04-27T23:11:05.586Z,1714259465.586 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-04-27T23:11:05.983Z,1714259465.983 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@46,archive_bac) 2024-04-27T23:13:56.051Z,1714259636.051 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@46,archive_bac) 2024-04-27T23:14:07.766Z,1714259647.766 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-04-27T23:14:08.173Z,1714259648.173 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2024-04-27T23:19:07.999Z,1714259947.999 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2024-04-27T23:19:08.843Z,1714259948.843 [sci2_sampling:SampleAtPeakChlDepOrTemp:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed. 2024-04-27T23:19:12.496Z,1714259952.496 [sci2_sampling:SendMessage:SendMsg:B](IMPORTANT): Set _.terminateMission to 1 bool 2024-04-27T23:19:37.130Z,1714259977.130 [sci2_sampling:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T23:19:37.135Z,1714259977.135 [sci2_sampling:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-04-27T23:24:42.575Z,1714260282.575 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.