2024-06-13T09:08:16.640Z,1718269696.640 [CommandExec](IMPORTANT): got command restart logs 2024-06-13T09:23:17.866Z,1718270597.866 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 15.000000 min minutes 2024-06-13T09:29:24.779Z,1718270964.779 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.864743 +/- 0.096530 deg (conf. level 99.90%, sigma: 0.038738 deg). 2024-06-13T09:57:15.736Z,1718272635.736 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.861984 +/- 0.056338 deg (conf. level 99.90%, sigma: 0.034438 deg). 2024-06-13T10:10:30.971Z,1718273430.971 [ThrusterHE](FAULT): Could not get status:serial timeout 2024-06-13T10:29:21.260Z,1718274561.260 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-13T10:35:56.063Z,1718274956.063 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21289, MT Status=1, MTMSN=1719 2024-06-13T10:35:56.742Z,1718274956.742 [CommandExec](IMPORTANT): got command stop 2024-06-13T10:35:56.743Z,1718274956.743 [CommandExec](IMPORTANT): Scheduling is paused for 2 commands 2024-06-13T10:35:56.743Z,1718274956.743 [CommandExec](IMPORTANT): got command schedule clear 2024-06-13T10:35:56.744Z,1718274956.744 [CommandExec](IMPORTANT): Cleared 2 scheduled commands. 2024-06-13T10:35:56.744Z,1718274956.744 [CommandExec](IMPORTANT): got command schedule resume 2024-06-13T10:35:56.744Z,1718274956.744 [CommandExec](IMPORTANT): Scheduling is resumed 2024-06-13T10:35:57.067Z,1718274957.067 [MissionManager](IMPORTANT): Started mission Default 2024-06-13T11:34:26.318Z,1718278466.318 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21293, MT Status=1, MTMSN=1720 2024-06-13T11:34:27.072Z,1718278467.072 [CommandExec](IMPORTANT): got command schedule asap "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4hzkg 1 3.000000 2024-06-13T11:34:27.072Z,1718278467.072 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4hzkg 2024-06-13T11:34:27.073Z,1718278467.073 [CommandExec](IMPORTANT): Scheduled #23 (#1 of 3 with id='4hzkg'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", ASAP 2024-06-13T11:34:50.338Z,1718278490.338 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21294, MT Status=1, MTMSN=1721 2024-06-13T11:34:51.132Z,1718278491.132 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4hzkg 2 3.000000 2024-06-13T11:34:51.132Z,1718278491.132 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4hzkg 2024-06-13T11:34:51.133Z,1718278491.133 [CommandExec](IMPORTANT): Scheduled #24 (#2 of 3 with id='4hzkg'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", ASAP 2024-06-13T11:42:05.566Z,1718278925.566 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21295, MT Status=1, MTMSN=1722 2024-06-13T11:42:06.234Z,1718278926.234 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 20 min;run" 4hzkg 3 3.000000 2024-06-13T11:42:06.234Z,1718278926.234 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4hzkg 2024-06-13T11:42:06.245Z,1718278926.245 [CommandExec](IMPORTANT): Scheduled #25 (#3 of 3 with id='4hzkg'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 20 min;run", ASAP 2024-06-13T11:42:06.407Z,1718278926.407 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-13T11:42:16.006Z,1718278936.006 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-13T11:42:30.906Z,1718278950.906 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-13T11:42:30.907Z,1718278950.907 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 1 bool 2024-06-13T11:42:30.907Z,1718278950.907 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-13T11:42:30.908Z,1718278950.908 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-13T11:42:47.136Z,1718278967.136 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-13T11:42:47.137Z,1718278967.137 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-13T11:42:47.138Z,1718278967.138 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-13T11:42:47.172Z,1718278967.172 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-13T11:42:47.173Z,1718278967.173 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-13T11:43:00.242Z,1718278980.242 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-13T11:43:00.242Z,1718278980.242 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-13T11:43:00.243Z,1718278980.243 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-13T11:43:00.244Z,1718278980.244 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 20 minute 2024-06-13T11:43:00.244Z,1718278980.244 [CommandExec](IMPORTANT): got command run 2024-06-13T11:43:00.246Z,1718278980.246 [CommandExec](IMPORTANT): Running 2024-06-13T11:43:01.329Z,1718278981.329 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-13T11:43:01.430Z,1718278981.430 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-13T11:43:01.606Z,1718278981.606 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.874208 +/- 0.041532 deg (conf. level 99.90%, sigma: 0.026844 deg). 2024-06-13T11:43:02.703Z,1718278982.703 [CircleSample:MissionStartComms:A](IMPORTANT): Mission start comms. 2024-06-13T12:03:20.903Z,1718280200.903 [CircleSample:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 20.000000 min minutes 2024-06-13T13:02:30.551Z,1718283750.551 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (2 messages): @12:16:39.39 Selecting Cartridge 22 @12:55:42.48 Sampled 1000.0ml 2024-06-13T13:02:30.557Z,1718283750.557 [ESPComponent](IMPORTANT): [sample #8] ESP Cartridge 22: S_FILTERING: start: 2024-06-13T12:19:47.910Z end: 2024-06-13T12:55:44.945Z 2024-06-13T14:50:38.061Z,1718290238.061 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (2 messages): @14:10:49.45 Selecting Cartridge 21 @14:43:50.68 Sampled 1000.0ml 2024-06-13T14:50:38.062Z,1718290238.062 [ESPComponent](IMPORTANT): [sample #9] ESP Cartridge 21: S_FILTERING: start: 2024-06-13T14:13:57.836Z end: 2024-06-13T14:43:52.849Z 2024-06-13T12:03:48.860Z,1718280228.860 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-13T12:03:50.886Z,1718280230.886 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-13T12:13:05.119Z,1718280785.119 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T12:13:49.962Z,1718280829.962 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.031884 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-13T12:14:48.967Z,1718280888.967 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.100250 m , settling for 0.500000 min 2024-06-13T12:15:31.848Z,1718280931.848 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-13T12:15:31.992Z,1718280931.992 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-13T12:16:25.891Z,1718280985.891 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-13T12:16:26.706Z,1718280986.706 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2024-06-13T12:16:26.707Z,1718280986.707 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-13T12:16:27.136Z,1718280987.136 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.173244 m in 13.347681 degC water with 1.182040 ug/l chlorophyll fluorescence. 2024-06-13T12:16:27.522Z,1718280987.522 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-13T12:16:27.919Z,1718280987.919 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-13T12:16:29.925Z,1718280989.925 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-13T12:17:01.846Z,1718281021.846 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@22,aRNAl_bac) 2024-06-13T12:17:04.307Z,1718281024.307 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2024-06-13T12:17:04.308Z,1718281024.308 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-13T12:17:04.675Z,1718281024.675 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-13T12:19:47.910Z,1718281187.910 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2024-06-13T12:19:48.307Z,1718281188.307 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-13T12:44:42.919Z,1718282682.919 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-13T12:44:53.750Z,1718282693.750 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001857 CHAN A1 (24V): -0.001241 CHAN A2 (12V): -0.000140 CHAN A3 (5V): 0.000229 CHAN B0 (3.3V): 0.000089 CHAN B1 (3.15aV): 0.000256 CHAN B2 (3.15bV): -0.000350 CHAN B3 (GND): -0.000218 OPEN: 0.007637 Full Scale: +/- 1 mA 2024-06-13T12:55:44.495Z,1718283344.495 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0) 2024-06-13T12:55:44.946Z,1718283344.946 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-13T12:55:46.933Z,1718283346.933 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-13T12:55:47.346Z,1718283347.346 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@22,aRNAl_bac) 2024-06-13T12:56:11.622Z,1718283371.622 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T12:57:22.665Z,1718283442.665 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@22,aRNAl_bac) 2024-06-13T12:57:29.940Z,1718283449.940 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-13T12:57:30.342Z,1718283450.342 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2024-06-13T13:02:30.122Z,1718283750.122 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2024-06-13T13:02:32.649Z,1718283752.649 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-13T13:07:36.447Z,1718284056.447 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-13T13:07:39.653Z,1718284059.653 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 53.814966 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-13T13:09:28.749Z,1718284168.749 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.161942 m , settling for 60.000000 min 2024-06-13T13:26:31.778Z,1718285191.778 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T13:58:04.212Z,1718287084.212 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T14:09:41.567Z,1718287781.567 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-13T14:09:41.672Z,1718287781.672 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-13T14:10:35.758Z,1718287835.758 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-13T14:10:36.583Z,1718287836.583 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2024-06-13T14:10:36.584Z,1718287836.584 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-13T14:10:37.008Z,1718287837.008 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 7.791129 m in 13.461054 degC water with 0.961640 ug/l chlorophyll fluorescence. 2024-06-13T14:10:37.374Z,1718287837.374 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-13T14:10:37.775Z,1718287837.775 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-13T14:10:39.798Z,1718287839.798 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-13T14:11:11.726Z,1718287871.726 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@21,aRNAl_bac) 2024-06-13T14:11:13.729Z,1718287873.729 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2024-06-13T14:11:14.166Z,1718287874.166 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-13T14:11:14.560Z,1718287874.560 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-13T14:13:57.837Z,1718288037.837 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2024-06-13T14:13:58.210Z,1718288038.210 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-13T14:30:04.279Z,1718289004.279 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T14:43:52.456Z,1718289832.456 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1000.0) 2024-06-13T14:43:52.850Z,1718289832.850 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-13T14:43:54.898Z,1718289834.898 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-13T14:43:55.326Z,1718289835.326 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@21,aRNAl_bac) 2024-06-13T14:45:30.232Z,1718289930.232 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@21,aRNAl_bac) 2024-06-13T14:45:37.509Z,1718289937.509 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-13T14:45:37.898Z,1718289937.898 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2024-06-13T14:50:37.667Z,1718290237.667 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2024-06-13T14:50:40.105Z,1718290240.105 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-13T14:50:41.910Z,1718290241.910 [MissionManager](IMPORTANT): Started mission Default 2024-06-13T14:56:05.100Z,1718290565.100 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-13T15:30:05.077Z,1718292605.077 [BPC1](FAULT): Failed to receive data from 19 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-13T16:49:10.608Z,1718297350.608 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21312, MT Status=1, MTMSN=1723 2024-06-13T16:49:11.140Z,1718297351.140 [CommandExec](IMPORTANT): got command schedule clear 2024-06-13T16:49:11.140Z,1718297351.140 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2024-06-13T16:49:11.140Z,1718297351.140 [CommandExec](IMPORTANT): got command schedule resume 2024-06-13T17:09:29.390Z,1718298569.390 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21313, MT Status=1, MTMSN=1724 2024-06-13T17:09:30.180Z,1718298570.180 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m" 4id49 1 2.000000 2024-06-13T17:09:30.180Z,1718298570.180 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4id49 2024-06-13T17:09:30.181Z,1718298570.181 [CommandExec](IMPORTANT): Scheduled #26 (#1 of 2 with id='4id49'): "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m", ASAP 2024-06-13T18:16:46.884Z,1718302606.884 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21314, MT Status=1, MTMSN=1725 2024-06-13T18:16:47.615Z,1718302607.615 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4id49 2 2.000000 2024-06-13T18:16:47.615Z,1718302607.615 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4id49 2024-06-13T18:16:47.616Z,1718302607.616 [CommandExec](IMPORTANT): Scheduled #27 (#2 of 2 with id='4id49'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", ASAP 2024-06-13T18:16:47.771Z,1718302607.771 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-13T18:16:56.026Z,1718302616.026 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-13T18:17:10.505Z,1718302630.505 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 3 hour 2024-06-13T18:17:10.505Z,1718302630.505 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2024-06-13T18:17:10.506Z,1718302630.506 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-13T18:17:10.507Z,1718302630.507 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-13T18:17:10.507Z,1718302630.507 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-13T18:17:17.034Z,1718302637.034 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21315, MT Status=1, MTMSN=1726 2024-06-13T18:17:28.133Z,1718302648.133 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-13T18:17:28.134Z,1718302648.134 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-13T18:17:28.135Z,1718302648.135 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-13T18:17:28.136Z,1718302648.136 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-13T18:17:28.136Z,1718302648.136 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-13T18:17:28.137Z,1718302648.137 [CommandExec](IMPORTANT): got command run 2024-06-13T18:17:28.203Z,1718302648.203 [CommandExec](IMPORTANT): Running 2024-06-13T18:17:28.213Z,1718302648.213 [CommandExec](IMPORTANT): got command schedule 20240613T180000 "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4ide3 1 3.000000 2024-06-13T18:17:28.213Z,1718302648.213 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4ide3 2024-06-13T18:17:28.215Z,1718302648.215 [CommandExec](IMPORTANT): Scheduled #28 (#1 of 3 with id='4ide3'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", AT TIME: 2024-06-13T18:00:00Z 2024-06-13T18:17:30.233Z,1718302650.233 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-13T18:17:46.852Z,1718302666.852 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-06-13T18:23:09.427Z,1718302989.427 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21316, MT Status=1, MTMSN=1727 2024-06-13T18:23:10.176Z,1718302990.176 [CommandExec](IMPORTANT): got command schedule 20240613T180000 "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4ide3 2 3.000000 2024-06-13T18:23:10.177Z,1718302990.177 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4ide3 2024-06-13T18:23:10.178Z,1718302990.178 [CommandExec](IMPORTANT): Scheduled #29 (#2 of 3 with id='4ide3'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", AT TIME: 2024-06-13T18:00:00Z 2024-06-13T18:32:02.948Z,1718303522.948 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-13T18:44:53.814Z,1718304293.814 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-13T18:44:58.300Z,1718304298.300 [RudderServo](FAULT): Reading outside of valid range:0.000000 2024-06-13T18:44:58.931Z,1718304298.931 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-06-13T18:45:04.810Z,1718304304.810 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.003858 CHAN A1 (24V): -0.001653 CHAN A2 (12V): -0.000607 CHAN A3 (5V): 0.000554 CHAN B0 (3.3V): -0.000549 CHAN B1 (3.15aV): 0.000045 CHAN B2 (3.15bV): -0.032352 CHAN B3 (GND): 0.000434 OPEN: 0.007595 Full Scale: +/- 1 mA 2024-06-13T18:45:22.767Z,1718304322.767 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-06-13T19:01:27.628Z,1718305287.628 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.865738 +/- 0.071765 deg (conf. level 99.90%, sigma: 0.040677 deg). 2024-06-13T19:04:40.978Z,1718305480.978 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21317, MT Status=1, MTMSN=1728 2024-06-13T19:04:41.728Z,1718305481.728 [CommandExec](IMPORTANT): got command schedule 20240613T180000 "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run" 4ide3 3 3.000000 2024-06-13T19:04:41.729Z,1718305481.729 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4ide3 2024-06-13T19:04:41.730Z,1718305481.730 [CommandExec](IMPORTANT): Scheduled #30 (#3 of 3 with id='4ide3'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run", AT TIME: 2024-06-13T18:00:00Z 2024-06-13T19:04:41.888Z,1718305481.888 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-13T19:04:49.801Z,1718305489.801 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-13T19:04:49.920Z,1718305489.920 [MissionManager](IMPORTANT): Started mission Default 2024-06-13T19:05:03.415Z,1718305503.415 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-13T19:05:03.416Z,1718305503.416 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 1 bool 2024-06-13T19:05:03.417Z,1718305503.417 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-13T19:05:03.418Z,1718305503.418 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-13T19:05:18.939Z,1718305518.939 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-13T19:05:18.940Z,1718305518.940 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-13T19:05:18.941Z,1718305518.941 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-13T19:05:18.941Z,1718305518.941 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-13T19:05:18.942Z,1718305518.942 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-13T19:05:32.557Z,1718305532.557 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-13T19:05:32.558Z,1718305532.558 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-13T19:05:32.558Z,1718305532.558 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-13T19:05:32.559Z,1718305532.559 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 10 minute 2024-06-13T19:05:32.560Z,1718305532.560 [CommandExec](IMPORTANT): got command run 2024-06-13T19:05:32.568Z,1718305532.568 [CommandExec](IMPORTANT): Running 2024-06-13T19:05:32.854Z,1718305532.854 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-13T19:05:32.939Z,1718305532.939 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-13T19:05:34.498Z,1718305534.498 [CircleSample:MissionStartComms:A](IMPORTANT): Mission start comms. 2024-06-13T20:09:52.077Z,1718309392.077 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (2 messages): @19:28:16.68 Selecting Cartridge 20 @20:03:03.82 Sampled 1000.0ml 2024-06-13T20:09:52.080Z,1718309392.080 [ESPComponent](IMPORTANT): [sample #10] ESP Cartridge 20: S_FILTERING: start: 2024-06-13T19:31:25.587Z end: 2024-06-13T20:03:06.429Z 2024-06-13T22:02:46.383Z,1718316166.383 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (2 messages): @21:17:23.11 Selecting Cartridge 19 @21:55:58.42 Sampled 1000.0ml 2024-06-13T22:02:46.384Z,1718316166.384 [ESPComponent](IMPORTANT): [sample #11] ESP Cartridge 19: S_FILTERING: start: 2024-06-13T21:20:31.645Z end: 2024-06-13T21:56:00.757Z 2024-06-13T19:14:40.741Z,1718306080.741 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21318, MT Status=1, MTMSN=1729 2024-06-13T19:14:41.573Z,1718306081.573 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m" 4iden 1 2.000000 2024-06-13T19:14:41.574Z,1718306081.574 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4iden 2024-06-13T19:14:41.584Z,1718306081.584 [CommandExec](IMPORTANT): Scheduled #31 (#1 of 2 with id='4iden'): "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-13T19:15:28.097Z,1718306128.097 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21319, MT Status=1, MTMSN=1730 2024-06-13T19:15:28.868Z,1718306128.868 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4iden 2 2.000000 2024-06-13T19:15:28.868Z,1718306128.868 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4iden 2024-06-13T19:15:28.869Z,1718306128.869 [CommandExec](IMPORTANT): Scheduled #32 (#2 of 2 with id='4iden'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", AFTER MISSION 2024-06-13T19:15:37.711Z,1718306137.711 [CircleSample:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-13T19:16:05.615Z,1718306165.615 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-13T19:16:07.758Z,1718306167.758 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-13T19:26:06.683Z,1718306766.683 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.031316 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-13T19:26:25.262Z,1718306785.262 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.771044 m , settling for 0.500000 min 2024-06-13T19:27:08.164Z,1718306828.164 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-13T19:27:08.339Z,1718306828.339 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-13T19:28:03.963Z,1718306883.963 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-13T19:28:04.766Z,1718306884.766 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2024-06-13T19:28:04.766Z,1718306884.766 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-13T19:28:05.197Z,1718306885.197 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.293438 m in 13.271875 degC water with 1.243520 ug/l chlorophyll fluorescence. 2024-06-13T19:28:05.569Z,1718306885.569 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-13T19:28:05.979Z,1718306885.979 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-13T19:28:07.999Z,1718306887.999 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-13T19:28:39.927Z,1718306919.927 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@20,aRNAl_bac) 2024-06-13T19:28:42.755Z,1718306922.755 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2024-06-13T19:28:42.756Z,1718306922.756 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-13T19:28:43.147Z,1718306923.147 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-13T19:31:25.588Z,1718307085.588 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2024-06-13T19:31:25.970Z,1718307085.970 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-13T20:03:06.011Z,1718308986.011 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1000.0) 2024-06-13T20:03:06.430Z,1718308986.430 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-13T20:03:08.439Z,1718308988.439 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-13T20:03:08.837Z,1718308988.837 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,aRNAl_bac) 2024-06-13T20:04:43.776Z,1718309083.776 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@20,aRNAl_bac) 2024-06-13T20:04:51.086Z,1718309091.086 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-13T20:04:51.509Z,1718309091.509 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2024-06-13T20:09:51.658Z,1718309391.658 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2024-06-13T20:09:54.101Z,1718309394.101 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-13T20:14:57.994Z,1718309697.994 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-13T20:15:01.249Z,1718309701.249 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 48.895850 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-13T20:16:01.814Z,1718309761.814 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.067882 m , settling for 60.000000 min 2024-06-13T20:24:03.528Z,1718310243.528 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T20:59:01.157Z,1718312341.157 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T21:16:14.764Z,1718313374.764 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-13T21:16:14.932Z,1718313374.932 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-13T21:17:09.653Z,1718313429.653 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-13T21:17:10.465Z,1718313430.465 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2024-06-13T21:17:10.466Z,1718313430.466 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-13T21:17:10.883Z,1718313430.883 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.714882 m in 12.755304 degC water with 1.904720 ug/l chlorophyll fluorescence. 2024-06-13T21:17:11.266Z,1718313431.266 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-13T21:17:11.665Z,1718313431.665 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-13T21:17:13.686Z,1718313433.686 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-13T21:17:45.623Z,1718313465.623 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@19,aRNAl_bac) 2024-06-13T21:17:48.458Z,1718313468.458 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2024-06-13T21:17:48.460Z,1718313468.460 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-13T21:17:48.835Z,1718313468.835 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-13T21:20:31.645Z,1718313631.645 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2024-06-13T21:20:32.072Z,1718313632.072 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-13T21:36:00.227Z,1718314560.227 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-13T21:36:00.551Z,1718314560.551 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-13T21:36:00.958Z,1718314560.958 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-13T21:36:01.379Z,1718314561.379 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-13T21:36:01.735Z,1718314561.735 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-13T21:39:14.017Z,1718314754.017 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T21:56:00.356Z,1718315760.356 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0) 2024-06-13T21:56:00.758Z,1718315760.758 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-13T21:56:02.778Z,1718315762.778 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-13T21:56:03.183Z,1718315763.183 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@19,aRNAl_bac) 2024-06-13T21:57:38.525Z,1718315858.525 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@19,aRNAl_bac) 2024-06-13T21:57:45.799Z,1718315865.799 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-13T21:57:46.221Z,1718315866.221 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2024-06-13T22:02:45.970Z,1718316165.970 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2024-06-13T22:02:48.431Z,1718316168.431 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-13T22:02:49.769Z,1718316169.769 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-13T22:02:55.986Z,1718316175.986 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-13T22:02:56.195Z,1718316176.195 [MissionManager](IMPORTANT): Started mission Default 2024-06-13T22:03:11.027Z,1718316191.027 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 3 hour 2024-06-13T22:03:11.028Z,1718316191.028 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2024-06-13T22:03:11.029Z,1718316191.029 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-13T22:03:11.029Z,1718316191.029 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-13T22:03:11.030Z,1718316191.030 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-13T22:03:28.360Z,1718316208.360 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-13T22:03:28.360Z,1718316208.360 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-13T22:03:28.361Z,1718316208.361 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-13T22:03:28.362Z,1718316208.362 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-13T22:03:28.362Z,1718316208.362 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-13T22:03:28.364Z,1718316208.364 [CommandExec](IMPORTANT): got command run 2024-06-13T22:03:28.366Z,1718316208.366 [CommandExec](IMPORTANT): Running 2024-06-13T22:03:28.506Z,1718316208.506 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-13T22:08:20.581Z,1718316500.581 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-13T22:12:08.736Z,1718316728.736 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21320, MT Status=1, MTMSN=1731 2024-06-13T22:12:09.532Z,1718316729.532 [CommandExec](IMPORTANT): got command schedule 20240613T220100 "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4iqet 1 3.000000 2024-06-13T22:12:09.532Z,1718316729.532 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4iqet 2024-06-13T22:12:09.535Z,1718316729.535 [CommandExec](IMPORTANT): Scheduled #33 (#1 of 3 with id='4iqet'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", AT TIME: 2024-06-13T22:01:00Z 2024-06-13T22:18:11.843Z,1718317091.843 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21321, MT Status=1, MTMSN=1732 2024-06-13T22:18:12.660Z,1718317092.660 [CommandExec](IMPORTANT): got command schedule 20240613T220100 "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4iqet 2 3.000000 2024-06-13T22:18:12.660Z,1718317092.660 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4iqet 2024-06-13T22:18:12.662Z,1718317092.662 [CommandExec](IMPORTANT): Scheduled #34 (#2 of 3 with id='4iqet'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", AT TIME: 2024-06-13T22:01:00Z 2024-06-13T22:18:30.462Z,1718317110.462 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-13T22:21:53.487Z,1718317313.487 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-06-13T22:21:53.487Z,1718317313.487 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-06-13T22:54:03.629Z,1718319243.629 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-13T22:54:52.156Z,1718319292.156 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21322, MT Status=1, MTMSN=1733 2024-06-13T22:54:52.888Z,1718319292.888 [CommandExec](IMPORTANT): got command schedule 20240613T220100 "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run" 4iqet 3 3.000000 2024-06-13T22:54:52.889Z,1718319292.889 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4iqet 2024-06-13T22:54:52.890Z,1718319292.890 [CommandExec](IMPORTANT): Scheduled #35 (#3 of 3 with id='4iqet'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run", AT TIME: 2024-06-13T22:01:00Z 2024-06-13T22:54:53.032Z,1718319293.032 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-13T22:55:00.892Z,1718319300.892 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-13T22:55:01.007Z,1718319301.007 [MissionManager](IMPORTANT): Started mission Default 2024-06-13T22:55:14.454Z,1718319314.454 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-13T22:55:14.455Z,1718319314.455 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 1 bool 2024-06-13T22:55:14.456Z,1718319314.456 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-13T22:55:14.457Z,1718319314.457 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-13T22:55:30.564Z,1718319330.564 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-13T22:55:30.565Z,1718319330.565 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-13T22:55:30.565Z,1718319330.565 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-13T22:55:30.566Z,1718319330.566 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-13T22:55:30.587Z,1718319330.587 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-13T22:55:42.264Z,1718319342.264 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-13T22:55:42.265Z,1718319342.265 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-13T22:55:42.265Z,1718319342.265 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-13T22:55:42.266Z,1718319342.266 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 10 minute 2024-06-13T22:55:42.266Z,1718319342.266 [CommandExec](IMPORTANT): got command run 2024-06-13T22:55:42.269Z,1718319342.269 [CommandExec](IMPORTANT): Running 2024-06-13T22:55:42.573Z,1718319342.573 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-13T22:55:42.666Z,1718319342.666 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-13T22:55:44.164Z,1718319344.164 [CircleSample:MissionStartComms:A](IMPORTANT): Mission start comms. 2024-06-14T00:03:49.211Z,1718323429.211 [ESPComponent](IMPORTANT): [sample #12] ESP log summary report (2 messages): @23:19:04.60 Selecting Cartridge 18 @23:57:01.12 Sampled 1000.0ml 2024-06-14T00:03:49.212Z,1718323429.212 [ESPComponent](IMPORTANT): [sample #12] ESP Cartridge 18: S_FILTERING: start: 2024-06-13T23:22:13.134Z end: 2024-06-13T23:57:03.553Z 2024-06-14T01:56:59.958Z,1718330219.958 [ESPComponent](IMPORTANT): [sample #13] ESP log summary report (2 messages): @01:13:10.47 Selecting Cartridge 17 @01:50:12.32 Sampled 1000.0ml 2024-06-14T01:56:59.961Z,1718330219.961 [ESPComponent](IMPORTANT): [sample #13] ESP Cartridge 17: S_FILTERING: start: 2024-06-14T01:16:20.114Z end: 2024-06-14T01:50:14.309Z 2024-06-13T22:58:21.914Z,1718319501.914 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21323, MT Status=1, MTMSN=1734 2024-06-13T22:58:22.685Z,1718319502.685 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m" 4iqgc 1 2.000000 2024-06-13T22:58:22.686Z,1718319502.686 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4iqgc 2024-06-13T22:58:22.687Z,1718319502.687 [CommandExec](IMPORTANT): Scheduled #36 (#1 of 2 with id='4iqgc'): "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-13T22:59:47.496Z,1718319587.496 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21324, MT Status=1, MTMSN=1735 2024-06-13T22:59:50.016Z,1718319590.016 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4iqgc 2 2.000000 2024-06-13T22:59:50.016Z,1718319590.016 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4iqgc 2024-06-13T22:59:50.017Z,1718319590.017 [CommandExec](IMPORTANT): Scheduled #37 (#2 of 2 with id='4iqgc'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", AFTER MISSION 2024-06-13T23:02:01.719Z,1718319721.719 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T23:02:01.724Z,1718319721.724 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T23:02:51.717Z,1718319771.717 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21325, MT Status=1, MTMSN=1736 2024-06-13T23:02:52.528Z,1718319772.528 [CommandExec](IMPORTANT): got command schedule 20240614T020000 "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4irkg 1 3.000000 2024-06-13T23:02:52.528Z,1718319772.528 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4irkg 2024-06-13T23:02:52.529Z,1718319772.529 [CommandExec](IMPORTANT): Scheduled #38 (#1 of 3 with id='4irkg'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", AT TIME: 2024-06-14T02:00:00Z 2024-06-13T23:05:08.207Z,1718319908.207 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21326, MT Status=1, MTMSN=1737 2024-06-13T23:05:08.977Z,1718319908.977 [CommandExec](IMPORTANT): got command schedule 20240614T020000 "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4irkg 2 3.000000 2024-06-13T23:05:08.977Z,1718319908.977 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4irkg 2024-06-13T23:05:08.978Z,1718319908.978 [CommandExec](IMPORTANT): Scheduled #39 (#2 of 3 with id='4irkg'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", AT TIME: 2024-06-14T02:00:00Z 2024-06-13T23:05:56.132Z,1718319956.132 [CircleSample:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-13T23:06:06.177Z,1718319966.177 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-13T23:06:08.177Z,1718319968.177 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-13T23:09:16.620Z,1718320156.620 [BPC1](FAULT): Failed to receive data from 17 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 62. 2024-06-13T23:16:07.118Z,1718320567.118 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.029607 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-13T23:17:06.100Z,1718320626.100 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.084714 m , settling for 0.500000 min 2024-06-13T23:17:49.025Z,1718320669.025 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-13T23:17:49.183Z,1718320669.183 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-13T23:18:51.542Z,1718320731.542 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-13T23:18:52.366Z,1718320732.366 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -1 2024-06-13T23:18:52.366Z,1718320732.366 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-13T23:18:52.802Z,1718320732.802 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 2.665175 m in 12.676111 degC water with 1.810760 ug/l chlorophyll fluorescence. 2024-06-13T23:18:53.166Z,1718320733.166 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-13T23:18:53.557Z,1718320733.557 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-13T23:18:55.595Z,1718320735.595 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-13T23:19:27.493Z,1718320767.493 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@18,aRNAl_bac) 2024-06-13T23:19:30.326Z,1718320770.326 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2024-06-13T23:19:30.327Z,1718320770.327 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-13T23:19:30.742Z,1718320770.742 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-13T23:22:13.134Z,1718320933.134 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2024-06-13T23:22:13.534Z,1718320933.534 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-13T23:32:01.828Z,1718321521.828 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-13T23:57:03.151Z,1718323023.151 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@1000.0) 2024-06-13T23:57:03.553Z,1718323023.553 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-13T23:57:05.595Z,1718323025.595 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-13T23:57:05.996Z,1718323025.996 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@18,aRNAl_bac) 2024-06-13T23:58:40.919Z,1718323120.919 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@18,aRNAl_bac) 2024-06-13T23:58:48.599Z,1718323128.599 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-13T23:58:49.012Z,1718323129.012 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2024-06-14T00:02:02.153Z,1718323322.153 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T00:03:48.816Z,1718323428.816 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2024-06-14T00:03:51.288Z,1718323431.288 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-14T00:08:55.086Z,1718323735.086 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-14T00:08:58.403Z,1718323738.403 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 52.839591 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-14T00:11:48.827Z,1718323908.827 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.040276 m , settling for 60.000000 min 2024-06-14T00:27:30.196Z,1718324850.196 [RudderServo](FAULT): Invalid EZ Servo response:"" 2024-06-14T00:32:02.457Z,1718325122.457 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T00:45:05.155Z,1718325905.155 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-14T00:45:16.098Z,1718325916.098 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002533 CHAN A1 (24V): -0.001132 CHAN A2 (12V): -0.000383 CHAN A3 (5V): 0.000434 CHAN B0 (3.3V): 0.000101 CHAN B1 (3.15aV): 0.000474 CHAN B2 (3.15bV): 0.000075 CHAN B3 (GND): -0.000222 OPEN: 0.007502 Full Scale: +/- 1 mA 2024-06-14T00:56:46.812Z,1718326606.812 [BPC1](FAULT): Failed to receive data from 18 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-14T01:05:37.303Z,1718327137.303 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T01:12:01.632Z,1718327521.632 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T01:12:01.752Z,1718327521.752 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T01:12:56.909Z,1718327576.909 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T01:12:57.717Z,1718327577.717 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -1 2024-06-14T01:12:57.717Z,1718327577.717 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T01:12:58.178Z,1718327578.178 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 0.661527 m in 12.785852 degC water with 1.755080 ug/l chlorophyll fluorescence. 2024-06-14T01:12:58.524Z,1718327578.524 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T01:12:58.916Z,1718327578.916 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T01:13:00.929Z,1718327580.929 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T01:13:32.854Z,1718327612.854 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@17,aRNAl_bac) 2024-06-14T01:13:35.678Z,1718327615.678 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T01:13:35.690Z,1718327615.690 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T01:13:36.110Z,1718327616.110 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T01:16:20.115Z,1718327780.115 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T01:16:20.526Z,1718327780.526 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T01:38:08.722Z,1718329088.722 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T01:50:13.911Z,1718329813.911 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T01:50:14.310Z,1718329814.310 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T01:50:16.332Z,1718329816.332 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T01:50:16.752Z,1718329816.752 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@17,aRNAl_bac) 2024-06-14T01:51:52.086Z,1718329912.086 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@17,aRNAl_bac) 2024-06-14T01:51:59.354Z,1718329919.354 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T01:51:59.767Z,1718329919.767 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2024-06-14T01:56:59.534Z,1718330219.534 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2024-06-14T01:57:01.995Z,1718330221.995 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-14T01:57:03.312Z,1718330223.312 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-14T01:57:09.620Z,1718330229.620 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-14T01:57:09.880Z,1718330229.880 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T01:57:24.484Z,1718330244.484 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 3 hour 2024-06-14T01:57:24.485Z,1718330244.485 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2024-06-14T01:57:24.486Z,1718330244.486 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-14T01:57:24.486Z,1718330244.486 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-14T01:57:24.487Z,1718330244.487 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-14T01:57:41.648Z,1718330261.648 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-14T01:57:41.649Z,1718330261.649 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-14T01:57:41.650Z,1718330261.650 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-14T01:57:41.650Z,1718330261.650 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-14T01:57:41.651Z,1718330261.651 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-14T01:57:41.652Z,1718330261.652 [CommandExec](IMPORTANT): got command run 2024-06-14T01:57:41.653Z,1718330261.653 [CommandExec](IMPORTANT): Running 2024-06-14T01:57:41.813Z,1718330261.813 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-14T01:59:23.133Z,1718330363.133 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=21327, MT Status=1, MTMSN=1738 2024-06-14T01:59:23.908Z,1718330363.908 [CommandExec](IMPORTANT): got command schedule 20240614T020000 "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run" 4irkg 3 3.000000 2024-06-14T01:59:23.908Z,1718330363.908 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4irkg 2024-06-14T01:59:23.909Z,1718330363.909 [CommandExec](IMPORTANT): Scheduled #40 (#3 of 3 with id='4irkg'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run", AT TIME: 2024-06-14T02:00:00Z 2024-06-14T01:59:51.568Z,1718330391.568 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21328, MT Status=1, MTMSN=1739 2024-06-14T01:59:52.364Z,1718330392.364 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m" 4irt4 1 2.000000 2024-06-14T01:59:52.365Z,1718330392.365 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4irt4 2024-06-14T01:59:52.366Z,1718330392.366 [CommandExec](IMPORTANT): Scheduled #41 (#1 of 2 with id='4irt4'): "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-14T02:00:00.093Z,1718330400.093 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-14T02:00:07.637Z,1718330407.637 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-14T02:00:07.811Z,1718330407.811 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T02:00:21.007Z,1718330421.007 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-14T02:00:21.008Z,1718330421.008 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 1 bool 2024-06-14T02:00:21.008Z,1718330421.008 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-14T02:00:21.009Z,1718330421.009 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-14T02:00:37.825Z,1718330437.825 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-14T02:00:37.826Z,1718330437.826 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-14T02:00:37.871Z,1718330437.871 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-14T02:00:37.872Z,1718330437.872 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-14T02:00:37.872Z,1718330437.872 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-14T02:00:52.393Z,1718330452.393 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-14T02:00:52.394Z,1718330452.394 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-14T02:00:52.395Z,1718330452.395 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-14T02:00:52.396Z,1718330452.396 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 10 minute 2024-06-14T02:00:52.396Z,1718330452.396 [CommandExec](IMPORTANT): got command run 2024-06-14T02:00:52.398Z,1718330452.398 [CommandExec](IMPORTANT): Running 2024-06-14T02:00:52.500Z,1718330452.500 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-14T02:00:52.580Z,1718330452.580 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-14T02:00:54.053Z,1718330454.053 [CircleSample:MissionStartComms:A](IMPORTANT): Mission start comms. 2024-06-14T03:07:48.574Z,1718334468.574 [ESPComponent](IMPORTANT): [sample #14] ESP log summary report (2 messages): @02:24:26.79 Selecting Cartridge 16 @03:01:00.52 Sampled 1000.0ml 2024-06-14T03:07:48.575Z,1718334468.575 [ESPComponent](IMPORTANT): [sample #14] ESP Cartridge 16: S_FILTERING: start: 2024-06-14T02:27:35.039Z end: 2024-06-14T03:01:02.943Z 2024-06-14T04:56:10.660Z,1718340970.660 [ESPComponent](IMPORTANT): [sample #15] ESP log summary report (2 messages): @04:14:33.14 Selecting Cartridge 15 @04:49:22.92 Sampled 1000.0ml 2024-06-14T04:56:10.662Z,1718340970.662 [ESPComponent](IMPORTANT): [sample #15] ESP Cartridge 15: S_FILTERING: start: 2024-06-14T04:17:40.754Z end: 2024-06-14T04:49:24.982Z 2024-06-14T02:11:03.881Z,1718331063.881 [CircleSample:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T02:11:11.272Z,1718331071.272 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-14T02:11:13.288Z,1718331073.288 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-14T02:18:10.239Z,1718331490.239 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T02:21:12.075Z,1718331672.075 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.028102 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-14T02:22:27.616Z,1718331747.616 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.120956 m , settling for 0.500000 min 2024-06-14T02:23:10.507Z,1718331790.507 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T02:23:10.611Z,1718331790.611 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T02:24:13.424Z,1718331853.424 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T02:24:14.229Z,1718331854.229 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -1 2024-06-14T02:24:14.229Z,1718331854.229 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T02:24:14.657Z,1718331854.657 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 2.179798 m in 12.956873 degC water with 1.441880 ug/l chlorophyll fluorescence. 2024-06-14T02:24:15.026Z,1718331855.026 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T02:24:15.446Z,1718331855.446 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T02:24:17.485Z,1718331857.485 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T02:24:49.396Z,1718331889.396 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@16,aRNAl_bac) 2024-06-14T02:24:52.218Z,1718331892.218 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T02:24:52.220Z,1718331892.220 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T02:24:52.606Z,1718331892.606 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T02:27:35.040Z,1718332055.040 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T02:27:35.417Z,1718332055.417 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T02:48:10.510Z,1718333290.510 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T03:01:02.529Z,1718334062.529 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T03:01:02.943Z,1718334062.943 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T03:01:04.972Z,1718334064.972 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T03:01:05.358Z,1718334065.358 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@16,aRNAl_bac) 2024-06-14T03:02:40.746Z,1718334160.746 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@16,aRNAl_bac) 2024-06-14T03:02:47.981Z,1718334167.981 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T03:02:48.387Z,1718334168.387 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED 2024-06-14T03:07:48.173Z,1718334468.173 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally. 2024-06-14T03:07:50.609Z,1718334470.609 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-14T03:12:54.458Z,1718334774.458 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-14T03:12:57.690Z,1718334777.690 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 51.747904 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-14T03:13:11.845Z,1718334791.845 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.799963 m , settling for 60.000000 min 2024-06-14T03:20:18.475Z,1718335218.475 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T03:50:18.782Z,1718337018.782 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T04:13:24.683Z,1718338404.683 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T04:13:24.828Z,1718338404.828 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T04:14:19.547Z,1718338459.547 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T04:14:20.358Z,1718338460.358 [ESPComponent](IMPORTANT): [sample #15] Setting cartridge argument to -1 2024-06-14T04:14:20.358Z,1718338460.358 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T04:14:20.781Z,1718338460.781 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.666728 m in 13.094995 degC water with 1.338640 ug/l chlorophyll fluorescence. 2024-06-14T04:14:21.162Z,1718338461.162 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T04:14:21.561Z,1718338461.561 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T04:14:23.586Z,1718338463.586 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T04:14:55.509Z,1718338495.509 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PRIMING (@15,aRNAl_bac) 2024-06-14T04:14:58.327Z,1718338498.327 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T04:14:58.328Z,1718338498.328 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T04:14:58.734Z,1718338498.734 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T04:17:40.755Z,1718338660.755 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T04:17:41.146Z,1718338661.146 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T04:49:24.546Z,1718340564.546 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T04:49:24.983Z,1718340564.983 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T04:49:26.986Z,1718340566.986 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T04:49:27.366Z,1718340567.366 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PROCESSED (@15,aRNAl_bac) 2024-06-14T04:51:02.768Z,1718340662.768 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPING (@15,aRNAl_bac) 2024-06-14T04:51:09.990Z,1718340669.990 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T04:51:10.400Z,1718340670.400 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPED 2024-06-14T04:53:58.171Z,1718340838.171 [ElevatorServo](FAULT): Reading outside of valid range:0.000000 2024-06-14T04:53:58.836Z,1718340838.836 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2024-06-14T04:56:10.267Z,1718340970.267 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence completed normally. 2024-06-14T04:56:11.068Z,1718340971.068 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Invalid Command. 2024-06-14T04:56:11.069Z,1718340971.069 [ESPComponent](FAULT): LCB 2 fault: Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated. 2024-06-14T04:56:12.757Z,1718340972.757 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-14T04:56:14.538Z,1718340974.538 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T04:58:29.026Z,1718341109.026 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21329, MT Status=1, MTMSN=1740 2024-06-14T04:58:29.804Z,1718341109.804 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4irt4 2 2.000000 2024-06-14T04:58:29.805Z,1718341109.805 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4irt4 2024-06-14T04:58:29.806Z,1718341109.806 [CommandExec](IMPORTANT): Scheduled #42 (#2 of 2 with id='4irt4'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", AFTER MISSION 2024-06-14T04:58:29.958Z,1718341109.958 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-14T04:58:35.743Z,1718341115.743 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-14T04:58:50.788Z,1718341130.788 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 3 hour 2024-06-14T04:58:50.790Z,1718341130.790 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2024-06-14T04:58:50.791Z,1718341130.791 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-14T04:58:50.792Z,1718341130.792 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-14T04:58:50.792Z,1718341130.792 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-14T04:59:11.002Z,1718341151.002 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-14T04:59:11.002Z,1718341151.002 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-14T04:59:11.007Z,1718341151.007 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-14T04:59:11.008Z,1718341151.008 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-14T04:59:11.008Z,1718341151.008 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-14T04:59:11.009Z,1718341151.009 [CommandExec](IMPORTANT): got command run 2024-06-14T04:59:11.019Z,1718341151.019 [CommandExec](IMPORTANT): Running 2024-06-14T04:59:11.121Z,1718341151.121 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-14T05:02:39.389Z,1718341359.389 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21330, MT Status=1, MTMSN=1741 2024-06-14T05:02:40.160Z,1718341360.160 [CommandExec](IMPORTANT): got command schedule 20240614T040000 "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4irwm 1 3.000000 2024-06-14T05:02:40.161Z,1718341360.161 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4irwm 2024-06-14T05:02:40.162Z,1718341360.162 [CommandExec](IMPORTANT): Scheduled #43 (#1 of 3 with id='4irwm'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 1 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", AT TIME: 2024-06-14T04:00:00Z 2024-06-14T05:04:07.214Z,1718341447.214 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21331, MT Status=1, MTMSN=1742 2024-06-14T05:04:09.976Z,1718341449.976 [CommandExec](IMPORTANT): got command schedule 20240614T040000 "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4irwm 2 3.000000 2024-06-14T05:04:09.977Z,1718341449.977 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4irwm 2024-06-14T05:04:09.978Z,1718341449.978 [CommandExec](IMPORTANT): Scheduled #44 (#2 of 3 with id='4irwm'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", AT TIME: 2024-06-14T04:00:00Z 2024-06-14T05:05:29.111Z,1718341529.111 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21332, MT Status=1, MTMSN=1743 2024-06-14T05:05:29.876Z,1718341529.876 [CommandExec](IMPORTANT): got command schedule 20240614T040000 "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run" 4irwm 3 3.000000 2024-06-14T05:05:29.877Z,1718341529.877 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4irwm 2024-06-14T05:05:29.878Z,1718341529.878 [CommandExec](IMPORTANT): Scheduled #45 (#3 of 3 with id='4irwm'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 10 min;run", AT TIME: 2024-06-14T04:00:00Z 2024-06-14T05:05:30.050Z,1718341530.050 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-14T05:05:37.784Z,1718341537.784 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-14T05:05:38.054Z,1718341538.054 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T05:05:51.183Z,1718341551.183 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-14T05:05:51.184Z,1718341551.184 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 1 bool 2024-06-14T05:05:51.184Z,1718341551.184 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-14T05:05:51.185Z,1718341551.185 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-14T05:06:07.525Z,1718341567.525 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-14T05:06:07.526Z,1718341567.526 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-14T05:06:07.555Z,1718341567.555 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-14T05:06:07.556Z,1718341567.556 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-14T05:06:07.556Z,1718341567.556 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-14T05:06:19.684Z,1718341579.684 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-14T05:06:19.685Z,1718341579.685 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-14T05:06:19.685Z,1718341579.685 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-14T05:06:19.686Z,1718341579.686 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 10 minute 2024-06-14T05:06:19.686Z,1718341579.686 [CommandExec](IMPORTANT): got command run 2024-06-14T05:06:19.689Z,1718341579.689 [CommandExec](IMPORTANT): Running 2024-06-14T05:06:19.697Z,1718341579.697 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-14T05:06:19.778Z,1718341579.778 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-14T05:06:21.237Z,1718341581.237 [CircleSample:MissionStartComms:A](IMPORTANT): Mission start comms. 2024-06-14T06:12:52.721Z,1718345572.721 [ESPComponent](IMPORTANT): [sample #16] ESP log summary report (2 messages): @05:31:06.61 Selecting Cartridge 14 @06:06:04.91 Sampled 1000.0ml 2024-06-14T06:12:52.723Z,1718345572.723 [ESPComponent](IMPORTANT): [sample #16] ESP Cartridge 14: S_FILTERING: start: 2024-06-14T05:34:14.390Z end: 2024-06-14T06:06:07.106Z 2024-06-14T08:01:10.565Z,1718352070.565 [ESPComponent](IMPORTANT): [sample #17] ESP log summary report (2 messages): @07:19:40.01 Selecting Cartridge 13 @07:54:22.93 Sampled 1000.0ml 2024-06-14T08:01:10.567Z,1718352070.567 [ESPComponent](IMPORTANT): [sample #17] ESP Cartridge 13: S_FILTERING: start: 2024-06-14T07:22:49.298Z end: 2024-06-14T07:54:24.963Z 2024-06-14T05:16:24.292Z,1718342184.292 [CircleSample:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T05:16:32.071Z,1718342192.071 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-14T05:16:34.067Z,1718342194.067 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-14T05:26:32.993Z,1718342792.993 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.029460 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-14T05:29:15.837Z,1718342955.837 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.079968 m , settling for 0.500000 min 2024-06-14T05:29:58.739Z,1718342998.739 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T05:29:58.858Z,1718342998.858 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T05:30:53.207Z,1718343053.207 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T05:30:54.008Z,1718343054.008 [ESPComponent](IMPORTANT): [sample #16] Setting cartridge argument to -1 2024-06-14T05:30:54.008Z,1718343054.008 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T05:30:54.466Z,1718343054.466 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.508034 m in 13.063227 degC water with 1.331680 ug/l chlorophyll fluorescence. 2024-06-14T05:30:54.801Z,1718343054.801 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T05:30:55.205Z,1718343055.205 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T05:30:56.048Z,1718343056.048 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T05:30:57.242Z,1718343057.242 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T05:31:29.143Z,1718343089.143 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PRIMING (@14,aRNAl_bac) 2024-06-14T05:31:31.161Z,1718343091.161 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T05:31:31.566Z,1718343091.566 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T05:31:31.983Z,1718343091.983 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T05:34:14.391Z,1718343254.391 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T05:34:14.804Z,1718343254.804 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T06:05:52.998Z,1718345152.998 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T06:06:06.693Z,1718345166.693 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T06:06:07.106Z,1718345167.106 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T06:06:09.132Z,1718345169.132 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T06:06:09.521Z,1718345169.521 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PROCESSED (@14,aRNAl_bac) 2024-06-14T06:07:44.879Z,1718345264.879 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPING (@14,aRNAl_bac) 2024-06-14T06:07:52.138Z,1718345272.138 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T06:07:52.548Z,1718345272.548 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPED 2024-06-14T06:12:52.317Z,1718345572.317 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence completed normally. 2024-06-14T06:12:54.796Z,1718345574.796 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-14T06:17:58.633Z,1718345878.633 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-14T06:18:01.849Z,1718345881.849 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 51.467879 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-14T06:18:11.136Z,1718345891.136 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.950536 m , settling for 60.000000 min 2024-06-14T06:38:02.988Z,1718347082.988 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T06:45:16.197Z,1718347516.197 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-14T06:45:27.446Z,1718347527.446 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003066 CHAN A1 (24V): -0.001308 CHAN A2 (12V): -0.000666 CHAN A3 (5V): 0.000446 CHAN B0 (3.3V): 0.000178 CHAN B1 (3.15aV): -0.000109 CHAN B2 (3.15bV): -0.000012 CHAN B3 (GND): 0.000363 OPEN: 0.007436 Full Scale: +/- 1 mA 2024-06-14T07:09:34.948Z,1718348974.948 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T07:18:23.915Z,1718349503.915 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T07:18:24.051Z,1718349504.051 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T07:19:26.418Z,1718349566.418 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T07:19:27.270Z,1718349567.270 [ESPComponent](IMPORTANT): [sample #17] Setting cartridge argument to -1 2024-06-14T07:19:27.271Z,1718349567.271 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T07:19:27.705Z,1718349567.705 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 0.986409 m in 13.293573 degC water with 1.035880 ug/l chlorophyll fluorescence. 2024-06-14T07:19:28.038Z,1718349568.038 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T07:19:28.437Z,1718349568.437 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T07:19:30.458Z,1718349570.458 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T07:20:02.504Z,1718349602.504 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PRIMING (@13,aRNAl_bac) 2024-06-14T07:20:05.279Z,1718349605.279 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T07:20:05.280Z,1718349605.280 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T07:20:05.688Z,1718349605.688 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T07:22:49.298Z,1718349769.298 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T07:22:49.697Z,1718349769.697 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T07:27:49.065Z,1718350069.065 [BPC1](FAULT): Failed to receive data from 19 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-14T07:40:44.796Z,1718350844.796 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T07:54:24.558Z,1718351664.558 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T07:54:24.964Z,1718351664.964 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T07:54:26.972Z,1718351666.972 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T07:54:27.370Z,1718351667.370 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PROCESSED (@13,aRNAl_bac) 2024-06-14T07:56:02.736Z,1718351762.736 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPING (@13,aRNAl_bac) 2024-06-14T07:56:10.003Z,1718351770.003 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T07:56:10.390Z,1718351770.390 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPED 2024-06-14T08:01:10.174Z,1718352070.174 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence completed normally. 2024-06-14T08:01:12.623Z,1718352072.623 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-14T08:01:14.360Z,1718352074.360 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T08:06:32.267Z,1718352392.267 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T08:07:29.791Z,1718352449.791 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21333, MT Status=1, MTMSN=1744 2024-06-14T08:07:30.577Z,1718352450.577 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m" 4is31 1 2.000000 2024-06-14T08:07:30.578Z,1718352450.578 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4is31 2024-06-14T08:07:30.608Z,1718352450.608 [CommandExec](IMPORTANT): Scheduled #46 (#1 of 2 with id='4is31'): "load Science/profile_station.tl;set profile_station.MissionTimeout 3 h;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-14T08:08:19.554Z,1718352499.554 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21334, MT Status=1, MTMSN=1745 2024-06-14T08:08:20.372Z,1718352500.372 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4is31 2 2.000000 2024-06-14T08:08:20.373Z,1718352500.373 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4is31 2024-06-14T08:08:20.374Z,1718352500.374 [CommandExec](IMPORTANT): Scheduled #47 (#2 of 2 with id='4is31'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", AFTER MISSION 2024-06-14T08:08:20.509Z,1718352500.509 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-14T08:08:26.808Z,1718352506.808 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-14T08:08:42.066Z,1718352522.066 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 3 hour 2024-06-14T08:08:42.066Z,1718352522.066 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2024-06-14T08:08:42.067Z,1718352522.067 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-14T08:08:42.068Z,1718352522.068 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-14T08:08:42.069Z,1718352522.069 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-14T08:08:51.079Z,1718352531.079 [BPC1](FAULT): Failed to receive data from 19 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-14T08:08:57.511Z,1718352537.511 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-14T08:08:57.512Z,1718352537.512 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-14T08:08:57.512Z,1718352537.512 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-14T08:08:57.513Z,1718352537.513 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-14T08:08:57.514Z,1718352537.514 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-14T08:08:57.514Z,1718352537.514 [CommandExec](IMPORTANT): got command run 2024-06-14T08:08:57.516Z,1718352537.516 [CommandExec](IMPORTANT): Running 2024-06-14T08:08:57.591Z,1718352537.591 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-14T08:11:02.598Z,1718352662.598 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T08:11:02.601Z,1718352662.601 [profile_station:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T08:19:04.595Z,1718353144.595 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T08:47:05.977Z,1718354825.977 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.864831 +/- 0.036807 deg (conf. level 99.90%, sigma: 0.023671 deg). 2024-06-14T09:02:22.660Z,1718355742.660 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T09:11:33.061Z,1718356293.061 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.868499 +/- 0.098562 deg (conf. level 99.90%, sigma: 0.033829 deg). 2024-06-14T09:38:27.897Z,1718357907.897 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T09:43:46.615Z,1718358226.615 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T09:43:46.618Z,1718358226.618 [profile_station:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T09:48:29.352Z,1718358509.352 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T09:48:35.387Z,1718358515.387 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.872336 +/- 0.080803 deg (conf. level 99.90%, sigma: 0.042834 deg). 2024-06-14T10:23:47.280Z,1718360627.280 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T10:33:55.057Z,1718361235.057 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T11:08:58.140Z,1718363338.140 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T11:10:03.981Z,1718363403.981 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T11:12:27.182Z,1718363547.182 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21346, MT Status=1, MTMSN=1746 2024-06-14T11:12:27.731Z,1718363547.731 [CommandExec](IMPORTANT): got command stop 2024-06-14T11:12:27.731Z,1718363547.731 [CommandExec](IMPORTANT): Scheduling is paused 2024-06-14T11:12:27.731Z,1718363547.731 [CommandExec](IMPORTANT): got command schedule clear 2024-06-14T11:12:27.731Z,1718363547.731 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2024-06-14T11:12:27.732Z,1718363547.732 [CommandExec](IMPORTANT): got command schedule resume 2024-06-14T11:12:27.732Z,1718363547.732 [CommandExec](IMPORTANT): Scheduling is resumed 2024-06-14T11:12:54.678Z,1718363574.678 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21347, MT Status=1, MTMSN=1747 2024-06-14T11:12:55.439Z,1718363575.439 [CommandExec](IMPORTANT): got command schedule asap "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4jqzj 1 3.000000 2024-06-14T11:12:55.439Z,1718363575.439 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4jqzj 2024-06-14T11:12:55.440Z,1718363575.440 [CommandExec](IMPORTANT): Scheduled #48 (#1 of 3 with id='4jqzj'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", ASAP 2024-06-14T11:21:36.354Z,1718364096.354 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21348, MT Status=1, MTMSN=1748 2024-06-14T11:21:37.112Z,1718364097.112 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min" 4jqzj 2 3.000000 2024-06-14T11:21:37.112Z,1718364097.112 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4jqzj 2024-06-14T11:21:37.113Z,1718364097.113 [CommandExec](IMPORTANT): Scheduled #49 (#2 of 3 with id='4jqzj'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 60 min", ASAP 2024-06-14T11:23:06.887Z,1718364186.887 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21349, MT Status=1, MTMSN=1749 2024-06-14T11:23:07.627Z,1718364187.627 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4jqzj 3 3.000000 2024-06-14T11:23:07.628Z,1718364187.628 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4jqzj 2024-06-14T11:23:07.629Z,1718364187.629 [CommandExec](IMPORTANT): Scheduled #50 (#3 of 3 with id='4jqzj'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 15 min;run", ASAP 2024-06-14T11:23:07.781Z,1718364187.781 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-14T11:23:17.228Z,1718364197.228 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-14T11:23:30.823Z,1718364210.823 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-14T11:23:30.824Z,1718364210.824 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-06-14T11:23:30.825Z,1718364210.825 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-14T11:23:30.825Z,1718364210.825 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-14T11:23:49.987Z,1718364229.987 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-14T11:23:49.988Z,1718364229.988 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-14T11:23:49.988Z,1718364229.988 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-14T11:23:49.989Z,1718364229.989 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-14T11:23:49.990Z,1718364229.990 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 60 minute 2024-06-14T11:24:00.769Z,1718364240.769 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-14T11:24:00.770Z,1718364240.770 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-14T11:24:00.770Z,1718364240.770 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-14T11:24:00.771Z,1718364240.771 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-06-14T11:24:00.772Z,1718364240.772 [CommandExec](IMPORTANT): got command run 2024-06-14T11:24:00.774Z,1718364240.774 [CommandExec](IMPORTANT): Running 2024-06-14T11:24:00.973Z,1718364240.973 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-14T11:24:01.003Z,1718364241.003 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-14T11:24:02.984Z,1718364242.984 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-06-14T11:24:04.227Z,1718364244.227 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-14T12:18:54.839Z,1718367534.839 [ESPComponent](IMPORTANT): [sample #18] ESP log summary report (2 messages): @11:36:03.16 Selecting Cartridge 12 @12:12:06.79 Sampled 1000.0ml 2024-06-14T12:18:54.840Z,1718367534.840 [ESPComponent](IMPORTANT): [sample #18] ESP Cartridge 12: S_FILTERING: start: 2024-06-14T11:39:11.144Z end: 2024-06-14T12:12:09.566Z 2024-06-14T14:07:34.130Z,1718374054.130 [ESPComponent](IMPORTANT): [sample #19] ESP log summary report (2 messages): @13:25:39.56 Selecting Cartridge 11 @14:00:46.51 Sampled 1000.0ml 2024-06-14T14:07:34.132Z,1718374054.132 [ESPComponent](IMPORTANT): [sample #19] ESP Cartridge 11: S_FILTERING: start: 2024-06-14T13:28:47.474Z end: 2024-06-14T14:00:48.199Z 2024-06-14T11:24:06.507Z,1718364246.507 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-14T11:25:33.653Z,1718364333.653 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T11:25:33.662Z,1718364333.662 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T11:34:05.526Z,1718364845.526 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.028709 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-14T11:34:12.375Z,1718364852.375 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.093339 m , settling for 0.500000 min 2024-06-14T11:34:55.364Z,1718364895.364 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T11:34:55.502Z,1718364895.502 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T11:35:49.926Z,1718364949.926 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T11:35:50.738Z,1718364950.738 [ESPComponent](IMPORTANT): [sample #18] Setting cartridge argument to -1 2024-06-14T11:35:50.738Z,1718364950.738 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T11:35:51.191Z,1718364951.191 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 2.406305 m in 13.017694 degC water with 1.278320 ug/l chlorophyll fluorescence. 2024-06-14T11:35:51.542Z,1718364951.542 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T11:35:51.944Z,1718364951.944 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T11:35:53.970Z,1718364953.970 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T11:36:25.882Z,1718364985.882 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PRIMING (@12,aRNAl_bac) 2024-06-14T11:36:27.906Z,1718364987.906 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T11:36:28.306Z,1718364988.306 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T11:36:28.715Z,1718364988.715 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T11:39:11.144Z,1718365151.144 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T11:39:11.522Z,1718365151.522 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T11:57:06.638Z,1718366226.638 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T12:12:09.162Z,1718367129.162 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T12:12:09.566Z,1718367129.566 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T12:12:11.583Z,1718367131.583 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T12:12:11.984Z,1718367131.984 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PROCESSED (@12,aRNAl_bac) 2024-06-14T12:13:46.945Z,1718367226.945 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPING (@12,aRNAl_bac) 2024-06-14T12:13:54.224Z,1718367234.224 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T12:13:54.675Z,1718367234.675 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPED 2024-06-14T12:18:54.452Z,1718367534.452 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence completed normally. 2024-06-14T12:18:56.885Z,1718367536.885 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-14T12:24:00.695Z,1718367840.695 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 60.000000 min . 2024-06-14T12:24:03.909Z,1718367843.909 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 49.960116 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-14T12:24:18.462Z,1718367858.462 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.824123 m , settling for 60.000000 min 2024-06-14T12:28:42.450Z,1718368122.450 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T12:45:27.883Z,1718369127.883 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-14T12:45:39.357Z,1718369139.357 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002835 CHAN A1 (24V): -0.000959 CHAN A2 (12V): -0.000534 CHAN A3 (5V): -0.000434 CHAN B0 (3.3V): -0.001191 CHAN B1 (3.15aV): -0.000103 CHAN B2 (3.15bV): -0.000003 CHAN B3 (GND): 0.000090 OPEN: 0.007684 Full Scale: +/- 1 mA 2024-06-14T13:01:34.473Z,1718370094.473 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T13:24:31.399Z,1718371471.399 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T13:24:31.518Z,1718371471.518 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T13:25:25.466Z,1718371525.466 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T13:25:26.276Z,1718371526.276 [ESPComponent](IMPORTANT): [sample #19] Setting cartridge argument to -1 2024-06-14T13:25:26.278Z,1718371526.278 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T13:25:26.778Z,1718371526.778 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 7.453314 m in 12.920954 degC water with 1.694760 ug/l chlorophyll fluorescence. 2024-06-14T13:25:27.087Z,1718371527.087 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T13:25:27.490Z,1718371527.490 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T13:25:29.505Z,1718371529.505 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T13:26:01.438Z,1718371561.438 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PRIMING (@11,aRNAl_bac) 2024-06-14T13:26:04.249Z,1718371564.249 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T13:26:04.250Z,1718371564.250 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T13:26:04.689Z,1718371564.689 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T13:28:47.474Z,1718371727.474 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T13:28:47.873Z,1718371727.873 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T13:31:46.078Z,1718371906.078 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T14:00:47.790Z,1718373647.790 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T14:00:48.200Z,1718373648.200 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T14:00:50.224Z,1718373650.224 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T14:00:50.613Z,1718373650.613 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PROCESSED (@11,aRNAl_bac) 2024-06-14T14:02:24.032Z,1718373744.032 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T14:02:26.018Z,1718373746.018 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPING (@11,aRNAl_bac) 2024-06-14T14:02:33.284Z,1718373753.284 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T14:02:33.710Z,1718373753.710 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPED 2024-06-14T14:07:33.712Z,1718374053.712 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence completed normally. 2024-06-14T14:07:36.141Z,1718374056.141 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-14T14:07:37.922Z,1718374057.922 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T14:32:46.294Z,1718375566.294 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21357, MT Status=1, MTMSN=1750 2024-06-14T14:32:46.890Z,1718375566.890 [CommandExec](IMPORTANT): got command schedule clear 2024-06-14T14:32:46.891Z,1718375566.891 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2024-06-14T14:32:46.891Z,1718375566.891 [CommandExec](IMPORTANT): got command schedule resume 2024-06-14T15:10:40.862Z,1718377840.862 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21358, MT Status=1, MTMSN=1751 2024-06-14T15:10:41.621Z,1718377841.621 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.tl;set profile_station.MissionTimeout 1 h;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m;set profile_station.YoYoMinAltitude 5 m" 4k2w0 1 2.000000 2024-06-14T15:10:41.621Z,1718377841.621 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4k2w0 2024-06-14T15:10:41.622Z,1718377841.622 [CommandExec](IMPORTANT): Scheduled #51 (#1 of 2 with id='4k2w0'): "load Science/profile_station.tl;set profile_station.MissionTimeout 1 h;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m;set profile_station.YoYoMinAltitude 5 m", ASAP 2024-06-14T15:49:16.953Z,1718380156.953 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21359, MT Status=1, MTMSN=1752 2024-06-14T15:49:17.698Z,1718380157.698 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4k2w0 2 2.000000 2024-06-14T15:49:17.699Z,1718380157.699 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4k2w0 2024-06-14T15:49:17.700Z,1718380157.700 [CommandExec](IMPORTANT): Scheduled #52 (#2 of 2 with id='4k2w0'): "set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", ASAP 2024-06-14T15:49:17.839Z,1718380157.839 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-14T15:49:23.922Z,1718380163.922 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-14T15:49:41.500Z,1718380181.500 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 1 hour 2024-06-14T15:49:41.501Z,1718380181.501 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-14T15:49:41.502Z,1718380181.502 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-14T15:49:41.502Z,1718380181.502 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-14T15:49:41.503Z,1718380181.503 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-14T15:49:53.623Z,1718380193.623 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-14T15:49:53.624Z,1718380193.624 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-14T15:49:53.624Z,1718380193.624 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-14T15:49:53.625Z,1718380193.625 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-14T15:49:53.625Z,1718380193.625 [CommandExec](IMPORTANT): got command run 2024-06-14T15:49:53.627Z,1718380193.627 [CommandExec](IMPORTANT): Running 2024-06-14T15:49:53.936Z,1718380193.936 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-14T15:49:56.369Z,1718380196.369 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21360, MT Status=1, MTMSN=1753 2024-06-14T15:49:57.190Z,1718380197.190 [CommandExec](IMPORTANT): got command schedule "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool;set CircleSample.DepthCircle 5 m" 4k2xb 1 3.000000 2024-06-14T15:49:57.199Z,1718380197.199 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4k2xb 2024-06-14T15:49:57.200Z,1718380197.200 [CommandExec](IMPORTANT): Scheduled #53 (#1 of 3 with id='4k2xb'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 4 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool;set CircleSample.DepthCircle 5 m", AFTER MISSION 2024-06-14T17:47:39.579Z,1718387259.579 [ESPComponent](IMPORTANT): [sample #20] ESP log summary report (2 messages): @17:02:50.74 Selecting Cartridge 10 @17:40:51.93 Sampled 1000.0ml 2024-06-14T17:47:39.580Z,1718387259.580 [ESPComponent](IMPORTANT): [sample #20] ESP Cartridge 10: S_FILTERING: start: 2024-06-14T17:05:59.961Z end: 2024-06-14T17:40:54.363Z 2024-06-14T19:09:54.741Z,1718392194.741 [ESPComponent](IMPORTANT): [sample #21] ESP log summary report (2 messages): @18:25:14.88 Selecting Cartridge 9 @19:03:06.62 Sampled 1000.0ml 2024-06-14T19:09:54.742Z,1718392194.742 [ESPComponent](IMPORTANT): [sample #21] ESP Cartridge 9: S_FILTERING: start: 2024-06-14T18:28:22.334Z end: 2024-06-14T19:03:09.140Z 2024-06-14T15:50:37.318Z,1718380237.318 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21361, MT Status=1, MTMSN=1754 2024-06-14T15:50:38.125Z,1718380238.125 [CommandExec](IMPORTANT): got command schedule "set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 30 min;set CircleSample.Settle1 0.5 min" 4k2xb 2 3.000000 2024-06-14T15:50:38.125Z,1718380238.125 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4k2xb 2024-06-14T15:50:38.126Z,1718380238.126 [CommandExec](IMPORTANT): Scheduled #54 (#2 of 3 with id='4k2xb'): "set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.613801 degree;set CircleSample.CenterLongitude 7.500824 degree;set CircleSample.SettleCommon 30 min;set CircleSample.Settle1 0.5 min", AFTER MISSION 2024-06-14T15:58:00.446Z,1718380680.446 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21362, MT Status=1, MTMSN=1755 2024-06-14T15:58:01.113Z,1718380681.113 [CommandExec](IMPORTANT): got command schedule "set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4k2xb 3 3.000000 2024-06-14T15:58:01.114Z,1718380681.114 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4k2xb 2024-06-14T15:58:01.115Z,1718380681.115 [CommandExec](IMPORTANT): Scheduled #55 (#3 of 3 with id='4k2xb'): "set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 0.7 km;set CircleSample:NeedComms.CommsTimeout 15 min;run", AFTER MISSION 2024-06-14T15:59:58.022Z,1718380798.022 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T16:27:25.313Z,1718382445.313 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.852325 +/- 0.062805 deg (conf. level 99.90%, sigma: 0.037564 deg). 2024-06-14T16:47:13.877Z,1718383633.877 [BPC1](FAULT): Failed to receive data from 19 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-14T16:49:54.378Z,1718383794.378 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-14T16:50:02.317Z,1718383802.317 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-14T16:50:02.380Z,1718383802.380 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T16:50:20.365Z,1718383820.365 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 4 hour 2024-06-14T16:50:20.366Z,1718383820.366 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-06-14T16:50:20.366Z,1718383820.366 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-14T16:50:20.367Z,1718383820.367 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-14T16:50:20.368Z,1718383820.368 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-14T16:50:36.067Z,1718383836.067 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-14T16:50:36.068Z,1718383836.068 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.613801 degree 2024-06-14T16:50:36.068Z,1718383836.068 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.500824 degree 2024-06-14T16:50:36.069Z,1718383836.069 [CommandExec](IMPORTANT): got command set CircleSample.SettleCommon 30 minute 2024-06-14T16:50:36.070Z,1718383836.070 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-14T16:50:45.319Z,1718383845.319 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-14T16:50:45.319Z,1718383845.319 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 0.7 kilometer 2024-06-14T16:50:45.321Z,1718383845.321 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-06-14T16:50:45.322Z,1718383845.322 [CommandExec](IMPORTANT): got command run 2024-06-14T16:50:45.324Z,1718383845.324 [CommandExec](IMPORTANT): Running 2024-06-14T16:50:45.593Z,1718383845.593 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-14T16:50:45.638Z,1718383845.638 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-14T16:50:47.615Z,1718383847.615 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-06-14T16:50:48.828Z,1718383848.828 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.613800 arcdeg , 7.500824 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-14T16:50:50.854Z,1718383850.854 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-14T16:52:16.548Z,1718383936.548 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.863027 +/- 0.067672 deg (conf. level 99.90%, sigma: 0.038423 deg). 2024-06-14T16:55:34.894Z,1718384134.894 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T16:55:47.040Z,1718384147.040 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T17:00:50.084Z,1718384450.084 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.027518 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-14T17:01:00.158Z,1718384460.158 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.022155 m , settling for 0.500000 min 2024-06-14T17:01:43.047Z,1718384503.047 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T17:01:43.152Z,1718384503.152 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T17:02:37.525Z,1718384557.525 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T17:02:38.348Z,1718384558.348 [ESPComponent](IMPORTANT): [sample #20] Setting cartridge argument to -1 2024-06-14T17:02:38.348Z,1718384558.348 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T17:02:38.825Z,1718384558.825 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.123112 m in 12.736872 degC water with 2.273600 ug/l chlorophyll fluorescence. 2024-06-14T17:02:39.160Z,1718384559.160 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T17:02:39.546Z,1718384559.546 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T17:02:41.565Z,1718384561.565 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T17:03:13.530Z,1718384593.530 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PRIMING (@10,aRNAl_bac) 2024-06-14T17:03:16.337Z,1718384596.337 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T17:03:16.338Z,1718384596.338 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T17:03:16.741Z,1718384596.741 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T17:05:59.962Z,1718384759.962 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T17:06:00.361Z,1718384760.361 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T17:29:06.557Z,1718386146.557 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T17:40:53.963Z,1718386853.963 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T17:40:54.364Z,1718386854.364 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T17:40:56.381Z,1718386856.381 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T17:40:56.777Z,1718386856.777 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PROCESSED (@10,aRNAl_bac) 2024-06-14T17:42:31.723Z,1718386951.723 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPING (@10,aRNAl_bac) 2024-06-14T17:42:39.004Z,1718386959.004 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T17:42:39.403Z,1718386959.403 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPED 2024-06-14T17:47:39.176Z,1718387259.176 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence completed normally. 2024-06-14T17:47:41.714Z,1718387261.714 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-14T17:52:45.549Z,1718387565.549 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 30.000000 min . 2024-06-14T17:52:48.758Z,1718387568.758 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 51.964189 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-14T17:53:54.198Z,1718387634.198 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.064867 m , settling for 30.000000 min 2024-06-14T17:59:34.808Z,1718387974.808 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T18:24:07.108Z,1718389447.108 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-14T18:24:07.246Z,1718389447.246 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-14T18:25:01.584Z,1718389501.584 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-14T18:25:02.356Z,1718389502.356 [ESPComponent](IMPORTANT): [sample #21] Setting cartridge argument to -1 2024-06-14T18:25:02.357Z,1718389502.357 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-14T18:25:02.797Z,1718389502.797 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 9.388298 m in 12.563227 degC water with 2.285200 ug/l chlorophyll fluorescence. 2024-06-14T18:25:03.157Z,1718389503.157 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-14T18:25:03.609Z,1718389503.609 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-14T18:25:05.609Z,1718389505.609 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-14T18:25:37.535Z,1718389537.535 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PRIMING (@9,aRNAl_bac) 2024-06-14T18:25:40.327Z,1718389540.327 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PRIMING (@100.0) 2024-06-14T18:25:40.328Z,1718389540.328 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-14T18:25:40.729Z,1718389540.729 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-14T18:28:22.334Z,1718389702.334 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_FILTERING (@0.0) 2024-06-14T18:28:22.733Z,1718389702.733 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-14T18:33:30.657Z,1718390010.657 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T18:45:39.599Z,1718390739.599 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-14T18:45:50.502Z,1718390750.502 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003187 CHAN A1 (24V): -0.001047 CHAN A2 (12V): -0.001076 CHAN A3 (5V): -0.000743 CHAN B0 (3.3V): -0.000099 CHAN B1 (3.15aV): -0.000903 CHAN B2 (3.15bV): 0.000550 CHAN B3 (GND): 0.000096 OPEN: 0.007839 Full Scale: +/- 1 mA 2024-06-14T19:03:08.726Z,1718391788.726 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PAUSED (@1000.0) 2024-06-14T19:03:09.140Z,1718391789.140 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-14T19:03:11.547Z,1718391791.547 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-14T19:03:11.951Z,1718391791.951 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PROCESSED (@9,aRNAl_bac) 2024-06-14T19:04:46.898Z,1718391886.898 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPING (@9,aRNAl_bac) 2024-06-14T19:04:54.167Z,1718391894.167 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-14T19:04:54.570Z,1718391894.570 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPED 2024-06-14T19:05:10.770Z,1718391910.770 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T19:09:54.345Z,1718392194.345 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence completed normally. 2024-06-14T19:09:56.786Z,1718392196.786 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-14T19:09:58.519Z,1718392198.519 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T19:10:15.856Z,1718392215.856 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-06-14T19:16:00.489Z,1718392560.489 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-14T19:16:32.123Z,1718392592.123 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21363, MT Status=1, MTMSN=1756 2024-06-14T19:16:32.944Z,1718392592.944 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.tl;set profile_station.MissionTimeout 1.5 h;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m;set profile_station.YoYoMinAltitude 5 m" 4k7wq 1 2.000000 2024-06-14T19:16:32.945Z,1718392592.945 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4k7wq 2024-06-14T19:16:32.946Z,1718392592.946 [CommandExec](IMPORTANT): Scheduled #56 (#1 of 2 with id='4k7wq'): "load Science/profile_station.tl;set profile_station.MissionTimeout 1.5 h;set profile_station.Lat 55.613801 degree;set profile_station.Lon 7.500824 degree;set profile_station.YoYoMaxDepth 5 m;set profile_station.YoYoMinAltitude 5 m", ASAP 2024-06-14T19:18:18.671Z,1718392698.671 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21364, MT Status=1, MTMSN=1757 2024-06-14T19:18:19.347Z,1718392699.347 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run" 4k7wq 2 2.000000 2024-06-14T19:18:19.347Z,1718392699.347 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4k7wq 2024-06-14T19:18:19.348Z,1718392699.348 [CommandExec](IMPORTANT): Scheduled #57 (#2 of 2 with id='4k7wq'): "set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 10 min;run", ASAP 2024-06-14T19:18:19.516Z,1718392699.516 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-14T19:18:25.840Z,1718392705.840 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-14T19:18:43.262Z,1718392723.262 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 1.5 hour 2024-06-14T19:18:43.269Z,1718392723.269 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.613801 degree 2024-06-14T19:18:43.270Z,1718392723.270 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.500824 degree 2024-06-14T19:18:43.271Z,1718392723.271 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-14T19:18:43.272Z,1718392723.272 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-14T19:18:45.803Z,1718392725.803 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21365, MT Status=1, MTMSN=1758 2024-06-14T19:18:57.656Z,1718392737.656 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-14T19:18:57.657Z,1718392737.657 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-14T19:18:57.657Z,1718392737.657 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-14T19:18:57.658Z,1718392737.658 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 10 minute 2024-06-14T19:18:57.659Z,1718392737.659 [CommandExec](IMPORTANT): got command run 2024-06-14T19:18:57.660Z,1718392737.660 [CommandExec](IMPORTANT): Running 2024-06-14T19:18:57.665Z,1718392737.665 [CommandExec](IMPORTANT): got command ! onESPclient heavyB -- 'slot [52,51,50]=>:dry' 2024-06-14T19:18:58.143Z,1718392738.143 [CommandExec](IMPORTANT): /mnt/mmc/started/2024-06-14UTC191858_onESPclient 2024-06-14T19:18:58.912Z,1718392738.912 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-14T19:29:31.569Z,1718393371.569 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T20:08:38.229Z,1718395718.229 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.852038 +/- 0.095206 deg (conf. level 99.90%, sigma: 0.070880 deg). 2024-06-14T20:41:40.238Z,1718397700.238 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 10.000000 min minutes 2024-06-14T20:41:49.014Z,1718397709.014 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.864458 +/- 0.049381 deg (conf. level 99.90%, sigma: 0.027028 deg). 2024-06-14T20:44:31.187Z,1718397871.187 [ThrusterHE](FAULT): Could not get status:serial timeout 2024-06-14T20:48:59.687Z,1718398139.687 [MissionManager](IMPORTANT): Started mission Default 2024-06-14T20:50:34.894Z,1718398234.894 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21371, MT Status=1, MTMSN=1759 2024-06-14T20:50:35.498Z,1718398235.498 [CommandExec](IMPORTANT): got command schedule clear 2024-06-14T20:50:35.499Z,1718398235.499 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2024-06-14T20:50:35.499Z,1718398235.499 [CommandExec](IMPORTANT): got command schedule resume 2024-06-14T20:59:56.820Z,1718398796.820 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21374, MT Status=1, MTMSN=1760 2024-06-14T20:59:57.504Z,1718398797.504 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2_flat_and_level.tl;set sci2_flat_and_level.MissionTimeout 5 h;set sci2_flat_and_level.Depth1 7 m;set sci2_flat_and_level.Lat1 55.609157 degree;set sci2_flat_and_level.Lon1 7.527718 degree" 4kkp2 1 4.000000 2024-06-14T20:59:57.504Z,1718398797.504 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4kkp2 2024-06-14T20:59:57.505Z,1718398797.505 [CommandExec](IMPORTANT): Scheduled #58 (#1 of 4 with id='4kkp2'): "load Science/sci2_flat_and_level.tl;set sci2_flat_and_level.MissionTimeout 5 h;set sci2_flat_and_level.Depth1 7 m;set sci2_flat_and_level.Lat1 55.609157 degree;set sci2_flat_and_level.Lon1 7.527718 degree", ASAP 2024-06-14T21:30:32.251Z,1718400632.251 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21375, MT Status=1, MTMSN=1761 2024-06-14T21:30:33.072Z,1718400633.072 [CommandExec](IMPORTANT): got command schedule asap "set sci2_flat_and_level.Lat2 55.60158 degree;set sci2_flat_and_level.Lon2 7.5501 degree;set sci2_flat_and_level.Lat3 55.59644 degree;set sci2_flat_and_level.Lon3 7.58074 degree;set sci2_flat_and_level.ApproachDepth 0.5 m" 4kkp2 2 4.000000 2024-06-14T21:30:33.072Z,1718400633.072 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4kkp2 2024-06-14T21:30:33.073Z,1718400633.073 [CommandExec](IMPORTANT): Scheduled #59 (#2 of 4 with id='4kkp2'): "set sci2_flat_and_level.Lat2 55.60158 degree;set sci2_flat_and_level.Lon2 7.5501 degree;set sci2_flat_and_level.Lat3 55.59644 degree;set sci2_flat_and_level.Lon3 7.58074 degree;set sci2_flat_and_level.ApproachDepth 0.5 m", ASAP 2024-06-14T22:19:59.423Z,1718403599.423 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21376, MT Status=1, MTMSN=1762 2024-06-14T22:20:00.225Z,1718403600.225 [CommandExec](IMPORTANT): got command schedule asap "set sci2_flat_and_level.ApproachSpeed 1 m/s;set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 min;set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 min;set sci2_flat_and_level.MinAltitude 3 m" 4kkp2 3 4.000000 2024-06-14T22:20:00.226Z,1718403600.226 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4kkp2 2024-06-14T22:20:00.228Z,1718403600.228 [CommandExec](IMPORTANT): Scheduled #60 (#3 of 4 with id='4kkp2'): "set sci2_flat_and_level.ApproachSpeed 1 m/s;set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 min;set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 min;set sci2_flat_and_level.MinAltitude 3 m", ASAP 2024-06-14T22:20:25.112Z,1718403625.112 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21377, MT Status=1, MTMSN=1763 2024-06-14T22:20:25.760Z,1718403625.760 [CommandExec](IMPORTANT): got command schedule asap "set sci2_flat_and_level.MaxDepth 10 m;set sci2_flat_and_level.MinOffshore 150 m;set sci2_flat_and_level:NeedComms.CommsTimeout 2 min;run" 4kkp2 4 4.000000 2024-06-14T22:20:25.760Z,1718403625.760 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4kkp2 2024-06-14T22:20:25.761Z,1718403625.761 [CommandExec](IMPORTANT): Scheduled #61 (#4 of 4 with id='4kkp2'): "set sci2_flat_and_level.MaxDepth 10 m;set sci2_flat_and_level.MinOffshore 150 m;set sci2_flat_and_level:NeedComms.CommsTimeout 2 min;run", ASAP 2024-06-14T22:20:25.932Z,1718403625.932 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_flat_and_level.tl 2024-06-14T22:20:39.555Z,1718403639.555 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_flat_and_level.tl id=sci2_flat_and_level 2024-06-14T22:20:53.458Z,1718403653.458 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MissionTimeout 5 hour 2024-06-14T22:20:53.458Z,1718403653.458 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Depth1 7 meter 2024-06-14T22:20:53.459Z,1718403653.459 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lat1 55.609157 degree 2024-06-14T22:20:53.460Z,1718403653.460 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lon1 7.527718 degree 2024-06-14T22:21:07.319Z,1718403667.319 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21378, MT Status=1, MTMSN=1764 2024-06-14T22:21:09.431Z,1718403669.431 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lat2 55.60158 degree 2024-06-14T22:21:09.432Z,1718403669.432 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lon2 7.5501 degree 2024-06-14T22:21:09.433Z,1718403669.433 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lat3 55.59644 degree 2024-06-14T22:21:09.433Z,1718403669.433 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lon3 7.58074 degree 2024-06-14T22:21:09.434Z,1718403669.434 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachDepth 0.5 meter 2024-06-14T22:21:09.464Z,1718403669.464 [CommandExec](IMPORTANT): got command schedule "load Science/circle_sample.xml;set CircleSample.NumSamplers 1 count;set CircleSample.MissionTimeout 1.5 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool" 4kksa 1 3.000000 2024-06-14T22:21:09.465Z,1718403669.465 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4kksa 2024-06-14T22:21:09.466Z,1718403669.466 [CommandExec](IMPORTANT): Scheduled #62 (#1 of 3 with id='4kksa'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 1 count;set CircleSample.MissionTimeout 1.5 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool", AFTER MISSION 2024-06-14T22:21:31.849Z,1718403691.849 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSpeed 1 meter_per_second 2024-06-14T22:21:31.849Z,1718403691.849 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 minute 2024-06-14T22:21:31.850Z,1718403691.850 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 minute 2024-06-14T22:21:31.851Z,1718403691.851 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MinAltitude 3 meter 2024-06-14T22:21:45.050Z,1718403705.050 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MaxDepth 10 meter 2024-06-14T22:21:45.059Z,1718403705.059 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MinOffshore 150 meter 2024-06-14T22:21:45.060Z,1718403705.060 [CommandExec](IMPORTANT): got command set sci2_flat_and_level:NeedComms.CommsTimeout 2 minute 2024-06-14T22:21:45.060Z,1718403705.060 [CommandExec](IMPORTANT): got command run 2024-06-14T22:21:45.069Z,1718403705.069 [CommandExec](IMPORTANT): Running 2024-06-14T22:21:46.986Z,1718403706.986 [MissionManager](IMPORTANT): Started mission sci2_flat_and_level 2024-06-14T22:21:47.188Z,1718403707.188 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T22:23:49.905Z,1718403829.905 [sci2_flat_and_level:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 2.000000 min minutes 2024-06-14T22:23:53.595Z,1718403833.595 [sci2_flat_and_level:Lap:Leg1:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 55.609158,7.527718 2024-06-14T23:07:54.822Z,1718406474.822 [sci2_flat_and_level:Lap:Leg1:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 55.609158,7.527718 2024-06-14T23:07:54.824Z,1718406474.824 [sci2_flat_and_level:Lap:Leg1:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 55.601582,7.550100 2024-06-14T23:13:44.337Z,1718406824.337 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-14T23:28:00.794Z,1718407680.794 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21379, MT Status=1, MTMSN=1765 2024-06-14T23:28:01.625Z,1718407681.625 [CommandExec](IMPORTANT): got command schedule "set CircleSample.SendSampleResultToWG 0 bool;set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.59644 degree;set CircleSample.CenterLongitude 7.58074 degree" 4kksa 2 3.000000 2024-06-14T23:28:01.625Z,1718407681.625 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4kksa 2024-06-14T23:28:01.626Z,1718407681.626 [CommandExec](IMPORTANT): Scheduled #63 (#2 of 3 with id='4kksa'): "set CircleSample.SendSampleResultToWG 0 bool;set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.59644 degree;set CircleSample.CenterLongitude 7.58074 degree", AFTER MISSION 2024-06-14T23:29:56.966Z,1718407796.966 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21380, MT Status=1, MTMSN=1766 2024-06-14T23:29:57.641Z,1718407797.641 [CommandExec](IMPORTANT): got command schedule "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 125 m;set CircleSample:NeedComms.CommsTimeout 2 min;run" 4kksa 3 3.000000 2024-06-14T23:29:57.642Z,1718407797.642 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4kksa 2024-06-14T23:29:57.643Z,1718407797.643 [CommandExec](IMPORTANT): Scheduled #64 (#3 of 3 with id='4kksa'): "set CircleSample.Settle1 0.5 min;set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 125 m;set CircleSample:NeedComms.CommsTimeout 2 min;run", AFTER MISSION 2024-06-14T23:30:03.413Z,1718407803.413 [sci2_flat_and_level:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 2.000000 min minutes 2024-06-14T23:30:07.956Z,1718407807.956 [BuoyancyServo](FAULT): Reading outside of valid range:9215.000000 2024-06-14T23:30:07.957Z,1718407807.957 [BuoyancyServo](FAULT): Buoyancy engine reporting null position 2024-06-14T23:37:59.808Z,1718408279.808 [sci2_flat_and_level:Lap:Leg1:Wpt2:A.Waypoint](IMPORTANT): Reached waypoint: 55.601582,7.550100 2024-06-14T23:37:59.810Z,1718408279.810 [sci2_flat_and_level:Lap:Leg1:Wpt3:A.Waypoint](IMPORTANT): Navigating to waypoint: 55.596439,7.580740 2024-06-14T23:46:45.039Z,1718408805.039 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T00:17:13.227Z,1718410633.227 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T00:31:48.300Z,1718411508.300 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=21381, MT Status=1, MTMSN=1767 2024-06-15T00:31:49.117Z,1718411509.117 [CommandExec](IMPORTANT): got command schedule "load Science/sci2_flat_and_level.tl;set sci2_flat_and_level.MissionTimeout 5 h;set sci2_flat_and_level.Depth1 7 m;set sci2_flat_and_level.Lat1 55.59299 degree;set sci2_flat_and_level.Lon1 7.6046 degree" 4kkwf 1 4.000000 2024-06-15T00:31:49.117Z,1718411509.117 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=4kkwf 2024-06-15T00:31:49.118Z,1718411509.118 [CommandExec](IMPORTANT): Scheduled #65 (#1 of 4 with id='4kkwf'): "load Science/sci2_flat_and_level.tl;set sci2_flat_and_level.MissionTimeout 5 h;set sci2_flat_and_level.Depth1 7 m;set sci2_flat_and_level.Lat1 55.59299 degree;set sci2_flat_and_level.Lon1 7.6046 degree", AFTER MISSION 2024-06-15T00:31:49.943Z,1718411509.943 [sci2_flat_and_level:Lap:Leg1:Wpt3:A.Waypoint](IMPORTANT): Reached waypoint: 55.596439,7.580740 2024-06-15T00:31:51.132Z,1718411511.132 [sci2_flat_and_level:StandardEnvelopes:C.OffshoreEnvelope](IMPORTANT): Offshore envelope ACTIVE 2024-06-15T00:31:58.898Z,1718411518.898 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-15T00:32:06.930Z,1718411526.930 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-15T00:32:07.227Z,1718411527.227 [MissionManager](IMPORTANT): Started mission Default 2024-06-15T00:32:14.492Z,1718411534.492 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21382, MT Status=1, MTMSN=1768 2024-06-15T00:32:21.524Z,1718411541.524 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 1 count 2024-06-15T00:32:21.525Z,1718411541.525 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 1.5 hour 2024-06-15T00:32:21.525Z,1718411541.525 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-06-15T00:32:21.526Z,1718411541.526 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-15T00:32:21.566Z,1718411541.566 [CommandExec](IMPORTANT): got command schedule "set sci2_flat_and_level.Lat2 55.591965 degree;set sci2_flat_and_level.Lon2 7.630687 degree;set sci2_flat_and_level.ApproachDepth 0.5 m;set sci2_flat_and_level.ApproachSpeed 1 m/s;set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 min" 4kkwf 2 4.000000 2024-06-15T00:32:21.566Z,1718411541.566 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=4kkwf 2024-06-15T00:32:21.567Z,1718411541.567 [CommandExec](IMPORTANT): Scheduled #66 (#2 of 4 with id='4kkwf'): "set sci2_flat_and_level.Lat2 55.591965 degree;set sci2_flat_and_level.Lon2 7.630687 degree;set sci2_flat_and_level.ApproachDepth 0.5 m;set sci2_flat_and_level.ApproachSpeed 1 m/s;set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 min", AFTER MISSION 2024-06-15T01:31:17.880Z,1718415077.880 [ESPComponent](IMPORTANT): [sample #22] ESP log summary report (2 messages): @00:46:35.82 Selecting Cartridge 52 @01:24:30.01 Sampled 1000.0ml 2024-06-15T01:31:17.881Z,1718415077.881 [ESPComponent](IMPORTANT): [sample #22] ESP Cartridge 52: S_FILTERING: start: 2024-06-15T00:49:43.697Z end: 2024-06-15T01:24:32.646Z 2024-06-15T00:32:42.870Z,1718411562.870 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-15T00:32:42.871Z,1718411562.871 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-15T00:32:42.872Z,1718411562.872 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-15T00:32:42.872Z,1718411562.872 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.59644 degree 2024-06-15T00:32:42.873Z,1718411562.873 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.58074 degree 2024-06-15T00:32:56.336Z,1718411576.336 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-15T00:32:56.337Z,1718411576.337 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-15T00:32:56.338Z,1718411576.338 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 125 meter 2024-06-15T00:32:56.338Z,1718411576.338 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 2 minute 2024-06-15T00:32:56.339Z,1718411576.339 [CommandExec](IMPORTANT): got command run 2024-06-15T00:32:56.341Z,1718411576.341 [CommandExec](IMPORTANT): Running 2024-06-15T00:32:56.454Z,1718411576.454 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-15T00:32:56.516Z,1718411576.516 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-15T00:32:56.517Z,1718411576.517 [CircleSample:StandardEnvelopes:C.OffshoreEnvelope](IMPORTANT): Offshore envelope ACTIVE 2024-06-15T00:32:58.445Z,1718411578.445 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-06-15T00:32:59.652Z,1718411579.652 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.596439 arcdeg , 7.580740 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-15T00:33:01.688Z,1718411581.688 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-15T00:33:05.682Z,1718411585.682 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21383, MT Status=1, MTMSN=1769 2024-06-15T00:33:06.570Z,1718411586.570 [CommandExec](IMPORTANT): got command schedule "set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 min;set sci2_flat_and_level.MinAltitude 3 m;set sci2_flat_and_level.MaxDepth 10 m;set sci2_flat_and_level.MinOffshore 150 m;set sci2_flat_and_level:NeedComms.CommsTimeout 2 min" 4kkwf 3 4.000000 2024-06-15T00:33:06.570Z,1718411586.570 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=4kkwf 2024-06-15T00:33:06.571Z,1718411586.571 [CommandExec](IMPORTANT): Scheduled #67 (#3 of 4 with id='4kkwf'): "set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 min;set sci2_flat_and_level.MinAltitude 3 m;set sci2_flat_and_level.MaxDepth 10 m;set sci2_flat_and_level.MinOffshore 150 m;set sci2_flat_and_level:NeedComms.CommsTimeout 2 min", AFTER MISSION 2024-06-15T00:34:58.126Z,1718411698.126 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T00:43:00.997Z,1718412180.997 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.028409 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-15T00:44:45.166Z,1718412285.166 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.121394 m , settling for 0.500000 min 2024-06-15T00:45:28.084Z,1718412328.084 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-15T00:45:28.239Z,1718412328.239 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-15T00:45:50.751Z,1718412350.751 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-15T00:46:02.034Z,1718412362.034 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003419 CHAN A1 (24V): -0.001138 CHAN A2 (12V): -0.000522 CHAN A3 (5V): -0.000089 CHAN B0 (3.3V): 0.000153 CHAN B1 (3.15aV): -0.000221 CHAN B2 (3.15bV): 0.000044 CHAN B3 (GND): 0.000146 OPEN: 0.007790 Full Scale: +/- 1 mA 2024-06-15T00:46:22.517Z,1718412382.517 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-15T00:46:23.311Z,1718412383.311 [ESPComponent](IMPORTANT): [sample #22] Setting cartridge argument to -1 2024-06-15T00:46:23.312Z,1718412383.312 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-15T00:46:23.741Z,1718412383.741 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.373788 m in 12.873011 degC water with 1.494080 ug/l chlorophyll fluorescence. 2024-06-15T00:46:24.119Z,1718412384.119 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-15T00:46:24.532Z,1718412384.532 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-15T00:46:26.555Z,1718412386.555 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-15T00:46:58.470Z,1718412418.470 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PRIMING (@52,aRNAl_bac) 2024-06-15T00:47:00.492Z,1718412420.492 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PRIMING (@100.0) 2024-06-15T00:47:00.887Z,1718412420.887 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-15T00:47:01.290Z,1718412421.290 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-15T00:49:43.698Z,1718412583.698 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_FILTERING (@0.0) 2024-06-15T00:49:44.108Z,1718412584.108 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-15T00:53:01.707Z,1718412781.707 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T01:04:58.510Z,1718413498.510 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T01:09:56.183Z,1718413796.183 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-06-15T01:10:13.415Z,1718413813.415 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-15T01:10:13.895Z,1718413813.895 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-15T01:10:14.206Z,1718413814.206 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-15T01:10:14.591Z,1718413814.591 [ThrusterHE](IMPORTANT): Status lock Detected. 2024-06-15T01:10:16.261Z,1718413816.261 [ThrusterHE](FAULT): Lock Detect. Motor stopped spinning or could not start spinning. 2024-06-15T01:12:19.643Z,1718413939.643 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-06-15T01:24:32.246Z,1718414672.246 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PAUSED (@1000.0) 2024-06-15T01:24:32.646Z,1718414672.646 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-15T01:24:34.667Z,1718414674.667 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-15T01:24:35.075Z,1718414675.075 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,aRNAl_bac) 2024-06-15T01:25:09.860Z,1718414709.860 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T01:26:10.009Z,1718414770.009 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPING (@52,aRNAl_bac) 2024-06-15T01:26:17.292Z,1718414777.292 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-15T01:26:17.686Z,1718414777.686 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPED 2024-06-15T01:31:17.462Z,1718415077.462 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence completed normally. 2024-06-15T01:31:19.931Z,1718415079.931 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-15T01:31:21.665Z,1718415081.665 [MissionManager](IMPORTANT): Started mission Default 2024-06-15T01:35:09.886Z,1718415309.886 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=21384, MT Status=1, MTMSN=1770 2024-06-15T01:35:10.438Z,1718415310.438 [CommandExec](IMPORTANT): got command schedule "run" 4kkwf 4 4.000000 2024-06-15T01:35:10.438Z,1718415310.438 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=4kkwf 2024-06-15T01:35:10.440Z,1718415310.440 [CommandExec](IMPORTANT): Scheduled #68 (#4 of 4 with id='4kkwf'): "run", AFTER MISSION 2024-06-15T01:35:10.586Z,1718415310.586 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2_flat_and_level.tl 2024-06-15T01:35:19.546Z,1718415319.546 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2_flat_and_level.tl id=sci2_flat_and_level 2024-06-15T01:35:34.548Z,1718415334.548 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MissionTimeout 5 hour 2024-06-15T01:35:34.549Z,1718415334.549 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Depth1 7 meter 2024-06-15T01:35:34.549Z,1718415334.549 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lat1 55.59299 degree 2024-06-15T01:35:34.550Z,1718415334.550 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lon1 7.6046 degree 2024-06-15T01:35:57.035Z,1718415357.035 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lat2 55.591965 degree 2024-06-15T01:35:57.036Z,1718415357.036 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.Lon2 7.630687 degree 2024-06-15T01:35:57.036Z,1718415357.036 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachDepth 0.5 meter 2024-06-15T01:35:57.037Z,1718415357.037 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSpeed 1 meter_per_second 2024-06-15T01:35:57.038Z,1718415357.038 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSettleTimePreDive 0.5 minute 2024-06-15T01:36:19.162Z,1718415379.162 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.ApproachSettleTimePostDive 0.001 minute 2024-06-15T01:36:19.163Z,1718415379.163 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MinAltitude 3 meter 2024-06-15T01:36:19.164Z,1718415379.164 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MaxDepth 10 meter 2024-06-15T01:36:19.165Z,1718415379.165 [CommandExec](IMPORTANT): got command set sci2_flat_and_level.MinOffshore 150 meter 2024-06-15T01:36:19.165Z,1718415379.165 [CommandExec](IMPORTANT): got command set sci2_flat_and_level:NeedComms.CommsTimeout 2 minute 2024-06-15T01:36:19.167Z,1718415379.167 [CommandExec](IMPORTANT): got command run 2024-06-15T01:36:19.169Z,1718415379.169 [CommandExec](IMPORTANT): Running 2024-06-15T01:36:19.264Z,1718415379.264 [MissionManager](IMPORTANT): Started mission sci2_flat_and_level 2024-06-15T01:36:19.317Z,1718415379.317 [sci2_flat_and_level:StandardEnvelopes:C.OffshoreEnvelope](IMPORTANT): Offshore envelope ACTIVE 2024-06-15T01:38:24.849Z,1718415504.849 [sci2_flat_and_level:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 2.000000 min minutes 2024-06-15T01:38:29.332Z,1718415509.332 [sci2_flat_and_level:Lap:Leg1:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 55.592991,7.604600 2024-06-15T01:48:57.301Z,1718416137.301 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T02:18:57.530Z,1718417937.530 [sci2_flat_and_level:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T02:33:39.896Z,1718418819.896 [BPC1](FAULT): Failed to receive data from 17 sticks prior to timeout. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-15T02:43:58.037Z,1718419438.037 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-15T02:44:07.755Z,1718419447.755 [sci2_flat_and_level:Lap:Leg1:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 55.592991,7.604600 2024-06-15T02:44:07.757Z,1718419447.757 [sci2_flat_and_level:Lap:Leg1:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 55.591965,7.630687 2024-06-15T02:46:08.594Z,1718419568.594 [sci2_flat_and_level:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 2.000000 min minutes 2024-06-15T02:55:56.082Z,1718420156.082 [sci2_flat_and_level:Lap:Leg1:Wpt2:A.Waypoint](IMPORTANT): Reached waypoint: 55.591965,7.630687 2024-06-15T02:56:02.159Z,1718420162.159 [MissionManager](IMPORTANT): Started mission Default 2024-06-15T03:01:28.172Z,1718420488.172 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-15T03:07:51.845Z,1718420871.845 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21385, MT Status=1, MTMSN=1771 2024-06-15T03:07:52.631Z,1718420872.631 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 55.591965 degree;set profile_station.Lon 7.630687 degree;set profile_station.YoYoMaxDepth 5 m" 4kl0t 1 2.000000 2024-06-15T03:07:52.632Z,1718420872.632 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4kl0t 2024-06-15T03:07:52.633Z,1718420872.633 [CommandExec](IMPORTANT): Scheduled #69 (#1 of 2 with id='4kl0t'): "load Science/profile_station.tl;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 55.591965 degree;set profile_station.Lon 7.630687 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-15T03:12:01.898Z,1718421121.898 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21386, MT Status=1, MTMSN=1772 2024-06-15T03:12:02.664Z,1718421122.664 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 15 min;run" 4kl0t 2 2.000000 2024-06-15T03:12:02.665Z,1718421122.665 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4kl0t 2024-06-15T03:12:02.666Z,1718421122.666 [CommandExec](IMPORTANT): Scheduled #70 (#2 of 2 with id='4kl0t'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 15 min;run", AFTER MISSION 2024-06-15T03:12:02.813Z,1718421122.813 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.tl 2024-06-15T03:12:09.268Z,1718421129.268 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.tl id=profile_station 2024-06-15T03:12:28.850Z,1718421148.850 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 6 hour 2024-06-15T03:12:28.850Z,1718421148.850 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 45 minute 2024-06-15T03:12:28.851Z,1718421148.851 [CommandExec](IMPORTANT): got command set profile_station.Lat 55.591965 degree 2024-06-15T03:12:28.852Z,1718421148.852 [CommandExec](IMPORTANT): got command set profile_station.Lon 7.630687 degree 2024-06-15T03:12:28.853Z,1718421148.853 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 5 meter 2024-06-15T03:12:51.869Z,1718421171.869 [CommandExec](IMPORTANT): got command set profile_station.YoYoMinAltitude 5 meter 2024-06-15T03:12:51.870Z,1718421171.870 [CommandExec](IMPORTANT): got command set profile_station.MinAltitude 3 meter 2024-06-15T03:12:51.871Z,1718421171.871 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2024-06-15T03:12:51.871Z,1718421171.871 [CommandExec](IMPORTANT): got command set profile_station.MinOffshore 1 kilometer 2024-06-15T03:12:51.872Z,1718421171.872 [CommandExec](IMPORTANT): got command set profile_station:NeedComms.CommsTimeout 15 minute 2024-06-15T03:12:51.872Z,1718421171.872 [CommandExec](IMPORTANT): got command run 2024-06-15T03:12:51.876Z,1718421171.876 [CommandExec](IMPORTANT): Running 2024-06-15T03:12:52.172Z,1718421172.172 [MissionManager](IMPORTANT): Started mission profile_station 2024-06-15T03:12:52.233Z,1718421172.233 [profile_station:StandardEnvelopes:C.OffshoreEnvelope](IMPORTANT): Offshore envelope ACTIVE 2024-06-15T03:13:01.010Z,1718421181.010 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-06-15T03:18:38.927Z,1718421518.927 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21387, MT Status=1, MTMSN=1773 2024-06-15T03:18:39.770Z,1718421519.770 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.tl;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 55.591965 degree;set profile_station.Lon 7.630687 degree;set profile_station.YoYoMaxDepth 5 m" 4kl6c 1 2.000000 2024-06-15T03:18:39.770Z,1718421519.770 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=4kl6c 2024-06-15T03:18:39.772Z,1718421519.772 [CommandExec](IMPORTANT): Scheduled #71 (#1 of 2 with id='4kl6c'): "load Science/profile_station.tl;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 55.591965 degree;set profile_station.Lon 7.630687 degree;set profile_station.YoYoMaxDepth 5 m", AFTER MISSION 2024-06-15T03:19:03.298Z,1718421543.298 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21388, MT Status=1, MTMSN=1774 2024-06-15T03:19:04.041Z,1718421544.041 [CommandExec](IMPORTANT): got command schedule "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 15 min;run" 4kl6c 2 2.000000 2024-06-15T03:19:04.041Z,1718421544.041 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=4kl6c 2024-06-15T03:19:04.042Z,1718421544.042 [CommandExec](IMPORTANT): Scheduled #72 (#2 of 2 with id='4kl6c'): "set profile_station.YoYoMinAltitude 5 m;set profile_station.MinAltitude 3 m;set profile_station.MaxDepth 10 m;set profile_station.MinOffshore 1 km;set profile_station:NeedComms.CommsTimeout 15 min;run", AFTER MISSION 2024-06-15T03:31:00.816Z,1718422260.816 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 15.000000 min minutes 2024-06-15T03:50:33.294Z,1718423433.294 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T03:56:46.624Z,1718423806.624 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.852024 +/- 0.086851 deg (conf. level 99.90%, sigma: 0.044295 deg). 2024-06-15T04:04:12.155Z,1718424252.155 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-06-15T04:23:53.911Z,1718425433.911 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T04:23:53.914Z,1718425433.914 [profile_station:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T04:36:11.814Z,1718426171.814 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 15.000000 min minutes 2024-06-15T04:36:40.143Z,1718426200.143 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.858190 +/- 0.087557 deg (conf. level 99.90%, sigma: 0.038442 deg). 2024-06-15T05:03:29.011Z,1718427809.011 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T05:04:43.727Z,1718427883.727 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.853218 +/- 0.064326 deg (conf. level 99.90%, sigma: 0.038019 deg). 2024-06-15T05:27:25.232Z,1718429245.232 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-15T05:40:50.002Z,1718430050.002 [profile_station:O.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T05:40:50.009Z,1718430050.009 [profile_station:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T05:42:29.974Z,1718430149.974 [profile_station:NeedComms:D:A_Timeout:A](IMPORTANT): Comms timed out after 15.000000 min minutes 2024-06-15T05:42:37.178Z,1718430157.178 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21402, MT Status=1, MTMSN=1775 2024-06-15T05:42:37.719Z,1718430157.719 [CommandExec](IMPORTANT): got command stop 2024-06-15T05:42:37.719Z,1718430157.719 [CommandExec](IMPORTANT): Scheduling is paused for 2 commands 2024-06-15T05:42:37.720Z,1718430157.720 [CommandExec](IMPORTANT): got command schedule clear 2024-06-15T05:42:37.720Z,1718430157.720 [CommandExec](IMPORTANT): Cleared 2 scheduled commands. 2024-06-15T05:42:37.720Z,1718430157.720 [CommandExec](IMPORTANT): got command schedule resume 2024-06-15T05:42:37.720Z,1718430157.720 [CommandExec](IMPORTANT): Scheduling is resumed 2024-06-15T05:42:38.022Z,1718430158.022 [MissionManager](IMPORTANT): Started mission Default 2024-06-15T06:50:46.369Z,1718434246.369 [ESPComponent](IMPORTANT): [sample #23] ESP log summary report (2 messages): @06:07:14.52 Selecting Cartridge 51 @06:43:58.41 Sampled 1000.0ml 2024-06-15T06:50:46.370Z,1718434246.370 [ESPComponent](IMPORTANT): [sample #23] ESP Cartridge 51: S_FILTERING: start: 2024-06-15T06:10:23.524Z end: 2024-06-15T06:44:01.138Z 2024-06-15T07:46:00.169Z,1718437560.169 [ESPComponent](IMPORTANT): [sample #24] ESP log summary report (2 messages): @07:01:01.77 Selecting Cartridge 50 @07:39:12.36 Sampled 1000.0ml 2024-06-15T07:46:00.170Z,1718437560.170 [ESPComponent](IMPORTANT): [sample #24] ESP Cartridge 50: S_FILTERING: start: 2024-06-15T07:04:09.969Z end: 2024-06-15T07:39:14.845Z 2024-06-15T05:51:23.924Z,1718430683.924 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21403, MT Status=1, MTMSN=1776 2024-06-15T05:51:25.004Z,1718430685.004 [CommandExec](IMPORTANT): got command schedule asap "load Science/circle_sample.xml;set CircleSample.MissionTimeout 2.5 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool" 4l78i 1 3.000000 2024-06-15T05:51:25.004Z,1718430685.004 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=4l78i 2024-06-15T05:51:25.005Z,1718430685.005 [CommandExec](IMPORTANT): Scheduled #73 (#1 of 3 with id='4l78i'): "load Science/circle_sample.xml;set CircleSample.MissionTimeout 2.5 h;set CircleSample.NeedCommsMissionStart 0 bool;set CircleSample.TrackAcousticTarget 0 bool;set CircleSample.SendSampleResultToWG 0 bool", ASAP 2024-06-15T05:51:48.310Z,1718430708.310 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21404, MT Status=1, MTMSN=1777 2024-06-15T05:51:49.072Z,1718430709.072 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.591965 degree;set CircleSample.CenterLongitude 7.630687 degree;set CircleSample.Settle1 0.5 min" 4l78i 2 3.000000 2024-06-15T05:51:49.072Z,1718430709.072 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=4l78i 2024-06-15T05:51:49.073Z,1718430709.073 [CommandExec](IMPORTANT): Scheduled #74 (#2 of 3 with id='4l78i'): "set CircleSample.DepthCircle 5 m;set CircleSample.CircleYoYoMaxDepth 10 m;set CircleSample.CenterLatitude 55.591965 degree;set CircleSample.CenterLongitude 7.630687 degree;set CircleSample.Settle1 0.5 min", ASAP 2024-06-15T05:52:15.640Z,1718430735.640 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21405, MT Status=1, MTMSN=1778 2024-06-15T05:52:16.279Z,1718430736.279 [CommandExec](IMPORTANT): got command schedule asap "set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 125 m;set CircleSample:NeedComms.CommsTimeout 2 min;run" 4l78i 3 3.000000 2024-06-15T05:52:16.279Z,1718430736.279 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=4l78i 2024-06-15T05:52:16.280Z,1718430736.280 [CommandExec](IMPORTANT): Scheduled #75 (#3 of 3 with id='4l78i'): "set CircleSample.MinAltitude 3 m;set CircleSample.MinOffshore 125 m;set CircleSample:NeedComms.CommsTimeout 2 min;run", ASAP 2024-06-15T05:52:16.443Z,1718430736.443 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-06-15T05:52:25.933Z,1718430745.933 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-06-15T05:52:44.729Z,1718430764.729 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 2.5 hour 2024-06-15T05:52:44.729Z,1718430764.729 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-06-15T05:52:44.730Z,1718430764.730 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 0 bool 2024-06-15T05:52:44.779Z,1718430764.779 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 0 bool 2024-06-15T05:53:04.293Z,1718430784.293 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 5 meter 2024-06-15T05:53:04.294Z,1718430784.294 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 10 meter 2024-06-15T05:53:04.294Z,1718430784.294 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 55.591965 degree 2024-06-15T05:53:04.299Z,1718430784.299 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude 7.630687 degree 2024-06-15T05:53:04.300Z,1718430784.300 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 0.5 minute 2024-06-15T05:53:18.638Z,1718430798.638 [CommandExec](IMPORTANT): got command set CircleSample.MinAltitude 3 meter 2024-06-15T05:53:18.639Z,1718430798.639 [CommandExec](IMPORTANT): got command set CircleSample.MinOffshore 125 meter 2024-06-15T05:53:18.651Z,1718430798.651 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 2 minute 2024-06-15T05:53:18.652Z,1718430798.652 [CommandExec](IMPORTANT): got command run 2024-06-15T05:53:18.654Z,1718430798.654 [CommandExec](IMPORTANT): Running 2024-06-15T05:53:20.734Z,1718430800.734 [MissionManager](IMPORTANT): Started mission CircleSample 2024-06-15T05:53:20.761Z,1718430800.761 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 24.00 hours. 2024-06-15T05:53:20.762Z,1718430800.762 [CircleSample:StandardEnvelopes:C.OffshoreEnvelope](IMPORTANT): Offshore envelope ACTIVE 2024-06-15T05:53:20.895Z,1718430800.895 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 1.00 m/s, pitch -20.00 deg, mass-position 12.06 mm. Average elevator angle=3.861407 +/- 0.051514 deg (conf. level 99.90%, sigma: 0.029976 deg). 2024-06-15T05:53:20.919Z,1718430800.919 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T05:53:20.984Z,1718430800.984 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T05:53:22.479Z,1718430802.479 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-06-15T05:53:23.719Z,1718430803.719 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 55.591965 arcdeg , 7.630687 arcdeg , at radius = 20.000000 m and depth = 5.000000 m . 2024-06-15T05:53:25.722Z,1718430805.722 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 0.500000 min . 2024-06-15T06:03:25.151Z,1718431405.151 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.029990 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-06-15T06:05:23.217Z,1718431523.217 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.038127 m , settling for 0.500000 min 2024-06-15T06:06:06.152Z,1718431566.152 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-15T06:06:06.311Z,1718431566.311 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-15T06:07:01.117Z,1718431621.117 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-15T06:07:01.908Z,1718431621.908 [ESPComponent](IMPORTANT): [sample #23] Setting cartridge argument to -1 2024-06-15T06:07:01.909Z,1718431621.909 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-15T06:07:02.345Z,1718431622.345 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.396139 m in 12.887262 degC water with 1.224960 ug/l chlorophyll fluorescence. 2024-06-15T06:07:02.710Z,1718431622.710 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-15T06:07:03.160Z,1718431623.160 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-15T06:07:05.173Z,1718431625.173 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-15T06:07:37.069Z,1718431657.069 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac) 2024-06-15T06:07:39.473Z,1718431659.473 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PRIMING (@100.0) 2024-06-15T06:07:39.474Z,1718431659.474 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-15T06:07:39.877Z,1718431659.877 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-15T06:10:23.524Z,1718431823.524 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_FILTERING (@0.0) 2024-06-15T06:10:23.909Z,1718431823.909 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-15T06:23:21.272Z,1718432601.272 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T06:44:00.733Z,1718433840.733 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PAUSED (@1000.0) 2024-06-15T06:44:01.139Z,1718433841.139 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-15T06:44:03.159Z,1718433843.159 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-15T06:44:03.578Z,1718433843.578 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac) 2024-06-15T06:45:38.511Z,1718433938.511 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPING (@51,aRNAl_bac) 2024-06-15T06:45:45.782Z,1718433945.782 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-15T06:45:46.183Z,1718433946.183 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPED 2024-06-15T06:46:02.054Z,1718433962.054 [CBIT](IMPORTANT): Beginning ground fault scan 2024-06-15T06:46:12.977Z,1718433972.977 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003759 CHAN A1 (24V): -0.001271 CHAN A2 (12V): -0.000311 CHAN A3 (5V): 0.000778 CHAN B0 (3.3V): -0.002927 CHAN B1 (3.15aV): 0.000548 CHAN B2 (3.15bV): -0.000288 CHAN B3 (GND): -0.000290 OPEN: 0.007693 Full Scale: +/- 1 mA 2024-06-15T06:47:25.220Z,1718434045.220 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T06:50:45.949Z,1718434245.949 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence completed normally. 2024-06-15T06:50:46.780Z,1718434246.780 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Invalid Command. Current Limiter Activated. 2024-06-15T06:50:48.411Z,1718434248.411 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-06-15T06:53:59.938Z,1718434439.938 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T06:55:52.247Z,1718434552.247 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-06-15T06:55:55.496Z,1718434555.496 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 52.490735 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 20.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-06-15T06:58:40.329Z,1718434720.329 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.072642 m , settling for 1.000000 min 2024-06-15T06:59:53.135Z,1718434793.135 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-06-15T06:59:53.262Z,1718434793.262 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-06-15T07:00:48.369Z,1718434848.369 [ESPComponent](IMPORTANT): ESP has connected as client 2024-06-15T07:00:49.194Z,1718434849.194 [ESPComponent](IMPORTANT): [sample #24] Setting cartridge argument to -1 2024-06-15T07:00:49.195Z,1718434849.195 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-06-15T07:00:49.614Z,1718434849.614 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.716183 m in 13.144709 degC water with 1.164640 ug/l chlorophyll fluorescence. 2024-06-15T07:00:49.990Z,1718434849.990 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_LOG 2024-06-15T07:00:50.390Z,1718434850.390 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-06-15T07:00:52.410Z,1718434852.410 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_LOADING_CARTRIDGE 2024-06-15T07:01:24.327Z,1718434884.327 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PRIMING (@50,aRNAl_bac) 2024-06-15T07:01:26.346Z,1718434886.346 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PRIMING (@100.0) 2024-06-15T07:01:26.772Z,1718434886.772 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-06-15T07:01:27.153Z,1718434887.153 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-06-15T07:04:09.970Z,1718435049.970 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_FILTERING (@0.0) 2024-06-15T07:04:10.390Z,1718435050.390 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-06-15T07:17:52.613Z,1718435872.613 [CircleSample:StandardEnvelopes:A.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T07:24:25.700Z,1718436265.700 [CircleSample:L.AltitudeEnvelope](IMPORTANT): Altitude envelope ACTIVE 2024-06-15T07:39:14.466Z,1718437154.466 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PAUSED (@1000.0) 2024-06-15T07:39:14.846Z,1718437154.846 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PROCESSING (@1000.0) 2024-06-15T07:39:16.865Z,1718437156.865 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-06-15T07:39:17.271Z,1718437157.271 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,aRNAl_bac) 2024-06-15T07:40:52.224Z,1718437252.224 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPING (@50,aRNAl_bac) 2024-06-15T07:40:59.499Z,1718437259.499 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-06-15T07:40:59.929Z,1718437259.929 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPED 2024-06-15T07:45:59.765Z,1718437559.765 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence completed normally. 2024-06-15T07:46:02.274Z,1718437562.274 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-06-15T07:46:03.893Z,1718437563.893 [MissionManager](IMPORTANT): Started mission Default 2024-06-15T11:51:25.058Z,1718452285.058 [Default:CheckIn:Read_Iridium:A_Timeout:B](CRITICAL): Dropped weight due to communications timeout. 2024-06-15T07:51:29.113Z,1718437889.113 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-06-15T07:53:14.276Z,1718437994.276 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21406, MT Status=1, MTMSN=1779 2024-06-15T07:53:15.828Z,1718437995.828 [CommandExec](IMPORTANT): got command configSet IBIT.batteryCapacityThreshold 20 ampere_hour persist 2024-06-15T07:53:15.837Z,1718437995.837 [CommandExec](IMPORTANT): got command configSet IBIT.batteryVoltageThreshold 12 volt persist 2024-06-15T07:53:38.702Z,1718438018.702 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21407, MT Status=1, MTMSN=1780 2024-06-15T07:53:39.268Z,1718438019.268 [CommandExec](IMPORTANT): got command stop 2024-06-15T07:53:39.268Z,1718438019.268 [CommandExec](IMPORTANT): Scheduling is paused 2024-06-15T07:54:34.309Z,1718438074.309 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21408, MT Status=1, MTMSN=1781 2024-06-15T07:54:34.669Z,1718438074.669 [CommandExec](IMPORTANT): got command ! echo 1 > /dev/loadB6 2024-06-15T08:54:49.336Z,1718441689.336 [BPC1](FAULT): Failed to receive data from 19 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 35, 62. 2024-06-15T11:50:35.784Z,1718452235.784 [BPC1](FAULT): Failed to receive data from 18 sticks prior to timeout. Missing stick IDs are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 34, 62. 2024-06-15T11:51:24.804Z,1718452284.804 [CommandExec](IMPORTANT): got command burn on 2024-06-15T11:51:24.804Z,1718452284.804 [CommandExec](IMPORTANT): Activating dropweight wire 2024-06-15T11:51:25.066Z,1718452285.066 [CommandExec](FAULT): Scheduling is paused 2024-06-15T11:56:26.480Z,1718452586.480 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 250.369222 min 2024-06-15T11:56:28.049Z,1718452588.049 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-06-15T12:10:10.800Z,1718453410.800 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=21523, MT Status=1, MTMSN=1782