2022-05-28T16:17:07.866Z,1653754627.866 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2022-05-28T16:17:11.777Z,1653754631.777 [SBIT](IMPORTANT): git: 2022-05-25A 2022-05-28T16:17:11.777Z,1653754631.777 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2022-05-28T16:19:01.284Z,1653754741.284 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47163, MT Status=1, MTMSN=3466 2022-05-28T16:19:02.034Z,1653754742.034 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-28T16:19:05.212Z,1653754745.212 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-28T16:19:15.187Z,1653754755.187 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 3 hour 2022-05-28T16:19:15.188Z,1653754755.188 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2022-05-28T16:19:15.189Z,1653754755.189 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.368 degree 2022-05-28T16:19:15.189Z,1653754755.189 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.851 degree 2022-05-28T16:19:15.190Z,1653754755.190 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-05-28T16:19:15.191Z,1653754755.191 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-05-28T16:19:15.191Z,1653754755.191 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-05-28T16:19:15.193Z,1653754755.193 [CommandExec](IMPORTANT): got command run 2022-05-28T16:19:15.196Z,1653754755.196 [CommandExec](IMPORTANT): Running 2022-05-28T16:19:23.558Z,1653754763.558 [SBIT](IMPORTANT): Beginning Startup BIT 2022-05-28T16:19:23.566Z,1653754763.566 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:19:34.501Z,1653754774.501 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.006073 CHAN A1 (24V): 0.001627 CHAN A2 (12V): -0.006000 CHAN A3 (5V): -0.072634 CHAN B0 (3.3V): -0.001514 CHAN B1 (3.15aV): -0.001596 CHAN B2 (3.15bV): 0.025772 CHAN B3 (GND): -0.000023 OPEN: 0.012445 Full Scale: +/- 1 mA 2022-05-28T16:20:16.859Z,1653754816.859 [SBIT](FAULT): Mass: EXPECTED:-0.027809 ACTUAL:-0.028000 2022-05-28T16:20:16.860Z,1653754816.860 [SBIT](FAULT): Control surface position failure. 2022-05-28T16:20:17.283Z,1653754817.283 [SBIT](CRITICAL): SBIT FAILED 2022-05-28T16:20:17.283Z,1653754817.283 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): BPC1.loadAtStartup=0 bool; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): DAT.sbdAddress=20 enum; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): Depth_Keller.offset=-0.41 decibar; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): Express linearApproximation CBIT.ampHoursUsed 5.0 ampere_hour; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): Express linearApproximation NAL9602.sigQuality 1.000000 count; 2022-05-28T16:20:17.284Z,1653754817.284 [SBIT](IMPORTANT): RudderServo.offsetAngle=0 degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex1=34.54 degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex2=34.46 degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex3=34.23 degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex4=34.31 degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex5=nan degree; 2022-05-28T16:20:17.285Z,1653754817.285 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.loadAtStartup=1 bool; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex1=-120.09 degree; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex2=-119.71 degree; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex3=-119.71 degree; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex4=-120.11 degree; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex5=nan degree; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=418.268050 cubic_centimeter; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): VerticalControl.massDefault=-27.808592 millimeter; 2022-05-28T16:20:17.286Z,1653754817.286 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-28 millimeter; 2022-05-28T16:20:17.288Z,1653754817.288 [CommandExec](FAULT): Scheduling is paused 2022-05-28T16:20:17.660Z,1653754817.660 [MissionManager](IMPORTANT): Started mission Startup 2022-05-28T16:21:20.436Z,1653754880.436 [MissionManager](IMPORTANT): Started mission sci2 2022-05-28T16:21:20.847Z,1653754880.847 [MissionManager](IMPORTANT): Started mission Default 2022-05-28T16:21:50.041Z,1653754910.041 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47164, MT Status=1, MTMSN=3467 2022-05-28T16:21:50.611Z,1653754910.611 [CommandExec](IMPORTANT): got command schedule clear 2022-05-28T16:21:50.611Z,1653754910.611 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2022-05-28T16:27:31.291Z,1653755251.291 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.167241 min 2022-05-28T16:28:16.258Z,1653755296.258 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47165, MT Status=1, MTMSN=3468 2022-05-28T16:28:17.046Z,1653755297.046 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.33379 degree;set sci2.Lon1 -119.71973 degree;set sci2.Lat2 34.368 degree;set sci2.Lon2 -119.851 degree;set sci2.YoYoMaxDepth 50 m" 22dr3 1 2.000000 2022-05-28T16:28:17.046Z,1653755297.046 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=22dr3 2022-05-28T16:28:17.047Z,1653755297.047 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 2 with id='22dr3'): "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.33379 degree;set sci2.Lon1 -119.71973 degree;set sci2.Lat2 34.368 degree;set sci2.Lon2 -119.851 degree;set sci2.YoYoMaxDepth 50 m", ASAP 2022-05-28T16:28:44.351Z,1653755324.351 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T162844 2022-05-28T16:28:52.362Z,1653755332.362 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadC4 2022-05-28T16:28:53.327Z,1653755333.327 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T162853 2022-05-28T16:29:01.111Z,1653755341.111 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:29:01.206Z,1653755341.206 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:29:12.139Z,1653755352.139 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.089224 CHAN A1 (24V): 0.002126 CHAN A2 (12V): -0.005655 CHAN A3 (5V): -0.003484 CHAN B0 (3.3V): -0.001551 CHAN B1 (3.15aV): -0.001387 CHAN B2 (3.15bV): -0.001141 CHAN B3 (GND): -0.031555 OPEN: 0.012077 Full Scale: +/- 1 mA 2022-05-28T16:30:58.710Z,1653755458.710 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:30:58.818Z,1653755458.818 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:31:09.675Z,1653755469.675 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.064020 CHAN A1 (24V): 0.001213 CHAN A2 (12V): -0.005663 CHAN A3 (5V): -0.003466 CHAN B0 (3.3V): -0.001620 CHAN B1 (3.15aV): -0.001505 CHAN B2 (3.15bV): -0.001384 CHAN B3 (GND): -0.016586 OPEN: 0.012559 Full Scale: +/- 1 mA 2022-05-28T16:31:16.835Z,1653755476.835 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:31:16.968Z,1653755476.968 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:31:27.898Z,1653755487.898 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.005590 CHAN A1 (24V): 0.001269 CHAN A2 (12V): -0.005828 CHAN A3 (5V): -0.004794 CHAN B0 (3.3V): -0.001864 CHAN B1 (3.15aV): -0.001679 CHAN B2 (3.15bV): -0.101215 CHAN B3 (GND): 0.000473 OPEN: 0.012521 Full Scale: +/- 1 mA 2022-05-28T16:31:37.334Z,1653755497.334 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:31:37.607Z,1653755497.607 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:31:48.479Z,1653755508.479 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.220586 CHAN A1 (24V): 0.001492 CHAN A2 (12V): -0.006153 CHAN A3 (5V): -0.003750 CHAN B0 (3.3V): -0.001316 CHAN B1 (3.15aV): -0.001384 CHAN B2 (3.15bV): -0.001323 CHAN B3 (GND): -0.000357 OPEN: 0.012197 Full Scale: +/- 1 mA 2022-05-28T16:32:05.634Z,1653755525.634 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:32:05.862Z,1653755525.862 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:32:16.793Z,1653755536.793 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.006173 CHAN A1 (24V): 0.000851 CHAN A2 (12V): -0.005575 CHAN A3 (5V): 0.003135 CHAN B0 (3.3V): -0.002140 CHAN B1 (3.15aV): -0.001222 CHAN B2 (3.15bV): 0.021104 CHAN B3 (GND): -0.000319 OPEN: 0.012450 Full Scale: +/- 1 mA 2022-05-28T16:32:59.006Z,1653755579.006 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:32:59.227Z,1653755579.227 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:33:04.867Z,1653755584.867 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.727346 min 2022-05-28T16:33:10.207Z,1653755590.207 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005611 CHAN A1 (24V): 0.002186 CHAN A2 (12V): -0.005808 CHAN A3 (5V): -0.003835 CHAN B0 (3.3V): -0.001039 CHAN B1 (3.15aV): -0.001638 CHAN B2 (3.15bV): -0.001273 CHAN B3 (GND): -0.000398 OPEN: 0.012272 Full Scale: +/- 1 mA 2022-05-28T16:33:25.694Z,1653755605.694 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:33:25.899Z,1653755605.899 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:33:36.896Z,1653755616.896 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.000601 CHAN A1 (24V): -0.057130 CHAN A2 (12V): -0.004929 CHAN A3 (5V): -0.003726 CHAN B0 (3.3V): 0.013395 CHAN B1 (3.15aV): -0.001059 CHAN B2 (3.15bV): -0.001464 CHAN B3 (GND): -0.000382 OPEN: 0.012254 Full Scale: +/- 1 mA 2022-05-28T16:34:14.519Z,1653755654.519 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:34:14.861Z,1653755654.861 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:34:25.758Z,1653755665.758 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.006389 CHAN A1 (24V): 0.014375 CHAN A2 (12V): -0.005317 CHAN A3 (5V): -0.003214 CHAN B0 (3.3V): 0.004104 CHAN B1 (3.15aV): -0.001061 CHAN B2 (3.15bV): -0.001134 CHAN B3 (GND): -0.000292 OPEN: 0.011945 Full Scale: +/- 1 mA 2022-05-28T16:34:39.620Z,1653755679.620 [CommandExec](IMPORTANT): got command failComponent hardware DAT 2022-05-28T16:34:39.620Z,1653755679.620 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault 2022-05-28T16:34:46.426Z,1653755686.426 [CommandExec](IMPORTANT): got command gfscan 2022-05-28T16:34:46.798Z,1653755686.798 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T16:34:58.139Z,1653755698.139 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.003712 CHAN A1 (24V): 0.000946 CHAN A2 (12V): -0.005603 CHAN A3 (5V): -0.040192 CHAN B0 (3.3V): -0.001879 CHAN B1 (3.15aV): -0.001338 CHAN B2 (3.15bV): 0.043686 CHAN B3 (GND): 0.000182 OPEN: 0.012410 Full Scale: +/- 1 mA 2022-05-28T16:35:15.066Z,1653755715.066 [DAT](FAULT): failed to set time 2022-05-28T16:37:46.019Z,1653755866.019 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T163745 2022-05-28T16:37:53.598Z,1653755873.598 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 2 h;set sci2.NeedCommsTime 20 min;set sci2.Lat1 34.3707 degree;set sci2.Lon1 -119.8586 degree;set sci2.Lat2 34.368 degree;set sci2.Lon2 -119.851 degree;set sci2.YoYoMaxDepth 50 m" 23cur 1 2.000000 2022-05-28T16:37:53.598Z,1653755873.598 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=23cur 2022-05-28T16:37:53.599Z,1653755873.599 [CommandExec](IMPORTANT): Scheduled #2 (#1 of 2 with id='23cur'): "load Science/sci2.xml;set sci2.MissionTimeout 2 h;set sci2.NeedCommsTime 20 min;set sci2.Lat1 34.3707 degree;set sci2.Lon1 -119.8586 degree;set sci2.Lat2 34.368 degree;set sci2.Lon2 -119.851 degree;set sci2.YoYoMaxDepth 50 m", ASAP 2022-05-28T16:37:54.499Z,1653755874.499 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T163754 2022-05-28T16:38:02.025Z,1653755882.025 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run" 23cur 2 2.000000 2022-05-28T16:38:02.025Z,1653755882.025 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=23cur 2022-05-28T16:38:02.026Z,1653755882.026 [CommandExec](IMPORTANT): Scheduled #3 (#2 of 2 with id='23cur'): "set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run", ASAP 2022-05-28T16:38:02.140Z,1653755882.140 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-28T16:38:04.084Z,1653755884.084 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-28T16:38:09.600Z,1653755889.600 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T16:38:10.539Z,1653755890.539 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 2 hour 2022-05-28T16:38:10.540Z,1653755890.540 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 20 minute 2022-05-28T16:38:10.541Z,1653755890.541 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.3707 degree 2022-05-28T16:38:10.578Z,1653755890.578 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.8586 degree 2022-05-28T16:38:10.579Z,1653755890.579 [CommandExec](IMPORTANT): got command set sci2.Lat2 34.368 degree 2022-05-28T16:38:10.579Z,1653755890.579 [CommandExec](IMPORTANT): got command set sci2.Lon2 -119.851 degree 2022-05-28T16:38:10.580Z,1653755890.580 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-05-28T16:38:12.734Z,1653755892.734 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-05-28T16:38:12.735Z,1653755892.735 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-05-28T16:38:12.736Z,1653755892.736 [CommandExec](IMPORTANT): got command run 2022-05-28T16:38:12.737Z,1653755892.737 [CommandExec](IMPORTANT): Running 2022-05-28T16:38:12.834Z,1653755892.834 [MissionManager](IMPORTANT): Started mission sci2 2022-05-28T16:44:30.564Z,1653756270.564 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T16:50:20.856Z,1653756620.856 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 34.370701,-119.858597 2022-05-28T17:09:25.543Z,1653757765.543 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T17:15:09.457Z,1653758109.457 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T17:18:34.706Z,1653758314.706 [sci2:Lap:Wpt2:A.Waypoint](IMPORTANT): Reached Waypoint: 34.368000,-119.850998 2022-05-28T17:18:37.139Z,1653758317.139 [MissionManager](IMPORTANT): Started mission Default 2022-05-28T17:24:54.460Z,1653758694.460 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T17:30:19.286Z,1653759019.286 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.695697 min 2022-05-28T17:30:20.893Z,1653759020.893 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-28T17:30:21.707Z,1653759021.707 [NAL9602](FAULT): LCB fault: Invalid Command. Current Limiter Activated. 2022-05-28T17:35:08.199Z,1653759308.199 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T173508 2022-05-28T17:35:15.701Z,1653759315.701 [CommandExec](IMPORTANT): got command ibit 2022-05-28T17:35:16.013Z,1653759316.013 [IBIT](IMPORTANT): Beginning Initiated BIT 2022-05-28T17:35:16.013Z,1653759316.013 [IBIT](IMPORTANT): Beginning control surface checks. 2022-05-28T17:35:16.020Z,1653759316.020 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T17:35:26.886Z,1653759326.886 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.005650 CHAN A1 (24V): 0.001134 CHAN A2 (12V): -0.015149 CHAN A3 (5V): -0.002345 CHAN B0 (3.3V): -0.001969 CHAN B1 (3.15aV): -0.036727 CHAN B2 (3.15bV): -0.000570 CHAN B3 (GND): -0.000007 OPEN: 0.012091 Full Scale: +/- 1 mA 2022-05-28T17:36:05.642Z,1653759365.642 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T17:37:07.074Z,1653759427.074 [IBIT](FAULT): Error acquiring IBIT communications status. Timeout expired. 2022-05-28T17:37:07.475Z,1653759427.475 [IBIT](FAULT): Warning: Battery Data not active. Expected only when running primaries. Threshold checking not active. 2022-05-28T17:37:07.891Z,1653759427.891 [IBIT](IMPORTANT): bitHumidityThreshold: 55.000000 % 2022-05-28T17:37:07.892Z,1653759427.892 [IBIT](IMPORTANT): bitPressureThreshold: 0.750000 psi 2022-05-28T17:37:07.892Z,1653759427.892 [IBIT](IMPORTANT): Pressure:9.331175 PSI 2022-05-28T17:37:07.892Z,1653759427.892 [IBIT](IMPORTANT): Humidity:19.478697 % 2022-05-28T17:37:08.298Z,1653759428.298 [IBIT](IMPORTANT): Vehicle Pitch:-2.934861 degrees 2022-05-28T17:37:08.299Z,1653759428.299 [IBIT](IMPORTANT): Vehicle Roll:0.316182 degrees 2022-05-28T17:37:08.299Z,1653759428.299 [IBIT](IMPORTANT): Vehicle Heading:113.177788 degrees 2022-05-28T17:37:08.719Z,1653759428.719 [IBIT](IMPORTANT): surfaceThreshold: 1.000000 m 2022-05-28T17:37:08.719Z,1653759428.719 [IBIT](IMPORTANT): buoyancyNeutral: 418.268036 cc 2022-05-28T17:37:08.720Z,1653759428.720 [IBIT](IMPORTANT): massDefault: -2.780859 cm 2022-05-28T17:37:08.720Z,1653759428.720 [IBIT](IMPORTANT): stopDepth: 215.000000 m 2022-05-28T17:37:08.720Z,1653759428.720 [IBIT](IMPORTANT): abortDepth: 250.000000 m 2022-05-28T17:37:08.720Z,1653759428.720 [IBIT](IMPORTANT): IBIT FAILED 2022-05-28T17:40:06.891Z,1653759606.891 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T174006 2022-05-28T17:40:14.609Z,1653759614.609 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-28T17:40:16.506Z,1653759616.506 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-28T17:40:23.836Z,1653759623.836 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 2 hour 2022-05-28T17:40:23.837Z,1653759623.837 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2022-05-28T17:40:23.842Z,1653759623.842 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.373 degree 2022-05-28T17:40:23.843Z,1653759623.843 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.8583 degree 2022-05-28T17:40:23.843Z,1653759623.843 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-05-28T17:40:23.844Z,1653759623.844 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-05-28T17:40:23.844Z,1653759623.844 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-05-28T17:40:23.849Z,1653759623.849 [CommandExec](IMPORTANT): got command run 2022-05-28T17:40:23.851Z,1653759623.851 [CommandExec](IMPORTANT): Running 2022-05-28T17:40:23.936Z,1653759623.936 [MissionManager](IMPORTANT): Started mission sci2 2022-05-28T17:40:50.367Z,1653759650.367 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T174050 2022-05-28T17:40:58.375Z,1653759658.375 [CommandExec](IMPORTANT): got command schedule clear 2022-05-28T17:40:58.376Z,1653759658.376 [CommandExec](IMPORTANT): Cleared 1 scheduled commands. 2022-05-28T17:40:59.662Z,1653759659.662 [DataOverHttps](IMPORTANT): SBD MOMSN=16838707, MTMSN=20220528T174059 2022-05-28T17:41:07.327Z,1653759667.327 [CommandExec](IMPORTANT): got command schedule resume 2022-05-28T17:41:07.327Z,1653759667.327 [CommandExec](IMPORTANT): Scheduling is resumed 2022-05-28T17:41:31.911Z,1653759691.911 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47166, MT Status=1, MTMSN=3469 2022-05-28T17:41:32.496Z,1653759692.496 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run" 22dr3 2 2.000000 2022-05-28T17:41:32.497Z,1653759692.497 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=22dr3 2022-05-28T17:41:32.498Z,1653759692.498 [CommandExec](IMPORTANT): Scheduled #4 (#2 of 2 with id='22dr3'): "set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run", ASAP 2022-05-28T17:47:22.635Z,1653760042.635 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T17:51:13.799Z,1653760273.799 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 34.373001,-119.858299 2022-05-28T17:52:03.883Z,1653760323.883 [MissionManager](IMPORTANT): Started mission Default 2022-05-28T17:56:40.634Z,1653760600.634 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47167, MT Status=1, MTMSN=3470 2022-05-28T17:56:41.194Z,1653760601.194 [CommandExec](IMPORTANT): got command stop 2022-05-28T17:56:41.195Z,1653760601.195 [CommandExec](IMPORTANT): Scheduling is paused for 1 commands 2022-05-28T18:02:10.299Z,1653760930.299 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.100473 min 2022-05-28T18:06:24.771Z,1653761184.771 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180624 2022-05-28T18:06:32.319Z,1653761192.319 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h" 23gy1 1 6.000000 2022-05-28T18:06:32.321Z,1653761192.321 [CommandExec](IMPORTANT): Scheduling command #1 of 6 with id=23gy1 2022-05-28T18:06:32.322Z,1653761192.322 [CommandExec](IMPORTANT): Scheduled #5 (#1 of 6 with id='23gy1'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h", ASAP 2022-05-28T18:06:33.236Z,1653761193.236 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180633 2022-05-28T18:06:44.078Z,1653761204.078 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h;set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min" 23gy1 2 6.000000 2022-05-28T18:06:44.079Z,1653761204.079 [CommandExec](IMPORTANT): Scheduling command #2 of 6 with id=23gy1 2022-05-28T18:06:44.080Z,1653761204.080 [CommandExec](IMPORTANT): Scheduled #6 (#2 of 6 with id='23gy1'): "set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h;set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min", ASAP 2022-05-28T18:06:45.015Z,1653761205.015 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180644 2022-05-28T18:06:52.766Z,1653761212.766 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SpeedSampling 1 m/s;set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree" 23gy1 3 6.000000 2022-05-28T18:06:52.766Z,1653761212.766 [CommandExec](IMPORTANT): Scheduling command #3 of 6 with id=23gy1 2022-05-28T18:06:52.767Z,1653761212.767 [CommandExec](IMPORTANT): Scheduled #7 (#3 of 6 with id='23gy1'): "set IsothermDepthSampling.SpeedSampling 1 m/s;set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree", ASAP 2022-05-28T18:06:53.715Z,1653761213.715 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180653 2022-05-28T18:07:01.314Z,1653761221.314 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.MaxDepth 40 m;set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DeepBoundChl 27 m;set IsothermDepthSampling.CartridgeTypeCommon NaN count" 23gy1 4 6.000000 2022-05-28T18:07:01.314Z,1653761221.314 [CommandExec](IMPORTANT): Scheduling command #4 of 6 with id=23gy1 2022-05-28T18:07:01.315Z,1653761221.315 [CommandExec](IMPORTANT): Scheduled #8 (#4 of 6 with id='23gy1'): "set IsothermDepthSampling.MaxDepth 40 m;set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DeepBoundChl 27 m;set IsothermDepthSampling.CartridgeTypeCommon NaN count", ASAP 2022-05-28T18:07:02.283Z,1653761222.283 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180702 2022-05-28T18:07:09.958Z,1653761229.958 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -6 count;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.TempDiffFromPeakChl1 0 K;set IsothermDepthSampling.TempDiffFromPeakChl2 0 K" 23gy1 5 6.000000 2022-05-28T18:07:09.958Z,1653761229.958 [CommandExec](IMPORTANT): Scheduling command #5 of 6 with id=23gy1 2022-05-28T18:07:09.959Z,1653761229.959 [CommandExec](IMPORTANT): Scheduled #9 (#5 of 6 with id='23gy1'): "set IsothermDepthSampling.CartridgeType1 -6 count;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.TempDiffFromPeakChl1 0 K;set IsothermDepthSampling.TempDiffFromPeakChl2 0 K", ASAP 2022-05-28T18:07:10.879Z,1653761230.879 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T180710 2022-05-28T18:07:17.444Z,1653761237.444 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T18:07:18.612Z,1653761238.612 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.WaitDepUndulation1 20 min;set IsothermDepthSampling.WaitDepUndulation2 5.0 min;run" 23gy1 6 6.000000 2022-05-28T18:07:18.613Z,1653761238.613 [CommandExec](IMPORTANT): Scheduling command #6 of 6 with id=23gy1 2022-05-28T18:07:18.614Z,1653761238.614 [CommandExec](IMPORTANT): Scheduled #10 (#6 of 6 with id='23gy1'): "set IsothermDepthSampling.WaitDepUndulation1 20 min;set IsothermDepthSampling.WaitDepUndulation2 5.0 min;run", ASAP 2022-05-28T18:07:18.772Z,1653761238.772 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2022-05-28T18:07:51.343Z,1653761271.343 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2022-05-28T18:07:52.424Z,1653761272.424 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 15.801917 min 2022-05-28T18:08:29.072Z,1653761309.072 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count 2022-05-28T18:08:29.073Z,1653761309.073 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour 2022-05-28T18:08:29.074Z,1653761309.074 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 3 hour 2022-05-28T18:09:13.383Z,1653761353.383 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 hour 2022-05-28T18:09:13.383Z,1653761353.383 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 1 bool 2022-05-28T18:09:13.384Z,1653761353.384 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2022-05-28T18:09:13.385Z,1653761353.385 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.InitialDiveDuration 3 minute 2022-05-28T18:09:52.289Z,1653761392.289 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SpeedSampling 1 meter_per_second 2022-05-28T18:09:52.290Z,1653761392.290 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide0 210 degree 2022-05-28T18:09:52.290Z,1653761392.290 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide2 30 degree 2022-05-28T18:09:52.291Z,1653761392.291 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide3 120 degree 2022-05-28T18:10:20.102Z,1653761420.102 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxDepth 40 meter 2022-05-28T18:10:20.103Z,1653761420.103 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MinOffshore 4 kilometer 2022-05-28T18:10:20.103Z,1653761420.103 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 2 meter 2022-05-28T18:10:20.104Z,1653761420.104 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 27 meter 2022-05-28T18:10:20.106Z,1653761420.106 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeTypeCommon NaN count 2022-05-28T18:10:53.530Z,1653761453.530 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -6 count 2022-05-28T18:10:53.531Z,1653761453.531 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count 2022-05-28T18:10:53.531Z,1653761453.531 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TempDiffFromPeakChl1 0 kelvin 2022-05-28T18:10:53.532Z,1653761453.532 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TempDiffFromPeakChl2 0 kelvin 2022-05-28T18:11:08.314Z,1653761468.314 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 20 minute 2022-05-28T18:11:08.315Z,1653761468.315 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation2 5.0 minute 2022-05-28T18:11:08.315Z,1653761468.315 [CommandExec](IMPORTANT): got command run 2022-05-28T18:11:08.345Z,1653761468.345 [CommandExec](IMPORTANT): Running 2022-05-28T18:11:08.600Z,1653761468.600 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2022-05-28T18:11:08.701Z,1653761468.701 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 21.00 hours. 2022-05-28T18:12:56.679Z,1653761576.679 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T18:13:55.932Z,1653761635.932 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.443079 m 2022-05-28T18:16:08.290Z,1653761768.290 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47168, MT Status=1, MTMSN=3471 2022-05-28T18:16:09.124Z,1653761769.124 [CommandExec](IMPORTANT): got command schedule "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h;set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h" 23gb9 1 6.000000 2022-05-28T18:16:09.124Z,1653761769.124 [CommandExec](IMPORTANT): Scheduling command #1 of 6 with id=23gb9 2022-05-28T18:16:09.126Z,1653761769.126 [CommandExec](IMPORTANT): Scheduled #11 (#1 of 6 with id='23gb9'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h;set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h", AFTER MISSION 2022-05-28T18:16:09.126Z,1653761769.126 [CommandExec](FAULT): Scheduling was paused by a command at 2022-05-28T17:56:41Z 2022-05-28T18:17:05.432Z,1653761825.432 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , 0.000000 K , nan m , nan degC , nan m , 20.000000 min . 2022-05-28T18:18:52.516Z,1653761932.516 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 30.079546 m 2022-05-28T18:20:35.102Z,1653762035.102 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 24.803007 m , 1.705600 ug/l , 21.602364 m , 12.682747 degC , 5.994557 ug/l 2022-05-28T18:20:37.970Z,1653762037.970 [IsothermDepthSampling:TakeSamples:FiringOnTemp:C](IMPORTANT): Taking sample No. 1.000000 count . Target temp = 12.682747 degC . 2022-05-28T18:20:38.334Z,1653762038.334 [IsothermDepthSampling:TakeSamples:FiringOnTemp:D](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-05-28T18:20:39.164Z,1653762039.164 [IsothermDepthSampling:DampDown:A](IMPORTANT): Wait 20.000000 min for the depth undulation to damp down. 2022-05-28T18:40:41.930Z,1653763241.930 [IsothermDepthSampling:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger sampling. DepthLogged, TempLogged, ChlLogged = 17.328465 m , 13.245386 degC , 3.380000 ug/l . 2022-05-28T18:40:43.478Z,1653763243.478 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2022-05-28T18:40:53.632Z,1653763253.632 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-28T18:40:53.779Z,1653763253.779 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-28T18:41:48.938Z,1653763308.938 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-28T18:41:49.751Z,1653763309.751 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2022-05-28T18:41:49.752Z,1653763309.752 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-28T18:41:50.222Z,1653763310.222 [IsothermDepthSampling:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-05-28T18:41:50.958Z,1653763310.958 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-28T18:41:51.364Z,1653763311.364 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-28T18:41:52.983Z,1653763312.983 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-28T18:42:32.577Z,1653763352.577 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@40,archive_bac) 2022-05-28T18:42:35.402Z,1653763355.402 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2022-05-28T18:42:35.403Z,1653763355.403 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-28T18:42:35.806Z,1653763355.806 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-28T18:44:46.728Z,1653763486.728 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2022-05-28T18:44:47.112Z,1653763487.112 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-28T19:37:23.287Z,1653766643.287 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@799.9) 2022-05-28T19:37:23.690Z,1653766643.690 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@799.9) 2022-05-28T19:37:25.712Z,1653766645.712 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-05-28T19:37:26.118Z,1653766646.118 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,archive_bac) 2022-05-28T19:49:30.511Z,1653767370.511 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@40,archive_bac) 2022-05-28T19:49:39.807Z,1653767379.807 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-28T19:49:40.214Z,1653767380.214 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2022-05-28T19:54:40.014Z,1653767680.014 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2022-05-28T19:54:41.309Z,1653767681.309 [IsothermDepthSampling:TakeSamples:FiringOnTemp:TriggerOnTemp:C](IMPORTANT): Sample No. 1.000000 count completed. 2022-05-28T19:54:43.343Z,1653767683.343 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataCntSamples:A](IMPORTANT): CntSamples = 1.000000 count 2022-05-28T19:54:44.527Z,1653767684.527 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleNumber:A](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2022-05-28T19:54:45.720Z,1653767685.720 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPCartridgeType:A](IMPORTANT): Science:ESPComponent.cartridge = -6 count 2022-05-28T19:54:46.945Z,1653767686.945 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleVolume:A](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2022-05-28T19:54:48.587Z,1653767688.587 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleDep:A](IMPORTANT): Universal:depth = 19.814926 m 2022-05-28T19:54:50.197Z,1653767690.197 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleChl:A](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 3.016000 ug/l 2022-05-28T19:54:57.974Z,1653767697.974 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:55:06.784Z,1653767706.784 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , 0.000000 K , nan m , nan degC , nan m , 5.000000 min . 2022-05-28T19:55:09.578Z,1653767709.578 [IsothermDepthSampling:TakeSamples:FiringOnTemp:C](IMPORTANT): Taking sample No. 2.000000 count . Target temp = 12.682747 degC . 2022-05-28T19:55:10.019Z,1653767710.019 [IsothermDepthSampling:TakeSamples:FiringOnTemp:D](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2022-05-28T19:55:10.805Z,1653767710.805 [IsothermDepthSampling:DampDown:A](IMPORTANT): Wait 5.000000 min for the depth undulation to damp down. 2022-05-28T19:55:11.838Z,1653767711.838 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:55:25.706Z,1653767725.706 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:55:39.568Z,1653767739.568 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:56:07.798Z,1653767767.798 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:56:21.674Z,1653767781.674 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:56:35.536Z,1653767795.536 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:57:11.097Z,1653767831.097 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:57:24.962Z,1653767844.962 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T19:57:38.829Z,1653767858.829 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T20:00:13.393Z,1653768013.393 [IsothermDepthSampling:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger sampling. DepthLogged, TempLogged, ChlLogged = 18.763346 m , 13.118585 degC , 3.577600 ug/l . 2022-05-28T20:00:25.194Z,1653768025.194 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-28T20:00:25.330Z,1653768025.330 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-28T20:01:21.222Z,1653768081.222 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-28T20:01:22.034Z,1653768082.034 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -10 2022-05-28T20:01:22.034Z,1653768082.034 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-28T20:01:22.519Z,1653768082.519 [IsothermDepthSampling:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-05-28T20:01:22.840Z,1653768082.840 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-28T20:01:23.243Z,1653768083.243 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-28T20:01:25.262Z,1653768085.262 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-28T20:02:05.663Z,1653768125.663 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@39,daSPR_phyto) 2022-05-28T20:02:08.090Z,1653768128.090 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2022-05-28T20:02:08.492Z,1653768128.492 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-28T20:02:08.895Z,1653768128.895 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-28T20:04:19.799Z,1653768259.799 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2022-05-28T20:04:20.203Z,1653768260.203 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-28T20:53:37.564Z,1653771217.564 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@748.6) 2022-05-28T20:53:37.967Z,1653771217.967 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@748.6) 2022-05-28T20:53:39.987Z,1653771219.987 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-28T20:53:40.390Z,1653771220.390 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,daSPR_phyto) 2022-05-28T21:54:09.283Z,1653774849.283 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@39,daSPR_phyto) 2022-05-28T21:54:20.191Z,1653774860.191 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-28T21:54:20.597Z,1653774860.597 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2022-05-28T21:59:20.366Z,1653775160.366 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2022-05-28T21:59:21.660Z,1653775161.660 [IsothermDepthSampling:TakeSamples:FiringOnTemp:TriggerOnTemp:C](IMPORTANT): Sample No. 2.000000 count completed. 2022-05-28T21:59:23.673Z,1653775163.673 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataCntSamples:A](IMPORTANT): CntSamples = 2.000000 count 2022-05-28T21:59:24.864Z,1653775164.864 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleNumber:A](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2022-05-28T21:59:26.089Z,1653775166.089 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPCartridgeType:A](IMPORTANT): Science:ESPComponent.cartridge = -10 count 2022-05-28T21:59:27.313Z,1653775167.313 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleVolume:A](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2022-05-28T21:59:28.916Z,1653775168.916 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleDep:A](IMPORTANT): Universal:depth = 18.775467 m 2022-05-28T21:59:30.555Z,1653775170.555 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleChl:A](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 4.253600 ug/l 2022-05-28T21:59:33.630Z,1653775173.630 [MissionManager](IMPORTANT): Started mission Default 2022-05-28T21:59:37.766Z,1653775177.766 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T21:59:51.638Z,1653775191.638 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:00:05.506Z,1653775205.506 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:05:30.137Z,1653775530.137 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T22:10:55.056Z,1653775855.056 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.350053 min 2022-05-28T22:10:56.663Z,1653775856.663 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-28T22:12:39.551Z,1653775959.551 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T221239 2022-05-28T22:12:47.188Z,1653775967.188 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.xml;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 34.3697 degree;set profile_station.Lon -119.9021 degree;set profile_station.YoYoMaxDepth 30 m;set profile_station.Speed 0.75 m/s" 23scj 1 2.000000 2022-05-28T22:12:47.189Z,1653775967.189 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=23scj 2022-05-28T22:12:47.190Z,1653775967.190 [CommandExec](IMPORTANT): Scheduled #12 (#1 of 2 with id='23scj'): "load Science/profile_station.xml;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 34.3697 degree;set profile_station.Lon -119.9021 degree;set profile_station.YoYoMaxDepth 30 m;set profile_station.Speed 0.75 m/s", AFTER MISSION 2022-05-28T22:12:47.190Z,1653775967.190 [CommandExec](FAULT): Scheduling was paused by a command at 2022-05-28T17:56:41Z 2022-05-28T22:12:48.082Z,1653775968.082 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T221247 2022-05-28T22:12:55.645Z,1653775975.645 [CommandExec](IMPORTANT): got command schedule "set profile_station.MaxDepth 60 m;run" 23scj 2 2.000000 2022-05-28T22:12:55.646Z,1653775975.646 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=23scj 2022-05-28T22:12:55.647Z,1653775975.647 [CommandExec](IMPORTANT): Scheduled #13 (#2 of 2 with id='23scj'): "set profile_station.MaxDepth 60 m;run", AFTER MISSION 2022-05-28T22:12:55.647Z,1653775975.647 [CommandExec](FAULT): Scheduling was paused by a command at 2022-05-28T17:56:41Z 2022-05-28T22:15:20.932Z,1653776120.932 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47169, MT Status=1, MTMSN=3472 2022-05-28T22:15:21.740Z,1653776121.740 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min;set IsothermDepthSampling.SpeedSampling 1 m/s" 23gb9 2 6.000000 2022-05-28T22:15:21.741Z,1653776121.741 [CommandExec](IMPORTANT): Scheduling command #2 of 6 with id=23gb9 2022-05-28T22:15:21.742Z,1653776121.742 [CommandExec](IMPORTANT): Scheduled #14 (#2 of 6 with id='23gb9'): "set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min;set IsothermDepthSampling.SpeedSampling 1 m/s", AFTER MISSION 2022-05-28T22:15:21.742Z,1653776121.742 [CommandExec](FAULT): Scheduling was paused by a command at 2022-05-28T17:56:41Z 2022-05-28T22:21:11.268Z,1653776471.268 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 21.620361 min 2022-05-28T22:26:15.566Z,1653776775.566 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T22:26:45.073Z,1653776805.073 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 27.183752 min 2022-05-28T22:32:15.139Z,1653777135.139 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.684979 min 2022-05-28T22:37:19.790Z,1653777439.790 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T22:37:50.086Z,1653777470.086 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 38.267578 min 2022-05-28T22:42:54.684Z,1653777774.684 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T22:43:20.173Z,1653777800.173 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 43.768815 min 2022-05-28T22:43:22.563Z,1653777802.563 [NAL9602](FAULT): LCB fault: Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated. 2022-05-28T22:48:53.966Z,1653778133.966 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 49.332247 min 2022-05-28T22:49:22.179Z,1653778162.179 [DataOverHttps](IMPORTANT): SBD MOMSN=16839167, MTMSN=20220528T224922 2022-05-28T22:49:29.780Z,1653778169.780 [CommandExec](IMPORTANT): got command schedule clear 2022-05-28T22:49:29.781Z,1653778169.781 [CommandExec](IMPORTANT): Cleared 5 scheduled commands. 2022-05-28T22:49:29.781Z,1653778169.781 [CommandExec](IMPORTANT): got command schedule resume 2022-05-28T22:49:29.781Z,1653778169.781 [CommandExec](IMPORTANT): Scheduling is resumed 2022-05-28T22:50:01.496Z,1653778201.496 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T225001 2022-05-28T22:50:09.250Z,1653778209.250 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.xml;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 34.3697 degree;set profile_station.Lon -119.9021 degree;set profile_station.YoYoMaxDepth 30 m;set profile_station.Speed 0.75 m/s" 23u2k 1 2.000000 2022-05-28T22:50:09.251Z,1653778209.251 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=23u2k 2022-05-28T22:50:09.252Z,1653778209.252 [CommandExec](IMPORTANT): Scheduled #15 (#1 of 2 with id='23u2k'): "load Science/profile_station.xml;set profile_station.NeedCommsTime 30 min;set profile_station.Lat 34.3697 degree;set profile_station.Lon -119.9021 degree;set profile_station.YoYoMaxDepth 30 m;set profile_station.Speed 0.75 m/s", ASAP 2022-05-28T22:50:10.222Z,1653778210.222 [DataOverHttps](IMPORTANT): SBD MTMSN=20220528T225010 2022-05-28T22:50:17.785Z,1653778217.785 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.MaxDepth 60 m;run" 23u2k 2 2.000000 2022-05-28T22:50:17.785Z,1653778217.785 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=23u2k 2022-05-28T22:50:17.786Z,1653778217.786 [CommandExec](IMPORTANT): Scheduled #16 (#2 of 2 with id='23u2k'): "set profile_station.MaxDepth 60 m;run", ASAP 2022-05-28T22:50:17.909Z,1653778217.909 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml 2022-05-28T22:50:20.302Z,1653778220.302 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml 2022-05-28T22:50:44.293Z,1653778244.293 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 30 minute 2022-05-28T22:50:44.294Z,1653778244.294 [CommandExec](IMPORTANT): got command set profile_station.Lat 34.3697 degree 2022-05-28T22:50:44.295Z,1653778244.295 [CommandExec](IMPORTANT): got command set profile_station.Lon -119.9021 degree 2022-05-28T22:50:44.295Z,1653778244.295 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 30 meter 2022-05-28T22:50:44.296Z,1653778244.296 [CommandExec](IMPORTANT): got command set profile_station.Speed 0.75 meter_per_second 2022-05-28T22:50:53.804Z,1653778253.804 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 60 meter 2022-05-28T22:50:53.804Z,1653778253.804 [CommandExec](IMPORTANT): got command run 2022-05-28T22:50:53.813Z,1653778253.813 [CommandExec](IMPORTANT): Running 2022-05-28T22:50:54.102Z,1653778254.102 [MissionManager](IMPORTANT): Started mission profile_station 2022-05-28T22:53:58.711Z,1653778438.711 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T22:56:28.418Z,1653778588.418 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:56:42.298Z,1653778602.298 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:56:56.158Z,1653778616.158 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:57:10.773Z,1653778630.773 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:57:24.894Z,1653778644.894 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:57:38.754Z,1653778658.754 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:58:03.458Z,1653778683.458 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T22:58:17.342Z,1653778697.342 [DAT](FAULT): Ack receipt timeout failure. 2022-05-28T23:00:03.609Z,1653778803.609 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T23:31:02.634Z,1653780662.634 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-28T23:35:27.081Z,1653780927.081 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-28T23:35:38.065Z,1653780938.065 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005966 CHAN A1 (24V): 0.001714 CHAN A2 (12V): -0.004192 CHAN A3 (5V): -0.003384 CHAN B0 (3.3V): -0.001946 CHAN B1 (3.15aV): 0.003045 CHAN B2 (3.15bV): -0.001481 CHAN B3 (GND): -0.000191 OPEN: 0.012739 Full Scale: +/- 1 mA 2022-05-28T23:36:49.772Z,1653781009.772 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T00:06:04.331Z,1653782764.331 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T000604 2022-05-29T00:06:16.741Z,1653782776.741 [CommandExec](IMPORTANT): got command stop 2022-05-29T00:06:16.742Z,1653782776.742 [CommandExec](IMPORTANT): Scheduling is paused 2022-05-29T00:06:17.434Z,1653782777.434 [MissionManager](IMPORTANT): Started mission Default 2022-05-29T00:07:26.478Z,1653782846.478 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T00:13:20.281Z,1653783200.281 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 7.040594 min 2022-05-29T00:18:24.543Z,1653783504.543 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T00:18:53.261Z,1653783533.261 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.590335 min 2022-05-29T00:20:30.952Z,1653783630.952 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002030 2022-05-29T00:20:41.567Z,1653783641.567 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h" 23y9q 1 6.000000 2022-05-29T00:20:41.568Z,1653783641.568 [CommandExec](IMPORTANT): Scheduling command #1 of 6 with id=23y9q 2022-05-29T00:20:41.569Z,1653783641.569 [CommandExec](IMPORTANT): Scheduled #17 (#1 of 6 with id='23y9q'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 3 h", ASAP 2022-05-29T00:20:42.559Z,1653783642.559 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002042 2022-05-29T00:20:50.298Z,1653783650.298 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h;set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min" 23y9q 2 6.000000 2022-05-29T00:20:50.298Z,1653783650.298 [CommandExec](IMPORTANT): Scheduling command #2 of 6 with id=23y9q 2022-05-29T00:20:50.299Z,1653783650.299 [CommandExec](IMPORTANT): Scheduled #18 (#2 of 6 with id='23y9q'): "set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 h;set IsothermDepthSampling.SendSampleStatusAndData 1 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.InitialDiveDuration 3 min", ASAP 2022-05-29T00:20:51.243Z,1653783651.243 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002051 2022-05-29T00:20:59.174Z,1653783659.174 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SpeedSampling 0.75 m/s;set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree" 23y9q 3 6.000000 2022-05-29T00:20:59.174Z,1653783659.174 [CommandExec](IMPORTANT): Scheduling command #3 of 6 with id=23y9q 2022-05-29T00:20:59.176Z,1653783659.176 [CommandExec](IMPORTANT): Scheduled #19 (#3 of 6 with id='23y9q'): "set IsothermDepthSampling.SpeedSampling 0.75 m/s;set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree", ASAP 2022-05-29T00:21:00.087Z,1653783660.087 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002100 2022-05-29T00:21:07.651Z,1653783667.651 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.MaxDepth 40 m;set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DeepBoundChl 27 m;set IsothermDepthSampling.CartridgeTypeCommon NaN count" 23y9q 4 6.000000 2022-05-29T00:21:07.651Z,1653783667.651 [CommandExec](IMPORTANT): Scheduling command #4 of 6 with id=23y9q 2022-05-29T00:21:07.653Z,1653783667.653 [CommandExec](IMPORTANT): Scheduled #20 (#4 of 6 with id='23y9q'): "set IsothermDepthSampling.MaxDepth 40 m;set IsothermDepthSampling.MinOffshore 4 km;set IsothermDepthSampling.ShallowBoundChl 2 m;set IsothermDepthSampling.DeepBoundChl 27 m;set IsothermDepthSampling.CartridgeTypeCommon NaN count", ASAP 2022-05-29T00:21:08.563Z,1653783668.563 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002108 2022-05-29T00:21:16.317Z,1653783676.317 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -6 count;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.TempDiffFromPeakChl1 0 K;set IsothermDepthSampling.TempDiffFromPeakChl2 0 K" 23y9q 5 6.000000 2022-05-29T00:21:16.318Z,1653783676.318 [CommandExec](IMPORTANT): Scheduling command #5 of 6 with id=23y9q 2022-05-29T00:21:16.319Z,1653783676.319 [CommandExec](IMPORTANT): Scheduled #21 (#5 of 6 with id='23y9q'): "set IsothermDepthSampling.CartridgeType1 -6 count;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.TempDiffFromPeakChl1 0 K;set IsothermDepthSampling.TempDiffFromPeakChl2 0 K", ASAP 2022-05-29T00:21:17.227Z,1653783677.227 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T002117 2022-05-29T00:21:24.750Z,1653783684.750 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.WaitDepUndulation1 20 min;set IsothermDepthSampling.WaitDepUndulation2 5.0 min;run" 23y9q 6 6.000000 2022-05-29T00:21:24.750Z,1653783684.750 [CommandExec](IMPORTANT): Scheduling command #6 of 6 with id=23y9q 2022-05-29T00:21:24.751Z,1653783684.751 [CommandExec](IMPORTANT): Scheduled #22 (#6 of 6 with id='23y9q'): "set IsothermDepthSampling.WaitDepUndulation1 20 min;set IsothermDepthSampling.WaitDepUndulation2 5.0 min;run", ASAP 2022-05-29T00:21:24.921Z,1653783684.921 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2022-05-29T00:21:49.079Z,1653783709.079 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2022-05-29T00:22:09.033Z,1653783729.033 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count 2022-05-29T00:22:09.034Z,1653783729.034 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour 2022-05-29T00:22:09.035Z,1653783729.035 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 3 hour 2022-05-29T00:22:43.607Z,1653783763.607 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNotAchievingDepth 0.5 hour 2022-05-29T00:22:43.608Z,1653783763.608 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 1 bool 2022-05-29T00:22:43.609Z,1653783763.609 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2022-05-29T00:22:43.610Z,1653783763.610 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.InitialDiveDuration 3 minute 2022-05-29T00:23:13.087Z,1653783793.087 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SpeedSampling 0.75 meter_per_second 2022-05-29T00:23:13.088Z,1653783793.088 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide0 210 degree 2022-05-29T00:23:13.089Z,1653783793.089 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide2 30 degree 2022-05-29T00:23:13.090Z,1653783793.090 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.HeadingLeavingSide3 120 degree 2022-05-29T00:23:51.300Z,1653783831.300 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxDepth 40 meter 2022-05-29T00:23:51.308Z,1653783831.308 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MinOffshore 4 kilometer 2022-05-29T00:23:51.308Z,1653783831.308 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 2 meter 2022-05-29T00:23:51.309Z,1653783831.309 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 27 meter 2022-05-29T00:23:51.310Z,1653783831.310 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeTypeCommon NaN count 2022-05-29T00:24:17.499Z,1653783857.499 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -6 count 2022-05-29T00:24:17.499Z,1653783857.499 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count 2022-05-29T00:24:17.500Z,1653783857.500 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TempDiffFromPeakChl1 0 kelvin 2022-05-29T00:24:17.501Z,1653783857.501 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TempDiffFromPeakChl2 0 kelvin 2022-05-29T00:24:23.906Z,1653783863.906 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 18.100920 min 2022-05-29T00:24:37.730Z,1653783877.730 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 20 minute 2022-05-29T00:24:37.731Z,1653783877.731 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation2 5.0 minute 2022-05-29T00:24:37.732Z,1653783877.732 [CommandExec](IMPORTANT): got command run 2022-05-29T00:24:37.746Z,1653783877.746 [CommandExec](IMPORTANT): Running 2022-05-29T00:24:37.914Z,1653783877.914 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2022-05-29T00:24:38.125Z,1653783878.125 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 21.00 hours. 2022-05-29T00:29:28.736Z,1653784168.736 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T00:30:49.484Z,1653784249.484 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.449363 m 2022-05-29T00:33:59.102Z,1653784439.102 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , 0.000000 K , nan m , nan degC , nan m , 20.000000 min . 2022-05-29T00:35:48.549Z,1653784548.549 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T00:36:22.526Z,1653784582.526 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 30.044096 m 2022-05-29T00:39:42.143Z,1653784782.143 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 24.859110 m , 1.664000 ug/l , 16.226444 m , 13.400915 degC , 9.930440 ug/l 2022-05-29T00:39:44.946Z,1653784784.946 [IsothermDepthSampling:TakeSamples:FiringOnTemp:C](IMPORTANT): Taking sample No. 1.000000 count . Target temp = 13.400915 degC . 2022-05-29T00:39:45.367Z,1653784785.367 [IsothermDepthSampling:TakeSamples:FiringOnTemp:D](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-05-29T00:39:46.164Z,1653784786.164 [IsothermDepthSampling:DampDown:A](IMPORTANT): Wait 20.000000 min for the depth undulation to damp down. 2022-05-29T00:59:48.937Z,1653785988.937 [IsothermDepthSampling:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger sampling. DepthLogged, TempLogged, ChlLogged = 14.251315 m , 13.683008 degC , 2.631200 ug/l . 2022-05-29T01:00:00.689Z,1653786000.689 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-29T01:00:00.829Z,1653786000.829 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-29T01:00:57.146Z,1653786057.146 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-29T01:00:57.956Z,1653786057.956 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2022-05-29T01:00:57.957Z,1653786057.957 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-29T01:00:58.432Z,1653786058.432 [IsothermDepthSampling:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-05-29T01:00:58.763Z,1653786058.763 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-29T01:00:59.166Z,1653786059.166 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-29T01:01:01.190Z,1653786061.190 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-29T01:01:40.778Z,1653786100.778 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@38,archive_bac) 2022-05-29T01:01:43.611Z,1653786103.611 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2022-05-29T01:01:43.613Z,1653786103.613 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-29T01:01:44.010Z,1653786104.010 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-29T01:03:54.503Z,1653786234.503 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2022-05-29T01:03:54.907Z,1653786234.907 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-29T01:20:07.367Z,1653787207.367 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@268.8) 2022-05-29T01:20:07.770Z,1653787207.770 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@268.8) 2022-05-29T01:20:09.813Z,1653787209.813 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-05-29T01:20:10.197Z,1653787210.197 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,archive_bac) 2022-05-29T01:32:14.607Z,1653787934.607 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@38,archive_bac) 2022-05-29T01:32:24.303Z,1653787944.303 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-29T01:32:24.707Z,1653787944.707 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2022-05-29T01:37:24.486Z,1653788244.486 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2022-05-29T01:37:25.749Z,1653788245.749 [IsothermDepthSampling:TakeSamples:FiringOnTemp:TriggerOnTemp:C](IMPORTANT): Sample No. 1.000000 count completed. 2022-05-29T01:37:27.776Z,1653788247.776 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataCntSamples:A](IMPORTANT): CntSamples = 1.000000 count 2022-05-29T01:37:28.989Z,1653788248.989 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleNumber:A](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2022-05-29T01:37:30.199Z,1653788250.199 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPCartridgeType:A](IMPORTANT): Science:ESPComponent.cartridge = -6 count 2022-05-29T01:37:31.426Z,1653788251.426 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleVolume:A](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2022-05-29T01:37:33.051Z,1653788253.051 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleDep:A](IMPORTANT): Universal:depth = 15.271049 m 2022-05-29T01:37:34.658Z,1653788254.658 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleChl:A](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 2.433600 ug/l 2022-05-29T01:37:41.922Z,1653788261.922 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:37:51.214Z,1653788271.214 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , 0.000000 K , nan m , nan degC , nan m , 5.000000 min . 2022-05-29T01:37:54.036Z,1653788274.036 [IsothermDepthSampling:TakeSamples:FiringOnTemp:C](IMPORTANT): Taking sample No. 2.000000 count . Target temp = 13.400915 degC . 2022-05-29T01:37:54.434Z,1653788274.434 [IsothermDepthSampling:TakeSamples:FiringOnTemp:D](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2022-05-29T01:37:55.268Z,1653788275.268 [IsothermDepthSampling:DampDown:A](IMPORTANT): Wait 5.000000 min for the depth undulation to damp down. 2022-05-29T01:37:55.785Z,1653788275.785 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:38:20.993Z,1653788300.993 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:38:34.857Z,1653788314.857 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:38:48.734Z,1653788328.734 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:39:02.626Z,1653788342.626 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:39:16.758Z,1653788356.758 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:39:30.634Z,1653788370.634 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:39:44.494Z,1653788384.494 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:40:09.210Z,1653788409.210 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:40:23.078Z,1653788423.078 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:40:36.940Z,1653788436.940 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:40:50.802Z,1653788450.802 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:41:15.770Z,1653788475.770 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:41:29.634Z,1653788489.634 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:41:43.506Z,1653788503.506 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:41:57.385Z,1653788517.385 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T01:42:57.897Z,1653788577.897 [IsothermDepthSampling:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger sampling. DepthLogged, TempLogged, ChlLogged = 13.505359 m , 14.299585 degC , 2.984800 ug/l . 2022-05-29T01:43:09.641Z,1653788589.641 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-29T01:43:09.791Z,1653788589.791 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-29T01:44:05.407Z,1653788645.407 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-29T01:44:06.218Z,1653788646.218 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -10 2022-05-29T01:44:06.218Z,1653788646.218 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-29T01:44:06.722Z,1653788646.722 [IsothermDepthSampling:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-05-29T01:44:07.023Z,1653788647.023 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-29T01:44:07.426Z,1653788647.426 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-29T01:44:09.446Z,1653788649.446 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-29T01:44:49.847Z,1653788689.847 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@37,daSPR_phyto) 2022-05-29T01:44:52.270Z,1653788692.270 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2022-05-29T01:44:52.271Z,1653788692.271 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-29T01:44:52.675Z,1653788692.675 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-29T01:47:03.166Z,1653788823.166 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2022-05-29T01:47:03.573Z,1653788823.573 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-29T02:05:45.915Z,1653789945.915 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@310.1) 2022-05-29T02:05:46.314Z,1653789946.314 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@310.1) 2022-05-29T02:05:48.336Z,1653789948.336 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-29T02:05:48.738Z,1653789948.738 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,daSPR_phyto) 2022-05-29T03:05:11.358Z,1653793511.358 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@37,daSPR_phyto) 2022-05-29T03:05:22.276Z,1653793522.276 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-29T03:05:22.682Z,1653793522.682 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2022-05-29T03:10:22.462Z,1653793822.462 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2022-05-29T03:10:23.732Z,1653793823.732 [IsothermDepthSampling:TakeSamples:FiringOnTemp:TriggerOnTemp:C](IMPORTANT): Sample No. 2.000000 count completed. 2022-05-29T03:10:25.767Z,1653793825.767 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataCntSamples:A](IMPORTANT): CntSamples = 2.000000 count 2022-05-29T03:10:26.980Z,1653793826.980 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleNumber:A](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2022-05-29T03:10:28.237Z,1653793828.237 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPCartridgeType:A](IMPORTANT): Science:ESPComponent.cartridge = -10 count 2022-05-29T03:10:29.385Z,1653793829.385 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataESPSampleVolume:A](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2022-05-29T03:10:31.006Z,1653793831.006 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleDep:A](IMPORTANT): Universal:depth = 17.235548 m 2022-05-29T03:10:32.627Z,1653793832.627 [IsothermDepthSampling:SendData:SendDataTo1Modem:SendDataSampleChl:A](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 5.002400 ug/l 2022-05-29T03:10:35.553Z,1653793835.553 [MissionManager](IMPORTANT): Started mission Default 2022-05-29T03:10:40.313Z,1653793840.313 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T03:10:54.173Z,1653793854.173 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T03:11:19.382Z,1653793879.382 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T03:16:31.414Z,1653794191.414 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T03:28:16.442Z,1653794896.442 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 17.674646 min 2022-05-29T03:28:18.035Z,1653794898.035 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-29T03:34:02.731Z,1653795242.731 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T03:38:49.575Z,1653795529.575 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 28.226955 min 2022-05-29T03:43:53.395Z,1653795833.395 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T03:44:19.643Z,1653795859.643 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 33.728648 min 2022-05-29T03:49:53.858Z,1653796193.858 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 39.298145 min 2022-05-29T03:54:58.509Z,1653796498.509 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T03:55:24.369Z,1653796524.369 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 44.807060 min 2022-05-29T03:55:36.753Z,1653796536.753 [DataOverHttps](IMPORTANT): SBD MOMSN=16840035, MTMSN=20220529T035536 2022-05-29T03:55:46.757Z,1653796546.757 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.379 degree;set sci2.Lon1 -119.843 degree;set sci2.Lat2 34.343 degree;set sci2.Lon2 -119.811 degree;set sci2.Lat3 34.355 degree" 2488e 1 2.000000 2022-05-29T03:55:46.758Z,1653796546.758 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=2488e 2022-05-29T03:55:46.759Z,1653796546.759 [CommandExec](IMPORTANT): Scheduled #23 (#1 of 2 with id='2488e'): "load Science/sci2.xml;set sci2.MissionTimeout 12 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.379 degree;set sci2.Lon1 -119.843 degree;set sci2.Lat2 34.343 degree;set sci2.Lon2 -119.811 degree;set sci2.Lat3 34.355 degree", ASAP 2022-05-29T03:55:47.954Z,1653796547.954 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T035547 2022-05-29T03:55:55.525Z,1653796555.525 [CommandExec](IMPORTANT): got command schedule asap "set sci2.Lon3 -119.858 degree;set sci2.Speed 0.75 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run" 2488e 2 2.000000 2022-05-29T03:55:55.525Z,1653796555.525 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=2488e 2022-05-29T03:55:55.526Z,1653796555.526 [CommandExec](IMPORTANT): Scheduled #24 (#2 of 2 with id='2488e'): "set sci2.Lon3 -119.858 degree;set sci2.Speed 0.75 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run", ASAP 2022-05-29T03:55:55.651Z,1653796555.651 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-29T03:55:57.894Z,1653796557.894 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-29T03:56:15.158Z,1653796575.158 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 12 hour 2022-05-29T03:56:15.159Z,1653796575.159 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2022-05-29T03:56:15.160Z,1653796575.160 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.379 degree 2022-05-29T03:56:15.161Z,1653796575.161 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.843 degree 2022-05-29T03:56:15.165Z,1653796575.165 [CommandExec](IMPORTANT): got command set sci2.Lat2 34.343 degree 2022-05-29T03:56:15.166Z,1653796575.166 [CommandExec](IMPORTANT): got command set sci2.Lon2 -119.811 degree 2022-05-29T03:56:15.167Z,1653796575.167 [CommandExec](IMPORTANT): got command set sci2.Lat3 34.355 degree 2022-05-29T03:56:28.964Z,1653796588.964 [CommandExec](IMPORTANT): got command set sci2.Lon3 -119.858 degree 2022-05-29T03:56:28.965Z,1653796588.965 [CommandExec](IMPORTANT): got command set sci2.Speed 0.75 meter_per_second 2022-05-29T03:56:28.970Z,1653796588.970 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-05-29T03:56:28.970Z,1653796588.970 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-05-29T03:56:28.971Z,1653796588.971 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-05-29T03:56:28.971Z,1653796588.971 [CommandExec](IMPORTANT): got command run 2022-05-29T03:56:28.977Z,1653796588.977 [CommandExec](IMPORTANT): Running 2022-05-29T03:56:29.102Z,1653796589.102 [MissionManager](IMPORTANT): Started mission sci2 2022-05-29T04:01:35.249Z,1653796895.249 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T04:03:03.473Z,1653796983.473 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:03:17.337Z,1653796997.337 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:03:31.206Z,1653797011.206 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:03:45.317Z,1653797025.317 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:03:59.182Z,1653797039.182 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:04:13.057Z,1653797053.057 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:04:26.922Z,1653797066.922 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:04:41.034Z,1653797081.034 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:04:54.909Z,1653797094.909 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:05:08.774Z,1653797108.774 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:05:22.638Z,1653797122.638 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:05:36.758Z,1653797136.758 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:05:50.630Z,1653797150.630 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:06:04.494Z,1653797164.494 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:06:18.366Z,1653797178.366 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:06:32.489Z,1653797192.489 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:06:46.361Z,1653797206.361 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:07:00.225Z,1653797220.225 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T04:07:14.093Z,1653797234.093 [DAT](FAULT): Ack receipt timeout failure. 2022-05-29T05:07:29.352Z,1653800849.352 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T05:35:38.532Z,1653802538.532 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-29T05:35:49.319Z,1653802549.319 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.005395 CHAN A1 (24V): 0.001696 CHAN A2 (12V): -0.005160 CHAN A3 (5V): -0.003301 CHAN B0 (3.3V): -0.052008 CHAN B1 (3.15aV): -0.001058 CHAN B2 (3.15bV): -0.001250 CHAN B3 (GND): -0.000210 OPEN: 0.012595 Full Scale: +/- 1 mA 2022-05-29T06:14:03.684Z,1653804843.684 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T06:16:44.058Z,1653805004.058 [Radio_Surface](FAULT): LCB fault: Invalid Command. Hardware Overcurrent Shutdown. 2022-05-29T06:19:49.637Z,1653805189.637 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T07:26:55.645Z,1653809215.645 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T07:32:52.626Z,1653809572.626 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T08:39:00.612Z,1653813540.612 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T08:44:48.634Z,1653813888.634 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T09:52:56.461Z,1653817976.461 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T09:58:45.663Z,1653818325.663 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T10:23:37.569Z,1653819817.569 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. 2022-05-29T10:59:38.302Z,1653821978.302 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47170, MT Status=1, MTMSN=3473 2022-05-29T10:59:39.100Z,1653821979.100 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree;set IsothermDepthSampling.MaxDepth 40 m" 23gb9 3 6.000000 2022-05-29T10:59:39.105Z,1653821979.105 [CommandExec](IMPORTANT): Scheduling command #3 of 6 with id=23gb9 2022-05-29T10:59:39.106Z,1653821979.106 [CommandExec](IMPORTANT): Scheduled #25 (#3 of 6 with id='23gb9'): "set IsothermDepthSampling.HeadingLeavingSide0 210 degree;set IsothermDepthSampling.HeadingLeavingSide2 30 degree;set IsothermDepthSampling.HeadingLeavingSide3 120 degree;set IsothermDepthSampling.MaxDepth 40 m", AFTER MISSION 2022-05-29T10:59:39.106Z,1653821979.106 [CommandExec](FAULT): Scheduling was paused by a command at 2022-05-29T00:06:16Z 2022-05-29T11:03:36.406Z,1653822216.406 [BuoyancyServo](FAULT): Buoyancy getPosition uart error.serial timeout 2022-05-29T11:08:08.763Z,1653822488.763 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T11:13:34.733Z,1653822814.733 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2022-05-29T11:35:49.514Z,1653824149.514 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-29T11:36:00.372Z,1653824160.372 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.004211 CHAN A1 (24V): 0.001545 CHAN A2 (12V): -0.005520 CHAN A3 (5V): -0.036229 CHAN B0 (3.3V): -0.002166 CHAN B1 (3.15aV): -0.001830 CHAN B2 (3.15bV): 0.021168 CHAN B3 (GND): -0.000310 OPEN: 0.012366 Full Scale: +/- 1 mA 2022-05-29T12:13:02.711Z,1653826382.711 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T12:18:50.697Z,1653826730.697 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T12:51:33.782Z,1653828693.782 [ElevatorOffsetCalculator](IMPORTANT): Completed estimation for commanded vars: speed 0.75 m/s, pitch -20.00 deg, mass-position -27.81 mm. Average elevator angle=5.783865 +/- 0.099816 deg (conf. level 99.90%, sigma: 0.375692 deg). 2022-05-29T13:23:29.678Z,1653830609.678 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T13:29:27.042Z,1653830967.042 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T13:32:03.817Z,1653831123.817 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 34.379002,-119.843002 2022-05-29T13:36:16.809Z,1653831376.809 [ThrusterHE](FAULT): Could not get status:serial timeout 2022-05-29T14:34:46.427Z,1653834886.427 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T14:40:29.479Z,1653835229.479 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T15:41:30.763Z,1653838890.763 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T15:47:22.473Z,1653839242.473 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T15:56:29.556Z,1653839789.556 [MissionManager](IMPORTANT): Started mission Default 2022-05-29T16:03:09.578Z,1653840189.578 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T16:07:26.275Z,1653840446.275 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T160726 2022-05-29T16:07:35.939Z,1653840455.939 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 3 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.355 degree;set sci2.Lon1 -119.858 degree;set sci2.Speed 0.75 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MaxDepth 60 m" 2563k 1 2.000000 2022-05-29T16:07:35.939Z,1653840455.939 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=2563k 2022-05-29T16:07:35.940Z,1653840455.940 [CommandExec](IMPORTANT): Scheduled #26 (#1 of 2 with id='2563k'): "load Science/sci2.xml;set sci2.MissionTimeout 3 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 34.355 degree;set sci2.Lon1 -119.858 degree;set sci2.Speed 0.75 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MaxDepth 60 m", ASAP 2022-05-29T16:07:36.906Z,1653840456.906 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T160736 2022-05-29T16:07:46.337Z,1653840466.337 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MinOffshore 2 km;run" 2563k 2 2.000000 2022-05-29T16:07:46.337Z,1653840466.337 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=2563k 2022-05-29T16:07:46.338Z,1653840466.338 [CommandExec](IMPORTANT): Scheduled #27 (#2 of 2 with id='2563k'): "set sci2.MinOffshore 2 km;run", ASAP 2022-05-29T16:07:46.457Z,1653840466.457 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.xml 2022-05-29T16:07:48.327Z,1653840468.327 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2022-05-29T16:08:01.721Z,1653840481.721 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 3 hour 2022-05-29T16:08:01.722Z,1653840481.722 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2022-05-29T16:08:01.723Z,1653840481.723 [CommandExec](IMPORTANT): got command set sci2.Lat1 34.355 degree 2022-05-29T16:08:01.723Z,1653840481.723 [CommandExec](IMPORTANT): got command set sci2.Lon1 -119.858 degree 2022-05-29T16:08:01.724Z,1653840481.724 [CommandExec](IMPORTANT): got command set sci2.Speed 0.75 meter_per_second 2022-05-29T16:08:01.725Z,1653840481.725 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-05-29T16:08:01.725Z,1653840481.725 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-05-29T16:08:04.070Z,1653840484.070 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-05-29T16:08:04.070Z,1653840484.070 [CommandExec](IMPORTANT): got command run 2022-05-29T16:08:04.072Z,1653840484.072 [CommandExec](IMPORTANT): Running 2022-05-29T16:08:04.207Z,1653840484.207 [MissionManager](IMPORTANT): Started mission sci2 2022-05-29T16:08:07.821Z,1653840487.821 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-29T16:13:56.475Z,1653840836.475 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T16:19:39.664Z,1653841179.664 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T17:10:26.288Z,1653844226.288 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 34.355000,-119.858002 2022-05-29T17:12:46.879Z,1653844366.879 [MissionManager](IMPORTANT): Started mission Default 2022-05-29T17:18:04.530Z,1653844684.530 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-29T17:22:24.630Z,1653844944.630 [DataOverHttps](IMPORTANT): SBD MTMSN=20220529T172224