2023-09-15T20:44:34.424Z,1694810674.424 [CommandExec](IMPORTANT): got command schedule clear 2023-09-15T20:44:34.424Z,1694810674.424 [CommandExec](IMPORTANT): Cleared 4 scheduled commands. 2023-09-15T20:44:34.425Z,1694810674.425 [CommandExec](IMPORTANT): got command restart logs 2023-09-15T20:52:12.298Z,1694811132.298 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10909, MT Status=1, MTMSN=3373 2023-09-15T20:52:13.150Z,1694811133.150 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 9 h;set altitudeServo_approach_sampling.NeedCommsTime 180 min;set altitudeServo_approach_sampling.Lat1 41.938104 degree" tdxy 1 7.000000 2023-09-15T20:52:13.151Z,1694811133.151 [CommandExec](IMPORTANT): Scheduling command #1 of 7 with id=tdxy 2023-09-15T20:52:13.152Z,1694811133.152 [CommandExec](IMPORTANT): Scheduled #35 (#1 of 7 with id='tdxy'): "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 9 h;set altitudeServo_approach_sampling.NeedCommsTime 180 min;set altitudeServo_approach_sampling.Lat1 41.938104 degree", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:53:00.177Z,1694811180.177 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=10912, MT Status=1, MTMSN=3374 2023-09-15T20:53:00.893Z,1694811180.893 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.Lon1 -81.091611 degree;set altitudeServo_approach_sampling.Lat2 41.93887 degree;set altitudeServo_approach_sampling.Lon2 -81.020543 degree" tdxy 2 7.000000 2023-09-15T20:53:00.893Z,1694811180.893 [CommandExec](IMPORTANT): Scheduling command #2 of 7 with id=tdxy 2023-09-15T20:53:00.894Z,1694811180.894 [CommandExec](IMPORTANT): Scheduled #36 (#2 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.Lon1 -81.091611 degree;set altitudeServo_approach_sampling.Lat2 41.93887 degree;set altitudeServo_approach_sampling.Lon2 -81.020543 degree", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:53:54.052Z,1694811234.052 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=10913, MT Status=1, MTMSN=3375 2023-09-15T20:53:54.833Z,1694811234.833 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.Lat3 41.938104 degree;set altitudeServo_approach_sampling.Lon3 -81.091611 degree;set altitudeServo_approach_sampling.Lat4 41.938615 degree" tdxy 3 7.000000 2023-09-15T20:53:54.834Z,1694811234.834 [CommandExec](IMPORTANT): Scheduling command #3 of 7 with id=tdxy 2023-09-15T20:53:54.835Z,1694811234.835 [CommandExec](IMPORTANT): Scheduled #37 (#3 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.Lat3 41.938104 degree;set altitudeServo_approach_sampling.Lon3 -81.091611 degree;set altitudeServo_approach_sampling.Lat4 41.938615 degree", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:54:13.098Z,1694811253.098 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10914, MT Status=1, MTMSN=3376 2023-09-15T20:54:13.913Z,1694811253.913 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.Lon4 -81.072385 degree;set altitudeServo_approach_sampling.Lat5 41.84222 degree;set altitudeServo_approach_sampling.Lon5 -81.06945 degree" tdxy 4 7.000000 2023-09-15T20:54:13.913Z,1694811253.913 [CommandExec](IMPORTANT): Scheduling command #4 of 7 with id=tdxy 2023-09-15T20:54:13.915Z,1694811253.915 [CommandExec](IMPORTANT): Scheduled #38 (#4 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.Lon4 -81.072385 degree;set altitudeServo_approach_sampling.Lat5 41.84222 degree;set altitudeServo_approach_sampling.Lon5 -81.06945 degree", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:57:06.546Z,1694811426.546 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10915, MT Status=1, MTMSN=3377 2023-09-15T20:57:07.278Z,1694811427.278 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.Lat6 41.85321 degree;set altitudeServo_approach_sampling.Lon6 -81.0427 degree;set altitudeServo_approach_sampling.Lat7 41.93887 degree" tdxy 5 7.000000 2023-09-15T20:57:07.278Z,1694811427.278 [CommandExec](IMPORTANT): Scheduling command #5 of 7 with id=tdxy 2023-09-15T20:57:07.279Z,1694811427.279 [CommandExec](IMPORTANT): Scheduled #39 (#5 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.Lat6 41.85321 degree;set altitudeServo_approach_sampling.Lon6 -81.0427 degree;set altitudeServo_approach_sampling.Lat7 41.93887 degree", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:57:17.217Z,1694811437.217 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10916, MT Status=1, MTMSN=3378 2023-09-15T20:57:18.009Z,1694811438.009 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.Lon7 -81.045949 degree;set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m" tdxy 6 7.000000 2023-09-15T20:57:18.010Z,1694811438.010 [CommandExec](IMPORTANT): Scheduling command #6 of 7 with id=tdxy 2023-09-15T20:57:18.011Z,1694811438.011 [CommandExec](IMPORTANT): Scheduled #40 (#6 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.Lon7 -81.045949 degree;set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:57:37.774Z,1694811457.774 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10917, MT Status=1, MTMSN=3379 2023-09-15T20:57:38.444Z,1694811458.444 [CommandExec](IMPORTANT): got command schedule 20230916T005000 "set altitudeServo_approach_sampling.MaxDepth 18 m;set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run" tdxy 7 7.000000 2023-09-15T20:57:38.445Z,1694811458.445 [CommandExec](IMPORTANT): Scheduling command #7 of 7 with id=tdxy 2023-09-15T20:57:38.446Z,1694811458.446 [CommandExec](IMPORTANT): Scheduled #41 (#7 of 7 with id='tdxy'): "set altitudeServo_approach_sampling.MaxDepth 18 m;set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run", AT TIME: 2023-09-16T00:50:00Z 2023-09-15T20:57:56.837Z,1694811476.837 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10918, MT Status=1, MTMSN=3380 2023-09-15T20:57:57.623Z,1694811477.623 [CommandExec](IMPORTANT): got command schedule 20230916T095000 "load Science/profile_station.xml;set profile_station.MissionTimeout 16 h;set profile_station.Lat 41.93887 degree;set profile_station.Lon -81.045949 degree;set profile_station.YoYoMaxDepth 8 m" te3r 1 2.000000 2023-09-15T20:57:57.624Z,1694811477.624 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=te3r 2023-09-15T20:57:57.626Z,1694811477.626 [CommandExec](IMPORTANT): Scheduled #42 (#1 of 2 with id='te3r'): "load Science/profile_station.xml;set profile_station.MissionTimeout 16 h;set profile_station.Lat 41.93887 degree;set profile_station.Lon -81.045949 degree;set profile_station.YoYoMaxDepth 8 m", AT TIME: 2023-09-16T09:50:00Z 2023-09-15T20:58:10.986Z,1694811490.986 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=10919, MT Status=1, MTMSN=3381 2023-09-15T20:58:11.601Z,1694811491.601 [CommandExec](IMPORTANT): got command schedule 20230916T095000 "set profile_station.MaxDepth 10 m;run" te3r 2 2.000000 2023-09-15T20:58:11.602Z,1694811491.602 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=te3r 2023-09-15T20:58:11.604Z,1694811491.604 [CommandExec](IMPORTANT): Scheduled #43 (#2 of 2 with id='te3r'): "set profile_station.MaxDepth 10 m;run", AT TIME: 2023-09-16T09:50:00Z 2023-09-16T00:21:11.660Z,1694823671.660 [Radio_Surface](FAULT): LCB fault: Hardware Overcurrent Shutdown. 2023-09-16T00:50:00.111Z,1694825400.111 [CommandExec](IMPORTANT): got command load ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-16T00:50:01.882Z,1694825401.882 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/StandardEnvelopes.tx 2023-09-16T00:50:02.118Z,1694825402.118 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/NeedComms.tx 2023-09-16T00:50:02.570Z,1694825402.570 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/Science.tx 2023-09-16T00:50:04.103Z,1694825404.103 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/BackseatDriver.tx 2023-09-16T00:50:04.738Z,1694825404.738 [CommandExec](IMPORTANT): Loaded ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-16T00:50:05.135Z,1694825405.135 [MissionManager](IMPORTANT): Started mission Default 2023-09-16T00:50:22.218Z,1694825422.218 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MissionTimeout 9 hour 2023-09-16T00:50:22.218Z,1694825422.218 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.NeedCommsTime 180 minute 2023-09-16T00:50:22.219Z,1694825422.219 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat1 41.938104 degree 2023-09-16T00:50:36.077Z,1694825436.077 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon1 -81.091611 degree 2023-09-16T00:50:36.078Z,1694825436.078 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat2 41.93887 degree 2023-09-16T00:50:36.079Z,1694825436.079 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon2 -81.020543 degree 2023-09-16T00:50:49.755Z,1694825449.755 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat3 41.938104 degree 2023-09-16T00:50:49.756Z,1694825449.756 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon3 -81.091611 degree 2023-09-16T00:50:49.758Z,1694825449.758 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat4 41.938615 degree 2023-09-16T00:51:03.388Z,1694825463.388 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon4 -81.072385 degree 2023-09-16T00:51:03.389Z,1694825463.389 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat5 41.84222 degree 2023-09-16T00:51:03.390Z,1694825463.390 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon5 -81.06945 degree 2023-09-16T00:51:20.039Z,1694825480.039 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat6 41.85321 degree 2023-09-16T00:51:20.043Z,1694825480.043 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon6 -81.0427 degree 2023-09-16T00:51:20.044Z,1694825480.044 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat7 41.93887 degree 2023-09-16T00:51:40.886Z,1694825500.886 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon7 -81.045949 degree 2023-09-16T00:51:40.887Z,1694825500.887 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.HoldAltitude 7 meter 2023-09-16T00:51:40.888Z,1694825500.888 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.InitDepth 5 meter 2023-09-16T00:51:40.889Z,1694825500.889 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MinAltitude 4 meter 2023-09-16T00:51:51.950Z,1694825511.950 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MaxDepth 18 meter 2023-09-16T00:51:51.951Z,1694825511.951 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.ESPCartridgeType -1 count 2023-09-16T00:51:51.952Z,1694825511.952 [CommandExec](IMPORTANT): got command run 2023-09-16T00:51:51.954Z,1694825511.954 [CommandExec](IMPORTANT): Running 2023-09-16T00:51:52.208Z,1694825512.208 [MissionManager](IMPORTANT): Started mission altitudeServo_approach_sampling 2023-09-16T00:56:04.347Z,1694825764.347 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938103,-81.091614 2023-09-16T01:01:34.516Z,1694826094.516 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T01:01:34.903Z,1694826094.903 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T01:02:08.995Z,1694826128.995 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. 2023-09-16T01:02:27.041Z,1694826147.041 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T01:02:27.870Z,1694826147.870 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -1 2023-09-16T01:02:27.870Z,1694826147.870 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T01:02:28.314Z,1694826148.314 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 7.385729 m in 21.808466 degC water with 1.475760 ug/l chlorophyll fluorescence. 2023-09-16T01:02:28.663Z,1694826148.663 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T01:02:29.063Z,1694826149.063 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T01:02:31.081Z,1694826151.081 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T01:03:02.997Z,1694826182.997 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@15,aRNAl_bac) 2023-09-16T01:03:05.835Z,1694826185.835 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T01:03:05.838Z,1694826185.838 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T01:03:06.231Z,1694826186.231 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T01:05:24.006Z,1694826324.006 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T01:05:24.410Z,1694826324.410 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T01:08:56.559Z,1694826536.559 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 41.938103,-81.091614 2023-09-16T01:08:56.571Z,1694826536.571 [altitudeServo_approach_sampling:Lap:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938869,-81.020546 2023-09-16T01:38:38.768Z,1694828318.768 [CBIT](IMPORTANT): Beginning ground fault scan 2023-09-16T01:38:49.687Z,1694828329.687 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003372 CHAN A1 (24V): 0.004853 CHAN A2 (12V): -0.004378 CHAN A3 (5V): -0.003089 CHAN B0 (3.3V): -0.002181 CHAN B1 (3.15aV): -0.001555 CHAN B2 (3.15bV): 0.001408 CHAN B3 (GND): -0.001039 OPEN: 0.012015 Full Scale: +/- 1 mA 2023-09-16T01:55:50.920Z,1694829350.920 [Power24vConverter](FAULT): LCB fault: Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated. 2023-09-16T02:05:33.490Z,1694829933.490 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1245.6) 2023-09-16T02:05:33.902Z,1694829933.902 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1245.6) 2023-09-16T02:05:35.919Z,1694829935.919 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T02:05:36.313Z,1694829936.313 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@15,aRNAl_bac) 2023-09-16T02:07:11.261Z,1694830031.261 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@15,aRNAl_bac) 2023-09-16T02:07:18.939Z,1694830038.939 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T02:07:19.351Z,1694830039.351 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2023-09-16T02:12:19.136Z,1694830339.136 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2023-09-16T02:12:31.012Z,1694830351.012 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T02:12:31.127Z,1694830351.127 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T02:13:24.242Z,1694830404.242 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T02:13:25.053Z,1694830405.053 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -1 2023-09-16T02:13:25.054Z,1694830405.054 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T02:13:25.513Z,1694830405.513 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 14.659577 m in 21.784266 degC water with 1.241760 ug/l chlorophyll fluorescence. 2023-09-16T02:13:25.858Z,1694830405.858 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T02:13:26.261Z,1694830406.261 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T02:13:28.282Z,1694830408.282 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T02:14:00.198Z,1694830440.198 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@14,aRNAl_bac) 2023-09-16T02:14:02.622Z,1694830442.622 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T02:14:02.623Z,1694830442.623 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T02:14:03.027Z,1694830443.027 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T02:16:28.474Z,1694830588.474 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T02:16:28.878Z,1694830588.878 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T02:48:21.515Z,1694832501.515 [altitudeServo_approach_sampling:Lap:Wpt2:A.Waypoint](IMPORTANT): Reached waypoint: 41.938869,-81.020546 2023-09-16T02:48:21.526Z,1694832501.526 [altitudeServo_approach_sampling:Lap:Wpt3:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938103,-81.091614 2023-09-16T03:05:50.892Z,1694833550.892 [ThrusterHE](IMPORTANT): Status lock Detected. 2023-09-16T03:05:51.212Z,1694833551.212 [ThrusterHE](IMPORTANT): Status lock Detected. 2023-09-16T03:05:51.588Z,1694833551.588 [ThrusterHE](IMPORTANT): Status lock Detected. 2023-09-16T03:05:52.055Z,1694833552.055 [ThrusterHE](IMPORTANT): Status lock Detected. 2023-09-16T03:05:52.439Z,1694833552.439 [ThrusterHE](IMPORTANT): Status lock Detected. 2023-09-16T03:13:07.845Z,1694833987.845 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1197.6) 2023-09-16T03:13:08.255Z,1694833988.255 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1197.6) 2023-09-16T03:13:10.274Z,1694833990.274 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T03:13:10.674Z,1694833990.674 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@14,aRNAl_bac) 2023-09-16T03:14:45.626Z,1694834085.626 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@14,aRNAl_bac) 2023-09-16T03:14:53.302Z,1694834093.302 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T03:14:53.713Z,1694834093.713 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2023-09-16T03:19:53.485Z,1694834393.485 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2023-09-16T03:20:05.308Z,1694834405.308 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T03:20:05.439Z,1694834405.439 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T03:20:57.791Z,1694834457.791 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T03:20:58.597Z,1694834458.597 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -1 2023-09-16T03:20:58.597Z,1694834458.597 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T03:20:59.073Z,1694834459.073 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 13.546336 m in 21.783197 degC water with 1.227200 ug/l chlorophyll fluorescence. 2023-09-16T03:20:59.397Z,1694834459.397 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T03:20:59.802Z,1694834459.802 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T03:21:01.821Z,1694834461.821 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T03:21:33.742Z,1694834493.742 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@13,aRNAl_bac) 2023-09-16T03:21:36.575Z,1694834496.575 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T03:21:36.586Z,1694834496.586 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T03:21:36.976Z,1694834496.976 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T03:24:02.422Z,1694834642.422 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T03:24:02.825Z,1694834642.825 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T03:26:19.781Z,1694834779.781 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1.9) 2023-09-16T03:26:20.185Z,1694834780.185 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1.9) 2023-09-16T03:26:22.206Z,1694834782.206 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T03:26:22.609Z,1694834782.609 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@13,aRNAl_bac) 2023-09-16T03:26:37.563Z,1694834797.563 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@13,aRNAl_bac) 2023-09-16T03:26:46.046Z,1694834806.046 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T03:26:46.456Z,1694834806.456 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2023-09-16T03:31:46.229Z,1694835106.229 [ESPComponent](CRITICAL): [sample #7] ESP sampling sequence terminated with error: In state S_WAITING_FOR_PROCESSED got exception while awaiting completion of cmd='Cmd.startProcessing' exception=<>. Elapsed time since command issued: 17.3764 2023-09-16T03:31:46.507Z,1694835106.507 [CommandExec](FAULT): Scheduling is paused for 2 commands 2023-09-16T03:31:47.348Z,1694835107.348 [MissionManager](IMPORTANT): Started mission Default 2023-09-16T03:42:54.973Z,1694835774.973 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.120121 min 2023-09-16T03:48:48.152Z,1694836128.152 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 17.008124 min 2023-09-16T03:55:46.202Z,1694836546.202 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 23.973954 min 2023-09-16T04:01:39.814Z,1694836899.814 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 29.867957 min 2023-09-16T04:07:38.025Z,1694837258.025 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 35.837724 min 2023-09-16T04:09:27.553Z,1694837367.553 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=11026, MT Status=1, MTMSN=3382 2023-09-16T04:09:28.281Z,1694837368.281 [CommandExec](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set smear_waypoint_sampling.CartridgeType -1 count;set smear_waypoint_sampling.NeedCommsTimeInTransit 120 min;set smear_waypoint_sampling.ReachEndWaypoint 0 bool" ty5z 1 3.000000 2023-09-16T04:09:28.281Z,1694837368.281 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=ty5z 2023-09-16T04:09:28.282Z,1694837368.282 [CommandExec](IMPORTANT): Scheduled #44 (#1 of 3 with id='ty5z'): "load Science/smear_waypoint_sampling.xml;set smear_waypoint_sampling.CartridgeType -1 count;set smear_waypoint_sampling.NeedCommsTimeInTransit 120 min;set smear_waypoint_sampling.ReachEndWaypoint 0 bool", ASAP 2023-09-16T04:09:47.196Z,1694837387.196 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=11027, MT Status=1, MTMSN=3383 2023-09-16T04:09:47.973Z,1694837387.973 [CommandExec](IMPORTANT): got command schedule asap "set smear_waypoint_sampling.Speed 0.8 m/s;set smear_waypoint_sampling.MaxDepth 18 m;set smear_waypoint_sampling.SamplingYoYoMinDepth 3 m;set smear_waypoint_sampling.SamplingYoYoMaxDepth 15 m" ty5z 2 3.000000 2023-09-16T04:09:47.973Z,1694837387.973 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=ty5z 2023-09-16T04:09:47.974Z,1694837387.974 [CommandExec](IMPORTANT): Scheduled #45 (#2 of 3 with id='ty5z'): "set smear_waypoint_sampling.Speed 0.8 m/s;set smear_waypoint_sampling.MaxDepth 18 m;set smear_waypoint_sampling.SamplingYoYoMinDepth 3 m;set smear_waypoint_sampling.SamplingYoYoMaxDepth 15 m", ASAP 2023-09-16T04:10:23.912Z,1694837423.912 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11029, MT Status=1, MTMSN=3384 2023-09-16T04:10:24.736Z,1694837424.736 [CommandExec](IMPORTANT): got command schedule asap "set smear_waypoint_sampling.SamplingEndLatitude 41.93887 degree;set smear_waypoint_sampling.SamplingEndLongitude -81.020543 degree;set smear_waypoint_sampling.YoYoMinAltitude 5 m;set smear_waypoint_sampling.MinAltitude 4 m;run" ty5z 3 3.000000 2023-09-16T04:10:24.737Z,1694837424.737 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=ty5z 2023-09-16T04:10:24.738Z,1694837424.738 [CommandExec](IMPORTANT): Scheduled #46 (#3 of 3 with id='ty5z'): "set smear_waypoint_sampling.SamplingEndLatitude 41.93887 degree;set smear_waypoint_sampling.SamplingEndLongitude -81.020543 degree;set smear_waypoint_sampling.YoYoMinAltitude 5 m;set smear_waypoint_sampling.MinAltitude 4 m;run", ASAP 2023-09-16T04:10:24.872Z,1694837424.872 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2023-09-16T04:10:26.666Z,1694837426.666 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/NeedComms.tx 2023-09-16T04:10:27.208Z,1694837427.208 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/StandardEnvelopes.tx 2023-09-16T04:10:27.555Z,1694837427.555 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/Science.tx 2023-09-16T04:10:30.406Z,1694837430.406 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2023-09-16T04:10:47.649Z,1694837447.649 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11030, MT Status=1, MTMSN=3385 2023-09-16T04:10:48.198Z,1694837448.198 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.CartridgeType -1 count 2023-09-16T04:10:48.198Z,1694837448.198 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.NeedCommsTimeInTransit 120 minute 2023-09-16T04:10:48.199Z,1694837448.199 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.ReachEndWaypoint 0 bool 2023-09-16T04:10:48.206Z,1694837448.206 [CommandExec](IMPORTANT): got command schedule resume 2023-09-16T04:10:48.206Z,1694837448.206 [CommandExec](IMPORTANT): Scheduling is resumed 2023-09-16T04:11:12.008Z,1694837472.008 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.Speed 0.8 meter_per_second 2023-09-16T04:11:12.008Z,1694837472.008 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.MaxDepth 18 meter 2023-09-16T04:11:12.009Z,1694837472.009 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.SamplingYoYoMinDepth 3 meter 2023-09-16T04:11:12.010Z,1694837472.010 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.SamplingYoYoMaxDepth 15 meter 2023-09-16T04:11:35.272Z,1694837495.272 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.SamplingEndLatitude 41.93887 degree 2023-09-16T04:11:35.273Z,1694837495.273 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.SamplingEndLongitude -81.020543 degree 2023-09-16T04:11:35.273Z,1694837495.273 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.YoYoMinAltitude 5 meter 2023-09-16T04:11:35.274Z,1694837495.274 [CommandExec](IMPORTANT): got command set smear_waypoint_sampling.MinAltitude 4 meter 2023-09-16T04:11:35.275Z,1694837495.275 [CommandExec](IMPORTANT): got command run 2023-09-16T04:11:35.279Z,1694837495.279 [CommandExec](IMPORTANT): Running 2023-09-16T04:11:35.436Z,1694837495.436 [MissionManager](IMPORTANT): Started mission smear_waypoint_sampling 2023-09-16T04:12:59.644Z,1694837579.644 [smear_waypoint_sampling:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 41.938869,-81.020546 2023-09-16T04:14:50.307Z,1694837690.307 [smear_waypoint_sampling:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2023-09-16T04:14:53.569Z,1694837693.569 [smear_waypoint_sampling:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 3.627821 m , 21.921991 degC , 1.414400 ug/l . 2023-09-16T04:15:05.333Z,1694837705.333 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T04:15:05.486Z,1694837705.486 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T04:15:57.442Z,1694837757.442 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T04:15:58.247Z,1694837758.247 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2023-09-16T04:15:58.247Z,1694837758.247 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T04:15:58.703Z,1694837758.703 [smear_waypoint_sampling:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2023-09-16T04:15:59.051Z,1694837759.051 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T04:15:59.454Z,1694837759.454 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T04:16:01.473Z,1694837761.473 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T04:16:33.396Z,1694837793.396 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@12,aRNAl_bac) 2023-09-16T04:16:35.813Z,1694837795.813 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T04:16:35.814Z,1694837795.814 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T04:16:36.231Z,1694837796.231 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T04:19:03.741Z,1694837943.741 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T04:19:04.146Z,1694837944.146 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T05:15:29.739Z,1694841329.739 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1195.8) 2023-09-16T05:15:30.141Z,1694841330.141 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1195.8) 2023-09-16T05:15:32.163Z,1694841332.163 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T05:15:32.567Z,1694841332.567 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@12,aRNAl_bac) 2023-09-16T05:17:07.513Z,1694841427.513 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@12,aRNAl_bac) 2023-09-16T05:17:15.244Z,1694841435.244 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T05:17:15.650Z,1694841435.650 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2023-09-16T05:22:15.429Z,1694841735.429 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2023-09-16T05:22:16.678Z,1694841736.678 [smear_waypoint_sampling:SamplingToWaypoint:TakeSamples:Firing:G](IMPORTANT): Sample No. 1.000000 count completed. 2023-09-16T05:22:57.911Z,1694841777.911 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=11040, MT Status=1, MTMSN=3386 2023-09-16T05:22:58.707Z,1694841778.707 [CommandExec](IMPORTANT): got command schedule "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 6 h;set altitudeServo_approach_sampling.Lat1 41.93887 degree;set altitudeServo_approach_sampling.Lon1 -81.020543 degree" tz4h 1 5.000000 2023-09-16T05:22:58.707Z,1694841778.707 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=tz4h 2023-09-16T05:22:58.736Z,1694841778.736 [CommandExec](IMPORTANT): Scheduled #47 (#1 of 5 with id='tz4h'): "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 6 h;set altitudeServo_approach_sampling.Lat1 41.93887 degree;set altitudeServo_approach_sampling.Lon1 -81.020543 degree", AFTER MISSION 2023-09-16T05:23:16.729Z,1694841796.729 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11041, MT Status=1, MTMSN=3387 2023-09-16T05:23:18.483Z,1694841798.483 [CommandExec](IMPORTANT): got command schedule "set altitudeServo_approach_sampling.Lat2 41.938615 degree;set altitudeServo_approach_sampling.Lon2 -81.072385 degree;set altitudeServo_approach_sampling.Lat3 41.84415 degree;set altitudeServo_approach_sampling.Lon3 -81.06951 degree" tz4h 2 5.000000 2023-09-16T05:23:18.497Z,1694841798.497 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=tz4h 2023-09-16T05:23:18.498Z,1694841798.498 [CommandExec](IMPORTANT): Scheduled #48 (#2 of 5 with id='tz4h'): "set altitudeServo_approach_sampling.Lat2 41.938615 degree;set altitudeServo_approach_sampling.Lon2 -81.072385 degree;set altitudeServo_approach_sampling.Lat3 41.84415 degree;set altitudeServo_approach_sampling.Lon3 -81.06951 degree", AFTER MISSION 2023-09-16T05:23:42.710Z,1694841822.710 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11042, MT Status=1, MTMSN=3388 2023-09-16T05:23:43.496Z,1694841823.496 [CommandExec](IMPORTANT): got command schedule "set altitudeServo_approach_sampling.Lat4 41.85445 degree;set altitudeServo_approach_sampling.Lon4 -81.04275 degree;set altitudeServo_approach_sampling.Lat5 41.93887 degree;set altitudeServo_approach_sampling.Lon5 -81.045949 degree" tz4h 3 5.000000 2023-09-16T05:23:43.500Z,1694841823.500 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=tz4h 2023-09-16T05:23:43.501Z,1694841823.501 [CommandExec](IMPORTANT): Scheduled #49 (#3 of 5 with id='tz4h'): "set altitudeServo_approach_sampling.Lat4 41.85445 degree;set altitudeServo_approach_sampling.Lon4 -81.04275 degree;set altitudeServo_approach_sampling.Lat5 41.93887 degree;set altitudeServo_approach_sampling.Lon5 -81.045949 degree", AFTER MISSION 2023-09-16T05:24:06.951Z,1694841846.951 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11043, MT Status=1, MTMSN=3389 2023-09-16T05:24:07.708Z,1694841847.708 [CommandExec](IMPORTANT): got command schedule "set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m;set altitudeServo_approach_sampling.MaxDepth 18 m" tz4h 4 5.000000 2023-09-16T05:24:07.721Z,1694841847.721 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=tz4h 2023-09-16T05:24:07.723Z,1694841847.723 [CommandExec](IMPORTANT): Scheduled #50 (#4 of 5 with id='tz4h'): "set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m;set altitudeServo_approach_sampling.MaxDepth 18 m", AFTER MISSION 2023-09-16T05:24:29.392Z,1694841869.392 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11044, MT Status=1, MTMSN=3390 2023-09-16T05:24:29.998Z,1694841869.998 [CommandExec](IMPORTANT): got command schedule "set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run" tz4h 5 5.000000 2023-09-16T05:24:29.998Z,1694841869.998 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=tz4h 2023-09-16T05:24:29.999Z,1694841869.999 [CommandExec](IMPORTANT): Scheduled #51 (#5 of 5 with id='tz4h'): "set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run", AFTER MISSION 2023-09-16T05:29:31.352Z,1694842171.352 [CommandExec](IMPORTANT): got command load ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-16T05:29:34.600Z,1694842174.600 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/StandardEnvelopes.tx 2023-09-16T05:29:34.750Z,1694842174.750 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/NeedComms.tx 2023-09-16T05:29:35.240Z,1694842175.240 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/Science.tx 2023-09-16T05:29:36.586Z,1694842176.586 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/BackseatDriver.tx 2023-09-16T05:29:37.347Z,1694842177.347 [CommandExec](IMPORTANT): Loaded ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-16T05:29:37.631Z,1694842177.631 [MissionManager](IMPORTANT): Started mission Default 2023-09-16T05:29:53.146Z,1694842193.146 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MissionTimeout 6 hour 2023-09-16T05:29:53.147Z,1694842193.147 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat1 41.93887 degree 2023-09-16T05:29:53.148Z,1694842193.148 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon1 -81.020543 degree 2023-09-16T05:30:13.239Z,1694842213.239 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat2 41.938615 degree 2023-09-16T05:30:13.240Z,1694842213.240 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon2 -81.072385 degree 2023-09-16T05:30:13.240Z,1694842213.240 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat3 41.84415 degree 2023-09-16T05:30:13.241Z,1694842213.241 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon3 -81.06951 degree 2023-09-16T05:30:34.625Z,1694842234.625 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat4 41.85445 degree 2023-09-16T05:30:34.626Z,1694842234.626 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon4 -81.04275 degree 2023-09-16T05:30:34.626Z,1694842234.626 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat5 41.93887 degree 2023-09-16T05:30:34.627Z,1694842234.627 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon5 -81.045949 degree 2023-09-16T05:30:58.493Z,1694842258.493 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.HoldAltitude 7 meter 2023-09-16T05:30:58.494Z,1694842258.494 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.InitDepth 5 meter 2023-09-16T05:30:58.495Z,1694842258.495 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MinAltitude 4 meter 2023-09-16T05:30:58.496Z,1694842258.496 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MaxDepth 18 meter 2023-09-16T05:31:04.821Z,1694842264.821 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.ESPCartridgeType -1 count 2023-09-16T05:31:04.821Z,1694842264.821 [CommandExec](IMPORTANT): got command run 2023-09-16T05:31:04.824Z,1694842264.824 [CommandExec](IMPORTANT): Running 2023-09-16T05:31:04.899Z,1694842264.899 [MissionManager](IMPORTANT): Started mission altitudeServo_approach_sampling 2023-09-16T05:31:55.121Z,1694842315.121 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938869,-81.020546 2023-09-16T05:36:17.162Z,1694842577.162 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T05:36:17.308Z,1694842577.308 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T05:37:10.058Z,1694842630.058 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T05:37:10.868Z,1694842630.868 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2023-09-16T05:37:10.869Z,1694842630.869 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T05:37:11.270Z,1694842631.270 [ESPComponent](IMPORTANT): [sample #9] Stop sampling requested. 2023-09-16T05:37:11.306Z,1694842631.306 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 5.471796 m in 21.846888 degC water with 1.270880 ug/l chlorophyll fluorescence. 2023-09-16T05:37:11.674Z,1694842631.674 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T05:37:11.674Z,1694842631.674 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.stop 2023-09-16T05:37:12.082Z,1694842632.082 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING 2023-09-16T05:37:14.101Z,1694842634.101 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T05:37:14.503Z,1694842634.503 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2023-09-16T05:42:14.269Z,1694842934.269 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2023-09-16T05:42:26.093Z,1694842946.093 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T05:42:26.229Z,1694842946.229 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T05:43:18.150Z,1694842998.150 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T05:43:18.962Z,1694842998.962 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2023-09-16T05:43:18.963Z,1694842998.963 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T05:43:19.398Z,1694842999.398 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 13.834292 m in 21.826563 degC water with 1.256320 ug/l chlorophyll fluorescence. 2023-09-16T05:43:19.770Z,1694842999.770 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T05:43:20.169Z,1694843000.169 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T05:43:22.189Z,1694843002.189 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T05:43:54.105Z,1694843034.105 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@11,aRNAl_bac) 2023-09-16T05:43:56.937Z,1694843036.937 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T05:43:56.939Z,1694843036.939 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T05:43:57.338Z,1694843037.338 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T05:46:22.374Z,1694843182.374 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T05:46:22.782Z,1694843182.782 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T05:58:05.482Z,1694843885.482 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 41.938869,-81.020546 2023-09-16T05:58:05.496Z,1694843885.496 [altitudeServo_approach_sampling:Lap:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938614,-81.072388 2023-09-16T06:39:11.166Z,1694846351.166 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1074.8) 2023-09-16T06:39:11.567Z,1694846351.567 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1074.8) 2023-09-16T06:39:13.587Z,1694846353.587 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T06:39:13.989Z,1694846353.989 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@11,aRNAl_bac) 2023-09-16T06:40:48.931Z,1694846448.931 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@11,aRNAl_bac) 2023-09-16T06:40:56.606Z,1694846456.606 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T06:40:57.013Z,1694846457.013 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2023-09-16T06:45:56.809Z,1694846756.809 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2023-09-16T06:46:08.633Z,1694846768.633 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T06:46:08.773Z,1694846768.773 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T06:47:01.449Z,1694846821.449 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T06:47:02.260Z,1694846822.260 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2023-09-16T06:47:02.261Z,1694846822.261 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T06:47:02.699Z,1694846822.699 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 14.755561 m in 21.827723 degC water with 1.278160 ug/l chlorophyll fluorescence. 2023-09-16T06:47:03.066Z,1694846823.066 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T06:47:03.469Z,1694846823.469 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T06:47:05.489Z,1694846825.489 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T06:47:37.418Z,1694846857.418 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@10,aRNAl_bac) 2023-09-16T06:47:40.246Z,1694846860.246 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T06:47:40.247Z,1694846860.247 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T06:47:40.649Z,1694846860.649 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T06:50:05.694Z,1694847005.694 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T06:50:06.105Z,1694847006.105 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T07:11:49.923Z,1694848309.923 [altitudeServo_approach_sampling:Lap:Wpt2:A.Waypoint](IMPORTANT): Reached waypoint: 41.938614,-81.072388 2023-09-16T07:11:49.928Z,1694848309.928 [altitudeServo_approach_sampling:Lap:Wpt3:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.844151,-81.069511 2023-09-16T07:38:49.914Z,1694849929.914 [CBIT](IMPORTANT): Beginning ground fault scan 2023-09-16T07:39:00.816Z,1694849940.816 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003138 CHAN A1 (24V): 0.006398 CHAN A2 (12V): -0.004926 CHAN A3 (5V): -0.004349 CHAN B0 (3.3V): -0.001970 CHAN B1 (3.15aV): -0.001902 CHAN B2 (3.15bV): -0.001488 CHAN B3 (GND): -0.000845 OPEN: 0.012071 Full Scale: +/- 1 mA 2023-09-16T07:41:20.096Z,1694850080.096 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1067.2) 2023-09-16T07:41:20.494Z,1694850080.494 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1067.2) 2023-09-16T07:41:22.514Z,1694850082.514 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T07:41:22.918Z,1694850082.918 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@10,aRNAl_bac) 2023-09-16T07:42:57.862Z,1694850177.862 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@10,aRNAl_bac) 2023-09-16T07:43:05.539Z,1694850185.539 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T07:43:05.942Z,1694850185.942 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2023-09-16T07:48:05.777Z,1694850485.777 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2023-09-16T07:48:06.433Z,1694850486.433 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 14.437128 m in 21.826014 degC water with 1.166880 ug/l chlorophyll fluorescence. 2023-09-16T09:47:54.020Z,1694857674.020 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-16T09:47:54.153Z,1694857674.153 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-16T09:48:46.022Z,1694857726.022 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-16T09:48:46.837Z,1694857726.837 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -1 2023-09-16T09:48:46.837Z,1694857726.837 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-16T09:48:47.286Z,1694857727.286 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 7.030527 m in 21.937891 degC water with 1.203280 ug/l chlorophyll fluorescence. 2023-09-16T09:48:47.641Z,1694857727.641 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-16T09:48:48.042Z,1694857728.042 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-16T09:48:50.061Z,1694857730.061 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-16T09:49:21.978Z,1694857761.978 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@9,aRNAl_bac) 2023-09-16T09:49:24.806Z,1694857764.806 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2023-09-16T09:49:24.807Z,1694857764.807 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-16T09:49:25.210Z,1694857765.210 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-16T09:50:00.132Z,1694857800.132 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml 2023-09-16T09:50:00.899Z,1694857800.899 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/Science.tx 2023-09-16T09:50:02.347Z,1694857802.347 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/NeedComms.tx 2023-09-16T09:50:02.582Z,1694857802.582 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/StandardEnvelopes.tx 2023-09-16T09:50:02.706Z,1694857802.706 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/BackseatDriver.tx 2023-09-16T09:50:02.923Z,1694857802.923 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml 2023-09-16T09:50:03.270Z,1694857803.270 [MissionManager](IMPORTANT): Started mission Default 2023-09-16T09:50:18.087Z,1694857818.087 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 16 hour 2023-09-16T09:50:18.088Z,1694857818.088 [CommandExec](IMPORTANT): got command set profile_station.Lat 41.93887 degree 2023-09-16T09:50:18.089Z,1694857818.089 [CommandExec](IMPORTANT): got command set profile_station.Lon -81.045949 degree 2023-09-16T09:50:18.090Z,1694857818.090 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 8 meter 2023-09-16T09:50:21.985Z,1694857821.985 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 10 meter 2023-09-16T09:50:21.986Z,1694857821.986 [CommandExec](IMPORTANT): got command run 2023-09-16T09:50:22.006Z,1694857822.006 [CommandExec](IMPORTANT): Running 2023-09-16T09:50:22.243Z,1694857822.243 [MissionManager](IMPORTANT): Started mission profile_station 2023-09-16T09:51:57.114Z,1694857917.114 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2023-09-16T09:51:57.517Z,1694857917.517 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-16T10:12:56.768Z,1694859176.768 [ElevatorServo](FAULT): Invalid EZ Servo response:"" 2023-09-16T10:39:42.226Z,1694860782.226 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@1015.5) 2023-09-16T10:39:42.636Z,1694860782.636 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@1015.5) 2023-09-16T10:39:44.646Z,1694860784.646 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-16T10:39:45.049Z,1694860785.049 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@9,aRNAl_bac) 2023-09-16T10:41:20.396Z,1694860880.396 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@9,aRNAl_bac) 2023-09-16T10:41:28.071Z,1694860888.071 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-16T10:41:28.481Z,1694860888.481 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2023-09-16T10:46:28.242Z,1694861188.242 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2023-09-16T13:39:01.132Z,1694871541.132 [CBIT](IMPORTANT): Beginning ground fault scan 2023-09-16T13:39:12.419Z,1694871552.419 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003367 CHAN A1 (24V): -0.002749 CHAN A2 (12V): -0.004155 CHAN A3 (5V): -0.004628 CHAN B0 (3.3V): 0.001541 CHAN B1 (3.15aV): -0.001484 CHAN B2 (3.15bV): -0.001249 CHAN B3 (GND): -0.001277 OPEN: 0.011929 Full Scale: +/- 1 mA 2023-09-16T19:39:12.565Z,1694893152.565 [CBIT](IMPORTANT): Beginning ground fault scan 2023-09-16T19:39:23.906Z,1694893163.906 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004968 CHAN A1 (24V): 0.013345 CHAN A2 (12V): -0.005246 CHAN A3 (5V): -0.004482 CHAN B0 (3.3V): -0.002259 CHAN B1 (3.15aV): -0.001799 CHAN B2 (3.15bV): -0.001673 CHAN B3 (GND): -0.001103 OPEN: 0.012113 Full Scale: +/- 1 mA 2023-09-16T21:55:51.790Z,1694901351.790 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=11235, MT Status=1, MTMSN=3391 2023-09-16T21:55:52.590Z,1694901352.590 [CommandExec](IMPORTANT): got command schedule 20230917T004700 "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 12 h;set altitudeServo_approach_sampling.Lat1 41.93887 degree;set altitudeServo_approach_sampling.Lon1 -81.045949 degree" va6y 1 5.000000 2023-09-16T21:55:52.590Z,1694901352.590 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=va6y 2023-09-16T21:55:52.592Z,1694901352.592 [CommandExec](IMPORTANT): Scheduled #52 (#1 of 5 with id='va6y'): "load Science/altitudeServo_approach_sampling.xml;set altitudeServo_approach_sampling.MissionTimeout 12 h;set altitudeServo_approach_sampling.Lat1 41.93887 degree;set altitudeServo_approach_sampling.Lon1 -81.045949 degree", AT TIME: 2023-09-17T00:47:00Z 2023-09-16T21:56:26.913Z,1694901386.913 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11236, MT Status=1, MTMSN=3392 2023-09-16T21:56:27.638Z,1694901387.638 [CommandExec](IMPORTANT): got command schedule 20230917T004700 "set altitudeServo_approach_sampling.Lat2 41.85445 degree;set altitudeServo_approach_sampling.Lon2 -81.04275 degree;set altitudeServo_approach_sampling.Lat3 41.86199 degree" va6y 2 5.000000 2023-09-16T21:56:27.639Z,1694901387.639 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=va6y 2023-09-16T21:56:27.640Z,1694901387.640 [CommandExec](IMPORTANT): Scheduled #53 (#2 of 5 with id='va6y'): "set altitudeServo_approach_sampling.Lat2 41.85445 degree;set altitudeServo_approach_sampling.Lon2 -81.04275 degree;set altitudeServo_approach_sampling.Lat3 41.86199 degree", AT TIME: 2023-09-17T00:47:00Z 2023-09-16T21:56:57.657Z,1694901417.657 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11237, MT Status=1, MTMSN=3393 2023-09-16T21:56:59.653Z,1694901419.653 [CommandExec](IMPORTANT): got command schedule 20230917T004700 "set altitudeServo_approach_sampling.Lon3 -81.01861 degree;set altitudeServo_approach_sampling.Lat4 41.93887 degree;set altitudeServo_approach_sampling.Lon4 -81.020543 degree" va6y 3 5.000000 2023-09-16T21:56:59.653Z,1694901419.653 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=va6y 2023-09-16T21:56:59.655Z,1694901419.655 [CommandExec](IMPORTANT): Scheduled #54 (#3 of 5 with id='va6y'): "set altitudeServo_approach_sampling.Lon3 -81.01861 degree;set altitudeServo_approach_sampling.Lat4 41.93887 degree;set altitudeServo_approach_sampling.Lon4 -81.020543 degree", AT TIME: 2023-09-17T00:47:00Z 2023-09-16T21:57:20.722Z,1694901440.722 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11238, MT Status=1, MTMSN=3394 2023-09-16T21:57:21.458Z,1694901441.458 [CommandExec](IMPORTANT): got command schedule 20230917T004700 "set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m;set altitudeServo_approach_sampling.MaxDepth 18 m" va6y 4 5.000000 2023-09-16T21:57:21.458Z,1694901441.458 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=va6y 2023-09-16T21:57:21.459Z,1694901441.459 [CommandExec](IMPORTANT): Scheduled #55 (#4 of 5 with id='va6y'): "set altitudeServo_approach_sampling.HoldAltitude 7 m;set altitudeServo_approach_sampling.InitDepth 5 m;set altitudeServo_approach_sampling.MinAltitude 4 m;set altitudeServo_approach_sampling.MaxDepth 18 m", AT TIME: 2023-09-17T00:47:00Z 2023-09-16T21:59:18.874Z,1694901558.874 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=11239, MT Status=1, MTMSN=3395 2023-09-16T21:59:19.502Z,1694901559.502 [CommandExec](IMPORTANT): got command schedule 20230917T004700 "set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run" va6y 5 5.000000 2023-09-16T21:59:19.502Z,1694901559.502 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=va6y 2023-09-16T21:59:19.503Z,1694901559.503 [CommandExec](IMPORTANT): Scheduled #56 (#5 of 5 with id='va6y'): "set altitudeServo_approach_sampling.ESPCartridgeType -1 count;run", AT TIME: 2023-09-17T00:47:00Z 2023-09-16T22:19:30.224Z,1694902770.224 [Radio_Surface](FAULT): LCB fault: Invalid Command. 2023-09-16T22:22:06.540Z,1694902926.540 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-09-16T22:32:56.436Z,1694903576.436 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2023-09-17T00:47:00.077Z,1694911620.077 [CommandExec](IMPORTANT): got command load ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-17T00:47:01.978Z,1694911621.978 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/StandardEnvelopes.tx 2023-09-17T00:47:02.115Z,1694911622.115 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/NeedComms.tx 2023-09-17T00:47:02.603Z,1694911622.603 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/Science.tx 2023-09-17T00:47:04.058Z,1694911624.058 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file Missions/Insert/BackseatDriver.tx 2023-09-17T00:47:04.854Z,1694911624.854 [CommandExec](IMPORTANT): Loaded ./Missions/Science/altitudeServo_approach_sampling.xml 2023-09-17T00:47:05.229Z,1694911625.229 [MissionManager](IMPORTANT): Started mission Default 2023-09-17T00:47:21.816Z,1694911641.816 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MissionTimeout 12 hour 2023-09-17T00:47:21.817Z,1694911641.817 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat1 41.93887 degree 2023-09-17T00:47:21.818Z,1694911641.818 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon1 -81.045949 degree 2023-09-17T00:47:38.162Z,1694911658.162 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat2 41.85445 degree 2023-09-17T00:47:38.163Z,1694911658.163 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon2 -81.04275 degree 2023-09-17T00:47:38.163Z,1694911658.163 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat3 41.86199 degree 2023-09-17T00:47:54.194Z,1694911674.194 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon3 -81.01861 degree 2023-09-17T00:47:54.195Z,1694911674.195 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lat4 41.93887 degree 2023-09-17T00:47:54.196Z,1694911674.196 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.Lon4 -81.020543 degree 2023-09-17T00:48:19.146Z,1694911699.146 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.HoldAltitude 7 meter 2023-09-17T00:48:19.147Z,1694911699.147 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.InitDepth 5 meter 2023-09-17T00:48:19.161Z,1694911699.161 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MinAltitude 4 meter 2023-09-17T00:48:19.161Z,1694911699.161 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.MaxDepth 18 meter 2023-09-17T00:48:25.898Z,1694911705.898 [CommandExec](IMPORTANT): got command set altitudeServo_approach_sampling.ESPCartridgeType -1 count 2023-09-17T00:48:25.899Z,1694911705.899 [CommandExec](IMPORTANT): got command run 2023-09-17T00:48:25.914Z,1694911705.914 [CommandExec](IMPORTANT): Running 2023-09-17T00:48:25.988Z,1694911705.988 [MissionManager](IMPORTANT): Started mission altitudeServo_approach_sampling 2023-09-17T00:52:03.135Z,1694911923.135 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.938869,-81.045952 2023-09-17T00:57:42.404Z,1694912262.404 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-17T00:57:42.725Z,1694912262.725 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-17T00:58:35.043Z,1694912315.043 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-17T00:58:35.854Z,1694912315.854 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -1 2023-09-17T00:58:35.855Z,1694912315.855 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-17T00:58:36.354Z,1694912316.354 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 5.163645 m in 22.017206 degC water with 1.352000 ug/l chlorophyll fluorescence. 2023-09-17T00:58:36.659Z,1694912316.659 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-17T00:58:37.061Z,1694912317.061 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-17T00:58:39.077Z,1694912319.077 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-17T00:59:10.995Z,1694912350.995 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@8,aRNAl_bac) 2023-09-17T00:59:13.418Z,1694912353.418 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2023-09-17T00:59:13.419Z,1694912353.419 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-17T00:59:13.827Z,1694912353.827 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-17T01:00:04.352Z,1694912404.352 [altitudeServo_approach_sampling:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 41.938869,-81.045952 2023-09-17T01:00:04.354Z,1694912404.354 [altitudeServo_approach_sampling:Lap:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 41.854450,-81.042747 2023-09-17T01:01:31.989Z,1694912491.989 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2023-09-17T01:01:32.395Z,1694912492.395 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-17T01:39:23.995Z,1694914763.995 [CBIT](IMPORTANT): Beginning ground fault scan 2023-09-17T01:39:35.750Z,1694914775.750 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002462 CHAN A1 (24V): 0.004824 CHAN A2 (12V): -0.004263 CHAN A3 (5V): -0.003691 CHAN B0 (3.3V): -0.002225 CHAN B1 (3.15aV): -0.001997 CHAN B2 (3.15bV): 0.001176 CHAN B3 (GND): -0.001104 OPEN: 0.012131 Full Scale: +/- 1 mA 2023-09-17T01:53:29.086Z,1694915609.086 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1078.2) 2023-09-17T01:53:29.489Z,1694915609.489 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1078.2) 2023-09-17T01:53:31.511Z,1694915611.511 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-09-17T01:53:31.913Z,1694915611.913 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@8,aRNAl_bac) 2023-09-17T01:55:06.865Z,1694915706.865 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@8,aRNAl_bac) 2023-09-17T01:55:14.539Z,1694915714.539 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-09-17T01:55:15.026Z,1694915715.026 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2023-09-17T02:00:14.797Z,1694916014.797 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2023-09-17T02:00:26.626Z,1694916026.626 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-09-17T02:00:26.781Z,1694916026.781 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-09-17T02:01:27.162Z,1694916087.162 [ESPComponent](IMPORTANT): ESP has connected as client 2023-09-17T02:01:27.984Z,1694916087.984 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -1 2023-09-17T02:01:27.985Z,1694916087.985 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-09-17T02:01:28.431Z,1694916088.431 [altitudeServo_approach_sampling:DiveCmd:AltitudeServoSampling:SampleESP:C](IMPORTANT): ESP sampling at 13.430655 m in 21.803980 degC water with 0.957840 ug/l chlorophyll fluorescence. 2023-09-17T02:01:28.775Z,1694916088.775 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2023-09-17T02:01:29.178Z,1694916089.178 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-09-17T02:01:31.198Z,1694916091.198 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2023-09-17T02:02:03.114Z,1694916123.114 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@7,aRNAl_bac) 2023-09-17T02:02:05.942Z,1694916125.942 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2023-09-17T02:02:05.943Z,1694916125.943 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2023-09-17T02:02:06.349Z,1694916126.349 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-09-17T02:04:41.485Z,1694916281.485 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2023-09-17T02:04:41.889Z,1694916281.889 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-09-17T02:53:08.027Z,1694919188.027 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=11300, MT Status=1, MTMSN=3396