2024-05-06T22:09:16.930Z,1715033356.930 [Supervisor](IMPORTANT): Reloading configuration overrides from Data/persisted.cfg 2024-05-06T22:09:21.775Z,1715033361.775 [SBIT](IMPORTANT): git: 2024-04-26_A 2024-05-06T22:09:21.776Z,1715033361.776 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2024-05-06T22:09:21.931Z,1715033361.931 [MissionManager](IMPORTANT): Using TethysL tl-to-tx parser 2024-05-06T22:09:34.931Z,1715033374.931 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:09:39.980Z,1715033379.980 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:09:40.782Z,1715033380.782 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:09:49.731Z,1715033389.731 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:09:55.395Z,1715033395.395 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:09:55.739Z,1715033395.739 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:09:58.951Z,1715033398.951 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:10:04.526Z,1715033404.526 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:10:04.679Z,1715033404.679 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:10:10.678Z,1715033410.678 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:10:17.959Z,1715033417.959 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:10:18.017Z,1715033418.017 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T22:10:18.537Z,1715033418.537 [CommandExec](FAULT): Scheduling is paused 2024-05-06T22:10:25.626Z,1715033425.626 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:10:28.319Z,1715033428.319 [CommandExec](IMPORTANT): got command strobe off 2024-05-06T22:10:28.319Z,1715033428.319 [CommandExec](IMPORTANT): Deactivating strobe 2024-05-06T22:10:40.573Z,1715033440.573 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:10:40.645Z,1715033440.645 [CBIT](CRITICAL): Data Fault in component: AHRS_M2 2024-05-06T22:11:21.412Z,1715033481.412 [SBIT](IMPORTANT): Beginning Startup BIT 2024-05-06T22:11:21.417Z,1715033481.417 [CBIT](IMPORTANT): Beginning ground fault scan 2024-05-06T22:11:21.803Z,1715033481.803 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:11:32.355Z,1715033492.355 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.004309 CHAN A1 (24V): -0.002291 CHAN A2 (12V): -0.000069 CHAN A3 (5V): 0.000063 CHAN B0 (3.3V): 0.000630 CHAN B1 (3.15aV): 0.000007 CHAN B2 (3.15bV): 0.000459 CHAN B3 (GND): 0.000962 OPEN: 0.004779 Full Scale: +/- 1 mA 2024-05-06T22:11:52.419Z,1715033512.419 [CommandExec](IMPORTANT): got command maintain sensor NAL9602.latitude_fix 36.797 degree 2024-05-06T22:11:52.423Z,1715033512.423 [CommandExec](IMPORTANT): got command maintain sensor NAL9602.longitude_fix -121.847 degree 2024-05-06T22:12:05.733Z,1715033525.733 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1715033512 second 2024-05-06T22:12:07.195Z,1715033527.195 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:12:07.197Z,1715033527.197 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:12:07.197Z,1715033527.197 [CommandExec](IMPORTANT): AHRS_M2: Data Fault 2024-05-06T22:12:07.197Z,1715033527.197 [CommandExec](IMPORTANT): DVL_micro: Communications Fault 2024-05-06T22:12:07.198Z,1715033527.198 [CommandExec](IMPORTANT): CTD_Seabird: Communications Fault 2024-05-06T22:12:07.198Z,1715033527.198 [CommandExec](IMPORTANT): WetLabsBB2FL: Communications Fault 2024-05-06T22:12:11.304Z,1715033531.304 [CommandExec](IMPORTANT): got command get latitude 2024-05-06T22:12:11.304Z,1715033531.304 [CommandExec](IMPORTANT): latitude 36.797001 arcdeg 2024-05-06T22:12:12.968Z,1715033532.968 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:12:12.968Z,1715033532.968 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:12:12.968Z,1715033532.968 [MissionManager](IMPORTANT): Mission loaded, but not running. 2024-05-06T22:12:15.529Z,1715033535.529 [SBIT](IMPORTANT): SBIT PASSED 2024-05-06T22:12:15.529Z,1715033535.529 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): AHRS_M2.loadAtStartup=1 bool; 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=16 count; 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): CANONSampler.loadAtStartup=0 bool; 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): CANONSampler.sampleTimeout=5 minute; 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): CANONSampler.simulateHardware=1 bool; 2024-05-06T22:12:15.530Z,1715033535.530 [SBIT](IMPORTANT): DVL_micro.baud=19200 bit_per_second; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): Depth_Keller.offset=3.6 decibar; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): DropWeight.loadAtStartup=0 bool; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): ESPComponent.loadAtStartup=1 bool; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=3 minute; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): ESPComponent.simulateHardware=0 bool; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): IBIT.batteryCapacityThreshold=2 ampere_hour; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): IBIT.batteryVoltageThreshold=10 volt; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=70.614532 cubic_centimeter; 2024-05-06T22:12:15.531Z,1715033535.531 [SBIT](IMPORTANT): VerticalControl.massDefault=24.832817 millimeter; 2024-05-06T22:12:15.950Z,1715033535.950 [MissionManager](IMPORTANT): Started mission Startup 2024-05-06T22:12:29.760Z,1715033549.760 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1715033535 second 2024-05-06T22:12:47.654Z,1715033567.654 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:12:50.984Z,1715033570.984 [CommandExec](IMPORTANT): got command set NAL9602.platform_communications 1 bool 2024-05-06T22:12:51.493Z,1715033571.493 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T22:12:59.206Z,1715033579.206 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:13:02.456Z,1715033582.456 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:13:17.254Z,1715033597.254 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:13:17.369Z,1715033597.369 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:13:23.010Z,1715033603.010 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:13:28.988Z,1715033608.988 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1715033603 second 2024-05-06T22:13:31.615Z,1715033611.615 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:13:31.615Z,1715033611.615 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:13:31.615Z,1715033611.615 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T22:13:31.615Z,1715033611.615 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:Read_Iridium 2024-05-06T22:13:35.899Z,1715033615.899 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:13:54.620Z,1715033634.620 [CommandExec](IMPORTANT): got command get ESPComponent.loadAtStartup 2024-05-06T22:13:54.621Z,1715033634.621 [CommandExec](IMPORTANT): ESPComponent.loadAtStartup 1 bool 2024-05-06T22:13:54.891Z,1715033634.891 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:14:03.245Z,1715033643.245 [CommandExec](IMPORTANT): got command get ESPComponent.simulateHardware 2024-05-06T22:14:03.245Z,1715033643.245 [CommandExec](IMPORTANT): ESPComponent.simulateHardware 0 bool 2024-05-06T22:14:06.259Z,1715033646.259 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:14:06.259Z,1715033646.259 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:14:06.260Z,1715033646.260 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T22:14:06.260Z,1715033646.260 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:Read_Iridium 2024-05-06T22:14:13.887Z,1715033653.887 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:14:13.974Z,1715033653.974 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T22:15:03.168Z,1715033703.168 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T22:15:24.202Z,1715033724.202 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:15:57.722Z,1715033757.722 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:16:00.446Z,1715033760.446 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:16:03.098Z,1715033763.098 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:16:12.674Z,1715033772.674 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:16:15.690Z,1715033775.690 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:16:27.617Z,1715033787.617 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:16:30.507Z,1715033790.507 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:16:30.876Z,1715033790.876 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:16:42.566Z,1715033802.566 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:16:47.315Z,1715033807.315 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:16:47.315Z,1715033807.315 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:16:47.315Z,1715033807.315 [CommandExec](IMPORTANT): DVL_micro: Communications Fault 2024-05-06T22:16:47.315Z,1715033807.315 [CommandExec](IMPORTANT): CTD_Seabird: Communications Fault 2024-05-06T22:16:47.315Z,1715033807.315 [CommandExec](IMPORTANT): WetLabsBB2FL: Communications Fault 2024-05-06T22:16:50.452Z,1715033810.452 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:16:50.452Z,1715033810.452 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:16:50.452Z,1715033810.452 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T22:16:50.452Z,1715033810.452 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:C.Wait 2024-05-06T22:16:57.514Z,1715033817.514 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:16:57.574Z,1715033817.574 [CBIT](CRITICAL): Data Fault in component: AHRS_M2 2024-05-06T22:17:02.260Z,1715033822.260 [CommandExec](IMPORTANT): got command get CANONSampler.loadAtStartup 2024-05-06T22:17:02.261Z,1715033822.261 [CommandExec](IMPORTANT): CANONSampler.loadAtStartup 0 bool 2024-05-06T22:17:25.389Z,1715033845.389 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:17:31.839Z,1715033851.839 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:17:50.847Z,1715033870.847 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:18:09.823Z,1715033889.823 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:18:09.862Z,1715033889.862 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T22:19:06.820Z,1715033946.820 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:19:14.006Z,1715033954.006 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:19:26.610Z,1715033966.610 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:19:28.810Z,1715033968.810 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:19:39.545Z,1715033979.545 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.794267 min 2024-05-06T22:19:43.610Z,1715033983.610 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:19:44.003Z,1715033984.003 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:21:26.695Z,1715034086.695 [DataOverHttps](IMPORTANT): SBD MTMSN=20240506T222125 2024-05-06T22:21:27.445Z,1715034087.445 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:21:27.819Z,1715034087.819 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:21:34.015Z,1715034094.015 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.tl 2024-05-06T22:21:34.161Z,1715034094.161 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.tl id=sample_lab 2024-05-06T22:21:37.715Z,1715034097.715 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -1 count 2024-05-06T22:21:37.716Z,1715034097.716 [CommandExec](IMPORTANT): got command set sample_lab.NumberOfSamples 2 count 2024-05-06T22:21:37.716Z,1715034097.716 [CommandExec](IMPORTANT): got command run 2024-05-06T22:21:37.718Z,1715034097.718 [CommandExec](IMPORTANT): Running 2024-05-06T22:21:37.924Z,1715034097.924 [MissionManager](IMPORTANT): Started mission sample_lab 2024-05-06T22:21:37.926Z,1715034097.926 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -1.000000 count 2024-05-06T22:21:39.524Z,1715034099.524 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2024-05-06T22:21:46.788Z,1715034106.788 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:21:49.671Z,1715034109.671 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-05-06T22:21:49.777Z,1715034109.777 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-05-06T22:22:05.859Z,1715034125.859 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:22:05.953Z,1715034125.953 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T22:22:07.154Z,1715034127.154 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T22:22:10.372Z,1715034130.372 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:22:11.446Z,1715034131.446 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:22:26.398Z,1715034146.398 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:22:27.082Z,1715034147.082 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:22:41.342Z,1715034161.342 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:22:41.879Z,1715034161.879 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:22:56.290Z,1715034176.290 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:22:56.685Z,1715034176.685 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:22:56.779Z,1715034176.779 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:23:00.329Z,1715034180.329 [ESPComponent](IMPORTANT): ESP has connected as client 2024-05-06T22:23:01.253Z,1715034181.253 [ESPComponent](IMPORTANT): [sample #1] Explicit cartridge argument not given 2024-05-06T22:23:01.254Z,1715034181.254 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-05-06T22:23:01.638Z,1715034181.638 [ESPComponent](IMPORTANT): [sample #1] Stop sampling requested. 2024-05-06T22:23:01.973Z,1715034181.973 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.stop 2024-05-06T22:23:02.368Z,1715034182.368 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING 2024-05-06T22:23:11.238Z,1715034191.238 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:23:11.322Z,1715034191.322 [CBIT](CRITICAL): Data Fault in component: AHRS_M2 2024-05-06T22:23:28.609Z,1715034208.609 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:23:40.840Z,1715034220.840 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:23:40.840Z,1715034220.840 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:23:40.840Z,1715034220.840 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T22:23:40.840Z,1715034220.840 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:Read_GPS 2024-05-06T22:24:07.195Z,1715034247.195 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:24:07.195Z,1715034247.195 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:24:07.196Z,1715034247.196 [CommandExec](IMPORTANT): AHRS_M2: Data Fault 2024-05-06T22:24:07.196Z,1715034247.196 [CommandExec](IMPORTANT): DVL_micro: Communications Fault 2024-05-06T22:24:07.196Z,1715034247.196 [CommandExec](IMPORTANT): CTD_Seabird: Communications Fault 2024-05-06T22:24:07.196Z,1715034247.196 [CommandExec](IMPORTANT): WetLabsBB2FL: Communications Fault 2024-05-06T22:24:32.548Z,1715034272.548 [DataOverHttps](IMPORTANT): SBD MTMSN=20240506T222431 2024-05-06T22:24:39.490Z,1715034279.490 [CommandExec](IMPORTANT): got command gfscan 2024-05-06T22:24:39.748Z,1715034279.748 [CBIT](IMPORTANT): Beginning ground fault scan 2024-05-06T22:24:40.036Z,1715034280.036 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.tl 2024-05-06T22:24:40.211Z,1715034280.211 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.tl id=sample_lab 2024-05-06T22:24:43.343Z,1715034283.343 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T22:24:43.942Z,1715034283.942 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -1 count 2024-05-06T22:24:43.943Z,1715034283.943 [CommandExec](IMPORTANT): got command set sample_lab.NumberOfSamples 2 count 2024-05-06T22:24:43.943Z,1715034283.943 [CommandExec](IMPORTANT): got command run 2024-05-06T22:24:43.950Z,1715034283.950 [CommandExec](IMPORTANT): Running 2024-05-06T22:24:44.143Z,1715034284.143 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-05-06T22:24:44.162Z,1715034284.162 [MissionManager](IMPORTANT): Started mission sample_lab 2024-05-06T22:24:44.164Z,1715034284.164 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -1.000000 count 2024-05-06T22:24:50.679Z,1715034290.679 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.006119 CHAN A1 (24V): -0.002367 CHAN A2 (12V): 0.002962 CHAN A3 (5V): -0.000058 CHAN B0 (3.3V): 0.001004 CHAN B1 (3.15aV): 0.000598 CHAN B2 (3.15bV): 0.000713 CHAN B3 (GND): 0.001362 OPEN: 0.004832 Full Scale: +/- 1 mA 2024-05-06T22:25:02.334Z,1715034302.334 [ESPComponent](FAULT): Timeout while awaiting result for cmd='Cmd.stop' 2024-05-06T22:25:02.335Z,1715034302.335 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-05-06T22:25:02.728Z,1715034302.728 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_ERROR 2024-05-06T22:25:13.691Z,1715034313.691 [CBIT](FAULT): Communications Fault in component: WetLabsBB2FL 2024-05-06T22:25:23.739Z,1715034323.739 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:25:29.819Z,1715034329.819 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:25:40.166Z,1715034340.166 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:25:42.727Z,1715034342.727 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:25:54.978Z,1715034354.978 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:26:01.716Z,1715034361.716 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T22:26:01.739Z,1715034361.739 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T22:26:02.941Z,1715034362.941 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T22:26:09.778Z,1715034369.778 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:26:09.867Z,1715034369.867 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:27:31.074Z,1715034451.074 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:28:24.894Z,1715034504.894 [AHRS_M2](FAULT): Failed to initialize within timeout. 2024-05-06T22:28:41.382Z,1715034521.382 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:28:53.266Z,1715034533.266 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:28:56.326Z,1715034536.326 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:29:08.068Z,1715034548.068 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:29:11.278Z,1715034551.278 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:29:22.866Z,1715034562.866 [CTD_Seabird](FAULT): Failed to initialize within timeout. 2024-05-06T22:29:23.071Z,1715034563.071 [CBIT](FAULT): Communications Fault in component: CTD_Seabird 2024-05-06T22:29:26.222Z,1715034566.222 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T22:29:26.255Z,1715034566.255 [CBIT](CRITICAL): Data Fault in component: AHRS_M2 2024-05-06T22:29:31.907Z,1715034571.907 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T22:30:02.577Z,1715034602.577 [ESPComponent](CRITICAL): [sample #1] ESP sampling sequence terminated with error: Timeout while awaiting result for cmd='Cmd.stop' 2024-05-06T22:30:38.779Z,1715034638.779 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:30:38.779Z,1715034638.779 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:30:38.779Z,1715034638.779 [CommandExec](IMPORTANT): AHRS_M2: Data Fault 2024-05-06T22:30:38.779Z,1715034638.779 [CommandExec](IMPORTANT): CTD_Seabird: Communications Fault 2024-05-06T22:30:44.005Z,1715034644.005 [CommandExec](IMPORTANT): got command failComponent none CTD_Seabird 2024-05-06T22:30:44.026Z,1715034644.026 [CommandExec](IMPORTANT): CTD_Seabird failureMode is No Fault 2024-05-06T22:30:47.439Z,1715034647.439 [CommandExec](IMPORTANT): got command failComponent none AHRS_M2 2024-05-06T22:30:47.439Z,1715034647.439 [CommandExec](IMPORTANT): AHRS_M2 failureMode is No Fault 2024-05-06T22:30:50.555Z,1715034650.555 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:30:50.555Z,1715034650.555 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:30:50.555Z,1715034650.555 [CommandExec](IMPORTANT): No failed Components. 2024-05-06T22:31:20.556Z,1715034680.556 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T22:32:00.923Z,1715034720.923 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T22:32:00.923Z,1715034720.923 [CommandExec](IMPORTANT): Failed components: 2024-05-06T22:32:00.923Z,1715034720.923 [CommandExec](IMPORTANT): No failed Components. 2024-05-06T22:32:10.805Z,1715034730.805 [DataOverHttps](IMPORTANT): SBD MOMSN=19986582, MTMSN=20240506T223210 2024-05-06T22:32:21.352Z,1715034741.352 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.tl 2024-05-06T22:32:21.458Z,1715034741.458 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.tl id=sample_lab 2024-05-06T22:32:25.922Z,1715034745.922 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -1 count 2024-05-06T22:32:25.927Z,1715034745.927 [CommandExec](IMPORTANT): got command set sample_lab.NumberOfSamples 2 count 2024-05-06T22:32:25.927Z,1715034745.927 [CommandExec](IMPORTANT): got command run 2024-05-06T22:32:25.929Z,1715034745.929 [CommandExec](IMPORTANT): Running 2024-05-06T22:32:26.041Z,1715034746.041 [MissionManager](IMPORTANT): Started mission sample_lab 2024-05-06T22:32:26.062Z,1715034746.062 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -1.000000 count 2024-05-06T22:32:37.835Z,1715034757.835 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-05-06T22:32:38.052Z,1715034758.052 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-05-06T22:33:04.775Z,1715034784.775 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:33:04.775Z,1715034784.775 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:33:04.775Z,1715034784.775 [sample_lab](IMPORTANT): Priority 0: sample_lab:A.AbortSample 2024-05-06T22:33:04.776Z,1715034784.776 [sample_lab:SampleRepeater](IMPORTANT): Priority 1: sample_lab:SampleRepeater:TriggerESP 2024-05-06T22:33:48.919Z,1715034828.919 [ESPComponent](IMPORTANT): ESP has connected as client 2024-05-06T22:33:49.758Z,1715034829.758 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -1 2024-05-06T22:33:49.759Z,1715034829.759 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-05-06T22:33:50.534Z,1715034830.534 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2024-05-06T22:33:50.946Z,1715034830.946 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-05-06T22:33:52.958Z,1715034832.958 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2024-05-06T22:34:24.081Z,1715034864.081 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@20,aRNAl_bac) 2024-05-06T22:34:26.893Z,1715034866.893 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2024-05-06T22:34:26.894Z,1715034866.894 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-05-06T22:34:27.305Z,1715034867.305 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-05-06T22:35:11.759Z,1715034911.759 [CommandExec](IMPORTANT): got command gfscan 2024-05-06T22:35:11.783Z,1715034911.783 [CBIT](IMPORTANT): Beginning ground fault scan 2024-05-06T22:35:22.678Z,1715034922.678 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.008770 CHAN A1 (24V): -0.001797 CHAN A2 (12V): -0.000401 CHAN A3 (5V): 0.000161 CHAN B0 (3.3V): 0.000969 CHAN B1 (3.15aV): -0.001332 CHAN B2 (3.15bV): 0.000659 CHAN B3 (GND): 0.000169 OPEN: 0.004862 Full Scale: +/- 1 mA 2024-05-06T22:37:20.242Z,1715035040.242 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2024-05-06T22:37:20.638Z,1715035040.638 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-05-06T22:40:46.972Z,1715035246.972 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:40:46.972Z,1715035246.972 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:40:46.972Z,1715035246.972 [sample_lab](IMPORTANT): Priority 0: sample_lab:A.AbortSample 2024-05-06T22:40:46.972Z,1715035246.972 [sample_lab:SampleRepeater](IMPORTANT): Priority 1: sample_lab:SampleRepeater:WaitForESP 2024-05-06T22:44:44.920Z,1715035484.920 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:44:44.920Z,1715035484.920 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:44:44.920Z,1715035484.920 [sample_lab](IMPORTANT): Priority 0: sample_lab:A.AbortSample 2024-05-06T22:44:44.920Z,1715035484.920 [sample_lab:SampleRepeater](IMPORTANT): Priority 1: sample_lab:SampleRepeater:WaitForESP 2024-05-06T22:49:14.211Z,1715035754.211 [CommandExec](IMPORTANT): got command show stack 2024-05-06T22:49:14.211Z,1715035754.211 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T22:49:14.212Z,1715035754.212 [sample_lab](IMPORTANT): Priority 0: sample_lab:A.AbortSample 2024-05-06T22:49:14.212Z,1715035754.212 [sample_lab:SampleRepeater](IMPORTANT): Priority 1: sample_lab:SampleRepeater:WaitForESP 2024-05-06T22:58:01.729Z,1715036281.729 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0) 2024-05-06T22:58:02.133Z,1715036282.133 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0) 2024-05-06T22:58:04.164Z,1715036284.164 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-05-06T22:58:04.560Z,1715036284.560 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,aRNAl_bac) 2024-05-06T22:59:39.902Z,1715036379.902 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@20,aRNAl_bac) 2024-05-06T22:59:47.176Z,1715036387.176 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-05-06T22:59:47.586Z,1715036387.586 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2024-05-06T23:02:34.645Z,1715036554.645 [CommandExec](IMPORTANT): got command show stack 2024-05-06T23:02:34.645Z,1715036554.645 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T23:02:34.645Z,1715036554.645 [sample_lab](IMPORTANT): Priority 0: sample_lab:A.AbortSample 2024-05-06T23:02:34.645Z,1715036554.645 [sample_lab:SampleRepeater](IMPORTANT): Priority 1: sample_lab:SampleRepeater:WaitForESP 2024-05-06T23:04:47.346Z,1715036687.346 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2024-05-06T23:04:47.965Z,1715036687.965 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -1.000000 count 2024-05-06T23:04:59.571Z,1715036699.571 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-05-06T23:04:59.715Z,1715036699.715 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-05-06T23:06:11.783Z,1715036771.783 [ESPComponent](IMPORTANT): ESP has connected as client 2024-05-06T23:06:12.597Z,1715036772.597 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1 2024-05-06T23:06:12.598Z,1715036772.598 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-05-06T23:06:13.410Z,1715036773.410 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2024-05-06T23:06:13.809Z,1715036773.809 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-05-06T23:06:15.838Z,1715036775.838 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2024-05-06T23:06:46.934Z,1715036806.934 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@19,aRNAl_bac) 2024-05-06T23:06:49.762Z,1715036809.762 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2024-05-06T23:06:49.763Z,1715036809.763 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-05-06T23:06:50.173Z,1715036810.173 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-05-06T23:07:37.477Z,1715036857.477 [CBIT](FAULT): WATER ALARM AUX. 2024-05-06T23:07:37.860Z,1715036857.860 [CBIT](FAULT): WATER ALARM AUX. 2024-05-06T23:07:38.280Z,1715036858.280 [CBIT](FAULT): WATER ALARM AUX. 2024-05-06T23:07:38.682Z,1715036858.682 [CBIT](FAULT): WATER ALARM AUX. 2024-05-06T23:07:38.694Z,1715036858.694 [CBIT](CRITICAL): WATER DETECTED IN PRESSURE HULL: AUX. BURNWIRE ACTIVATED 2024-05-06T23:07:38.695Z,1715036858.695 [CBIT](IMPORTANT): Powered off ESP with command: ! echo 0 > /dev/loadA6 2024-05-06T23:07:38.772Z,1715036858.772 [CommandExec](IMPORTANT): got command ! echo 0 > /dev/loadA6 2024-05-06T23:07:39.473Z,1715036859.473 [ESPComponent](IMPORTANT): [sample #3] Stop sampling requested. 2024-05-06T23:07:39.496Z,1715036859.496 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T23:07:39.861Z,1715036859.861 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.stop 2024-05-06T23:07:40.257Z,1715036860.257 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@100.0) 2024-05-06T23:07:50.373Z,1715036870.373 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-05-06T23:07:50.769Z,1715036870.769 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2024-05-06T23:07:54.809Z,1715036874.809 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-05-06T23:08:15.387Z,1715036895.387 [CommandExec](IMPORTANT): got command burn off 2024-05-06T23:08:15.388Z,1715036895.388 [CommandExec](IMPORTANT): Deactivating dropweight wire 2024-05-06T23:09:35.511Z,1715036975.511 [CommandExec](IMPORTANT): got command show stack 2024-05-06T23:09:35.511Z,1715036975.511 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T23:09:35.512Z,1715036975.512 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T23:09:35.512Z,1715036975.512 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:Read_GPS 2024-05-06T23:09:39.918Z,1715036979.918 [CommandExec](IMPORTANT): got command run ./Missions/Insert/Science.tl 2024-05-06T23:09:42.853Z,1715036982.853 [CommandExec](IMPORTANT): Running ./Missions/Insert/Science.tl 2024-05-06T23:09:43.115Z,1715036983.115 [MissionManager](IMPORTANT): Started mission Science 2024-05-06T23:10:08.899Z,1715037008.899 [CommandExec](IMPORTANT): got command show stack 2024-05-06T23:10:08.899Z,1715037008.899 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T23:10:08.900Z,1715037008.900 [Science](IMPORTANT): Priority 0: Science:B 2024-05-06T23:10:08.901Z,1715037008.901 [Science](IMPORTANT): Priority 1: Science:C 2024-05-06T23:10:08.901Z,1715037008.901 [Science](IMPORTANT): Priority 2: Science:E 2024-05-06T23:10:08.901Z,1715037008.901 [Science:OceanCurrent](IMPORTANT): Priority 3: Science:OceanCurrent:A. 2024-05-06T23:10:41.755Z,1715037041.755 [CommandExec](IMPORTANT): got command stop 2024-05-06T23:10:41.755Z,1715037041.755 [CommandExec](IMPORTANT): Scheduling is paused 2024-05-06T23:10:42.485Z,1715037042.485 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T23:12:50.547Z,1715037170.547 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2024-05-06T23:13:17.319Z,1715037197.319 [CommandExec](IMPORTANT): got command show stack 2024-05-06T23:13:17.319Z,1715037197.319 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T23:13:17.319Z,1715037197.319 [Default](IMPORTANT): Priority 0: Default:B.GoToSurface 2024-05-06T23:13:17.321Z,1715037197.321 [Default:CheckIn](IMPORTANT): Priority 1: Default:CheckIn:Read_GPS 2024-05-06T23:13:39.011Z,1715037219.011 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T23:14:03.254Z,1715037243.254 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:14:10.531Z,1715037250.531 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T23:14:18.202Z,1715037258.202 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:14:29.523Z,1715037269.523 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T23:14:29.548Z,1715037269.548 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2024-05-06T23:14:30.102Z,1715037270.102 [CommandExec](FAULT): Scheduling is paused 2024-05-06T23:14:33.150Z,1715037273.150 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:14:48.098Z,1715037288.098 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:15:03.051Z,1715037303.051 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:15:03.126Z,1715037303.126 [CBIT](CRITICAL): Data Fault in component: AHRS_M2 2024-05-06T23:17:03.034Z,1715037423.034 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T23:17:47.460Z,1715037467.460 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T23:18:06.449Z,1715037486.449 [DVL_micro](FAULT): Failed to initialize within timeout. 2024-05-06T23:18:21.396Z,1715037501.396 [Sonardyne_Nano](FAULT): Failed hardware self-test:>CKHW:FAIL;LBK 2024-05-06T23:18:23.011Z,1715037503.011 [DVL_micro](FAULT): DVL failed to acquire valid data within timeout. 2024-05-06T23:18:23.076Z,1715037503.076 [CBIT](CRITICAL): Data Fault in component: DVL_micro 2024-05-06T23:18:26.251Z,1715037506.251 [Sonardyne_Nano](IMPORTANT): Found beacon with acoustic ID 2207 2024-05-06T23:19:04.231Z,1715037544.231 [DeadReckonUsingMultipleVelocitySources](FAULT): Unable to read the rotation from vehicle frame to navigation frame for more than 120 seconds. 2024-05-06T23:20:20.996Z,1715037620.996 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:20:35.942Z,1715037635.942 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:20:50.890Z,1715037650.890 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:21:05.838Z,1715037665.838 [AHRS_M2](FAULT): Failed to acquire valid data within timeout. 2024-05-06T23:22:07.274Z,1715037727.274 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T23:22:07.274Z,1715037727.274 [CommandExec](IMPORTANT): Failed components: 2024-05-06T23:22:07.275Z,1715037727.275 [CommandExec](IMPORTANT): No failed Components. 2024-05-06T23:22:08.113Z,1715037728.113 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.419807 min 2024-05-06T23:22:08.522Z,1715037728.522 [CommandExec](IMPORTANT): got command 2024-05-06T23:22:08.522Z,1715037728.522 [CommandExec](FAULT): Incomplete syntax. Try: help 2024-05-06T23:22:09.664Z,1715037729.664 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-05-06T23:22:12.163Z,1715037732.163 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T23:22:12.163Z,1715037732.163 [CommandExec](IMPORTANT): Failed components: 2024-05-06T23:22:12.163Z,1715037732.163 [CommandExec](IMPORTANT): No failed Components. 2024-05-06T23:22:24.357Z,1715037744.357 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1715037732 second 2024-05-06T23:27:53.872Z,1715038073.872 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T23:32:44.845Z,1715038364.845 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.032552 min 2024-05-06T23:37:49.032Z,1715038669.032 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T23:43:21.167Z,1715039001.167 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.638647 min 2024-05-06T23:43:22.768Z,1715039002.768 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-05-06T23:43:29.667Z,1715039009.667 [CommandExec](IMPORTANT): got command failComponent 2024-05-06T23:43:29.667Z,1715039009.667 [CommandExec](IMPORTANT): Failed components: 2024-05-06T23:43:29.667Z,1715039009.667 [CommandExec](IMPORTANT): No failed Components. 2024-05-06T23:45:10.995Z,1715039110.995 [DataOverHttps](IMPORTANT): SBD MTMSN=20240506T234509 2024-05-06T23:45:18.608Z,1715039118.608 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.tl 2024-05-06T23:45:18.800Z,1715039118.800 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.tl id=sample_lab 2024-05-06T23:45:23.022Z,1715039123.022 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -1 count 2024-05-06T23:45:23.023Z,1715039123.023 [CommandExec](IMPORTANT): got command set sample_lab.NumberOfSamples 2 count 2024-05-06T23:45:23.023Z,1715039123.023 [CommandExec](IMPORTANT): got command run 2024-05-06T23:45:23.025Z,1715039123.025 [CommandExec](IMPORTANT): Running 2024-05-06T23:45:23.179Z,1715039123.179 [MissionManager](IMPORTANT): Started mission sample_lab 2024-05-06T23:45:23.181Z,1715039123.181 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -1.000000 count 2024-05-06T23:45:35.002Z,1715039135.002 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-05-06T23:45:35.144Z,1715039135.144 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-05-06T23:46:40.319Z,1715039200.319 [ESPComponent](IMPORTANT): ESP has connected as client 2024-05-06T23:46:41.159Z,1715039201.159 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -1 2024-05-06T23:46:41.160Z,1715039201.160 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-05-06T23:46:41.935Z,1715039201.935 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2024-05-06T23:46:42.344Z,1715039202.344 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-05-06T23:46:44.379Z,1715039204.379 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2024-05-06T23:47:15.479Z,1715039235.479 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@20,aRNAl_bac) 2024-05-06T23:47:18.308Z,1715039238.308 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2024-05-06T23:47:18.310Z,1715039238.310 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-05-06T23:47:18.709Z,1715039238.709 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-05-06T23:49:04.755Z,1715039344.755 [CommandExec](IMPORTANT): got command gfscan 2024-05-06T23:49:04.994Z,1715039344.994 [CBIT](IMPORTANT): Beginning ground fault scan 2024-05-06T23:49:06.968Z,1715039346.968 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-05-06T23:49:15.899Z,1715039355.899 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.006284 CHAN A1 (24V): -0.002195 CHAN A2 (12V): -0.000445 CHAN A3 (5V): 0.000501 CHAN B0 (3.3V): 0.000580 CHAN B1 (3.15aV): 0.000731 CHAN B2 (3.15bV): 0.000129 CHAN B3 (GND): 0.000838 OPEN: 0.004823 Full Scale: +/- 1 mA 2024-05-06T23:50:06.785Z,1715039406.785 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2024-05-06T23:50:07.186Z,1715039407.186 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-05-06T23:55:35.115Z,1715039735.115 [CommandExec](IMPORTANT): got command stop 2024-05-06T23:55:35.115Z,1715039735.115 [CommandExec](IMPORTANT): Scheduling is paused 2024-05-06T23:55:35.648Z,1715039735.648 [ESPComponent](IMPORTANT): [sample #4] Stop sampling requested. 2024-05-06T23:55:35.657Z,1715039735.657 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T23:55:36.061Z,1715039736.061 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.pauseFiltering 2024-05-06T23:55:38.485Z,1715039738.485 [CommandExec](IMPORTANT): got command run ./Missions/Insert/Science.tl 2024-05-06T23:55:41.132Z,1715039741.132 [CommandExec](IMPORTANT): Running ./Missions/Insert/Science.tl 2024-05-06T23:55:41.331Z,1715039741.331 [MissionManager](IMPORTANT): Started mission Science 2024-05-06T23:55:42.108Z,1715039742.108 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@273.4) 2024-05-06T23:55:42.532Z,1715039742.532 [ESPComponent](IMPORTANT): In S_PAUSED received result: 273.4 2024-05-06T23:55:42.915Z,1715039742.915 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@273.4) 2024-05-06T23:55:44.938Z,1715039744.938 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-05-06T23:55:45.347Z,1715039745.347 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,aRNAl_bac) 2024-05-06T23:56:20.759Z,1715039780.759 [CommandExec](IMPORTANT): got command burn off 2024-05-06T23:56:20.759Z,1715039780.759 [CommandExec](IMPORTANT): Deactivating dropweight wire 2024-05-06T23:56:43.808Z,1715039803.808 [CommandExec](IMPORTANT): got command show stack 2024-05-06T23:56:43.808Z,1715039803.808 [CommandExec](IMPORTANT): Behavior Stack: 2024-05-06T23:56:43.809Z,1715039803.809 [Science](IMPORTANT): Priority 0: Science:B 2024-05-06T23:56:43.810Z,1715039803.810 [Science](IMPORTANT): Priority 1: Science:C 2024-05-06T23:56:43.810Z,1715039803.810 [Science](IMPORTANT): Priority 2: Science:E 2024-05-06T23:56:43.810Z,1715039803.810 [Science:OceanCurrent](IMPORTANT): Priority 3: Science:OceanCurrent:A. 2024-05-06T23:56:44.957Z,1715039804.957 [CommandExec](IMPORTANT): got command stop 2024-05-06T23:56:45.547Z,1715039805.547 [MissionManager](IMPORTANT): Started mission Default 2024-05-06T23:57:00.892Z,1715039820.892 [NAL9602](FAULT): GPS failed to acquire within timeout. 2024-05-06T23:57:20.691Z,1715039840.691 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@20,aRNAl_bac) 2024-05-06T23:57:27.962Z,1715039847.962 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-05-06T23:57:28.370Z,1715039848.370 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2024-05-06T23:58:30.880Z,1715039910.880 [CommandExec](IMPORTANT): got command restart application