2023-04-18T15:29:50.769Z,1681831790.769 [CommandExec](IMPORTANT): got command restart logs 2023-04-18T15:31:02.502Z,1681831862.502 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153102 2023-04-18T15:31:10.065Z,1681831870.065 [CommandExec](IMPORTANT): got command schedule asap "load Science/front_sampling.xml;set FrontSampling.NumSamplers 5 count;set FrontSampling.MissionTimeout 18 h;set FrontSampling.MaxDepth 60 m;set FrontSampling.RepeatTransects 2 count;set FrontSampling.FrontAutonomousDetection 0 bool" 3dkey 1 7.000000 2023-04-18T15:31:10.065Z,1681831870.065 [CommandExec](IMPORTANT): Scheduling command #1 of 7 with id=3dkey 2023-04-18T15:31:10.066Z,1681831870.066 [CommandExec](IMPORTANT): Scheduled #17 (#1 of 7 with id='3dkey'): "load Science/front_sampling.xml;set FrontSampling.NumSamplers 5 count;set FrontSampling.MissionTimeout 18 h;set FrontSampling.MaxDepth 60 m;set FrontSampling.RepeatTransects 2 count;set FrontSampling.FrontAutonomousDetection 0 bool", ASAP 2023-04-18T15:31:10.982Z,1681831870.982 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153110 2023-04-18T15:31:18.397Z,1681831878.397 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.LonFrontMinDesignated -121.978 degree;set FrontSampling.LonFrontMaxDesignated -121.977 degree;set FrontSampling.Lat1 36.863 degree;set FrontSampling.Lon1 -121.975 degree;set FrontSampling.Lat2 36.863 degree" 3dkey 2 7.000000 2023-04-18T15:31:18.397Z,1681831878.397 [CommandExec](IMPORTANT): Scheduling command #2 of 7 with id=3dkey 2023-04-18T15:31:18.398Z,1681831878.398 [CommandExec](IMPORTANT): Scheduled #18 (#2 of 7 with id='3dkey'): "set FrontSampling.LonFrontMinDesignated -121.978 degree;set FrontSampling.LonFrontMaxDesignated -121.977 degree;set FrontSampling.Lat1 36.863 degree;set FrontSampling.Lon1 -121.975 degree;set FrontSampling.Lat2 36.863 degree", ASAP 2023-04-18T15:31:19.386Z,1681831879.386 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153119 2023-04-18T15:31:26.956Z,1681831886.956 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.Lon2 -122.06 degree;set FrontSampling.LatUpwellingEnd 36.863 degree;set FrontSampling.LonUpwellingEnd -122.06 degree;set FrontSampling.DeltaLatUpwellingSide1 0 degree" 3dkey 3 7.000000 2023-04-18T15:31:26.956Z,1681831886.956 [CommandExec](IMPORTANT): Scheduling command #3 of 7 with id=3dkey 2023-04-18T15:31:26.957Z,1681831886.957 [CommandExec](IMPORTANT): Scheduled #19 (#3 of 7 with id='3dkey'): "set FrontSampling.Lon2 -122.06 degree;set FrontSampling.LatUpwellingEnd 36.863 degree;set FrontSampling.LonUpwellingEnd -122.06 degree;set FrontSampling.DeltaLatUpwellingSide1 0 degree", ASAP 2023-04-18T15:31:27.923Z,1681831887.923 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153127 2023-04-18T15:31:35.393Z,1681831895.393 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.DeltaLonUpwellingSide1 -0.017 degree;set FrontSampling.DeltaLatUpwellingSide2 0 degree;set FrontSampling.DeltaLonUpwellingSide2 -0.034 degree;set FrontSampling.DeltaLatUpwellingSide3 0 degree" 3dkey 4 7.000000 2023-04-18T15:31:35.393Z,1681831895.393 [CommandExec](IMPORTANT): Scheduling command #4 of 7 with id=3dkey 2023-04-18T15:31:35.394Z,1681831895.394 [CommandExec](IMPORTANT): Scheduled #20 (#4 of 7 with id='3dkey'): "set FrontSampling.DeltaLonUpwellingSide1 -0.017 degree;set FrontSampling.DeltaLatUpwellingSide2 0 degree;set FrontSampling.DeltaLonUpwellingSide2 -0.034 degree;set FrontSampling.DeltaLatUpwellingSide3 0 degree", ASAP 2023-04-18T15:31:36.370Z,1681831896.370 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153136 2023-04-18T15:31:43.980Z,1681831903.980 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.DeltaLonUpwellingSide3 -0.051 degree;set FrontSampling.DeltaLatUpwellingSide4 0 degree;set FrontSampling.DeltaLonUpwellingSide4 -0.068 degree;set FrontSampling.LatStratifiedEnd 36.863 degree" 3dkey 5 7.000000 2023-04-18T15:31:43.981Z,1681831903.981 [CommandExec](IMPORTANT): Scheduling command #5 of 7 with id=3dkey 2023-04-18T15:31:43.982Z,1681831903.982 [CommandExec](IMPORTANT): Scheduled #21 (#5 of 7 with id='3dkey'): "set FrontSampling.DeltaLonUpwellingSide3 -0.051 degree;set FrontSampling.DeltaLatUpwellingSide4 0 degree;set FrontSampling.DeltaLonUpwellingSide4 -0.068 degree;set FrontSampling.LatStratifiedEnd 36.863 degree", ASAP 2023-04-18T15:31:44.906Z,1681831904.906 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153144 2023-04-18T15:31:52.416Z,1681831912.416 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.LonStratifiedEnd -121.97 degree;set FrontSampling.SpeedTransit 0.8 m/s;set FrontSampling.SpeedFindChlPeakAndSample 0.8 m/s;set FrontSampling.ShallowBoundChl 3 m;set FrontSampling.DeepBoundChl 20 m" 3dkey 6 7.000000 2023-04-18T15:31:52.417Z,1681831912.417 [CommandExec](IMPORTANT): Scheduling command #6 of 7 with id=3dkey 2023-04-18T15:31:52.418Z,1681831912.418 [CommandExec](IMPORTANT): Scheduled #22 (#6 of 7 with id='3dkey'): "set FrontSampling.LonStratifiedEnd -121.97 degree;set FrontSampling.SpeedTransit 0.8 m/s;set FrontSampling.SpeedFindChlPeakAndSample 0.8 m/s;set FrontSampling.ShallowBoundChl 3 m;set FrontSampling.DeepBoundChl 20 m", ASAP 2023-04-18T15:31:53.401Z,1681831913.401 [DataOverHttps](IMPORTANT): SBD MTMSN=20230418T153153 2023-04-18T15:32:00.947Z,1681831920.947 [CommandExec](IMPORTANT): got command schedule asap "set FrontSampling.YoYoMinDepth 2 m;set FrontSampling.YoYoMaxDepth 50 m;set FrontSampling.WaitDepthUndulation 5 min;run" 3dkey 7 7.000000 2023-04-18T15:32:00.948Z,1681831920.948 [CommandExec](IMPORTANT): Scheduling command #7 of 7 with id=3dkey 2023-04-18T15:32:00.949Z,1681831920.949 [CommandExec](IMPORTANT): Scheduled #23 (#7 of 7 with id='3dkey'): "set FrontSampling.YoYoMinDepth 2 m;set FrontSampling.YoYoMaxDepth 50 m;set FrontSampling.WaitDepthUndulation 5 min;run", ASAP 2023-04-18T15:32:01.090Z,1681831921.090 [CommandExec](IMPORTANT): got command load ./Missions/Science/front_sampling.xml 2023-04-18T15:32:10.565Z,1681831930.565 [CommandExec](IMPORTANT): Loaded ./Missions/Science/front_sampling.xml 2023-04-18T15:32:39.580Z,1681831959.580 [CommandExec](IMPORTANT): got command set FrontSampling.NumSamplers 5 count 2023-04-18T15:32:39.581Z,1681831959.581 [CommandExec](IMPORTANT): got command set FrontSampling.MissionTimeout 18 hour 2023-04-18T15:32:39.582Z,1681831959.582 [CommandExec](IMPORTANT): got command set FrontSampling.MaxDepth 60 meter 2023-04-18T15:32:39.582Z,1681831959.582 [CommandExec](IMPORTANT): got command set FrontSampling.RepeatTransects 2 count 2023-04-18T15:32:39.583Z,1681831959.583 [CommandExec](IMPORTANT): got command set FrontSampling.FrontAutonomousDetection 0 bool 2023-04-18T15:33:01.252Z,1681831981.252 [CommandExec](IMPORTANT): got command set FrontSampling.LonFrontMinDesignated -121.978 degree 2023-04-18T15:33:01.253Z,1681831981.253 [CommandExec](IMPORTANT): got command set FrontSampling.LonFrontMaxDesignated -121.977 degree 2023-04-18T15:33:01.253Z,1681831981.253 [CommandExec](IMPORTANT): got command set FrontSampling.Lat1 36.863 degree 2023-04-18T15:33:01.254Z,1681831981.254 [CommandExec](IMPORTANT): got command set FrontSampling.Lon1 -121.975 degree 2023-04-18T15:33:01.255Z,1681831981.255 [CommandExec](IMPORTANT): got command set FrontSampling.Lat2 36.863 degree 2023-04-18T15:33:24.299Z,1681832004.299 [CommandExec](IMPORTANT): got command set FrontSampling.Lon2 -122.06 degree 2023-04-18T15:33:24.300Z,1681832004.300 [CommandExec](IMPORTANT): got command set FrontSampling.LatUpwellingEnd 36.863 degree 2023-04-18T15:33:24.301Z,1681832004.301 [CommandExec](IMPORTANT): got command set FrontSampling.LonUpwellingEnd -122.06 degree 2023-04-18T15:33:24.301Z,1681832004.301 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLatUpwellingSide1 0 degree 2023-04-18T15:33:52.934Z,1681832032.934 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLonUpwellingSide1 -0.017 degree 2023-04-18T15:33:52.934Z,1681832032.934 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLatUpwellingSide2 0 degree 2023-04-18T15:33:52.935Z,1681832032.935 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLonUpwellingSide2 -0.034 degree 2023-04-18T15:33:52.936Z,1681832032.936 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLatUpwellingSide3 0 degree 2023-04-18T15:34:01.718Z,1681832041.718 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.861058 min 2023-04-18T15:34:03.310Z,1681832043.310 [NAL9602](FAULT): GPS failed to acquire within timeout. 2023-04-18T15:34:16.974Z,1681832056.974 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLonUpwellingSide3 -0.051 degree 2023-04-18T15:34:16.975Z,1681832056.975 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLatUpwellingSide4 0 degree 2023-04-18T15:34:16.976Z,1681832056.976 [CommandExec](IMPORTANT): got command set FrontSampling.DeltaLonUpwellingSide4 -0.068 degree 2023-04-18T15:34:16.988Z,1681832056.988 [CommandExec](IMPORTANT): got command set FrontSampling.LatStratifiedEnd 36.863 degree 2023-04-18T15:34:46.702Z,1681832086.702 [CommandExec](IMPORTANT): got command set FrontSampling.LonStratifiedEnd -121.97 degree 2023-04-18T15:34:46.703Z,1681832086.703 [CommandExec](IMPORTANT): got command set FrontSampling.SpeedTransit 0.8 meter_per_second 2023-04-18T15:34:46.703Z,1681832086.703 [CommandExec](IMPORTANT): got command set FrontSampling.SpeedFindChlPeakAndSample 0.8 meter_per_second 2023-04-18T15:34:46.705Z,1681832086.705 [CommandExec](IMPORTANT): got command set FrontSampling.ShallowBoundChl 3 meter 2023-04-18T15:34:46.706Z,1681832086.706 [CommandExec](IMPORTANT): got command set FrontSampling.DeepBoundChl 20 meter 2023-04-18T15:35:05.243Z,1681832105.243 [CommandExec](IMPORTANT): got command set FrontSampling.YoYoMinDepth 2 meter 2023-04-18T15:35:05.243Z,1681832105.243 [CommandExec](IMPORTANT): got command set FrontSampling.YoYoMaxDepth 50 meter 2023-04-18T15:35:05.245Z,1681832105.245 [CommandExec](IMPORTANT): got command set FrontSampling.WaitDepthUndulation 5 minute 2023-04-18T15:35:05.245Z,1681832105.245 [CommandExec](IMPORTANT): got command run 2023-04-18T15:35:05.250Z,1681832105.250 [CommandExec](IMPORTANT): Running 2023-04-18T15:35:05.535Z,1681832105.535 [MissionManager](IMPORTANT): Started mission FrontSampling 2023-04-18T15:39:48.733Z,1681832388.733 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T15:41:00.813Z,1681832460.813 [FrontSampling:BackAndForth:Waypoints:SampleFrontFirstPass:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862999,-121.974998 2023-04-18T15:46:01.835Z,1681832761.835 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T15:54:42.392Z,1681833282.392 [RudderServo](FAULT): Invalid EZ Servo response:"" 2023-04-18T16:47:32.329Z,1681836452.329 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-18T17:13:14.544Z,1681837994.544 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-04-18T17:25:43.895Z,1681838743.895 [FrontSampling:BackAndForth:Waypoints:SampleFrontFirstPass:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 36.862999,-121.974998 2023-04-18T17:25:43.897Z,1681838743.897 [FrontSampling:BackAndForth:Waypoints:SampleFrontFirstPass:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862999,-122.059998 2023-04-18T17:37:03.428Z,1681839423.428 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T17:37:56.796Z,1681839476.796 [FrontSampling:CrossFrontFromStratifiedToUpwelling:P](IMPORTANT): Crossed the front from stratified to upwelling. VTHI = 0.638635 degC . VSHI = 0.170368 psu . Front latitude, longitude = 36.862768 arcdeg , -121.977743 arcdeg . Lat1, Lon1 = 36.862768 arcdeg , -121.994743 arcdeg . Lat2, Lon2 = 36.862768 arcdeg , -122.011743 arcdeg . Lat3, Lon3 = 36.862768 arcdeg , -122.028743 arcdeg . Lat4, Lon4 = 36.862768 arcdeg , -122.045743 arcdeg . Lat5, Lon5 = 36.862999 arcdeg , -122.059998 arcdeg . Lat6, Lon6 = 36.862999 arcdeg , -121.970001 arcdeg . Sample count = 1.000000 count . 2023-04-18T17:38:00.009Z,1681839480.009 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt5:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862999,-122.059998 2023-04-18T17:38:01.219Z,1681839481.219 [FrontSampling:BackAndForth:Waypoints:SampleFrontFirstPass:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862768,-121.994743 2023-04-18T17:38:01.627Z,1681839481.627 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt5:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862999,-122.059998 2023-04-18T17:38:02.025Z,1681839482.025 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862768,-121.994743 2023-04-18T17:38:03.655Z,1681839483.655 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoDown:B](IMPORTANT): Go down from 0.246890 m 2023-04-18T17:39:45.212Z,1681839585.212 [Radio_Surface](FAULT): LCB fault: Hardware Overcurrent Shutdown. 2023-04-18T17:41:22.027Z,1681839682.027 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoUp:B](IMPORTANT): Go up from 23.136301 m 2023-04-18T17:42:52.502Z,1681839772.502 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T17:43:17.954Z,1681839797.954 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:SetChlPeakDepth:J](IMPORTANT): Set peak-chl depth. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlSet = 20.245718 m , 0.278400 ug/l , 7.662421 m , 9.573477 ug/l 2023-04-18T17:43:21.603Z,1681839801.603 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Latitude, longitude = 36.862683 arcdeg , -121.977833 arcdeg . Target depth = 7.662421 m . 2023-04-18T17:43:22.011Z,1681839802.011 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:F](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2023-04-18T17:46:59.403Z,1681840019.403 [FrontSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 7.636260 m , settling for 5.000000 min 2023-04-18T17:52:12.140Z,1681840332.140 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-18T17:52:12.272Z,1681840332.272 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-18T17:53:05.030Z,1681840385.030 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-18T17:53:05.842Z,1681840385.842 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2023-04-18T17:53:05.843Z,1681840385.843 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-18T17:53:06.289Z,1681840386.289 [FrontSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 7.761589 m in 10.914667 degC water with 10.513080 ug/l chlorophyll fluorescence. 2023-04-18T17:53:06.654Z,1681840386.654 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-18T17:53:07.045Z,1681840387.045 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-18T17:53:09.066Z,1681840389.066 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-18T17:53:40.987Z,1681840420.987 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@40,archive_bac) 2023-04-18T17:53:43.006Z,1681840423.006 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2023-04-18T17:53:43.412Z,1681840423.412 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-18T17:53:43.814Z,1681840423.814 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-18T17:55:00.280Z,1681840500.280 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-18T17:55:11.211Z,1681840511.211 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.002945 CHAN A1 (24V): -0.001561 CHAN A2 (12V): -0.000533 CHAN A3 (5V): -0.033116 CHAN B0 (3.3V): 0.000024 CHAN B1 (3.15aV): -0.000011 CHAN B2 (3.15bV): 0.020942 CHAN B3 (GND): -0.000186 OPEN: 0.007481 Full Scale: +/- 1 mA 2023-04-18T17:56:27.026Z,1681840587.026 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2023-04-18T17:56:27.434Z,1681840587.434 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-18T18:56:00.847Z,1681844160.847 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@938.6) 2023-04-18T18:56:01.246Z,1681844161.246 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@938.6) 2023-04-18T18:56:03.265Z,1681844163.265 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-18T18:56:03.686Z,1681844163.686 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,archive_bac) 2023-04-18T19:08:07.267Z,1681844887.267 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@40,archive_bac) 2023-04-18T19:08:17.755Z,1681844897.755 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-18T19:08:18.282Z,1681844898.282 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2023-04-18T19:13:17.922Z,1681845197.922 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2023-04-18T19:13:18.758Z,1681845198.758 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2023-04-18T19:19:09.426Z,1681845549.426 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T19:50:09.710Z,1681847409.710 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T20:57:05.302Z,1681851425.302 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 36.862768,-121.994743 2023-04-18T20:59:40.262Z,1681851580.262 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoDown:B](IMPORTANT): Go down from 0.242136 m 2023-04-18T21:04:12.120Z,1681851852.120 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoUp:B](IMPORTANT): Go up from 23.131117 m 2023-04-18T21:06:08.505Z,1681851968.505 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:SetChlPeakDepth:J](IMPORTANT): Set peak-chl depth. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlSet = 20.291958 m , 0.255200 ug/l , 3.849845 m , 11.606346 ug/l 2023-04-18T21:06:10.908Z,1681851970.908 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Latitude, longitude = 36.862204 arcdeg , -121.994847 arcdeg . Target depth = 3.849845 m . 2023-04-18T21:06:11.322Z,1681851971.322 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:F](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2023-04-18T21:10:08.466Z,1681852208.466 [FrontSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.829242 m , settling for 5.000000 min 2023-04-18T21:15:21.260Z,1681852521.260 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-18T21:15:21.419Z,1681852521.419 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-18T21:16:22.617Z,1681852582.617 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-18T21:16:23.437Z,1681852583.437 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2023-04-18T21:16:23.437Z,1681852583.437 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-18T21:16:23.865Z,1681852583.865 [FrontSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.532336 m in 11.278315 degC water with 10.880800 ug/l chlorophyll fluorescence. 2023-04-18T21:16:24.234Z,1681852584.234 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-18T21:16:24.637Z,1681852584.637 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-18T21:16:26.657Z,1681852586.657 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-18T21:16:58.582Z,1681852618.582 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@39,archive_bac) 2023-04-18T21:17:01.410Z,1681852621.410 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2023-04-18T21:17:01.411Z,1681852621.411 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-18T21:17:01.806Z,1681852621.806 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-18T21:19:45.830Z,1681852785.830 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2023-04-18T21:19:46.240Z,1681852786.240 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-18T22:07:44.377Z,1681855664.377 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@758.5) 2023-04-18T22:07:44.786Z,1681855664.786 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@758.5) 2023-04-18T22:07:46.808Z,1681855666.808 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-18T22:07:47.203Z,1681855667.203 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,archive_bac) 2023-04-18T22:19:50.784Z,1681856390.784 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@39,archive_bac) 2023-04-18T22:20:00.886Z,1681856400.886 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-18T22:20:01.291Z,1681856401.291 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2023-04-18T22:25:01.065Z,1681856701.065 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2023-04-18T22:25:01.898Z,1681856701.898 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed. 2023-04-18T22:25:06.341Z,1681856706.341 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt2:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862768,-122.011743 2023-04-18T22:38:11.463Z,1681857491.463 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-18T23:31:21.910Z,1681860681.910 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt2:A.Waypoint](IMPORTANT): Reached waypoint: 36.862768,-122.011743 2023-04-18T23:36:39.690Z,1681860999.690 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoDown:B](IMPORTANT): Go down from 0.280600 m 2023-04-18T23:41:53.785Z,1681861313.785 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoUp:B](IMPORTANT): Go up from 23.170441 m 2023-04-18T23:43:47.399Z,1681861427.399 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:SetChlPeakDepth:J](IMPORTANT): Set peak-chl depth. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlSet = 20.314430 m , 0.510400 ug/l , 4.065703 m , 5.683369 ug/l 2023-04-18T23:43:49.754Z,1681861429.754 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Latitude, longitude = 36.862293 arcdeg , -122.014334 arcdeg . Target depth = 4.065703 m . 2023-04-18T23:43:50.143Z,1681861430.143 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:F](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2023-04-18T23:44:58.844Z,1681861498.844 [FrontSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.971861 m , settling for 5.000000 min 2023-04-18T23:50:11.625Z,1681861811.625 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-18T23:50:11.744Z,1681861811.744 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-18T23:51:00.838Z,1681861860.838 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-18T23:51:01.653Z,1681861861.653 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6 2023-04-18T23:51:01.654Z,1681861861.654 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-18T23:51:02.100Z,1681861862.100 [FrontSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.286054 m in 10.266565 degC water with 5.508840 ug/l chlorophyll fluorescence. 2023-04-18T23:51:02.453Z,1681861862.453 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-18T23:51:02.853Z,1681861862.853 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-18T23:51:04.879Z,1681861864.879 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-18T23:51:36.803Z,1681861896.803 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@38,archive_bac) 2023-04-18T23:51:39.213Z,1681861899.213 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2023-04-18T23:51:39.214Z,1681861899.214 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-18T23:51:39.616Z,1681861899.616 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-18T23:54:23.651Z,1681862063.651 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2023-04-18T23:54:24.047Z,1681862064.047 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-18T23:55:11.469Z,1681862111.469 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-18T23:55:22.702Z,1681862122.702 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002649 CHAN A1 (24V): -0.001206 CHAN A2 (12V): -0.000343 CHAN A3 (5V): 0.010995 CHAN B0 (3.3V): 0.000139 CHAN B1 (3.15aV): 0.000079 CHAN B2 (3.15bV): 0.000656 CHAN B3 (GND): -0.000304 OPEN: 0.007504 Full Scale: +/- 1 mA 2023-04-19T00:47:51.065Z,1681865271.065 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-19T00:52:28.622Z,1681865548.622 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2023-04-19T00:52:29.018Z,1681865549.018 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2023-04-19T00:52:31.037Z,1681865551.037 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-19T00:52:31.446Z,1681865551.446 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,archive_bac) 2023-04-19T01:04:35.015Z,1681866275.015 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@38,archive_bac) 2023-04-19T01:04:45.129Z,1681866285.129 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-19T01:04:45.559Z,1681866285.559 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2023-04-19T01:09:45.302Z,1681866585.302 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2023-04-19T01:09:46.200Z,1681866586.200 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed. 2023-04-19T01:09:50.587Z,1681866590.587 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt3:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862768,-122.028743 2023-04-19T02:01:21.124Z,1681869681.124 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt3:A.Waypoint](IMPORTANT): Reached waypoint: 36.862768,-122.028743 2023-04-19T02:09:21.523Z,1681870161.523 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoDown:B](IMPORTANT): Go down from 0.247322 m 2023-04-19T02:13:19.864Z,1681870399.864 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoUp:B](IMPORTANT): Go up from 23.169579 m 2023-04-19T02:15:08.535Z,1681870508.535 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:SetChlPeakDepth:J](IMPORTANT): Set peak-chl depth. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlSet = 20.292822 m , 0.672800 ug/l , 4.288235 m , 5.601572 ug/l 2023-04-19T02:15:10.970Z,1681870510.970 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 4.000000 count . Latitude, longitude = 36.862529 arcdeg , -122.030803 arcdeg . Target depth = 4.288235 m . 2023-04-19T02:15:11.390Z,1681870511.390 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:F](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2023-04-19T02:16:22.479Z,1681870582.479 [FrontSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.174119 m , settling for 5.000000 min 2023-04-19T02:21:35.264Z,1681870895.264 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-19T02:21:35.417Z,1681870895.417 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-19T02:22:28.525Z,1681870948.525 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-19T02:22:29.345Z,1681870949.345 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6 2023-04-19T02:22:29.346Z,1681870949.346 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-19T02:22:29.803Z,1681870949.803 [FrontSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.175416 m in 10.102594 degC water with 5.289600 ug/l chlorophyll fluorescence. 2023-04-19T02:22:30.140Z,1681870950.140 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-19T02:22:30.550Z,1681870950.550 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-19T02:22:32.566Z,1681870952.566 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-19T02:23:04.490Z,1681870984.490 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@37,archive_bac) 2023-04-19T02:23:07.313Z,1681870987.313 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2023-04-19T02:23:07.314Z,1681870987.314 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-19T02:23:07.722Z,1681870987.722 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-19T02:25:49.730Z,1681871149.730 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2023-04-19T02:25:50.147Z,1681871150.147 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-19T03:20:19.333Z,1681874419.333 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0) 2023-04-19T03:20:19.737Z,1681874419.737 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0) 2023-04-19T03:20:21.758Z,1681874421.758 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-19T03:20:22.175Z,1681874422.175 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,archive_bac) 2023-04-19T03:32:25.738Z,1681875145.738 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@37,archive_bac) 2023-04-19T03:32:35.847Z,1681875155.847 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-19T03:32:36.311Z,1681875156.311 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2023-04-19T03:37:36.005Z,1681875456.005 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2023-04-19T03:37:36.850Z,1681875456.850 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 4.000000 count completed. 2023-04-19T03:37:41.306Z,1681875461.306 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt4:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862768,-122.045743 2023-04-19T03:44:58.857Z,1681875898.857 [FrontSampling:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout 2023-04-19T04:25:06.853Z,1681878306.853 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt4:A.Waypoint](IMPORTANT): Reached waypoint: 36.862768,-122.045743 2023-04-19T04:30:40.507Z,1681878640.507 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoDown:B](IMPORTANT): Go down from 0.380003 m 2023-04-19T04:35:20.499Z,1681878920.499 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:LogGoUp:B](IMPORTANT): Go up from 23.104322 m 2023-04-19T04:37:09.189Z,1681879029.189 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:SetChlPeakDepth:J](IMPORTANT): Set peak-chl depth. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlSet = 20.338196 m , 0.232000 ug/l , 3.309875 m , 4.133014 ug/l 2023-04-19T04:37:11.594Z,1681879031.594 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 5.000000 count . Latitude, longitude = 36.862799 arcdeg , -122.047197 arcdeg . Target depth = 3.309875 m . 2023-04-19T04:37:12.013Z,1681879032.013 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:F](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2023-04-19T04:38:32.435Z,1681879112.435 [FrontSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.162823 m , settling for 5.000000 min 2023-04-19T04:43:45.184Z,1681879425.184 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-19T04:43:45.351Z,1681879425.351 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-19T04:44:34.838Z,1681879474.838 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-19T04:44:35.653Z,1681879475.653 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2023-04-19T04:44:35.654Z,1681879475.654 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-19T04:44:36.076Z,1681879476.076 [FrontSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.595434 m in 9.920771 degC water with 3.886000 ug/l chlorophyll fluorescence. 2023-04-19T04:44:36.458Z,1681879476.458 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-19T04:44:36.854Z,1681879476.854 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-19T04:44:38.869Z,1681879478.869 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-19T04:45:10.785Z,1681879510.785 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@36,archive_bac) 2023-04-19T04:45:13.625Z,1681879513.625 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2023-04-19T04:45:13.626Z,1681879513.626 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-19T04:45:14.017Z,1681879514.017 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-19T04:47:57.254Z,1681879677.254 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2023-04-19T04:47:57.637Z,1681879677.637 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-19T05:43:23.415Z,1681883003.415 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2023-04-19T05:43:23.817Z,1681883003.817 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2023-04-19T05:43:25.839Z,1681883005.839 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-19T05:43:26.245Z,1681883006.245 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@36,archive_bac) 2023-04-19T05:55:22.739Z,1681883722.739 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-19T05:55:29.818Z,1681883729.818 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@36,archive_bac) 2023-04-19T05:55:33.583Z,1681883733.583 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001335 CHAN A1 (24V): -0.001006 CHAN A2 (12V): -0.000040 CHAN A3 (5V): -0.001366 CHAN B0 (3.3V): -0.000400 CHAN B1 (3.15aV): 0.000331 CHAN B2 (3.15bV): 0.000301 CHAN B3 (GND): -0.000002 OPEN: 0.008077 Full Scale: +/- 1 mA 2023-04-19T05:55:39.921Z,1681883739.921 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-19T05:55:40.327Z,1681883740.327 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2023-04-19T06:00:40.098Z,1681884040.098 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2023-04-19T06:00:40.922Z,1681884040.922 [FrontSampling:FindPeakChlVerticalAndSample:FindPeakTakeSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 5.000000 count completed. 2023-04-19T06:06:11.032Z,1681884371.032 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T06:11:57.909Z,1681884717.909 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T06:36:17.595Z,1681886177.595 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt5:A.Waypoint](IMPORTANT): Reached waypoint: 36.862999,-122.059998 2023-04-19T06:36:17.997Z,1681886177.997 [FrontSampling:BackAndForth:Waypoints:SampleFrontNotFirstPass:Wpt6:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.862999,-121.970001 2023-04-19T07:12:55.817Z,1681888375.817 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T08:07:48.500Z,1681891668.500 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-04-19T08:23:58.981Z,1681892638.981 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T08:57:03.728Z,1681894623.728 [ThrusterHE](FAULT): getSpeed error serial timeout 2023-04-19T08:57:03.728Z,1681894623.728 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2023-04-19T09:35:06.344Z,1681896906.344 [MissionManager](IMPORTANT): Started mission Default 2023-04-19T09:36:29.104Z,1681896989.104 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T09:42:00.912Z,1681897320.912 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.902714 min 2023-04-19T09:47:04.701Z,1681897624.701 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T09:47:22.573Z,1681897642.573 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T094723 2023-04-19T09:47:30.356Z,1681897650.356 [CommandExec](IMPORTANT): got command ibit 2023-04-19T09:47:30.578Z,1681897650.578 [IBIT](IMPORTANT): Beginning Initiated BIT 2023-04-19T09:47:30.578Z,1681897650.578 [IBIT](IMPORTANT): Beginning control surface checks. 2023-04-19T09:47:30.581Z,1681897650.581 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-19T09:47:41.554Z,1681897661.554 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003478 CHAN A1 (24V): -0.020943 CHAN A2 (12V): -0.000475 CHAN A3 (5V): 0.000461 CHAN B0 (3.3V): -0.006909 CHAN B1 (3.15aV): -0.000055 CHAN B2 (3.15bV): 0.000209 CHAN B3 (GND): 0.000034 OPEN: 0.007539 Full Scale: +/- 1 mA 2023-04-19T09:49:21.714Z,1681897761.714 [IBIT](FAULT): Error acquiring IBIT communications status. Timeout expired. 2023-04-19T09:49:22.102Z,1681897762.102 [IBIT](IMPORTANT): Battery Status: Battery Charge (AH): 0.000000 Voltage: 14.232000 2023-04-19T09:49:22.103Z,1681897762.103 [IBIT](IMPORTANT): batteryCapacityThreshold: 50.000000 Ah 2023-04-19T09:49:22.103Z,1681897762.103 [IBIT](IMPORTANT): batteryVoltageThreshold: 13.700000 V 2023-04-19T09:49:22.489Z,1681897762.489 [IBIT](IMPORTANT): bitHumidityThreshold: 55.000000 % 2023-04-19T09:49:22.490Z,1681897762.490 [IBIT](IMPORTANT): bitPressureThreshold: 0.750000 psi 2023-04-19T09:49:22.503Z,1681897762.503 [IBIT](IMPORTANT): Pressure:9.389691 PSI 2023-04-19T09:49:22.504Z,1681897762.504 [IBIT](IMPORTANT): Humidity:16.768404 % 2023-04-19T09:49:22.903Z,1681897762.903 [IBIT](IMPORTANT): Vehicle Pitch:-3.678300 degrees 2023-04-19T09:49:22.903Z,1681897762.903 [IBIT](IMPORTANT): Vehicle Roll:1.711764 degrees 2023-04-19T09:49:22.904Z,1681897762.904 [IBIT](IMPORTANT): Vehicle Heading:315.025085 degrees 2023-04-19T09:49:23.315Z,1681897763.315 [IBIT](IMPORTANT): surfaceThreshold: 1.500000 m 2023-04-19T09:49:23.316Z,1681897763.316 [IBIT](IMPORTANT): buoyancyNeutral: 99.782722 cc 2023-04-19T09:49:23.316Z,1681897763.316 [IBIT](IMPORTANT): massDefault: 2.887236 cm 2023-04-19T09:49:23.316Z,1681897763.316 [IBIT](IMPORTANT): stopDepth: 265.000000 m 2023-04-19T09:49:23.316Z,1681897763.316 [IBIT](IMPORTANT): abortDepth: 300.000000 m 2023-04-19T09:49:23.317Z,1681897763.317 [IBIT](IMPORTANT): IBIT FAILED 2023-04-19T09:49:24.140Z,1681897764.140 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.289144 min 2023-04-19T09:55:11.526Z,1681898111.526 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.080178 min 2023-04-19T09:55:39.106Z,1681898139.106 [DataOverHttps](IMPORTANT): SBD MOMSN=18238091, MTMSN=20230419T095539 2023-04-19T09:55:46.772Z,1681898146.772 [CommandExec](IMPORTANT): got command ibit 2023-04-19T09:55:47.114Z,1681898147.114 [IBIT](IMPORTANT): Beginning Initiated BIT 2023-04-19T09:55:47.114Z,1681898147.114 [IBIT](IMPORTANT): Beginning control surface checks. 2023-04-19T09:55:47.117Z,1681898147.117 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-19T09:55:58.097Z,1681898158.097 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002859 CHAN A1 (24V): -0.001020 CHAN A2 (12V): -0.000277 CHAN A3 (5V): -0.000034 CHAN B0 (3.3V): 0.025255 CHAN B1 (3.15aV): 0.000397 CHAN B2 (3.15bV): 0.000298 CHAN B3 (GND): 0.000344 OPEN: 0.007922 Full Scale: +/- 1 mA 2023-04-19T09:57:37.848Z,1681898257.848 [IBIT](FAULT): Error acquiring IBIT communications status. Timeout expired. 2023-04-19T09:57:38.237Z,1681898258.237 [IBIT](IMPORTANT): Battery Status: Battery Charge (AH): 0.000000 Voltage: 14.193000 2023-04-19T09:57:38.237Z,1681898258.237 [IBIT](IMPORTANT): batteryCapacityThreshold: 50.000000 Ah 2023-04-19T09:57:38.238Z,1681898258.238 [IBIT](IMPORTANT): batteryVoltageThreshold: 13.700000 V 2023-04-19T09:57:38.628Z,1681898258.628 [IBIT](IMPORTANT): bitHumidityThreshold: 55.000000 % 2023-04-19T09:57:38.628Z,1681898258.628 [IBIT](IMPORTANT): bitPressureThreshold: 0.750000 psi 2023-04-19T09:57:38.628Z,1681898258.628 [IBIT](IMPORTANT): Pressure:9.439550 PSI 2023-04-19T09:57:38.629Z,1681898258.629 [IBIT](IMPORTANT): Humidity:16.652424 % 2023-04-19T09:57:39.040Z,1681898259.040 [IBIT](IMPORTANT): Vehicle Pitch:1.415046 degrees 2023-04-19T09:57:39.041Z,1681898259.041 [IBIT](IMPORTANT): Vehicle Roll:4.089065 degrees 2023-04-19T09:57:39.041Z,1681898259.041 [IBIT](IMPORTANT): Vehicle Heading:40.551647 degrees 2023-04-19T09:57:39.460Z,1681898259.460 [IBIT](IMPORTANT): surfaceThreshold: 1.500000 m 2023-04-19T09:57:39.461Z,1681898259.461 [IBIT](IMPORTANT): buoyancyNeutral: 99.782722 cc 2023-04-19T09:57:39.461Z,1681898259.461 [IBIT](IMPORTANT): massDefault: 2.887236 cm 2023-04-19T09:57:39.461Z,1681898259.461 [IBIT](IMPORTANT): stopDepth: 265.000000 m 2023-04-19T09:57:39.461Z,1681898259.461 [IBIT](IMPORTANT): abortDepth: 300.000000 m 2023-04-19T09:57:39.461Z,1681898259.461 [IBIT](IMPORTANT): IBIT FAILED 2023-04-19T10:00:15.776Z,1681898415.776 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T10:00:42.046Z,1681898442.046 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.588778 min 2023-04-19T10:02:10.371Z,1681898530.371 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T100209 2023-04-19T10:02:17.767Z,1681898537.767 [CommandExec](IMPORTANT): got command ibit 2023-04-19T10:02:17.801Z,1681898537.801 [IBIT](IMPORTANT): Beginning Initiated BIT 2023-04-19T10:02:17.801Z,1681898537.801 [IBIT](IMPORTANT): Beginning control surface checks. 2023-04-19T10:02:17.804Z,1681898537.804 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-19T10:02:28.718Z,1681898548.718 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003210 CHAN A1 (24V): -0.001141 CHAN A2 (12V): -0.000263 CHAN A3 (5V): 0.000143 CHAN B0 (3.3V): -0.000076 CHAN B1 (3.15aV): 0.000442 CHAN B2 (3.15bV): 0.000139 CHAN B3 (GND): 0.000638 OPEN: 0.007680 Full Scale: +/- 1 mA 2023-04-19T10:03:09.124Z,1681898589.124 [IBIT](IMPORTANT): Communications Status: Fix Status: 1 Iridium Signal Strength: 1 Latitude: 36.863911 Longitude: -121.990997 2023-04-19T10:03:09.548Z,1681898589.548 [IBIT](IMPORTANT): Battery Status: Battery Charge (AH): 0.000000 Voltage: 14.232000 2023-04-19T10:03:09.548Z,1681898589.548 [IBIT](IMPORTANT): batteryCapacityThreshold: 50.000000 Ah 2023-04-19T10:03:09.549Z,1681898589.549 [IBIT](IMPORTANT): batteryVoltageThreshold: 13.700000 V 2023-04-19T10:03:09.911Z,1681898589.911 [IBIT](IMPORTANT): bitHumidityThreshold: 55.000000 % 2023-04-19T10:03:09.911Z,1681898589.911 [IBIT](IMPORTANT): bitPressureThreshold: 0.750000 psi 2023-04-19T10:03:09.912Z,1681898589.912 [IBIT](IMPORTANT): Pressure:9.455802 PSI 2023-04-19T10:03:09.912Z,1681898589.912 [IBIT](IMPORTANT): Humidity:16.566963 % 2023-04-19T10:03:10.321Z,1681898590.321 [IBIT](IMPORTANT): Vehicle Pitch:-4.078627 degrees 2023-04-19T10:03:10.321Z,1681898590.321 [IBIT](IMPORTANT): Vehicle Roll:-0.324045 degrees 2023-04-19T10:03:10.321Z,1681898590.321 [IBIT](IMPORTANT): Vehicle Heading:42.071609 degrees 2023-04-19T10:03:10.730Z,1681898590.730 [IBIT](IMPORTANT): surfaceThreshold: 1.500000 m 2023-04-19T10:03:10.730Z,1681898590.730 [IBIT](IMPORTANT): buoyancyNeutral: 99.782722 cc 2023-04-19T10:03:10.730Z,1681898590.730 [IBIT](IMPORTANT): massDefault: 2.887236 cm 2023-04-19T10:03:10.731Z,1681898590.731 [IBIT](IMPORTANT): stopDepth: 265.000000 m 2023-04-19T10:03:10.731Z,1681898590.731 [IBIT](IMPORTANT): abortDepth: 300.000000 m 2023-04-19T10:03:10.731Z,1681898590.731 [IBIT](IMPORTANT): IBIT PASSED 2023-04-19T10:06:29.103Z,1681898789.103 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.372087 min 2023-04-19T10:07:21.588Z,1681898841.588 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T10:10:00.575Z,1681899000.575 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T100959 2023-04-19T10:10:08.102Z,1681899008.102 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.xml;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 36.8695 degree;set profile_station.Lon -121.9545 degree;set profile_station.YoYoMaxDepth 50 m" 3f085 1 2.000000 2023-04-19T10:10:08.102Z,1681899008.102 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=3f085 2023-04-19T10:10:08.103Z,1681899008.103 [CommandExec](IMPORTANT): Scheduled #24 (#1 of 2 with id='3f085'): "load Science/profile_station.xml;set profile_station.MissionTimeout 6 h;set profile_station.NeedCommsTime 45 min;set profile_station.Lat 36.8695 degree;set profile_station.Lon -121.9545 degree;set profile_station.YoYoMaxDepth 50 m", ASAP 2023-04-19T10:10:09.028Z,1681899009.028 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T101008 2023-04-19T10:10:16.549Z,1681899016.549 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.Speed 0.8 m/s;set profile_station.MaxDepth 60 m;run" 3f085 2 2.000000 2023-04-19T10:10:16.550Z,1681899016.550 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=3f085 2023-04-19T10:10:16.551Z,1681899016.551 [CommandExec](IMPORTANT): Scheduled #25 (#2 of 2 with id='3f085'): "set profile_station.Speed 0.8 m/s;set profile_station.MaxDepth 60 m;run", ASAP 2023-04-19T10:10:16.675Z,1681899016.675 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml 2023-04-19T10:10:18.293Z,1681899018.293 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml 2023-04-19T10:10:40.123Z,1681899040.123 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 6 hour 2023-04-19T10:10:40.123Z,1681899040.123 [CommandExec](IMPORTANT): got command set profile_station.NeedCommsTime 45 minute 2023-04-19T10:10:40.124Z,1681899040.124 [CommandExec](IMPORTANT): got command set profile_station.Lat 36.8695 degree 2023-04-19T10:10:40.125Z,1681899040.125 [CommandExec](IMPORTANT): got command set profile_station.Lon -121.9545 degree 2023-04-19T10:10:40.125Z,1681899040.125 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 50 meter 2023-04-19T10:10:48.342Z,1681899048.342 [CommandExec](IMPORTANT): got command set profile_station.Speed 0.8 meter_per_second 2023-04-19T10:10:48.343Z,1681899048.343 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 60 meter 2023-04-19T10:10:48.344Z,1681899048.344 [CommandExec](IMPORTANT): got command run 2023-04-19T10:10:48.350Z,1681899048.350 [CommandExec](IMPORTANT): Running 2023-04-19T10:10:48.476Z,1681899048.476 [MissionManager](IMPORTANT): Started mission profile_station 2023-04-19T10:15:54.393Z,1681899354.393 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T11:10:15.180Z,1681902615.180 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T11:16:07.470Z,1681902967.470 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T12:04:31.147Z,1681905871.147 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T12:07:14.870Z,1681906034.870 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-04-19T12:59:24.315Z,1681909164.315 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T13:54:07.307Z,1681912447.307 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T14:41:50.304Z,1681915310.304 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2023-04-19T14:49:26.393Z,1681915766.393 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-19T14:50:02.532Z,1681915802.532 [DataOverHttps](IMPORTANT): SBD MOMSN=18238499, MTMSN=20230419T145002 2023-04-19T14:50:10.588Z,1681915810.588 [CommandExec](IMPORTANT): got command stop 2023-04-19T14:50:10.588Z,1681915810.588 [CommandExec](IMPORTANT): Scheduling is paused 2023-04-19T14:50:11.255Z,1681915811.255 [MissionManager](IMPORTANT): Started mission Default 2023-04-19T14:52:28.775Z,1681915948.775 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T145228 2023-04-19T14:52:36.385Z,1681915956.385 [CommandExec](IMPORTANT): got command schedule clear 2023-04-19T14:52:36.386Z,1681915956.386 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2023-04-19T14:52:36.386Z,1681915956.386 [CommandExec](IMPORTANT): got command schedule resume 2023-04-19T14:52:36.386Z,1681915956.386 [CommandExec](IMPORTANT): Scheduling is resumed 2023-04-19T14:52:37.339Z,1681915957.339 [DataOverHttps](IMPORTANT): SBD MTMSN=20230419T145236