2023-04-10T03:50:09.657Z,1681098609.657 [CommandExec](IMPORTANT): got command restart logs 2023-04-10T03:52:54.586Z,1681098774.586 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.983670 min 2023-04-10T03:54:35.119Z,1681098875.119 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T035434 2023-04-10T03:54:46.625Z,1681098886.625 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.xml;set profile_station.MissionTimeout 12 h;set profile_station.Lat 36.67 degree;set profile_station.Lon -121.88 degree;set profile_station.YoYoMaxDepth 60 m;set profile_station.Speed 0.8 m/s" 2xuup 1 2.000000 2023-04-10T03:54:46.625Z,1681098886.625 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=2xuup 2023-04-10T03:54:46.626Z,1681098886.626 [CommandExec](IMPORTANT): Scheduled #19 (#1 of 2 with id='2xuup'): "load Science/profile_station.xml;set profile_station.MissionTimeout 12 h;set profile_station.Lat 36.67 degree;set profile_station.Lon -121.88 degree;set profile_station.YoYoMaxDepth 60 m;set profile_station.Speed 0.8 m/s", ASAP 2023-04-10T03:54:47.873Z,1681098887.873 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T035446 2023-04-10T03:54:55.479Z,1681098895.479 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.MaxDepth 65 m;run" 2xuup 2 2.000000 2023-04-10T03:54:55.479Z,1681098895.479 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=2xuup 2023-04-10T03:54:55.480Z,1681098895.480 [CommandExec](IMPORTANT): Scheduled #20 (#2 of 2 with id='2xuup'): "set profile_station.MaxDepth 65 m;run", ASAP 2023-04-10T03:54:55.604Z,1681098895.604 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml 2023-04-10T03:54:58.323Z,1681098898.323 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml 2023-04-10T03:55:27.934Z,1681098927.934 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 12 hour 2023-04-10T03:55:27.935Z,1681098927.935 [CommandExec](IMPORTANT): got command set profile_station.Lat 36.67 degree 2023-04-10T03:55:27.941Z,1681098927.941 [CommandExec](IMPORTANT): got command set profile_station.Lon -121.88 degree 2023-04-10T03:55:27.941Z,1681098927.941 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 60 meter 2023-04-10T03:55:27.942Z,1681098927.942 [CommandExec](IMPORTANT): got command set profile_station.Speed 0.8 meter_per_second 2023-04-10T03:55:34.297Z,1681098934.297 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 65 meter 2023-04-10T03:55:34.297Z,1681098934.297 [CommandExec](IMPORTANT): got command run 2023-04-10T03:55:34.302Z,1681098934.302 [MissionManager](IMPORTANT): Started mission profile_station 2023-04-10T03:55:34.305Z,1681098934.305 [CommandExec](IMPORTANT): Running 2023-04-10T04:30:25.019Z,1681101025.019 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T04:30:35.915Z,1681101035.915 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003307 CHAN A1 (24V): 0.005133 CHAN A2 (12V): -0.004505 CHAN A3 (5V): 0.003846 CHAN B0 (3.3V): -0.001199 CHAN B1 (3.15aV): -0.001246 CHAN B2 (3.15bV): 0.000119 CHAN B3 (GND): -0.000734 OPEN: 0.012471 Full Scale: +/- 1 mA 2023-04-10T05:30:36.051Z,1681104636.051 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T05:30:47.438Z,1681104647.438 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003677 CHAN A1 (24V): 0.005401 CHAN A2 (12V): -0.003651 CHAN A3 (5V): -0.003011 CHAN B0 (3.3V): -0.014589 CHAN B1 (3.15aV): -0.000626 CHAN B2 (3.15bV): -0.000891 CHAN B3 (GND): -0.000725 OPEN: 0.012459 Full Scale: +/- 1 mA 2023-04-10T05:36:56.676Z,1681105016.676 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-04-10T05:40:03.308Z,1681105203.308 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-04-10T05:47:22.727Z,1681105642.727 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-10T06:23:23.705Z,1681107803.705 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. 2023-04-10T06:30:47.611Z,1681108247.611 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T06:30:58.939Z,1681108258.939 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.110522 CHAN A1 (24V): 0.006075 CHAN A2 (12V): -0.003178 CHAN A3 (5V): 0.017132 CHAN B0 (3.3V): -0.001564 CHAN B1 (3.15aV): -0.001443 CHAN B2 (3.15bV): 0.010007 CHAN B3 (GND): -0.000810 OPEN: 0.012395 Full Scale: +/- 1 mA 2023-04-10T07:30:59.039Z,1681111859.039 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T07:31:09.979Z,1681111869.979 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003231 CHAN A1 (24V): 0.005754 CHAN A2 (12V): -0.004299 CHAN A3 (5V): -0.003897 CHAN B0 (3.3V): -0.001429 CHAN B1 (3.15aV): -0.001571 CHAN B2 (3.15bV): -0.001325 CHAN B3 (GND): 0.011415 OPEN: 0.012606 Full Scale: +/- 1 mA 2023-04-10T08:31:10.191Z,1681115470.191 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T08:31:21.083Z,1681115481.083 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002273 CHAN A1 (24V): 0.008181 CHAN A2 (12V): -0.004108 CHAN A3 (5V): -0.003451 CHAN B0 (3.3V): 0.001332 CHAN B1 (3.15aV): -0.001134 CHAN B2 (3.15bV): -0.000978 CHAN B3 (GND): -0.000954 OPEN: 0.012444 Full Scale: +/- 1 mA 2023-04-10T09:31:21.335Z,1681119081.335 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T09:31:32.163Z,1681119092.163 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.093826 CHAN A1 (24V): 0.009771 CHAN A2 (12V): -0.003048 CHAN A3 (5V): -0.003204 CHAN B0 (3.3V): -0.000136 CHAN B1 (3.15aV): -0.001135 CHAN B2 (3.15bV): -0.000560 CHAN B3 (GND): -0.001377 OPEN: 0.012622 Full Scale: +/- 1 mA 2023-04-10T10:31:32.221Z,1681122692.221 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T10:31:43.115Z,1681122703.115 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.001802 CHAN A1 (24V): 0.007092 CHAN A2 (12V): -0.003415 CHAN A3 (5V): -0.000559 CHAN B0 (3.3V): -0.000923 CHAN B1 (3.15aV): -0.116062 CHAN B2 (3.15bV): -0.000123 CHAN B3 (GND): -0.001184 OPEN: 0.012687 Full Scale: +/- 1 mA 2023-04-10T10:34:57.529Z,1681122897.529 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-10T11:31:43.172Z,1681126303.172 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T11:31:54.111Z,1681126314.111 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001435 CHAN A1 (24V): 0.004894 CHAN A2 (12V): -0.002933 CHAN A3 (5V): -0.003198 CHAN B0 (3.3V): 0.009546 CHAN B1 (3.15aV): -0.001208 CHAN B2 (3.15bV): -0.001112 CHAN B3 (GND): -0.001867 OPEN: 0.012656 Full Scale: +/- 1 mA 2023-04-10T11:42:43.649Z,1681126963.649 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-10T12:31:54.125Z,1681129914.125 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T12:32:05.079Z,1681129925.079 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001137 CHAN A1 (24V): 0.008511 CHAN A2 (12V): -0.003518 CHAN A3 (5V): -0.003429 CHAN B0 (3.3V): -0.001535 CHAN B1 (3.15aV): -0.000988 CHAN B2 (3.15bV): -0.000652 CHAN B3 (GND): -0.000853 OPEN: 0.012751 Full Scale: +/- 1 mA 2023-04-10T13:32:05.380Z,1681133525.380 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T13:32:16.606Z,1681133536.606 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.000444 CHAN A1 (24V): 0.009356 CHAN A2 (12V): -0.002563 CHAN A3 (5V): -0.053419 CHAN B0 (3.3V): -0.001360 CHAN B1 (3.15aV): -0.000658 CHAN B2 (3.15bV): -0.000390 CHAN B3 (GND): -0.001404 OPEN: 0.012472 Full Scale: +/- 1 mA 2023-04-10T14:32:16.620Z,1681137136.620 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T14:32:27.715Z,1681137147.715 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001178 CHAN A1 (24V): 0.009430 CHAN A2 (12V): -0.003107 CHAN A3 (5V): -0.002357 CHAN B0 (3.3V): -0.001307 CHAN B1 (3.15aV): -0.001022 CHAN B2 (3.15bV): -0.000242 CHAN B3 (GND): -0.001560 OPEN: 0.012565 Full Scale: +/- 1 mA 2023-04-10T15:32:27.727Z,1681140747.727 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T15:32:39.152Z,1681140759.152 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001356 CHAN A1 (24V): 0.008750 CHAN A2 (12V): -0.002760 CHAN A3 (5V): -0.003189 CHAN B0 (3.3V): -0.001676 CHAN B1 (3.15aV): -0.001144 CHAN B2 (3.15bV): 0.000951 CHAN B3 (GND): -0.001174 OPEN: 0.012236 Full Scale: +/- 1 mA 2023-04-10T15:55:34.696Z,1681142134.696 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T16:00:04.198Z,1681142404.198 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T160003 2023-04-10T16:00:17.241Z,1681142417.241 [CommandExec](IMPORTANT): got command configSet Express linearApproximation VerticalHomogeneityIndexCalculator.vertical_salinity_homogeneity_index 0.05 practical_salinity_unit persist 2023-04-10T16:03:17.214Z,1681142597.214 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T160316 2023-04-10T16:03:24.740Z,1681142604.740 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2yslf 1 3.000000 2023-04-10T16:03:24.740Z,1681142604.740 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2yslf 2023-04-10T16:03:24.741Z,1681142604.741 [CommandExec](IMPORTANT): Scheduled #21 (#1 of 3 with id='2yslf'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-10T16:03:25.714Z,1681142605.714 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T160324 2023-04-10T16:03:33.912Z,1681142613.912 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2yslf 2 3.000000 2023-04-10T16:03:33.912Z,1681142613.912 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2yslf 2023-04-10T16:03:33.913Z,1681142613.913 [CommandExec](IMPORTANT): Scheduled #22 (#2 of 3 with id='2yslf'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-10T16:03:34.826Z,1681142614.826 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T160333 2023-04-10T16:03:45.496Z,1681142625.496 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2yslf 3 3.000000 2023-04-10T16:03:45.497Z,1681142625.497 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2yslf 2023-04-10T16:03:45.498Z,1681142625.498 [CommandExec](IMPORTANT): Scheduled #23 (#3 of 3 with id='2yslf'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-10T16:03:45.631Z,1681142625.631 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T16:04:01.467Z,1681142641.467 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T16:04:24.008Z,1681142664.008 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-10T16:04:24.009Z,1681142664.009 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-10T16:04:24.010Z,1681142664.010 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-10T16:04:27.876Z,1681142667.876 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.879995 min 2023-04-10T16:05:07.876Z,1681142707.876 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-10T16:05:07.881Z,1681142707.881 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-10T16:05:07.881Z,1681142707.881 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-10T16:05:07.882Z,1681142707.882 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-10T16:05:40.916Z,1681142740.916 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-10T16:05:40.916Z,1681142740.916 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-10T16:05:40.918Z,1681142740.918 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-10T16:05:40.919Z,1681142740.919 [CommandExec](IMPORTANT): got command run 2023-04-10T16:05:41.029Z,1681142741.029 [CommandExec](IMPORTANT): Running 2023-04-10T16:05:41.102Z,1681142741.102 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-10T16:05:41.219Z,1681142741.219 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-10T16:07:00.227Z,1681142820.227 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.669699 arcdeg , longitude = -121.877272 arcdeg . Distance from shore = 4645.296875 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-10T16:07:02.232Z,1681142822.232 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.070224 m 2023-04-10T16:08:11.779Z,1681142891.779 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-10T16:10:13.833Z,1681143013.833 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.110882 m 2023-04-10T16:11:56.740Z,1681143116.740 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-10T16:13:02.246Z,1681143182.246 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.844063 m , 0.936000 ug/l , 4.375072 m , 11.585863 degC , 6.992961 ug/l 2023-04-10T16:13:02.628Z,1681143182.628 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 4.375072 m . 2023-04-10T16:13:03.069Z,1681143183.069 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-10T16:15:47.504Z,1681143347.504 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.337084 m , settling for 10.000000 min 2023-04-10T16:26:00.392Z,1681143960.392 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-10T16:26:00.708Z,1681143960.708 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-10T16:26:51.457Z,1681144011.457 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-10T16:26:52.279Z,1681144012.279 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -10 2023-04-10T16:26:52.281Z,1681144012.281 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-10T16:26:52.753Z,1681144012.753 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.598711 m in 11.566706 degC water with 7.944560 ug/l chlorophyll fluorescence. 2023-04-10T16:26:53.083Z,1681144013.083 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-10T16:26:53.477Z,1681144013.477 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-10T16:26:55.497Z,1681144015.497 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-10T16:28:21.549Z,1681144101.549 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING 2023-04-10T16:28:31.650Z,1681144111.650 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-10T16:28:32.135Z,1681144112.135 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2023-04-10T16:32:39.159Z,1681144359.159 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T16:32:50.068Z,1681144370.068 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.000325 CHAN A1 (24V): 0.008888 CHAN A2 (12V): -0.002547 CHAN A3 (5V): -0.003192 CHAN B0 (3.3V): -0.000972 CHAN B1 (3.15aV): -0.001025 CHAN B2 (3.15bV): -0.000722 CHAN B3 (GND): 0.005427 OPEN: 0.012715 Full Scale: +/- 1 mA 2023-04-10T16:33:31.909Z,1681144411.909 [ESPComponent](CRITICAL): [sample #4] ESP sampling sequence terminated with error: In state S_LOADING_CARTRIDGE got exception while awaiting result for cmd='Cmd.loadCartridge -10' exception=<>. Elapsed time since command issued: 86.0465 2023-04-10T16:33:32.105Z,1681144412.105 [CommandExec](FAULT): Scheduling is paused 2023-04-10T16:33:33.633Z,1681144413.633 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T16:42:27.546Z,1681144947.546 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.891618 min 2023-04-10T16:48:03.931Z,1681145283.931 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.498039 min 2023-04-10T16:53:35.396Z,1681145615.396 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.022662 min 2023-04-10T16:59:08.342Z,1681145948.342 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.572017 min 2023-04-10T17:04:42.975Z,1681146282.975 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.149432 min 2023-04-10T17:10:13.567Z,1681146613.567 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 36.658675 min 2023-04-10T17:15:49.126Z,1681146949.126 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 42.251762 min 2023-04-10T17:18:28.478Z,1681147108.478 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T171827 2023-04-10T17:18:39.450Z,1681147119.450 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2yw2e 1 3.000000 2023-04-10T17:18:39.450Z,1681147119.450 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2yw2e 2023-04-10T17:18:39.451Z,1681147119.451 [CommandExec](IMPORTANT): Scheduled #24 (#1 of 3 with id='2yw2e'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-10T17:18:40.386Z,1681147120.386 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T171839 2023-04-10T17:18:50.197Z,1681147130.197 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2yw2e 2 3.000000 2023-04-10T17:18:50.197Z,1681147130.197 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2yw2e 2023-04-10T17:18:50.198Z,1681147130.198 [CommandExec](IMPORTANT): Scheduled #25 (#2 of 3 with id='2yw2e'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-10T17:18:51.185Z,1681147131.185 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T171850 2023-04-10T17:18:58.838Z,1681147138.838 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2yw2e 3 3.000000 2023-04-10T17:18:58.838Z,1681147138.838 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2yw2e 2023-04-10T17:18:58.839Z,1681147138.839 [CommandExec](IMPORTANT): Scheduled #26 (#3 of 3 with id='2yw2e'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-10T17:18:58.988Z,1681147138.988 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T17:19:14.877Z,1681147154.877 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T17:19:43.443Z,1681147183.443 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-10T17:19:43.444Z,1681147183.444 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-10T17:19:43.444Z,1681147183.444 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-10T17:20:15.446Z,1681147215.446 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-10T17:20:15.447Z,1681147215.447 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-10T17:20:15.449Z,1681147215.449 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-10T17:20:15.450Z,1681147215.450 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-10T17:20:42.998Z,1681147242.998 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-10T17:20:42.999Z,1681147242.999 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-10T17:20:42.000Z,1681147243.000 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-10T17:20:43.001Z,1681147243.001 [CommandExec](IMPORTANT): got command run 2023-04-10T17:20:43.003Z,1681147243.003 [CommandExec](IMPORTANT): Running 2023-04-10T17:20:43.168Z,1681147243.168 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-10T17:20:43.273Z,1681147243.273 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-10T17:22:33.579Z,1681147353.579 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.678074 arcdeg , longitude = -121.873957 arcdeg . Distance from shore = 4576.210938 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-10T17:22:35.650Z,1681147355.650 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.037463 m 2023-04-10T17:23:45.166Z,1681147425.166 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-10T17:25:49.642Z,1681147549.642 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.054777 m 2023-04-10T17:28:30.005Z,1681147710.005 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.844502 m , 0.863200 ug/l , 3.801058 m , 12.143309 degC , 9.099998 ug/l 2023-04-10T17:28:30.382Z,1681147710.382 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.801058 m . 2023-04-10T17:28:30.782Z,1681147710.782 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-10T17:31:21.300Z,1681147881.300 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.794529 m , settling for 10.000000 min 2023-04-10T17:32:50.272Z,1681147970.272 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T17:33:01.207Z,1681147981.207 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.000977 CHAN A1 (24V): 0.007876 CHAN A2 (12V): -0.003110 CHAN A3 (5V): -0.000942 CHAN B0 (3.3V): -0.001484 CHAN B1 (3.15aV): 0.019306 CHAN B2 (3.15bV): -0.000790 CHAN B3 (GND): -0.000560 OPEN: 0.012450 Full Scale: +/- 1 mA 2023-04-10T17:41:34.204Z,1681148494.204 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-10T17:41:34.381Z,1681148494.381 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-10T17:42:25.525Z,1681148545.525 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-10T17:42:26.338Z,1681148546.338 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -10 2023-04-10T17:42:26.338Z,1681148546.338 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-10T17:42:26.823Z,1681148546.823 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.946210 m in 11.736963 degC water with 9.055280 ug/l chlorophyll fluorescence. 2023-04-10T17:42:27.141Z,1681148547.141 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-10T17:42:27.546Z,1681148547.546 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-10T17:42:29.565Z,1681148549.565 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-10T17:43:07.146Z,1681148587.146 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@49,daSPR_phyto) 2023-04-10T17:43:09.970Z,1681148589.970 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2023-04-10T17:43:09.971Z,1681148589.971 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-10T17:43:10.375Z,1681148590.375 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-10T17:45:29.754Z,1681148729.754 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2023-04-10T17:45:30.158Z,1681148730.158 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-10T17:45:53.590Z,1681148753.590 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@2.5) 2023-04-10T17:45:53.999Z,1681148753.999 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@2.5) 2023-04-10T17:45:56.018Z,1681148756.018 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2023-04-10T17:45:56.419Z,1681148756.419 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,daSPR_phyto) 2023-04-10T18:13:52.796Z,1681150432.796 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-04-10T18:33:01.559Z,1681151581.559 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T18:33:12.314Z,1681151592.314 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.002297 CHAN A1 (24V): 0.055799 CHAN A2 (12V): 0.015788 CHAN A3 (5V): 0.012410 CHAN B0 (3.3V): -0.000677 CHAN B1 (3.15aV): -0.000338 CHAN B2 (3.15bV): 0.032334 CHAN B3 (GND): -0.003352 OPEN: 0.012487 Full Scale: +/- 1 mA 2023-04-10T18:44:21.590Z,1681152261.590 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@49,daSPR_phyto) 2023-04-10T18:44:32.902Z,1681152272.902 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-10T18:44:33.358Z,1681152273.358 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2023-04-10T18:49:33.129Z,1681152573.129 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2023-04-10T18:49:34.001Z,1681152574.001 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2023-04-10T18:49:37.136Z,1681152577.136 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T18:58:33.529Z,1681153113.529 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.933459 min 2023-04-10T19:00:22.978Z,1681153222.978 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T190022 2023-04-10T19:00:30.600Z,1681153230.600 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2z0sf 1 3.000000 2023-04-10T19:00:30.600Z,1681153230.600 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2z0sf 2023-04-10T19:00:30.601Z,1681153230.601 [CommandExec](IMPORTANT): Scheduled #27 (#1 of 3 with id='2z0sf'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-10T19:00:31.586Z,1681153231.586 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T190030 2023-04-10T19:00:42.480Z,1681153242.480 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2z0sf 2 3.000000 2023-04-10T19:00:42.480Z,1681153242.480 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2z0sf 2023-04-10T19:00:42.481Z,1681153242.481 [CommandExec](IMPORTANT): Scheduled #28 (#2 of 3 with id='2z0sf'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-10T19:00:43.414Z,1681153243.414 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T190042 2023-04-10T19:00:52.392Z,1681153252.392 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2z0sf 3 3.000000 2023-04-10T19:00:52.392Z,1681153252.392 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2z0sf 2023-04-10T19:00:52.393Z,1681153252.393 [CommandExec](IMPORTANT): Scheduled #29 (#3 of 3 with id='2z0sf'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-10T19:00:52.534Z,1681153252.534 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T19:01:08.501Z,1681153268.501 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T19:01:31.508Z,1681153291.508 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-10T19:01:31.509Z,1681153291.509 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-10T19:01:31.510Z,1681153291.510 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-10T19:02:08.723Z,1681153328.723 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-10T19:02:08.726Z,1681153328.726 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-10T19:02:08.729Z,1681153328.729 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-10T19:02:08.731Z,1681153328.731 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-10T19:02:30.929Z,1681153350.929 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-10T19:02:30.930Z,1681153350.930 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-10T19:02:30.931Z,1681153350.931 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-10T19:02:30.931Z,1681153350.931 [CommandExec](IMPORTANT): got command run 2023-04-10T19:02:30.957Z,1681153350.957 [CommandExec](IMPORTANT): Running 2023-04-10T19:02:31.112Z,1681153351.112 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-10T19:02:31.176Z,1681153351.176 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-10T19:03:19.781Z,1681153399.781 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.682536 arcdeg , longitude = -121.865429 arcdeg . Distance from shore = 3984.483398 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-10T19:03:21.830Z,1681153401.830 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.096252 m 2023-04-10T19:04:31.410Z,1681153471.410 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-10T19:06:20.839Z,1681153580.839 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.123425 m 2023-04-10T19:07:36.379Z,1681153656.379 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.783468 m , 2.080000 ug/l , 14.042589 m , 12.700384 degC , 5.893159 ug/l 2023-04-10T19:07:36.777Z,1681153656.777 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 14.042589 m . 2023-04-10T19:07:37.199Z,1681153657.199 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-10T19:09:19.038Z,1681153759.038 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 14.109496 m , settling for 10.000000 min 2023-04-10T19:19:31.912Z,1681154371.912 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-10T19:19:32.081Z,1681154372.081 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-10T19:20:23.242Z,1681154423.242 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-10T19:20:24.056Z,1681154424.056 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -10 2023-04-10T19:20:24.057Z,1681154424.057 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-10T19:20:24.531Z,1681154424.531 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 13.324202 m in 10.571192 degC water with 2.004080 ug/l chlorophyll fluorescence. 2023-04-10T19:20:24.865Z,1681154424.865 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-10T19:20:25.265Z,1681154425.265 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-10T19:20:27.285Z,1681154427.285 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-10T19:21:04.861Z,1681154464.861 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@48,daSPR_phyto) 2023-04-10T19:21:07.689Z,1681154467.689 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2023-04-10T19:21:07.690Z,1681154467.690 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-10T19:21:08.093Z,1681154468.093 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-10T19:23:22.225Z,1681154602.225 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2023-04-10T19:23:22.630Z,1681154602.630 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-10T19:33:12.575Z,1681155192.575 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T19:33:23.526Z,1681155203.526 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.093669 CHAN A1 (24V): 0.147406 CHAN A2 (12V): 0.065307 CHAN A3 (5V): 0.019964 CHAN B0 (3.3V): -0.007231 CHAN B1 (3.15aV): 0.013015 CHAN B2 (3.15bV): 0.013218 CHAN B3 (GND): -0.005878 OPEN: 0.012531 Full Scale: +/- 1 mA 2023-04-10T19:44:41.328Z,1681155881.328 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-10T20:31:27.126Z,1681158687.126 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@914.8) 2023-04-10T20:31:27.530Z,1681158687.530 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@914.8) 2023-04-10T20:31:29.551Z,1681158689.551 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2023-04-10T20:31:29.955Z,1681158689.955 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,daSPR_phyto) 2023-04-10T20:33:23.575Z,1681158803.575 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T20:33:34.483Z,1681158814.483 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.070699 CHAN A1 (24V): 0.123390 CHAN A2 (12V): 0.041564 CHAN A3 (5V): 0.001545 CHAN B0 (3.3V): -0.026392 CHAN B1 (3.15aV): -0.000025 CHAN B2 (3.15bV): -0.000670 CHAN B3 (GND): -0.001439 OPEN: 0.012660 Full Scale: +/- 1 mA 2023-04-10T21:29:30.170Z,1681162170.170 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@48,daSPR_phyto) 2023-04-10T21:29:41.482Z,1681162181.482 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-10T21:29:41.902Z,1681162181.902 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2023-04-10T21:33:34.767Z,1681162414.767 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T21:33:45.627Z,1681162425.627 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002167 CHAN A1 (24V): 0.013516 CHAN A2 (12V): -0.002977 CHAN A3 (5V): -0.002937 CHAN B0 (3.3V): 0.000993 CHAN B1 (3.15aV): -0.001135 CHAN B2 (3.15bV): -0.000673 CHAN B3 (GND): -0.001071 OPEN: 0.012375 Full Scale: +/- 1 mA 2023-04-10T21:34:41.689Z,1681162481.689 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2023-04-10T21:34:42.560Z,1681162482.560 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2023-04-10T21:34:45.604Z,1681162485.604 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T21:46:06.179Z,1681163166.179 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.336130 min 2023-04-10T21:46:07.787Z,1681163167.787 [NAL9602](FAULT): GPS failed to acquire within timeout. 2023-04-10T21:55:16.477Z,1681163716.477 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.507570 min 2023-04-10T22:00:59.727Z,1681164059.727 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 26.228381 min 2023-04-10T22:06:42.386Z,1681164402.386 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.939071 min 2023-04-10T22:09:30.214Z,1681164570.214 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T220929 2023-04-10T22:09:40.961Z,1681164580.961 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.tl;set sci2.MissionTimeout 2 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 36.6778 degree;set sci2.Lon1 -121.8742 degree;set sci2.Speed 0.9 m/s;set sci2.YoYoMaxDepth 40 m;set sci2.YoYoMinAltitude 9 m" 2z9jl 1 2.000000 2023-04-10T22:09:40.961Z,1681164580.961 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=2z9jl 2023-04-10T22:09:40.962Z,1681164580.962 [CommandExec](IMPORTANT): Scheduled #30 (#1 of 2 with id='2z9jl'): "load Science/sci2.tl;set sci2.MissionTimeout 2 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 36.6778 degree;set sci2.Lon1 -121.8742 degree;set sci2.Speed 0.9 m/s;set sci2.YoYoMaxDepth 40 m;set sci2.YoYoMinAltitude 9 m", ASAP 2023-04-10T22:09:41.890Z,1681164581.890 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T220940 2023-04-10T22:09:49.711Z,1681164589.711 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MaxDepth 45 m;run" 2z9jl 2 2.000000 2023-04-10T22:09:49.711Z,1681164589.711 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=2z9jl 2023-04-10T22:09:49.713Z,1681164589.713 [CommandExec](IMPORTANT): Scheduled #31 (#2 of 2 with id='2z9jl'): "set sci2.MaxDepth 45 m;run", ASAP 2023-04-10T22:09:49.859Z,1681164589.859 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl 2023-04-10T22:09:49.869Z,1681164589.869 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx 2023-04-10T22:09:51.776Z,1681164591.776 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl 2023-04-10T22:10:07.604Z,1681164607.604 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 2 hour 2023-04-10T22:10:07.604Z,1681164607.604 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 30 minute 2023-04-10T22:10:07.605Z,1681164607.605 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.6778 degree 2023-04-10T22:10:07.606Z,1681164607.606 [CommandExec](IMPORTANT): got command set sci2.Lon1 -121.8742 degree 2023-04-10T22:10:07.606Z,1681164607.606 [CommandExec](IMPORTANT): got command set sci2.Speed 0.9 meter_per_second 2023-04-10T22:10:07.607Z,1681164607.607 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 40 meter 2023-04-10T22:10:07.609Z,1681164607.609 [CommandExec](IMPORTANT): got command set sci2.YoYoMinAltitude 9 meter 2023-04-10T22:10:09.752Z,1681164609.752 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 45 meter 2023-04-10T22:10:09.753Z,1681164609.753 [CommandExec](IMPORTANT): got command run 2023-04-10T22:10:09.754Z,1681164609.754 [CommandExec](IMPORTANT): Running 2023-04-10T22:10:10.097Z,1681164610.097 [MissionManager](IMPORTANT): Started mission sci2 2023-04-10T22:10:58.677Z,1681164658.677 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.677799,-121.874199 2023-04-10T22:33:46.011Z,1681166026.011 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T22:33:56.983Z,1681166036.983 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.000809 CHAN A1 (24V): 0.008669 CHAN A2 (12V): -0.002944 CHAN A3 (5V): 0.005888 CHAN B0 (3.3V): -0.001023 CHAN B1 (3.15aV): -0.001122 CHAN B2 (3.15bV): 0.003312 CHAN B3 (GND): -0.001428 OPEN: 0.012547 Full Scale: +/- 1 mA 2023-04-10T22:36:12.307Z,1681166172.307 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 36.677799,-121.874199 2023-04-10T22:37:12.487Z,1681166232.487 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T22:45:59.687Z,1681166759.687 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.779577 min 2023-04-10T22:46:09.945Z,1681166769.945 [DataOverHttps](IMPORTANT): SBD MOMSN=18180783, MTMSN=20230410T224608 2023-04-10T22:46:17.612Z,1681166777.612 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2zb8l 1 3.000000 2023-04-10T22:46:17.612Z,1681166777.612 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2zb8l 2023-04-10T22:46:17.613Z,1681166777.613 [CommandExec](IMPORTANT): Scheduled #32 (#1 of 3 with id='2zb8l'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-10T22:46:18.759Z,1681166778.759 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T224617 2023-04-10T22:46:26.157Z,1681166786.157 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2zb8l 2 3.000000 2023-04-10T22:46:26.157Z,1681166786.157 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2zb8l 2023-04-10T22:46:26.158Z,1681166786.158 [CommandExec](IMPORTANT): Scheduled #33 (#2 of 3 with id='2zb8l'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-10T22:46:27.280Z,1681166787.280 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T224625 2023-04-10T22:46:36.146Z,1681166796.146 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2zb8l 3 3.000000 2023-04-10T22:46:36.147Z,1681166796.147 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2zb8l 2023-04-10T22:46:36.148Z,1681166796.148 [CommandExec](IMPORTANT): Scheduled #34 (#3 of 3 with id='2zb8l'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-10T22:46:36.332Z,1681166796.332 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T22:47:06.419Z,1681166826.419 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T22:47:35.307Z,1681166855.307 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-10T22:47:35.323Z,1681166855.323 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-10T22:47:35.324Z,1681166855.324 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-10T22:48:19.543Z,1681166899.543 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-10T22:48:19.544Z,1681166899.544 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-10T22:48:19.545Z,1681166899.545 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-10T22:48:19.545Z,1681166899.545 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-10T22:48:55.313Z,1681166935.313 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-10T22:48:55.314Z,1681166935.314 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-10T22:48:55.314Z,1681166935.314 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-10T22:48:55.315Z,1681166935.315 [CommandExec](IMPORTANT): got command run 2023-04-10T22:48:55.323Z,1681166935.323 [CommandExec](IMPORTANT): Running 2023-04-10T22:48:55.333Z,1681166935.333 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-10T22:48:55.415Z,1681166935.415 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-10T22:50:33.001Z,1681167033.001 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.676973 arcdeg , longitude = -121.872343 arcdeg . Distance from shore = 4403.539062 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-10T22:50:34.996Z,1681167034.996 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.111062 m 2023-04-10T22:51:44.803Z,1681167104.803 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-10T22:54:01.361Z,1681167241.361 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.097858 m 2023-04-10T22:55:29.369Z,1681167329.369 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-10T22:56:31.228Z,1681167391.228 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.871428 m , 1.778400 ug/l , 4.514587 m , 11.713709 degC , 11.378118 ug/l 2023-04-10T22:56:31.617Z,1681167391.617 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 4.514587 m . 2023-04-10T22:56:32.026Z,1681167392.026 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-10T22:59:16.086Z,1681167556.086 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.507610 m , settling for 10.000000 min 2023-04-10T23:09:28.608Z,1681168168.608 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-10T23:09:28.785Z,1681168168.785 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-10T23:10:20.669Z,1681168220.669 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-10T23:10:21.480Z,1681168221.480 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -10 2023-04-10T23:10:21.481Z,1681168221.481 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-10T23:10:21.992Z,1681168221.992 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.429527 m in 11.605402 degC water with 10.904400 ug/l chlorophyll fluorescence. 2023-04-10T23:10:22.285Z,1681168222.285 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-10T23:10:22.690Z,1681168222.690 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-10T23:10:24.710Z,1681168224.710 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-10T23:11:02.283Z,1681168262.283 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@47,daSPR_phyto) 2023-04-10T23:11:04.706Z,1681168264.706 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2023-04-10T23:11:04.707Z,1681168264.707 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-10T23:11:05.110Z,1681168265.110 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-10T23:13:40.653Z,1681168420.653 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@0.0) 2023-04-10T23:13:49.542Z,1681168429.542 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-10T23:13:49.968Z,1681168429.968 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2023-04-10T23:18:49.736Z,1681168729.736 [ESPComponent](CRITICAL): [sample #7] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 158.3733 2023-04-10T23:18:51.642Z,1681168731.642 [MissionManager](IMPORTANT): Started mission Default 2023-04-10T23:22:45.037Z,1681168965.037 [DataOverHttps](FAULT): Server acknowledged file Logs/20230410T035009/Courier0138.lzma, notLogs/20230410T035009/Express0139.lzma 2023-04-10T23:28:14.461Z,1681169294.461 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 9.373228 min 2023-04-10T23:29:34.082Z,1681169374.082 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T232932 2023-04-10T23:29:45.010Z,1681169385.010 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2zd8s 1 3.000000 2023-04-10T23:29:45.011Z,1681169385.011 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2zd8s 2023-04-10T23:29:45.022Z,1681169385.022 [CommandExec](IMPORTANT): Scheduled #35 (#1 of 3 with id='2zd8s'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-10T23:29:45.906Z,1681169385.906 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T232944 2023-04-10T23:29:53.640Z,1681169393.640 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2zd8s 2 3.000000 2023-04-10T23:29:53.641Z,1681169393.641 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2zd8s 2023-04-10T23:29:53.642Z,1681169393.642 [CommandExec](IMPORTANT): Scheduled #36 (#2 of 3 with id='2zd8s'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-10T23:29:54.554Z,1681169394.554 [DataOverHttps](IMPORTANT): SBD MTMSN=20230410T232953 2023-04-10T23:30:02.080Z,1681169402.080 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2zd8s 3 3.000000 2023-04-10T23:30:02.081Z,1681169402.081 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2zd8s 2023-04-10T23:30:02.082Z,1681169402.082 [CommandExec](IMPORTANT): Scheduled #37 (#3 of 3 with id='2zd8s'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-10T23:30:02.206Z,1681169402.206 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T23:30:18.246Z,1681169418.246 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-10T23:30:48.402Z,1681169448.402 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-10T23:30:48.403Z,1681169448.403 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-10T23:30:48.404Z,1681169448.404 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-10T23:31:17.589Z,1681169477.589 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-10T23:31:17.590Z,1681169477.590 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-10T23:31:17.591Z,1681169477.591 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-10T23:31:17.591Z,1681169477.591 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-10T23:31:45.565Z,1681169505.565 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-10T23:31:45.569Z,1681169505.569 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-10T23:31:45.595Z,1681169505.595 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-10T23:31:45.595Z,1681169505.595 [CommandExec](IMPORTANT): got command run 2023-04-10T23:31:45.598Z,1681169505.598 [CommandExec](IMPORTANT): Running 2023-04-10T23:31:45.762Z,1681169505.762 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-10T23:31:45.883Z,1681169505.883 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-10T23:33:57.191Z,1681169637.191 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-10T23:34:08.131Z,1681169648.131 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001852 CHAN A1 (24V): 0.008313 CHAN A2 (12V): -0.003993 CHAN A3 (5V): -0.002537 CHAN B0 (3.3V): -0.001275 CHAN B1 (3.15aV): -0.000797 CHAN B2 (3.15bV): -0.000316 CHAN B3 (GND): 0.005384 OPEN: 0.012565 Full Scale: +/- 1 mA 2023-04-10T23:35:33.650Z,1681169733.650 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.674065 arcdeg , longitude = -121.871658 arcdeg . Distance from shore = 4264.798828 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-10T23:35:35.668Z,1681169735.668 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.248834 m 2023-04-10T23:36:45.191Z,1681169805.191 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-10T23:38:44.761Z,1681169924.761 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.074976 m 2023-04-10T23:41:07.361Z,1681170067.361 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 15.409499 m , 1.580800 ug/l , 4.114741 m , 12.192739 degC , 10.898158 ug/l 2023-04-10T23:41:07.775Z,1681170067.775 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 4.114741 m . 2023-04-10T23:41:08.146Z,1681170068.146 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-10T23:44:24.526Z,1681170264.526 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.122126 m , settling for 10.000000 min 2023-04-10T23:54:37.437Z,1681170877.437 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-10T23:54:37.589Z,1681170877.589 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-10T23:55:28.673Z,1681170928.673 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-10T23:55:29.486Z,1681170929.486 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -10 2023-04-10T23:55:29.486Z,1681170929.486 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-10T23:55:29.999Z,1681170929.999 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.481295 m in 12.294214 degC water with 10.873200 ug/l chlorophyll fluorescence. 2023-04-10T23:55:30.289Z,1681170930.289 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-10T23:55:30.694Z,1681170930.694 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-10T23:55:32.715Z,1681170932.715 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-10T23:56:10.286Z,1681170970.286 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@45,daSPR_phyto) 2023-04-10T23:56:13.115Z,1681170973.115 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2023-04-10T23:56:13.116Z,1681170973.116 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-10T23:56:13.517Z,1681170973.517 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-10T23:58:38.150Z,1681171118.150 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@0.0) 2023-04-10T23:58:47.039Z,1681171127.039 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-10T23:58:47.482Z,1681171127.482 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2023-04-11T00:03:47.269Z,1681171427.269 [ESPComponent](CRITICAL): [sample #8] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 147.8607 2023-04-11T00:03:49.095Z,1681171429.095 [MissionManager](IMPORTANT): Started mission Default 2023-04-11T00:09:17.510Z,1681171757.510 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-11T00:15:24.477Z,1681172124.477 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.582895 min 2023-04-11T00:15:26.086Z,1681172126.086 [NAL9602](FAULT): GPS failed to acquire within timeout. 2023-04-11T00:21:10.636Z,1681172470.636 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-11T00:26:14.912Z,1681172774.912 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.423739 min 2023-04-11T00:26:16.524Z,1681172776.524 [NAL9602](FAULT): GPS failed to acquire within timeout. 2023-04-11T00:31:46.084Z,1681173106.084 [DataOverHttps](IMPORTANT): SBD MOMSN=18182537, MTMSN=20230411T003144 2023-04-11T00:31:53.668Z,1681173113.668 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 2zg4w 1 3.000000 2023-04-11T00:31:53.669Z,1681173113.669 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=2zg4w 2023-04-11T00:31:53.670Z,1681173113.670 [CommandExec](IMPORTANT): Scheduled #38 (#1 of 3 with id='2zg4w'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-11T00:31:54.657Z,1681173114.657 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T003153 2023-04-11T00:32:02.532Z,1681173122.532 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m" 2zg4w 2 3.000000 2023-04-11T00:32:02.532Z,1681173122.532 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=2zg4w 2023-04-11T00:32:02.534Z,1681173122.534 [CommandExec](IMPORTANT): Scheduled #39 (#2 of 3 with id='2zg4w'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 15 m", ASAP 2023-04-11T00:32:03.454Z,1681173123.454 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T003202 2023-04-11T00:32:10.920Z,1681173130.920 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 2zg4w 3 3.000000 2023-04-11T00:32:10.921Z,1681173130.921 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=2zg4w 2023-04-11T00:32:10.922Z,1681173130.922 [CommandExec](IMPORTANT): Scheduled #40 (#3 of 3 with id='2zg4w'): "set IsothermDepthSampling.CartridgeType1 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP 2023-04-11T00:32:11.059Z,1681173131.059 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-11T00:32:27.566Z,1681173147.566 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-11T00:32:59.194Z,1681173179.194 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-11T00:32:59.195Z,1681173179.195 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-11T00:32:59.196Z,1681173179.196 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-11T00:33:27.950Z,1681173207.950 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-11T00:33:27.951Z,1681173207.951 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-11T00:33:27.952Z,1681173207.952 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-11T00:33:27.953Z,1681173207.953 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-11T00:33:59.567Z,1681173239.567 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -10 count 2023-04-11T00:33:59.568Z,1681173239.568 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-11T00:33:59.568Z,1681173239.568 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-11T00:33:59.569Z,1681173239.569 [CommandExec](IMPORTANT): got command run 2023-04-11T00:33:59.572Z,1681173239.572 [CommandExec](IMPORTANT): Running 2023-04-11T00:33:59.835Z,1681173239.835 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-11T00:33:59.923Z,1681173239.923 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-11T00:34:08.476Z,1681173248.476 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T00:34:19.366Z,1681173259.366 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001705 CHAN A1 (24V): 0.007823 CHAN A2 (12V): -0.002786 CHAN A3 (5V): -0.003404 CHAN B0 (3.3V): -0.001125 CHAN B1 (3.15aV): -0.001173 CHAN B2 (3.15bV): -0.000908 CHAN B3 (GND): 0.002278 OPEN: 0.012136 Full Scale: +/- 1 mA 2023-04-11T00:35:24.129Z,1681173324.129 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.669330 arcdeg , longitude = -121.872851 arcdeg . Distance from shore = 4253.901367 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-11T00:35:26.125Z,1681173326.125 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.123178 m 2023-04-11T00:36:35.730Z,1681173395.730 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-11T00:38:38.549Z,1681173518.549 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.084856 m 2023-04-11T00:41:28.653Z,1681173688.653 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.875028 m , 3.629600 ug/l , 3.054971 m , 12.503412 degC , 8.996000 ug/l 2023-04-11T00:41:29.043Z,1681173689.043 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.054971 m . 2023-04-11T00:41:29.415Z,1681173689.415 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-11T00:44:41.354Z,1681173881.354 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.054074 m , settling for 10.000000 min 2023-04-11T00:54:54.261Z,1681174494.261 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-11T00:54:54.471Z,1681174494.471 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-11T00:55:45.189Z,1681174545.189 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-11T00:55:45.000Z,1681174546.000 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -10 2023-04-11T00:55:46.001Z,1681174546.001 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-11T00:55:46.493Z,1681174546.493 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.132608 m in 12.558405 degC water with 8.834800 ug/l chlorophyll fluorescence. 2023-04-11T00:55:46.805Z,1681174546.805 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-11T00:55:47.209Z,1681174547.209 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-11T00:55:49.233Z,1681174549.233 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-11T00:56:26.829Z,1681174586.829 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@45,daSPR_phyto) 2023-04-11T00:56:29.653Z,1681174589.653 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2023-04-11T00:56:29.654Z,1681174589.654 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-11T00:56:30.057Z,1681174590.057 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-11T00:58:49.857Z,1681174729.857 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@0.0) 2023-04-11T00:58:58.755Z,1681174738.755 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-11T00:58:59.158Z,1681174739.158 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2023-04-11T01:03:58.968Z,1681175038.968 [ESPComponent](CRITICAL): [sample #9] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 143.0322 2023-04-11T01:03:59.168Z,1681175039.168 [CBIT](CRITICAL): Communications Fault in component: ESPComponent 2023-04-11T01:04:00.736Z,1681175040.736 [MissionManager](IMPORTANT): Started mission Default 2023-04-11T01:13:08.190Z,1681175588.190 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 9.117563 min 2023-04-11T01:16:17.617Z,1681175777.617 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T011616 2023-04-11T01:16:25.108Z,1681175785.108 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2023-04-11T01:16:25.111Z,1681175785.111 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/keepstation.tx 2023-04-11T01:16:25.775Z,1681175785.775 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl 2023-04-11T01:16:48.928Z,1681175808.928 [CommandExec](IMPORTANT): got command set keepstation.MissionTimeout 1 hour 2023-04-11T01:16:48.929Z,1681175808.929 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 30 minute 2023-04-11T01:16:48.930Z,1681175808.930 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.667344833083035 degree 2023-04-11T01:16:48.930Z,1681175808.930 [CommandExec](IMPORTANT): got command set keepstation.Longitude -121.87551866555421 degree 2023-04-11T01:16:48.931Z,1681175808.931 [CommandExec](IMPORTANT): got command run 2023-04-11T01:16:48.933Z,1681175808.933 [CommandExec](IMPORTANT): Running 2023-04-11T01:16:49.078Z,1681175809.078 [MissionManager](IMPORTANT): Started mission keepstation 2023-04-11T01:17:55.432Z,1681175875.432 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.667343,-121.875519 2023-04-11T01:21:11.718Z,1681176071.718 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.667343,-121.875519 2023-04-11T01:34:19.578Z,1681176859.578 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T01:34:30.501Z,1681176870.501 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.026732 CHAN A1 (24V): 0.052874 CHAN A2 (12V): 0.013778 CHAN A3 (5V): -0.001592 CHAN B0 (3.3V): -0.000517 CHAN B1 (3.15aV): -0.000940 CHAN B2 (3.15bV): -0.116024 CHAN B3 (GND): -0.002584 OPEN: 0.012051 Full Scale: +/- 1 mA 2023-04-11T02:16:49.501Z,1681179409.501 [MissionManager](IMPORTANT): Started mission Default 2023-04-11T02:25:35.057Z,1681179935.057 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.752130 min 2023-04-11T02:31:16.016Z,1681180276.016 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.435488 min 2023-04-11T02:34:30.522Z,1681180470.522 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T02:34:41.568Z,1681180481.568 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.024702 CHAN A1 (24V): 0.026173 CHAN A2 (12V): 0.013466 CHAN A3 (5V): -0.000848 CHAN B0 (3.3V): 0.000753 CHAN B1 (3.15aV): -0.000707 CHAN B2 (3.15bV): -0.000363 CHAN B3 (GND): -0.003291 OPEN: 0.012572 Full Scale: +/- 1 mA 2023-04-11T02:37:01.394Z,1681180621.394 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.191156 min 2023-04-11T02:42:45.229Z,1681180965.229 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.922103 min 2023-04-11T02:46:11.466Z,1681181171.466 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T024610 2023-04-11T02:46:19.032Z,1681181179.032 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h" 2zmcq 1 5.000000 2023-04-11T02:46:19.033Z,1681181179.033 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=2zmcq 2023-04-11T02:46:19.034Z,1681181179.034 [CommandExec](IMPORTANT): Scheduled #41 (#1 of 5 with id='2zmcq'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h", ASAP 2023-04-11T02:46:19.986Z,1681181179.986 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T024618 2023-04-11T02:46:27.568Z,1681181187.568 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m" 2zmcq 2 5.000000 2023-04-11T02:46:27.568Z,1681181187.568 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=2zmcq 2023-04-11T02:46:27.569Z,1681181187.569 [CommandExec](IMPORTANT): Scheduled #42 (#2 of 5 with id='2zmcq'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m", ASAP 2023-04-11T02:46:28.481Z,1681181188.481 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T024627 2023-04-11T02:46:39.724Z,1681181199.724 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DeepBoundChl 15 m;set IsothermDepthSampling.Lat1 36.671 degree;set IsothermDepthSampling.Lon1 -121.892 degree;set IsothermDepthSampling.CartridgeType1 -6 count" 2zmcq 3 5.000000 2023-04-11T02:46:39.725Z,1681181199.725 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=2zmcq 2023-04-11T02:46:39.726Z,1681181199.726 [CommandExec](IMPORTANT): Scheduled #43 (#3 of 5 with id='2zmcq'): "set IsothermDepthSampling.DeepBoundChl 15 m;set IsothermDepthSampling.Lat1 36.671 degree;set IsothermDepthSampling.Lon1 -121.892 degree;set IsothermDepthSampling.CartridgeType1 -6 count", ASAP 2023-04-11T02:46:40.678Z,1681181200.678 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T024639 2023-04-11T02:46:50.260Z,1681181210.260 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.DepDiffFromPeakChl2 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min" 2zmcq 4 5.000000 2023-04-11T02:46:50.261Z,1681181210.261 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=2zmcq 2023-04-11T02:46:50.262Z,1681181210.262 [CommandExec](IMPORTANT): Scheduled #44 (#4 of 5 with id='2zmcq'): "set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChl1 0 m;set IsothermDepthSampling.DepDiffFromPeakChl2 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min", ASAP 2023-04-11T02:46:51.213Z,1681181211.213 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T024650 2023-04-11T02:46:59.207Z,1681181219.207 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.WaitDepUndulation2 5 min;run" 2zmcq 5 5.000000 2023-04-11T02:46:59.207Z,1681181219.207 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=2zmcq 2023-04-11T02:46:59.209Z,1681181219.209 [CommandExec](IMPORTANT): Scheduled #45 (#5 of 5 with id='2zmcq'): "set IsothermDepthSampling.WaitDepUndulation2 5 min;run", ASAP 2023-04-11T02:46:59.340Z,1681181219.340 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-11T02:47:15.951Z,1681181235.951 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-11T02:47:42.622Z,1681181262.622 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count 2023-04-11T02:47:42.623Z,1681181262.623 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour 2023-04-11T02:47:42.624Z,1681181262.624 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-11T02:48:17.988Z,1681181297.988 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-11T02:48:17.989Z,1681181297.989 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-11T02:48:17.990Z,1681181297.990 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-11T02:48:17.991Z,1681181297.991 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-11T02:48:32.706Z,1681181312.706 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.712728 min 2023-04-11T02:48:55.339Z,1681181335.339 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter 2023-04-11T02:48:55.339Z,1681181335.339 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lat1 36.671 degree 2023-04-11T02:48:55.348Z,1681181335.348 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lon1 -121.892 degree 2023-04-11T02:48:55.349Z,1681181335.349 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType1 -6 count 2023-04-11T02:49:37.698Z,1681181377.698 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count 2023-04-11T02:49:37.699Z,1681181377.699 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl1 0 meter 2023-04-11T02:49:37.712Z,1681181377.712 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChl2 0 meter 2023-04-11T02:49:37.713Z,1681181377.713 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute 2023-04-11T02:49:54.614Z,1681181394.614 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation2 5 minute 2023-04-11T02:49:54.614Z,1681181394.614 [CommandExec](IMPORTANT): got command run 2023-04-11T02:49:54.617Z,1681181394.617 [CommandExec](IMPORTANT): Running 2023-04-11T02:49:54.805Z,1681181394.805 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-11T02:49:54.898Z,1681181394.898 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-11T02:50:43.169Z,1681181443.169 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.671001,-121.891998 2023-04-11T03:16:02.274Z,1681182962.274 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Reached waypoint: 36.671001,-121.891998 2023-04-11T03:16:59.667Z,1681183019.667 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.670805 arcdeg , longitude = -121.892168 arcdeg . Distance from shore = 4468.430664 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-11T03:17:01.676Z,1681183021.676 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.347114 m 2023-04-11T03:18:11.173Z,1681183091.173 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min . 2023-04-11T03:20:02.284Z,1681183202.284 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.082609 m 2023-04-11T03:22:06.720Z,1681183326.720 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 15.327385 m , 0.852800 ug/l , 3.246154 m , 11.936053 degC , 8.037123 ug/l 2023-04-11T03:22:07.119Z,1681183327.119 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.246154 m . 2023-04-11T03:22:07.546Z,1681183327.546 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2023-04-11T03:25:34.811Z,1681183534.811 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.222359 m , settling for 10.000000 min 2023-04-11T03:30:44.605Z,1681183844.605 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-11T03:34:41.924Z,1681184081.924 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T03:34:52.843Z,1681184092.843 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001493 CHAN A1 (24V): 0.007587 CHAN A2 (12V): 0.012882 CHAN A3 (5V): -0.001342 CHAN B0 (3.3V): 0.009942 CHAN B1 (3.15aV): -0.000623 CHAN B2 (3.15bV): -0.000885 CHAN B3 (GND): -0.003617 OPEN: 0.012640 Full Scale: +/- 1 mA 2023-04-11T03:35:47.706Z,1681184147.706 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-11T03:35:47.886Z,1681184147.886 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-11T03:36:38.629Z,1681184198.629 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-11T03:36:39.443Z,1681184199.443 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -6 2023-04-11T03:36:39.443Z,1681184199.443 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-11T03:36:39.965Z,1681184199.965 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.516747 m in 11.964289 degC water with 7.749041 ug/l chlorophyll fluorescence. 2023-04-11T03:36:40.245Z,1681184200.245 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-11T03:36:40.655Z,1681184200.655 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-11T03:36:42.673Z,1681184202.673 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-11T03:37:19.434Z,1681184239.434 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@44,archive_bac) 2023-04-11T03:37:21.857Z,1681184241.857 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2023-04-11T03:37:21.858Z,1681184241.858 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-11T03:37:22.262Z,1681184242.262 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-11T03:39:37.605Z,1681184377.605 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2023-04-11T03:39:38.012Z,1681184378.012 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-11T03:40:01.441Z,1681184401.441 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@2.3) 2023-04-11T03:40:01.845Z,1681184401.845 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@2.3) 2023-04-11T03:40:03.869Z,1681184403.869 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-11T03:40:04.270Z,1681184404.270 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@44,archive_bac) 2023-04-11T03:52:08.662Z,1681185128.662 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@44,archive_bac) 2023-04-11T03:52:18.763Z,1681185138.763 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-11T03:52:19.299Z,1681185139.299 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2023-04-11T03:57:19.067Z,1681185439.067 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2023-04-11T03:57:19.955Z,1681185439.955 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2023-04-11T03:57:36.538Z,1681185456.538 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min . 2023-04-11T03:57:40.143Z,1681185460.143 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 3.246154 m . 2023-04-11T03:57:40.554Z,1681185460.554 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 2023-04-11T03:58:43.592Z,1681185523.592 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.243451 m , settling for 5.000000 min 2023-04-11T04:03:56.308Z,1681185836.308 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-11T04:03:56.494Z,1681185836.494 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-11T04:04:47.121Z,1681185887.121 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-11T04:04:47.928Z,1681185887.928 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -10 2023-04-11T04:04:47.929Z,1681185887.929 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-11T04:04:48.434Z,1681185888.434 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.107924 m in 11.957453 degC water with 7.494240 ug/l chlorophyll fluorescence. 2023-04-11T04:04:48.733Z,1681185888.733 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-11T04:04:49.137Z,1681185889.137 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-11T04:04:51.158Z,1681185891.158 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-11T04:05:28.730Z,1681185928.730 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@43,daSPR_phyto) 2023-04-11T04:05:31.558Z,1681185931.558 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2023-04-11T04:05:31.559Z,1681185931.559 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-11T04:05:31.964Z,1681185931.964 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-11T04:07:50.937Z,1681186070.937 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2023-04-11T04:07:51.342Z,1681186071.342 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-11T04:34:53.120Z,1681187693.120 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T04:35:04.111Z,1681187704.111 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.155819 CHAN A1 (24V): 0.147942 CHAN A2 (12V): 0.062331 CHAN A3 (5V): 0.018717 CHAN B0 (3.3V): 0.011719 CHAN B1 (3.15aV): 0.012022 CHAN B2 (3.15bV): 0.012261 CHAN B3 (GND): -0.010429 OPEN: 0.012503 Full Scale: +/- 1 mA 2023-04-11T04:46:34.770Z,1681188394.770 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@551.1) 2023-04-11T04:46:35.173Z,1681188395.173 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@551.1) 2023-04-11T04:46:37.195Z,1681188397.195 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2023-04-11T04:46:37.598Z,1681188397.598 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@43,daSPR_phyto) 2023-04-11T05:35:04.174Z,1681191304.174 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T05:35:15.067Z,1681191315.067 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.068506 CHAN A1 (24V): 0.152536 CHAN A2 (12V): 0.040290 CHAN A3 (5V): 0.001634 CHAN B0 (3.3V): 0.004880 CHAN B1 (3.15aV): -0.000251 CHAN B2 (3.15bV): -0.000369 CHAN B3 (GND): -0.009120 OPEN: 0.012727 Full Scale: +/- 1 mA 2023-04-11T05:41:08.442Z,1681191668.442 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@43,daSPR_phyto) 2023-04-11T05:41:19.758Z,1681191679.758 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-11T05:41:20.204Z,1681191680.204 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2023-04-11T05:46:19.983Z,1681191979.983 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2023-04-11T05:46:20.862Z,1681191980.862 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed. 2023-04-11T05:46:23.878Z,1681191983.878 [MissionManager](IMPORTANT): Started mission Default 2023-04-11T05:49:15.523Z,1681192155.523 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2359, MT Status=1, MTMSN=2513 2023-04-11T05:49:16.044Z,1681192156.044 [CommandExec](IMPORTANT): got command stop 2023-04-11T05:49:16.045Z,1681192156.045 [CommandExec](IMPORTANT): Scheduling is paused 2023-04-11T05:54:56.624Z,1681192496.624 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T055455 2023-04-11T05:55:04.276Z,1681192504.276 [CommandExec](IMPORTANT): got command schedule clear 2023-04-11T05:55:04.276Z,1681192504.276 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2023-04-11T05:55:04.276Z,1681192504.276 [CommandExec](IMPORTANT): got command schedule resume 2023-04-11T05:55:04.276Z,1681192504.276 [CommandExec](IMPORTANT): Scheduling is resumed 2023-04-11T05:56:16.768Z,1681192576.768 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 9.875292 min 2023-04-11T06:02:09.730Z,1681192929.730 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T060208 2023-04-11T06:02:17.381Z,1681192937.381 [CommandExec](IMPORTANT): got command schedule asap "load Science/profile_station.xml;set profile_station.MissionTimeout 10 h;set profile_station.Lat 36.685 degree;set profile_station.Lon -121.898 degree;set profile_station.YoYoMaxDepth 60 m;set profile_station.Speed 0.8 m/s" 2zvfc 1 2.000000 2023-04-11T06:02:17.381Z,1681192937.381 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=2zvfc 2023-04-11T06:02:17.382Z,1681192937.382 [CommandExec](IMPORTANT): Scheduled #46 (#1 of 2 with id='2zvfc'): "load Science/profile_station.xml;set profile_station.MissionTimeout 10 h;set profile_station.Lat 36.685 degree;set profile_station.Lon -121.898 degree;set profile_station.YoYoMaxDepth 60 m;set profile_station.Speed 0.8 m/s", ASAP 2023-04-11T06:02:18.289Z,1681192938.289 [DataOverHttps](IMPORTANT): SBD MTMSN=20230411T060217 2023-04-11T06:02:25.826Z,1681192945.826 [CommandExec](IMPORTANT): got command schedule asap "set profile_station.MaxDepth 65 m;run" 2zvfc 2 2.000000 2023-04-11T06:02:25.827Z,1681192945.827 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=2zvfc 2023-04-11T06:02:25.828Z,1681192945.828 [CommandExec](IMPORTANT): Scheduled #47 (#2 of 2 with id='2zvfc'): "set profile_station.MaxDepth 65 m;run", ASAP 2023-04-11T06:02:25.955Z,1681192945.955 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml 2023-04-11T06:02:27.752Z,1681192947.752 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml 2023-04-11T06:02:28.653Z,1681192948.653 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.073229 min 2023-04-11T06:03:04.370Z,1681192984.370 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 10 hour 2023-04-11T06:03:04.371Z,1681192984.371 [CommandExec](IMPORTANT): got command set profile_station.Lat 36.685 degree 2023-04-11T06:03:04.372Z,1681192984.372 [CommandExec](IMPORTANT): got command set profile_station.Lon -121.898 degree 2023-04-11T06:03:04.373Z,1681192984.373 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 60 meter 2023-04-11T06:03:04.373Z,1681192984.373 [CommandExec](IMPORTANT): got command set profile_station.Speed 0.8 meter_per_second 2023-04-11T06:03:09.984Z,1681192989.984 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 65 meter 2023-04-11T06:03:09.985Z,1681192989.985 [CommandExec](IMPORTANT): got command run 2023-04-11T06:03:09.987Z,1681192989.987 [CommandExec](IMPORTANT): Running 2023-04-11T06:03:10.150Z,1681192990.150 [MissionManager](IMPORTANT): Started mission profile_station 2023-04-11T06:35:15.412Z,1681194915.412 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T06:35:26.383Z,1681194926.383 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.000451 CHAN A1 (24V): 0.007781 CHAN A2 (12V): -0.003422 CHAN A3 (5V): -0.003264 CHAN B0 (3.3V): -0.001838 CHAN B1 (3.15aV): -0.000972 CHAN B2 (3.15bV): -0.000716 CHAN B3 (GND): 0.018637 OPEN: 0.012587 Full Scale: +/- 1 mA 2023-04-11T07:35:26.629Z,1681198526.629 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T07:35:37.539Z,1681198537.539 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001782 CHAN A1 (24V): 0.009108 CHAN A2 (12V): -0.003168 CHAN A3 (5V): -0.003831 CHAN B0 (3.3V): -0.000628 CHAN B1 (3.15aV): -0.001415 CHAN B2 (3.15bV): -0.000878 CHAN B3 (GND): -0.001118 OPEN: 0.012341 Full Scale: +/- 1 mA 2023-04-11T08:35:37.759Z,1681202137.759 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T08:35:48.715Z,1681202148.715 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.001571 CHAN A1 (24V): 0.007758 CHAN A2 (12V): -0.057557 CHAN A3 (5V): -0.002508 CHAN B0 (3.3V): -0.001641 CHAN B1 (3.15aV): 0.004195 CHAN B2 (3.15bV): -0.000805 CHAN B3 (GND): -0.001399 OPEN: 0.012583 Full Scale: +/- 1 mA 2023-04-11T09:35:49.147Z,1681205749.147 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T09:36:00.109Z,1681205760.109 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.015800 CHAN A1 (24V): 0.008366 CHAN A2 (12V): -0.003230 CHAN A3 (5V): -0.002963 CHAN B0 (3.3V): -0.001026 CHAN B1 (3.15aV): -0.001255 CHAN B2 (3.15bV): -0.000837 CHAN B3 (GND): -0.045736 OPEN: 0.012468 Full Scale: +/- 1 mA 2023-04-11T10:21:29.140Z,1681208489.140 [ThrusterHE](FAULT): Could not get status:serial timeout 2023-04-11T10:36:00.560Z,1681209360.560 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T10:36:11.782Z,1681209371.782 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001699 CHAN A1 (24V): 0.006954 CHAN A2 (12V): -0.000406 CHAN A3 (5V): -0.003075 CHAN B0 (3.3V): -0.001517 CHAN B1 (3.15aV): 0.014011 CHAN B2 (3.15bV): -0.000923 CHAN B3 (GND): -0.001007 OPEN: 0.012463 Full Scale: +/- 1 mA 2023-04-11T11:36:12.172Z,1681212972.172 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T11:36:23.471Z,1681212983.471 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.011682 CHAN A1 (24V): 0.008435 CHAN A2 (12V): -0.003287 CHAN A3 (5V): -0.003255 CHAN B0 (3.3V): -0.000693 CHAN B1 (3.15aV): -0.000960 CHAN B2 (3.15bV): 0.013786 CHAN B3 (GND): -0.000547 OPEN: 0.012418 Full Scale: +/- 1 mA 2023-04-11T11:44:09.168Z,1681213449.168 [DVL_micro](FAULT): Failed to initialize within timeout. 2023-04-11T12:36:23.589Z,1681216583.589 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T12:36:34.571Z,1681216594.571 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001029 CHAN A1 (24V): 0.007008 CHAN A2 (12V): -0.003884 CHAN A3 (5V): 0.000385 CHAN B0 (3.3V): -0.001226 CHAN B1 (3.15aV): -0.000599 CHAN B2 (3.15bV): 0.010868 CHAN B3 (GND): -0.001087 OPEN: 0.012472 Full Scale: +/- 1 mA 2023-04-11T13:36:34.964Z,1681220194.964 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T13:36:45.866Z,1681220205.866 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001032 CHAN A1 (24V): 0.007487 CHAN A2 (12V): -0.003463 CHAN A3 (5V): -0.003728 CHAN B0 (3.3V): -0.002265 CHAN B1 (3.15aV): -0.001240 CHAN B2 (3.15bV): 0.001132 CHAN B3 (GND): 0.017950 OPEN: 0.012736 Full Scale: +/- 1 mA 2023-04-11T13:52:57.709Z,1681221177.709 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-11T14:36:46.340Z,1681223806.340 [CBIT](IMPORTANT): Beginning ground fault scan 2023-04-11T14:36:57.226Z,1681223817.226 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.001733 CHAN A1 (24V): 0.006798 CHAN A2 (12V): -0.003802 CHAN A3 (5V): -0.003676 CHAN B0 (3.3V): -0.001582 CHAN B1 (3.15aV): -0.001091 CHAN B2 (3.15bV): -0.047800 CHAN B3 (GND): -0.001080 OPEN: 0.012559 Full Scale: +/- 1 mA 2023-04-11T14:52:25.507Z,1681224745.507 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2404, MT Status=1, MTMSN=2514