2023-04-14T18:44:27.601Z,1681497867.601 [CommandExec](IMPORTANT): got command restart logs
2023-04-14T20:54:19.342Z,1681505659.342 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Reached waypoint: 36.868000,-121.967003
2023-04-14T20:55:21.988Z,1681505721.988 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.868009 arcdeg , longitude = -121.967193 arcdeg . Distance from shore = 9600.499023 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-14T20:55:23.981Z,1681505723.981 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.580104 m 
2023-04-14T20:56:33.495Z,1681505793.495 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-14T20:58:18.157Z,1681505898.157 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.062395 m 
2023-04-14T21:01:34.870Z,1681506094.870 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 20.441902 m , 1.368800 ug/l , 8.072393 m , 9.882103 degC , 1.611820 ug/l 
2023-04-14T21:01:35.316Z,1681506095.316 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 8.072393 m .
2023-04-14T21:01:35.706Z,1681506095.706 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-14T21:05:25.558Z,1681506325.558 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 8.053731 m , settling for 5.000000 min 
2023-04-14T21:10:38.333Z,1681506638.333 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-14T21:10:38.457Z,1681506638.457 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-14T21:11:38.426Z,1681506698.426 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-14T21:11:39.231Z,1681506699.231 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6
2023-04-14T21:11:39.232Z,1681506699.232 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-14T21:11:39.678Z,1681506699.678 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 7.820793 m in 9.862543 degC water with 1.671560 ug/l chlorophyll fluorescence.
2023-04-14T21:11:40.058Z,1681506700.058 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-14T21:11:40.435Z,1681506700.435 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-14T21:11:42.455Z,1681506702.455 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-14T21:12:14.367Z,1681506734.367 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@55,archive_bac)
2023-04-14T21:12:17.197Z,1681506737.197 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0)
2023-04-14T21:12:17.198Z,1681506737.198 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-14T21:12:17.597Z,1681506737.597 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-14T21:15:00.009Z,1681506900.009 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2023-04-14T21:15:00.413Z,1681506900.413 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-14T22:05:42.565Z,1681509942.565 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0)
2023-04-14T22:05:42.974Z,1681509942.974 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-14T22:05:45.003Z,1681509945.003 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-14T22:05:45.397Z,1681509945.397 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,archive_bac)
2023-04-14T22:17:48.967Z,1681510668.967 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@55,archive_bac)
2023-04-14T22:17:59.072Z,1681510679.072 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-14T22:17:59.540Z,1681510679.540 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED
2023-04-14T22:22:59.264Z,1681510979.264 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2023-04-14T22:23:00.113Z,1681510980.113 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-14T22:23:03.092Z,1681510983.092 [MissionManager](IMPORTANT): Started mission Default
2023-04-14T22:28:31.733Z,1681511311.733 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-14T22:33:56.252Z,1681511636.252 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.879007 min 
2023-04-14T22:33:57.818Z,1681511637.818 [NAL9602](FAULT): GPS failed to acquire within timeout.
2023-04-14T22:34:56.602Z,1681511696.602 [DataOverHttps](IMPORTANT): SBD MTMSN=20230414T223455
2023-04-14T22:35:04.156Z,1681511704.156 [CommandExec](IMPORTANT): got command schedule "load Science/sci2.tl;set sci2.MissionTimeout 8 h;set sci2.Lat1 36.835 degree;set sci2.Lon1 -121.855 degree;set sci2.YoYoMaxDepth 50 m;set sci2.MinAltitude 7 m;set sci2.MaxDepth 60 m;run"
2023-04-14T22:35:04.164Z,1681511704.164 [CommandExec](IMPORTANT): Scheduled #24: "load Science/sci2.tl;set sci2.MissionTimeout 8 h;set sci2.Lat1 36.835 degree;set sci2.Lon1 -121.855 degree;set sci2.YoYoMaxDepth 50 m;set sci2.MinAltitude 7 m;set sci2.MaxDepth 60 m;run", AFTER MISSION
2023-04-14T22:35:04.261Z,1681511704.261 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl
2023-04-14T22:35:04.263Z,1681511704.263 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx
2023-04-14T22:35:06.099Z,1681511706.099 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl
2023-04-14T22:35:19.761Z,1681511719.761 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 8 hour
2023-04-14T22:35:19.762Z,1681511719.762 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.835 degree
2023-04-14T22:35:19.763Z,1681511719.763 [CommandExec](IMPORTANT): got command set sci2.Lon1 -121.855 degree
2023-04-14T22:35:19.763Z,1681511719.763 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter
2023-04-14T22:35:19.764Z,1681511719.764 [CommandExec](IMPORTANT): got command set sci2.MinAltitude 7 meter
2023-04-14T22:35:19.765Z,1681511719.765 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter
2023-04-14T22:35:19.766Z,1681511719.766 [CommandExec](IMPORTANT): got command run
2023-04-14T22:35:19.767Z,1681511719.767 [CommandExec](IMPORTANT): Running 
2023-04-14T22:35:19.870Z,1681511719.870 [MissionManager](IMPORTANT): Started mission sci2
2023-04-14T22:42:20.957Z,1681512140.957 [sci2:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout
2023-04-14T22:44:10.415Z,1681512250.415 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.834999,-121.855003
2023-04-14T22:49:11.019Z,1681512551.019 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-14T22:49:22.011Z,1681512562.011 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): -0.003290
CHAN A1 (24V): -0.001361
CHAN A2 (12V): -0.000228
CHAN A3 (5V): -0.000944
CHAN B0 (3.3V): 0.000380
CHAN B1 (3.15aV): -0.086064
CHAN B2 (3.15bV): 0.000335
CHAN B3 (GND): 0.000291
OPEN: 0.007795
Full Scale: +/- 1 mA
2023-04-14T23:50:33.941Z,1681516233.941 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-14T23:56:17.365Z,1681516577.365 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T01:35:42.181Z,1681522542.181 [Radio_Surface](FAULT): LCB fault:  Invalid Command.
2023-04-15T01:42:49.358Z,1681522969.358 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 36.834999,-121.855003
2023-04-15T01:42:52.168Z,1681522972.168 [MissionManager](IMPORTANT): Started mission Default
2023-04-15T01:49:21.317Z,1681523361.317 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T01:59:43.352Z,1681523983.352 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.846324 min 
2023-04-15T01:59:44.962Z,1681523984.962 [NAL9602](FAULT): GPS failed to acquire within timeout.
2023-04-15T02:11:36.318Z,1681524696.318 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 28.729053 min 
2023-04-15T02:11:37.922Z,1681524697.922 [NAL9602](FAULT): GPS failed to acquire within timeout.
2023-04-15T02:13:13.370Z,1681524793.370 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T021312
2023-04-15T02:14:24.929Z,1681524864.929 [CommandExec](IMPORTANT): got command schedule "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 3 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 36zdz 1 3.000000
2023-04-15T02:14:24.930Z,1681524864.930 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=36zdz
2023-04-15T02:14:24.931Z,1681524864.931 [CommandExec](IMPORTANT): Scheduled #25 (#1 of 3 with id='36zdz'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 3 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", AFTER MISSION
2023-04-15T02:14:25.882Z,1681524865.882 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T021425
2023-04-15T02:20:37.364Z,1681525237.364 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15501, MT Status=1, MTMSN=1131
2023-04-15T02:20:38.162Z,1681525238.162 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m" 36zdz 2 3.000000
2023-04-15T02:20:38.162Z,1681525238.162 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=36zdz
2023-04-15T02:20:38.163Z,1681525238.163 [CommandExec](IMPORTANT): Scheduled #26 (#2 of 3 with id='36zdz'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m", AFTER MISSION
2023-04-15T02:20:59.215Z,1681525259.215 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15502, MT Status=1, MTMSN=1132
2023-04-15T02:20:59.933Z,1681525259.933 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run" 36zdz 3 3.000000
2023-04-15T02:20:59.933Z,1681525259.933 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=36zdz
2023-04-15T02:20:59.934Z,1681525259.934 [CommandExec](IMPORTANT): Scheduled #27 (#3 of 3 with id='36zdz'): "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run", AFTER MISSION
2023-04-15T02:21:00.090Z,1681525260.090 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T02:21:15.791Z,1681525275.791 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T02:21:35.614Z,1681525295.614 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 3 hour
2023-04-15T02:21:35.615Z,1681525295.615 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-15T02:21:35.616Z,1681525295.616 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-15T02:22:02.105Z,1681525322.105 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-15T02:22:02.106Z,1681525322.106 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-15T02:22:02.108Z,1681525322.108 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-15T02:22:02.109Z,1681525322.109 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter
2023-04-15T02:22:16.216Z,1681525336.216 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-15T02:22:16.217Z,1681525336.217 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 5 minute
2023-04-15T02:22:16.218Z,1681525336.218 [CommandExec](IMPORTANT): got command run
2023-04-15T02:22:16.220Z,1681525336.220 [CommandExec](IMPORTANT): Running 
2023-04-15T02:22:16.546Z,1681525336.546 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-15T02:22:16.610Z,1681525336.610 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-15T02:22:22.184Z,1681525342.184 [NAL9602](FAULT): GPS failed to acquire within timeout.
2023-04-15T02:24:47.281Z,1681525487.281 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m" 36zdz 2 3.000000
2023-04-15T02:24:47.282Z,1681525487.282 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=36zdz
2023-04-15T02:24:47.282Z,1681525487.282 [CommandExec](IMPORTANT): Scheduled #28 (#2 of 3 with id='36zdz'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m", AFTER MISSION
2023-04-15T02:32:25.241Z,1681525945.241 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.832065 arcdeg , longitude = -121.854505 arcdeg . Distance from shore = 4306.732422 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-15T02:32:27.269Z,1681525947.269 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.486754 m 
2023-04-15T02:33:36.786Z,1681526016.786 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-15T02:35:07.702Z,1681526107.702 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.069815 m 
2023-04-15T02:39:01.578Z,1681526341.578 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 20.415173 m , 5.289600 ug/l , 3.588924 m , 10.984451 degC , 7.287709 ug/l 
2023-04-15T02:39:01.984Z,1681526341.984 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.588924 m .
2023-04-15T02:39:02.391Z,1681526342.391 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-15T02:43:29.050Z,1681526609.050 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.569513 m , settling for 5.000000 min 
2023-04-15T02:48:41.800Z,1681526921.800 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-15T02:48:42.126Z,1681526922.126 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-15T02:49:34.398Z,1681526974.398 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-15T02:49:35.238Z,1681526975.238 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6
2023-04-15T02:49:35.238Z,1681526975.238 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-15T02:49:35.702Z,1681526975.702 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.053490 m in 11.117090 degC water with 6.351000 ug/l chlorophyll fluorescence.
2023-04-15T02:49:36.013Z,1681526976.013 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-15T02:49:36.430Z,1681526976.430 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-15T02:49:38.435Z,1681526978.435 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-15T02:50:10.341Z,1681527010.341 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@54,archive_bac)
2023-04-15T02:50:13.201Z,1681527013.201 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0)
2023-04-15T02:50:13.202Z,1681527013.202 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-15T02:50:13.593Z,1681527013.593 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-15T02:52:56.826Z,1681527176.826 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2023-04-15T02:52:57.204Z,1681527177.204 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-15T03:32:33.595Z,1681529553.595 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@621.8)
2023-04-15T03:32:33.969Z,1681529553.969 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@621.8)
2023-04-15T03:32:35.991Z,1681529555.991 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-15T03:32:36.402Z,1681529556.402 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,archive_bac)
2023-04-15T03:44:39.975Z,1681530279.975 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@54,archive_bac)
2023-04-15T03:44:50.070Z,1681530290.070 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-15T03:44:50.502Z,1681530290.502 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED
2023-04-15T03:49:50.243Z,1681530590.243 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally.
2023-04-15T03:49:51.075Z,1681530591.075 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-15T03:49:54.142Z,1681530594.142 [MissionManager](IMPORTANT): Started mission Default
2023-04-15T03:54:15.485Z,1681530855.485 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T035414
2023-04-15T03:54:26.515Z,1681530866.515 [CommandExec](IMPORTANT): got command schedule clear
2023-04-15T03:54:26.515Z,1681530866.515 [CommandExec](IMPORTANT): Cleared 1 scheduled commands.
2023-04-15T03:54:26.516Z,1681530866.516 [CommandExec](IMPORTANT): got command schedule resume
2023-04-15T03:59:05.959Z,1681531145.959 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 9.190353 min 
2023-04-15T04:00:40.720Z,1681531240.720 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15514, MT Status=1, MTMSN=1133
2023-04-15T04:00:41.490Z,1681531241.490 [CommandExec](IMPORTANT): got command schedule "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 8 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 374bg 1 4.000000
2023-04-15T04:00:41.490Z,1681531241.490 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=374bg
2023-04-15T04:00:41.516Z,1681531241.516 [CommandExec](IMPORTANT): Scheduled #29 (#1 of 4 with id='374bg'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 8 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", AFTER MISSION
2023-04-15T04:01:06.242Z,1681531266.242 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15515, MT Status=1, MTMSN=1134
2023-04-15T04:01:07.073Z,1681531267.073 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.MaxDepth 60 m;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m" 374bg 2 4.000000
2023-04-15T04:01:07.073Z,1681531267.073 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=374bg
2023-04-15T04:01:07.074Z,1681531267.074 [CommandExec](IMPORTANT): Scheduled #30 (#2 of 4 with id='374bg'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.MaxDepth 60 m;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m", AFTER MISSION
2023-04-15T04:02:11.679Z,1681531331.679 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=15516, MT Status=1, MTMSN=1135
2023-04-15T04:02:12.449Z,1681531332.449 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 50 m;set IsothermDepthSampling.Lat1 36.92 degree;set IsothermDepthSampling.Lon1 -121.9 degree" 374bg 3 4.000000
2023-04-15T04:02:12.449Z,1681531332.449 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=374bg
2023-04-15T04:02:12.450Z,1681531332.450 [CommandExec](IMPORTANT): Scheduled #31 (#3 of 4 with id='374bg'): "set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 50 m;set IsothermDepthSampling.Lat1 36.92 degree;set IsothermDepthSampling.Lon1 -121.9 degree", AFTER MISSION
2023-04-15T04:02:22.591Z,1681531342.591 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=15517, MT Status=1, MTMSN=1136
2023-04-15T04:02:23.281Z,1681531343.281 [CommandExec](IMPORTANT): got command schedule "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run" 374bg 4 4.000000
2023-04-15T04:02:23.282Z,1681531343.282 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=374bg
2023-04-15T04:02:23.282Z,1681531343.282 [CommandExec](IMPORTANT): Scheduled #32 (#4 of 4 with id='374bg'): "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run", AFTER MISSION
2023-04-15T04:02:23.458Z,1681531343.458 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T04:02:39.261Z,1681531359.261 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T04:03:04.287Z,1681531384.287 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 8 hour
2023-04-15T04:03:04.288Z,1681531384.288 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-15T04:03:04.304Z,1681531384.304 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-15T04:03:35.981Z,1681531415.981 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-15T04:03:35.982Z,1681531415.982 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-15T04:03:35.983Z,1681531415.983 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxDepth 60 meter
2023-04-15T04:03:35.984Z,1681531415.984 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-15T04:03:35.985Z,1681531415.985 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter
2023-04-15T04:03:58.666Z,1681531438.666 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMinDepth 2 meter
2023-04-15T04:03:58.666Z,1681531438.666 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMaxDepth 50 meter
2023-04-15T04:03:58.667Z,1681531438.667 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lat1 36.92 degree
2023-04-15T04:03:58.668Z,1681531438.668 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lon1 -121.9 degree
2023-04-15T04:04:13.800Z,1681531453.800 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-15T04:04:13.801Z,1681531453.801 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 5 minute
2023-04-15T04:04:13.802Z,1681531453.802 [CommandExec](IMPORTANT): got command run
2023-04-15T04:04:13.806Z,1681531453.806 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-15T04:04:13.808Z,1681531453.808 [CommandExec](IMPORTANT): Running 
2023-04-15T04:04:13.909Z,1681531453.909 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-15T04:14:11.139Z,1681532051.139 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.919998,-121.900002
2023-04-15T04:19:11.250Z,1681532351.250 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T04:49:22.080Z,1681534162.080 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-15T04:49:33.011Z,1681534173.011 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002785
CHAN A1 (24V): -0.000667
CHAN A2 (12V): -0.000286
CHAN A3 (5V): -0.000655
CHAN B0 (3.3V): 0.000046
CHAN B1 (3.15aV): -0.000087
CHAN B2 (3.15bV): 0.000342
CHAN B3 (GND): 0.027213
OPEN: 0.007624
Full Scale: +/- 1 mA
2023-04-15T05:19:34.954Z,1681535974.954 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15521, MT Status=1, MTMSN=1137
2023-04-15T05:19:35.694Z,1681535975.694 [CommandExec](IMPORTANT): got command schedule "load Science/profile_station.xml;set profile_station.MissionTimeout 12 h;set profile_station.Lat 36.874 degree;set profile_station.Lon -121.951 degree;set profile_station.YoYoMaxDepth 50 m;set profile_station.MaxDepth 60 m;run"
2023-04-15T05:19:35.694Z,1681535975.694 [CommandExec](IMPORTANT): Scheduled #33: "load Science/profile_station.xml;set profile_station.MissionTimeout 12 h;set profile_station.Lat 36.874 degree;set profile_station.Lon -121.951 degree;set profile_station.YoYoMaxDepth 50 m;set profile_station.MaxDepth 60 m;run", AFTER MISSION
2023-04-15T06:17:21.692Z,1681539441.692 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2023-04-15T06:47:07.137Z,1681541227.137 [Radio_Surface](FAULT): LCB fault:  Invalid Command. Hardware Overcurrent Shutdown.
2023-04-15T07:34:42.965Z,1681544082.965 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T07:39:04.884Z,1681544344.884 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Reached waypoint: 36.919998,-121.900002
2023-04-15T07:39:31.977Z,1681544371.977 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.920151 arcdeg , longitude = -121.900059 arcdeg . Distance from shore = 3450.868896 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-15T07:39:33.989Z,1681544373.989 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.913746 m 
2023-04-15T07:40:43.503Z,1681544443.503 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-15T08:39:33.326Z,1681547973.326 [IsothermDepthSampling:TakeSamples:NoFiringForTooLong:B](IMPORTANT): No firing for too long. Stopping mission. ElapsedTime, MaxWaitNoFiring = 60.002608 min , 60.000000 min 
2023-04-15T08:39:33.732Z,1681547973.732 [CommandExec](IMPORTANT): got command stop
2023-04-15T08:39:33.733Z,1681547973.733 [CommandExec](IMPORTANT): Scheduling is paused for 1 commands
2023-04-15T08:39:35.105Z,1681547975.105 [MissionManager](IMPORTANT): Started mission Default
2023-04-15T08:48:03.101Z,1681548483.101 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.459413 min 
2023-04-15T08:53:38.600Z,1681548818.600 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.051547 min 
2023-04-15T08:58:43.254Z,1681549123.254 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T08:59:09.181Z,1681549149.181 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.560260 min 
2023-04-15T09:04:40.040Z,1681549480.040 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.075682 min 
2023-04-15T09:10:10.557Z,1681549810.557 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.584172 min 
2023-04-15T09:15:15.298Z,1681550115.298 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T09:15:44.800Z,1681550144.800 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 36.155042 min 
2023-04-15T09:21:15.305Z,1681550475.305 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 41.663167 min 
2023-04-15T09:26:45.426Z,1681550805.426 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 47.165316 min 
2023-04-15T09:32:15.525Z,1681551135.525 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 52.666903 min 
2023-04-15T09:37:48.912Z,1681551468.912 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 58.223360 min 
2023-04-15T09:43:19.040Z,1681551799.040 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 63.725557 min 
2023-04-15T09:48:49.113Z,1681552129.113 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 69.227059 min 
2023-04-15T09:54:18.847Z,1681552458.847 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 74.723112 min 
2023-04-15T09:59:50.652Z,1681552790.652 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 80.252710 min 
2023-04-15T10:05:21.573Z,1681553121.573 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 85.767725 min 
2023-04-15T10:10:26.201Z,1681553426.201 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T10:10:52.093Z,1681553452.093 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 91.276090 min 
2023-04-15T10:16:22.253Z,1681553782.253 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 96.779159 min 
2023-04-15T10:21:53.177Z,1681554113.177 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 102.294425 min 
2023-04-15T10:27:26.144Z,1681554446.144 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 107.843896 min 
2023-04-15T10:32:30.003Z,1681554750.003 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T10:32:56.665Z,1681554776.665 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 113.352547 min 
2023-04-15T10:38:26.736Z,1681555106.736 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 118.853792 min 
2023-04-15T10:43:56.489Z,1681555436.489 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 124.349707 min 
2023-04-15T10:49:01.218Z,1681555741.218 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T10:49:30.702Z,1681555770.702 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 129.919743 min 
2023-04-15T10:49:33.166Z,1681555773.166 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-15T10:49:44.095Z,1681555784.095 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.020452
CHAN A1 (24V): -0.001253
CHAN A2 (12V): -0.000146
CHAN A3 (5V): 0.000025
CHAN B0 (3.3V): -0.000297
CHAN B1 (3.15aV): 0.000795
CHAN B2 (3.15bV): -0.000237
CHAN B3 (GND): 0.000019
OPEN: 0.007778
Full Scale: +/- 1 mA
2023-04-15T10:55:00.807Z,1681556100.807 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 135.421411 min 
2023-04-15T11:00:30.909Z,1681556430.909 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 140.923372 min 
2023-04-15T11:06:01.407Z,1681556761.407 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 146.431641 min 
2023-04-15T11:11:06.110Z,1681557066.110 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T11:11:36.830Z,1681557096.830 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 152.021956 min 
2023-04-15T11:16:46.445Z,1681557406.445 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T11:17:09.476Z,1681557429.476 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 157.566081 min 
2023-04-15T11:18:23.561Z,1681557503.561 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T111823
2023-04-15T11:18:32.375Z,1681557512.375 [CommandExec](IMPORTANT): got command schedule asap "schedule resume" 37oqf 1 2.000000
2023-04-15T11:18:32.375Z,1681557512.375 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=37oqf
2023-04-15T11:18:32.376Z,1681557512.376 [CommandExec](IMPORTANT): Scheduled #34 (#1 of 2 with id='37oqf'): "schedule resume", ASAP
2023-04-15T11:18:33.366Z,1681557513.366 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T111832
2023-04-15T11:18:40.814Z,1681557520.814 [CommandExec](IMPORTANT): got command schedule asap "" 37oqf 2 2.000000
2023-04-15T11:18:40.815Z,1681557520.815 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=37oqf
2023-04-15T11:18:40.816Z,1681557520.816 [CommandExec](IMPORTANT): Scheduled #35 (#2 of 2 with id='37oqf'): "", ASAP
2023-04-15T11:18:40.918Z,1681557520.918 [CommandExec](IMPORTANT): got command schedule resume
2023-04-15T11:18:40.918Z,1681557520.918 [CommandExec](IMPORTANT): Scheduling is resumed
2023-04-15T11:18:41.151Z,1681557521.151 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml
2023-04-15T11:18:43.471Z,1681557523.471 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml
2023-04-15T11:19:05.165Z,1681557545.165 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 12 hour
2023-04-15T11:19:05.166Z,1681557545.166 [CommandExec](IMPORTANT): got command set profile_station.Lat 36.874 degree
2023-04-15T11:19:05.167Z,1681557545.167 [CommandExec](IMPORTANT): got command set profile_station.Lon -121.951 degree
2023-04-15T11:19:05.168Z,1681557545.168 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 50 meter
2023-04-15T11:19:05.169Z,1681557545.169 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 60 meter
2023-04-15T11:19:05.169Z,1681557545.169 [CommandExec](IMPORTANT): got command run
2023-04-15T11:19:05.174Z,1681557545.174 [CommandExec](IMPORTANT): Running 
2023-04-15T11:19:05.481Z,1681557545.481 [MissionManager](IMPORTANT): Started mission profile_station
2023-04-15T11:24:16.597Z,1681557856.597 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T12:05:04.420Z,1681560304.420 [Radio_Surface](FAULT): LCB fault:  Hardware Overcurrent Shutdown.
2023-04-15T12:22:26.088Z,1681561346.088 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2023-04-15T13:44:11.866Z,1681566251.866 [Radio_Surface](FAULT): LCB fault:  Invalid Command.
2023-04-15T14:23:16.888Z,1681568596.888 [ThrusterHE](IMPORTANT): Status lock Detected.
2023-04-15T14:23:17.364Z,1681568597.364 [ThrusterHE](IMPORTANT): Status lock Detected.
2023-04-15T14:23:17.764Z,1681568597.764 [ThrusterHE](IMPORTANT): Status lock Detected.
2023-04-15T14:23:18.319Z,1681568598.319 [ThrusterHE](IMPORTANT): Status lock Detected.
2023-04-15T14:23:18.527Z,1681568598.527 [ThrusterHE](IMPORTANT): Status lock Detected.
2023-04-15T14:35:32.947Z,1681569332.947 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15545, MT Status=1, MTMSN=1138
2023-04-15T14:35:33.433Z,1681569333.433 [CommandExec](IMPORTANT): got command stop
2023-04-15T14:35:33.433Z,1681569333.433 [CommandExec](IMPORTANT): Scheduling is paused
2023-04-15T14:35:33.750Z,1681569333.750 [MissionManager](IMPORTANT): Started mission Default
2023-04-15T14:36:34.382Z,1681569394.382 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=15546, MT Status=1, MTMSN=1139
2023-04-15T14:36:34.963Z,1681569394.963 [CommandExec](IMPORTANT): got command schedule clear
2023-04-15T14:36:34.964Z,1681569394.964 [CommandExec](IMPORTANT): Cleared 0 scheduled commands.
2023-04-15T14:36:34.964Z,1681569394.964 [CommandExec](IMPORTANT): got command schedule resume
2023-04-15T14:36:34.964Z,1681569394.964 [CommandExec](IMPORTANT): Scheduling is resumed
2023-04-15T14:39:06.614Z,1681569546.614 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T143906
2023-04-15T14:39:14.100Z,1681569554.100 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 37y0a 1 4.000000
2023-04-15T14:39:14.101Z,1681569554.101 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=37y0a
2023-04-15T14:39:14.102Z,1681569554.102 [CommandExec](IMPORTANT): Scheduled #36 (#1 of 4 with id='37y0a'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP
2023-04-15T14:39:15.030Z,1681569555.030 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T143914
2023-04-15T14:39:22.560Z,1681569562.560 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m" 37y0a 2 4.000000
2023-04-15T14:39:22.561Z,1681569562.561 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=37y0a
2023-04-15T14:39:22.562Z,1681569562.562 [CommandExec](IMPORTANT): Scheduled #37 (#2 of 4 with id='37y0a'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m", ASAP
2023-04-15T14:39:23.506Z,1681569563.506 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T143923
2023-04-15T14:39:31.004Z,1681569571.004 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 30 m;set IsothermDepthSampling.Lat1 36.874 degree;set IsothermDepthSampling.Lon1 -121.961 degree" 37y0a 3 4.000000
2023-04-15T14:39:31.004Z,1681569571.004 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=37y0a
2023-04-15T14:39:31.005Z,1681569571.005 [CommandExec](IMPORTANT): Scheduled #38 (#3 of 4 with id='37y0a'): "set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 30 m;set IsothermDepthSampling.Lat1 36.874 degree;set IsothermDepthSampling.Lon1 -121.961 degree", ASAP
2023-04-15T14:39:31.962Z,1681569571.962 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T143931
2023-04-15T14:39:39.643Z,1681569579.643 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run" 37y0a 4 4.000000
2023-04-15T14:39:39.644Z,1681569579.644 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=37y0a
2023-04-15T14:39:39.645Z,1681569579.645 [CommandExec](IMPORTANT): Scheduled #39 (#4 of 4 with id='37y0a'): "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run", ASAP
2023-04-15T14:39:39.788Z,1681569579.788 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T14:39:55.793Z,1681569595.793 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-15T14:40:23.723Z,1681569623.723 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour
2023-04-15T14:40:23.723Z,1681569623.723 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-15T14:40:23.743Z,1681569623.743 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-15T14:40:59.926Z,1681569659.926 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-15T14:40:59.926Z,1681569659.926 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-15T14:40:59.927Z,1681569659.927 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-15T14:40:59.928Z,1681569659.928 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter
2023-04-15T14:41:26.112Z,1681569686.112 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMinDepth 2 meter
2023-04-15T14:41:26.117Z,1681569686.117 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMaxDepth 30 meter
2023-04-15T14:41:26.117Z,1681569686.117 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lat1 36.874 degree
2023-04-15T14:41:26.118Z,1681569686.118 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.Lon1 -121.961 degree
2023-04-15T14:41:42.334Z,1681569702.334 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-15T14:41:42.335Z,1681569702.335 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 5 minute
2023-04-15T14:41:42.336Z,1681569702.336 [CommandExec](IMPORTANT): got command run
2023-04-15T14:41:42.339Z,1681569702.339 [CommandExec](IMPORTANT): Running 
2023-04-15T14:41:42.433Z,1681569702.433 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-15T14:41:42.524Z,1681569702.524 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-15T14:42:04.282Z,1681569724.282 [DataOverHttps](IMPORTANT): SBD MTMSN=20230415T144203
2023-04-15T14:42:11.991Z,1681569731.991 [CommandExec](IMPORTANT): got command schedule clear
2023-04-15T14:42:11.991Z,1681569731.991 [CommandExec](IMPORTANT): Cleared 0 scheduled commands.
2023-04-15T14:42:11.991Z,1681569731.991 [CommandExec](IMPORTANT): got command schedule resume
2023-04-15T14:43:33.387Z,1681569813.387 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.874001,-121.960999
2023-04-15T14:48:33.991Z,1681570113.991 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-15T15:10:16.177Z,1681571416.177 [IsothermDepthSampling:TransitToSamplingLocationNoCommsOrNoTransit:Wpt1NoComms:Wpt1:G.Waypoint](IMPORTANT): Reached waypoint: 36.874001,-121.960999
2023-04-15T15:10:41.635Z,1681571441.635 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.873911 arcdeg , longitude = -121.961141 arcdeg . Distance from shore = 9006.054688 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-15T15:10:43.655Z,1681571443.655 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.883929 m 
2023-04-15T15:11:53.160Z,1681571513.160 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-15T15:13:43.902Z,1681571623.902 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.081831 m 
2023-04-15T15:19:13.523Z,1681571953.523 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 20.398676 m , 1.044000 ug/l , 3.347252 m , 10.077446 degC , 2.231262 ug/l 
2023-04-15T15:19:13.927Z,1681571953.927 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.347252 m .
2023-04-15T15:19:14.342Z,1681571954.342 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-15T15:23:26.071Z,1681572206.071 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.328777 m , settling for 5.000000 min 
2023-04-15T15:28:38.797Z,1681572518.797 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-15T15:28:39.125Z,1681572519.125 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-15T15:29:30.967Z,1681572570.967 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-15T15:29:31.772Z,1681572571.772 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6
2023-04-15T15:29:31.773Z,1681572571.773 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-15T15:29:32.233Z,1681572572.233 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.422559 m in 10.175989 degC water with 2.739920 ug/l chlorophyll fluorescence.
2023-04-15T15:29:32.590Z,1681572572.590 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-15T15:29:32.981Z,1681572572.981 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-15T15:29:35.006Z,1681572575.006 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-15T15:30:06.918Z,1681572606.918 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@53,archive_bac)
2023-04-15T15:30:08.946Z,1681572608.946 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0)
2023-04-15T15:30:09.358Z,1681572609.358 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-15T15:30:09.767Z,1681572609.767 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-15T15:32:54.182Z,1681572774.182 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0)
2023-04-15T15:32:54.582Z,1681572774.582 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-15T16:29:08.039Z,1681576148.039 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0)
2023-04-15T16:29:08.442Z,1681576148.442 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-15T16:29:10.458Z,1681576150.458 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-15T16:29:10.861Z,1681576150.861 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,archive_bac)
2023-04-15T16:41:14.433Z,1681576874.433 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@53,archive_bac)
2023-04-15T16:41:24.542Z,1681576884.542 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-15T16:41:24.964Z,1681576884.964 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED
2023-04-15T16:46:24.706Z,1681577184.706 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally.
2023-04-15T16:46:25.559Z,1681577185.559 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-15T16:46:28.463Z,1681577188.463 [MissionManager](IMPORTANT): Started mission Default
2023-04-15T16:49:44.403Z,1681577384.403 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-15T16:49:55.309Z,1681577395.309 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): -0.055376
CHAN A1 (24V): -0.000894
CHAN A2 (12V): -0.000555
CHAN A3 (5V): -0.000455
CHAN B0 (3.3V): 0.000057
CHAN B1 (3.15aV): 0.000121
CHAN B2 (3.15bV): 0.000219
CHAN B3 (GND): 0.000217
OPEN: 0.007687
Full Scale: +/- 1 mA