2023-04-16T13:40:52.777Z,1681652452.777 [CommandExec](IMPORTANT): got command restart logs 2023-04-16T13:42:31.270Z,1681652551.270 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=15650, MT Status=1, MTMSN=1154 2023-04-16T13:42:31.840Z,1681652551.840 [CommandExec](IMPORTANT): got command stop 2023-04-16T13:42:31.840Z,1681652551.840 [CommandExec](IMPORTANT): Scheduling is paused 2023-04-16T13:42:31.840Z,1681652551.840 [CommandExec](IMPORTANT): got command schedule clear 2023-04-16T13:42:31.841Z,1681652551.841 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2023-04-16T13:42:31.841Z,1681652551.841 [CommandExec](IMPORTANT): got command schedule resume 2023-04-16T13:42:31.841Z,1681652551.841 [CommandExec](IMPORTANT): Scheduling is resumed 2023-04-16T13:44:19.174Z,1681652659.174 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.161379 min 2023-04-16T13:46:05.603Z,1681652765.603 [DataOverHttps](IMPORTANT): SBD MOMSN=18228250, MTMSN=20230416T134606 2023-04-16T13:46:13.110Z,1681652773.110 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool" 39q8s 1 3.000000 2023-04-16T13:46:13.111Z,1681652773.111 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=39q8s 2023-04-16T13:46:13.112Z,1681652773.112 [CommandExec](IMPORTANT): Scheduled #59 (#1 of 3 with id='39q8s'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.MissionTimeout 4 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h;set IsothermDepthSampling.SendSampleStatusAndData 0 bool", ASAP 2023-04-16T13:46:14.098Z,1681652774.098 [DataOverHttps](IMPORTANT): SBD MTMSN=20230416T134614 2023-04-16T13:46:24.560Z,1681652784.560 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m" 39q8s 2 3.000000 2023-04-16T13:46:24.561Z,1681652784.561 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=39q8s 2023-04-16T13:46:24.562Z,1681652784.562 [CommandExec](IMPORTANT): Scheduled #60 (#2 of 3 with id='39q8s'): "set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m;set IsothermDepthSampling.DeepBoundChl 20 m", ASAP 2023-04-16T13:46:25.474Z,1681652785.474 [DataOverHttps](IMPORTANT): SBD MTMSN=20230416T134626 2023-04-16T13:46:33.227Z,1681652793.227 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run" 39q8s 3 3.000000 2023-04-16T13:46:33.228Z,1681652793.228 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=39q8s 2023-04-16T13:46:33.229Z,1681652793.229 [CommandExec](IMPORTANT): Scheduled #61 (#3 of 3 with id='39q8s'): "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 5 min;run", ASAP 2023-04-16T13:46:33.365Z,1681652793.365 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml 2023-04-16T13:46:49.517Z,1681652809.517 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml 2023-04-16T13:47:13.958Z,1681652833.958 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 4 hour 2023-04-16T13:47:13.958Z,1681652833.958 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour 2023-04-16T13:47:13.959Z,1681652833.959 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool 2023-04-16T13:47:42.376Z,1681652862.376 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool 2023-04-16T13:47:42.377Z,1681652862.377 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool 2023-04-16T13:47:42.378Z,1681652862.378 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter 2023-04-16T13:47:42.378Z,1681652862.378 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter 2023-04-16T13:48:07.569Z,1681652887.569 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter 2023-04-16T13:48:07.569Z,1681652887.569 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 5 minute 2023-04-16T13:48:07.570Z,1681652887.570 [CommandExec](IMPORTANT): got command run 2023-04-16T13:48:07.573Z,1681652887.573 [CommandExec](IMPORTANT): Running 2023-04-16T13:48:07.778Z,1681652887.778 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling 2023-04-16T13:48:07.843Z,1681652887.843 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2023-04-16T13:48:51.060Z,1681652931.060 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.869715 arcdeg , longitude = -121.950880 arcdeg . Distance from shore = 9617.932617 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m . 2023-04-16T13:48:53.098Z,1681652933.098 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.293566 m 2023-04-16T13:50:02.583Z,1681653002.583 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min . 2023-04-16T13:52:07.813Z,1681653127.813 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.055479 m 2023-04-16T13:59:16.481Z,1681653556.481 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 17.897427 m , 0.754000 ug/l , 3.441514 m , 10.430876 degC , 5.962400 ug/l 2023-04-16T13:59:16.885Z,1681653556.885 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.441514 m . 2023-04-16T13:59:17.303Z,1681653557.303 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2023-04-16T14:03:24.942Z,1681653804.942 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.436390 m , settling for 5.000000 min 2023-04-16T14:08:37.709Z,1681654117.709 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2023-04-16T14:08:38.069Z,1681654118.069 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2023-04-16T14:09:29.644Z,1681654169.644 [ESPComponent](IMPORTANT): ESP has connected as client 2023-04-16T14:09:30.468Z,1681654170.468 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6 2023-04-16T14:09:30.469Z,1681654170.469 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2023-04-16T14:09:30.945Z,1681654170.945 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.413484 m in 10.423487 degC water with 6.286040 ug/l chlorophyll fluorescence. 2023-04-16T14:09:31.266Z,1681654171.266 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2023-04-16T14:09:31.668Z,1681654171.668 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2023-04-16T14:09:33.695Z,1681654173.695 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2023-04-16T14:10:05.618Z,1681654205.618 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@49,archive_bac) 2023-04-16T14:10:08.067Z,1681654208.067 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2023-04-16T14:10:08.072Z,1681654208.072 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2023-04-16T14:10:08.451Z,1681654208.451 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2023-04-16T14:12:52.087Z,1681654372.087 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2023-04-16T14:12:52.499Z,1681654372.499 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2023-04-16T15:09:41.973Z,1681657781.973 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@931.3) 2023-04-16T15:09:42.357Z,1681657782.357 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@931.3) 2023-04-16T15:09:44.379Z,1681657784.379 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2023-04-16T15:09:44.782Z,1681657784.782 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,archive_bac) 2023-04-16T15:21:48.368Z,1681658508.368 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@49,archive_bac) 2023-04-16T15:21:58.463Z,1681658518.463 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2023-04-16T15:21:58.971Z,1681658518.971 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2023-04-16T15:26:58.739Z,1681658818.739 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2023-04-16T15:26:59.727Z,1681658819.727 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2023-04-16T15:27:02.631Z,1681658822.631 [MissionManager](IMPORTANT): Started mission Default 2023-04-16T15:32:44.317Z,1681659164.317 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2023-04-16T15:36:29.878Z,1681659389.878 [DataOverHttps](IMPORTANT): SBD MTMSN=20230416T153630 2023-04-16T15:36:37.458Z,1681659397.458 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.tl;set sci2.MissionTimeout 8 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 36.91 degree;set sci2.Lon1 -122.1 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MinAltitude 7 m;set sci2.MaxDepth 55 m" 39vcr 1 2.000000 2023-04-16T15:36:37.459Z,1681659397.459 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=39vcr 2023-04-16T15:36:37.460Z,1681659397.460 [CommandExec](IMPORTANT): Scheduled #62 (#1 of 2 with id='39vcr'): "load Science/sci2.tl;set sci2.MissionTimeout 8 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 36.91 degree;set sci2.Lon1 -122.1 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.MinAltitude 7 m;set sci2.MaxDepth 55 m", ASAP 2023-04-16T15:36:38.429Z,1681659398.429 [DataOverHttps](IMPORTANT): SBD MTMSN=20230416T153639 2023-04-16T15:36:45.998Z,1681659405.998 [CommandExec](IMPORTANT): got command schedule asap "run" 39vcr 2 2.000000 2023-04-16T15:36:45.998Z,1681659405.998 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=39vcr 2023-04-16T15:36:46.000Z,1681659406.000 [CommandExec](IMPORTANT): Scheduled #63 (#2 of 2 with id='39vcr'): "run", ASAP 2023-04-16T15:36:46.113Z,1681659406.113 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl 2023-04-16T15:36:46.115Z,1681659406.115 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx 2023-04-16T15:36:48.108Z,1681659408.108 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl 2023-04-16T15:37:07.029Z,1681659427.029 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 8 hour 2023-04-16T15:37:07.030Z,1681659427.030 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2023-04-16T15:37:07.031Z,1681659427.031 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.91 degree 2023-04-16T15:37:07.031Z,1681659427.031 [CommandExec](IMPORTANT): got command set sci2.Lon1 -122.1 degree 2023-04-16T15:37:07.032Z,1681659427.032 [CommandExec](IMPORTANT): got command set sci2.Speed 0.8 meter_per_second 2023-04-16T15:37:07.033Z,1681659427.033 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2023-04-16T15:37:07.034Z,1681659427.034 [CommandExec](IMPORTANT): got command set sci2.MinAltitude 7 meter 2023-04-16T15:37:07.034Z,1681659427.034 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 55 meter 2023-04-16T15:37:07.035Z,1681659427.035 [CommandExec](IMPORTANT): got command run 2023-04-16T15:37:07.087Z,1681659427.087 [CommandExec](IMPORTANT): Running 2023-04-16T15:37:07.378Z,1681659427.378 [MissionManager](IMPORTANT): Started mission sci2