2022-08-07T14:16:23.075Z,1659881783.075 [CommandExec](IMPORTANT): got command schedule resume 2022-08-07T14:16:23.075Z,1659881783.075 [CommandExec](IMPORTANT): Scheduling is resumed 2022-08-07T14:16:23.075Z,1659881783.075 [CommandExec](IMPORTANT): got command restart logs 2022-08-07T14:23:49.473Z,1659882229.473 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.823645 min 2022-08-07T14:24:11.201Z,1659882251.201 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9729, MT Status=1, MTMSN=763 2022-08-07T14:24:11.933Z,1659882251.933 [CommandExec](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" lh2i 1 3.000000 2022-08-07T14:24:11.933Z,1659882251.933 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=lh2i 2022-08-07T14:24:11.934Z,1659882251.934 [CommandExec](IMPORTANT): Scheduled #5 (#1 of 3 with id='lh2i'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-07T14:25:01.143Z,1659882301.143 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9730, MT Status=1, MTMSN=764 2022-08-07T14:25:01.884Z,1659882301.884 [CommandExec](IMPORTANT): got command schedule "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 30 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree" lh2i 2 3.000000 2022-08-07T14:25:01.885Z,1659882301.885 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=lh2i 2022-08-07T14:25:01.886Z,1659882301.886 [CommandExec](IMPORTANT): Scheduled #6 (#2 of 3 with id='lh2i'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 30 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-07T14:25:28.539Z,1659882328.539 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9731, MT Status=1, MTMSN=765 2022-08-07T14:25:29.178Z,1659882329.178 [CommandExec](IMPORTANT): got command schedule "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" lh2i 3 3.000000 2022-08-07T14:25:29.178Z,1659882329.178 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=lh2i 2022-08-07T14:25:29.179Z,1659882329.179 [CommandExec](IMPORTANT): Scheduled #7 (#3 of 3 with id='lh2i'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-07T14:25:29.324Z,1659882329.324 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T14:25:32.390Z,1659882332.390 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T14:25:45.828Z,1659882345.828 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-07T14:25:45.829Z,1659882345.829 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-07T14:25:45.829Z,1659882345.829 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-07T14:25:45.830Z,1659882345.830 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-07T14:25:45.831Z,1659882345.831 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-07T14:25:54.567Z,1659882354.567 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9732, MT Status=1, MTMSN=766 2022-08-07T14:26:00.711Z,1659882360.711 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-07T14:26:00.712Z,1659882360.712 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 15 meter 2022-08-07T14:26:00.717Z,1659882360.717 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 30 meter 2022-08-07T14:26:00.717Z,1659882360.717 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-07T14:26:00.718Z,1659882360.718 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -87.200960 degree 2022-08-07T14:26:00.719Z,1659882360.719 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-07T14:26:00.750Z,1659882360.750 [CommandExec](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" lh7n 1 3.000000 2022-08-07T14:26:00.750Z,1659882360.750 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=lh7n 2022-08-07T14:26:00.751Z,1659882360.751 [CommandExec](IMPORTANT): Scheduled #8 (#1 of 3 with id='lh7n'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-07T14:26:06.595Z,1659882366.595 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-07T14:26:06.596Z,1659882366.596 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-07T14:26:06.597Z,1659882366.597 [CommandExec](IMPORTANT): got command run 2022-08-07T14:26:06.599Z,1659882366.599 [CommandExec](IMPORTANT): Running 2022-08-07T14:26:06.966Z,1659882366.966 [MissionManager](IMPORTANT): Started mission Smear 2022-08-07T14:26:19.407Z,1659882379.407 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9733, MT Status=1, MTMSN=767 2022-08-07T14:26:20.181Z,1659882380.181 [CommandExec](IMPORTANT): got command schedule "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree" lh7n 2 3.000000 2022-08-07T14:26:20.182Z,1659882380.182 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=lh7n 2022-08-07T14:26:20.183Z,1659882380.183 [CommandExec](IMPORTANT): Scheduled #9 (#2 of 3 with id='lh7n'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-07T14:26:38.466Z,1659882398.466 [BPC1](FAULT): Failed to receive data from 8 sticks prior to timeout. Missing stick IDs are: 2, 4, 6, 7, 8, 16, 42, 43. 2022-08-07T14:26:38.856Z,1659882398.856 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9734, MT Status=1, MTMSN=768 2022-08-07T14:26:39.526Z,1659882399.526 [CommandExec](IMPORTANT): got command schedule "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" lh7n 3 3.000000 2022-08-07T14:26:39.526Z,1659882399.526 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=lh7n 2022-08-07T14:26:39.527Z,1659882399.527 [CommandExec](IMPORTANT): Scheduled #10 (#3 of 3 with id='lh7n'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-07T14:32:28.798Z,1659882748.798 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-87.200958 2022-08-07T14:34:45.357Z,1659882885.357 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-07T14:34:48.198Z,1659882888.198 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 15.407476 m , 12.894800 degC , 0.290000 ug/l . 2022-08-07T14:34:49.777Z,1659882889.777 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2022-08-07T14:34:59.929Z,1659882899.929 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-07T14:35:00.220Z,1659882900.220 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-07T14:35:59.420Z,1659882959.420 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-07T14:36:00.332Z,1659882960.332 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -1 2022-08-07T14:36:00.333Z,1659882960.333 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-07T14:36:00.637Z,1659882960.637 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-07T14:36:01.434Z,1659882961.434 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-07T14:36:01.834Z,1659882961.834 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-07T14:36:03.455Z,1659882963.455 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-07T14:36:37.798Z,1659882997.798 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@55,aRNAl_bac) 2022-08-07T14:36:40.618Z,1659883000.618 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2022-08-07T14:36:40.620Z,1659883000.620 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-07T14:36:41.041Z,1659883001.041 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-07T14:38:55.165Z,1659883135.165 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2022-08-07T14:38:55.564Z,1659883135.564 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-07T15:07:40.267Z,1659884860.267 [BPC1](FAULT): Failed to receive data from 8 sticks prior to timeout. Missing stick IDs are: 1, 2, 4, 6, 8, 16, 42, 43. 2022-08-07T15:10:21.870Z,1659885021.870 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@576.2) 2022-08-07T15:10:22.269Z,1659885022.269 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@576.2) 2022-08-07T15:10:24.290Z,1659885024.290 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-07T15:10:24.725Z,1659885024.725 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,aRNAl_bac) 2022-08-07T15:11:59.647Z,1659885119.647 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@55,aRNAl_bac) 2022-08-07T15:12:06.922Z,1659885126.922 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-07T15:12:07.323Z,1659885127.323 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2022-08-07T15:17:07.098Z,1659885427.098 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2022-08-07T15:17:08.333Z,1659885428.333 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-07T15:20:17.792Z,1659885617.792 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9744, MT Status=1, MTMSN=769 2022-08-07T15:20:18.360Z,1659885618.360 [CommandExec](IMPORTANT): got command stop 2022-08-07T15:20:18.360Z,1659885618.360 [CommandExec](IMPORTANT): Scheduling is paused for 3 commands 2022-08-07T15:20:19.057Z,1659885619.057 [MissionManager](IMPORTANT): Started mission Default 2022-08-07T15:20:19.621Z,1659885619.621 [WetLabsBB2FL](FAULT): LCB fault: LCB Watchdog Reset. 2022-08-07T15:29:10.880Z,1659886150.880 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.856921 min 2022-08-07T15:36:46.343Z,1659886606.343 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.448241 min 2022-08-07T15:42:25.996Z,1659886945.996 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.108824 min 2022-08-07T15:48:10.408Z,1659887290.408 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 27.849280 min 2022-08-07T15:54:22.903Z,1659887662.903 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 34.057357 min 2022-08-07T16:00:05.939Z,1659888005.939 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 39.774727 min 2022-08-07T16:05:49.638Z,1659888349.638 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 45.502954 min 2022-08-07T16:12:28.295Z,1659888748.295 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 52.147241 min 2022-08-07T16:18:13.580Z,1659889093.580 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 57.901953 min 2022-08-07T16:21:10.711Z,1659889270.711 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9785, MT Status=1, MTMSN=770 2022-08-07T16:21:11.227Z,1659889271.227 [CommandExec](IMPORTANT): got command schedule clear 2022-08-07T16:21:11.227Z,1659889271.227 [CommandExec](IMPORTANT): Cleared 3 scheduled commands. 2022-08-07T16:21:11.228Z,1659889271.228 [CommandExec](IMPORTANT): got command schedule resume 2022-08-07T16:21:11.228Z,1659889271.228 [CommandExec](IMPORTANT): Scheduling is resumed 2022-08-07T16:27:08.272Z,1659889628.272 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 66.813289 min 2022-08-07T16:27:35.647Z,1659889655.647 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9790, MT Status=1, MTMSN=771 2022-08-07T16:27:36.404Z,1659889656.404 [CommandExec](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" lmtm 1 3.000000 2022-08-07T16:27:36.404Z,1659889656.404 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=lmtm 2022-08-07T16:27:36.406Z,1659889656.406 [CommandExec](IMPORTANT): Scheduled #11 (#1 of 3 with id='lmtm'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", ASAP 2022-08-07T16:28:01.971Z,1659889681.971 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9791, MT Status=1, MTMSN=772 2022-08-07T16:28:02.772Z,1659889682.772 [CommandExec](IMPORTANT): got command schedule asap "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree" lmtm 2 3.000000 2022-08-07T16:28:02.773Z,1659889682.773 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=lmtm 2022-08-07T16:28:02.774Z,1659889682.774 [CommandExec](IMPORTANT): Scheduled #12 (#2 of 3 with id='lmtm'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree", ASAP 2022-08-07T16:28:32.094Z,1659889712.094 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9792, MT Status=1, MTMSN=773 2022-08-07T16:28:32.688Z,1659889712.688 [CommandExec](IMPORTANT): got command schedule asap "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" lmtm 3 3.000000 2022-08-07T16:28:32.688Z,1659889712.688 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=lmtm 2022-08-07T16:28:32.689Z,1659889712.689 [CommandExec](IMPORTANT): Scheduled #13 (#3 of 3 with id='lmtm'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", ASAP 2022-08-07T16:28:32.825Z,1659889712.825 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T16:28:35.125Z,1659889715.125 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T16:28:47.457Z,1659889727.457 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-07T16:28:47.458Z,1659889727.458 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-07T16:28:47.459Z,1659889727.459 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-07T16:28:47.459Z,1659889727.459 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-07T16:28:47.460Z,1659889727.460 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-07T16:29:04.745Z,1659889744.745 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-07T16:29:04.746Z,1659889744.746 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 5 meter 2022-08-07T16:29:04.746Z,1659889744.746 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 15 meter 2022-08-07T16:29:04.748Z,1659889744.748 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-07T16:29:04.749Z,1659889744.749 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -87.200960 degree 2022-08-07T16:29:04.749Z,1659889744.749 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-07T16:29:10.196Z,1659889750.196 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-07T16:29:10.201Z,1659889750.201 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-07T16:29:10.201Z,1659889750.201 [CommandExec](IMPORTANT): got command run 2022-08-07T16:29:10.203Z,1659889750.203 [CommandExec](IMPORTANT): Running 2022-08-07T16:29:10.215Z,1659889750.215 [MissionManager](IMPORTANT): Started mission Smear 2022-08-07T16:30:26.031Z,1659889826.031 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9793, MT Status=1, MTMSN=774 2022-08-07T16:30:26.773Z,1659889826.773 [CommandExec](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" lmxs 1 3.000000 2022-08-07T16:30:26.773Z,1659889826.773 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=lmxs 2022-08-07T16:30:26.774Z,1659889826.774 [CommandExec](IMPORTANT): Scheduled #14 (#1 of 3 with id='lmxs'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-07T16:30:49.635Z,1659889849.635 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9794, MT Status=1, MTMSN=775 2022-08-07T16:30:50.411Z,1659889850.411 [CommandExec](IMPORTANT): got command schedule "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree" lmxs 2 3.000000 2022-08-07T16:30:50.416Z,1659889850.416 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=lmxs 2022-08-07T16:30:50.417Z,1659889850.417 [CommandExec](IMPORTANT): Scheduled #15 (#2 of 3 with id='lmxs'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-07T16:31:24.543Z,1659889884.543 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9795, MT Status=1, MTMSN=776 2022-08-07T16:31:25.286Z,1659889885.286 [CommandExec](IMPORTANT): got command schedule "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" lmxs 3 3.000000 2022-08-07T16:31:25.286Z,1659889885.286 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=lmxs 2022-08-07T16:31:25.287Z,1659889885.287 [CommandExec](IMPORTANT): Scheduled #16 (#3 of 3 with id='lmxs'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-07T16:31:47.642Z,1659889907.642 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9796, MT Status=1, MTMSN=777 2022-08-07T16:31:48.198Z,1659889908.198 [CommandExec](IMPORTANT): got command schedule list 2022-08-07T16:31:48.199Z,1659889908.199 [CommandExec](IMPORTANT): Scheduled #14 (#1 of 3 with id='lmxs'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-07T16:31:48.199Z,1659889908.199 [CommandExec](IMPORTANT): Scheduled #15 (#2 of 3 with id='lmxs'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-07T16:31:48.200Z,1659889908.200 [CommandExec](IMPORTANT): Scheduled #16 (#3 of 3 with id='lmxs'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-07T16:33:18.313Z,1659889998.313 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-87.200958 2022-08-07T16:34:48.425Z,1659890088.425 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-07T16:34:51.262Z,1659890091.262 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 5.369689 m , 22.353204 degC , 0.197200 ug/l . 2022-08-07T16:35:03.077Z,1659890103.077 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-07T16:35:03.221Z,1659890103.221 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-07T16:36:01.945Z,1659890161.945 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-07T16:36:02.775Z,1659890162.775 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -1 2022-08-07T16:36:02.776Z,1659890162.776 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-07T16:36:03.173Z,1659890163.173 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-07T16:36:03.558Z,1659890163.558 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-07T16:36:03.968Z,1659890163.968 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-07T16:36:05.986Z,1659890165.986 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-07T16:36:40.339Z,1659890200.339 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@54,aRNAl_bac) 2022-08-07T16:36:42.370Z,1659890202.370 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2022-08-07T16:36:42.780Z,1659890202.780 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-07T16:36:43.167Z,1659890203.167 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-07T16:38:58.490Z,1659890338.490 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2022-08-07T16:38:58.893Z,1659890338.893 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-07T17:21:20.136Z,1659892880.136 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@834.6) 2022-08-07T17:21:20.546Z,1659892880.546 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@834.6) 2022-08-07T17:21:22.559Z,1659892882.559 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-07T17:21:22.966Z,1659892882.966 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,aRNAl_bac) 2022-08-07T17:22:57.909Z,1659892977.909 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@54,aRNAl_bac) 2022-08-07T17:23:05.182Z,1659892985.182 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-07T17:23:05.620Z,1659892985.620 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2022-08-07T17:28:05.362Z,1659893285.362 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2022-08-07T17:28:06.168Z,1659893286.168 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. 2022-08-07T17:28:06.169Z,1659893286.169 [ESPComponent](FAULT): LCB 2 fault: Invalid Command. 2022-08-07T17:28:06.591Z,1659893286.591 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-07T17:29:52.897Z,1659893392.897 [CBIT](CRITICAL): Navigation error has exceeded threshold. Set navErrorCriticalReporting to false to squeltch critical failure. 2022-08-07T17:29:53.398Z,1659893393.398 [CommandExec](FAULT): Scheduling is paused for 3 commands 2022-08-07T17:29:54.111Z,1659893394.111 [MissionManager](IMPORTANT): Started mission Default 2022-08-07T17:42:13.337Z,1659894133.337 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.313465 min 2022-08-07T17:50:25.364Z,1659894625.364 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.514134 min 2022-08-07T17:51:20.389Z,1659894680.389 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9819, MT Status=1, MTMSN=778 2022-08-07T17:51:21.133Z,1659894681.133 [CommandExec](IMPORTANT): got command load ./Missions/Transport/transit.tl 2022-08-07T17:51:21.134Z,1659894681.134 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/transit.tx 2022-08-07T17:51:21.812Z,1659894681.812 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/transit.tl 2022-08-07T17:51:31.549Z,1659894691.549 [CommandExec](IMPORTANT): got command set transit.MissionTimeout 20 minute 2022-08-07T17:51:31.550Z,1659894691.550 [CommandExec](IMPORTANT): got command set transit.NeedCommsTime 18 minute 2022-08-07T17:51:31.551Z,1659894691.551 [CommandExec](IMPORTANT): got command set transit.Latitude 43.615626 degree 2022-08-07T17:51:31.551Z,1659894691.551 [CommandExec](IMPORTANT): got command set transit.Longitude -86.931045 degree 2022-08-07T17:51:31.552Z,1659894691.552 [CommandExec](IMPORTANT): got command run 2022-08-07T17:51:31.554Z,1659894691.554 [CommandExec](IMPORTANT): Running 2022-08-07T17:51:31.770Z,1659894691.770 [MissionManager](IMPORTANT): Started mission transit 2022-08-07T17:51:33.052Z,1659894693.052 [transit:Transit:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-86.931046 2022-08-07T18:00:51.103Z,1659895251.103 [CBIT](IMPORTANT): Beginning ground fault scan 2022-08-07T18:01:01.987Z,1659895261.987 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004069 CHAN A1 (24V): -0.001000 CHAN A2 (12V): -0.000333 CHAN A3 (5V): 0.000252 CHAN B0 (3.3V): 0.000780 CHAN B1 (3.15aV): -0.000124 CHAN B2 (3.15bV): -0.000265 CHAN B3 (GND): -0.000973 OPEN: 0.007755 Full Scale: +/- 1 mA 2022-08-07T18:11:32.527Z,1659895892.527 [MissionManager](IMPORTANT): Started mission Default 2022-08-07T18:13:30.642Z,1659896010.642 [BPC1](FAULT): Failed to receive data from 7 sticks prior to timeout. Missing stick IDs are: 1, 2, 4, 6, 8, 16, 42. 2022-08-07T18:18:47.371Z,1659896327.371 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 7.240567 min 2022-08-07T18:19:29.123Z,1659896369.123 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9827, MT Status=1, MTMSN=779 2022-08-07T18:19:29.723Z,1659896369.723 [CommandExec](IMPORTANT): got command schedule clear 2022-08-07T18:19:29.723Z,1659896369.723 [CommandExec](IMPORTANT): Cleared 3 scheduled commands. 2022-08-07T18:19:29.724Z,1659896369.724 [CommandExec](IMPORTANT): got command schedule resume 2022-08-07T18:19:29.724Z,1659896369.724 [CommandExec](IMPORTANT): Scheduling is resumed 2022-08-07T18:20:24.190Z,1659896424.190 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9828, MT Status=1, MTMSN=780 2022-08-07T18:20:25.016Z,1659896425.016 [CommandExec](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" ls3r 1 3.000000 2022-08-07T18:20:25.016Z,1659896425.016 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=ls3r 2022-08-07T18:20:25.017Z,1659896425.017 [CommandExec](IMPORTANT): Scheduled #17 (#1 of 3 with id='ls3r'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", ASAP 2022-08-07T18:20:42.393Z,1659896442.393 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9829, MT Status=1, MTMSN=781 2022-08-07T18:20:43.136Z,1659896443.136 [CommandExec](IMPORTANT): got command schedule asap "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree" ls3r 2 3.000000 2022-08-07T18:20:43.136Z,1659896443.136 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=ls3r 2022-08-07T18:20:43.137Z,1659896443.137 [CommandExec](IMPORTANT): Scheduled #18 (#2 of 3 with id='ls3r'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -87.200960 degree", ASAP 2022-08-07T18:20:54.879Z,1659896454.879 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=9830, MT Status=1, MTMSN=782 2022-08-07T18:20:55.515Z,1659896455.515 [CommandExec](IMPORTANT): got command schedule asap "set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" ls3r 3 3.000000 2022-08-07T18:20:55.516Z,1659896455.516 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=ls3r 2022-08-07T18:20:55.517Z,1659896455.517 [CommandExec](IMPORTANT): Scheduled #19 (#3 of 3 with id='ls3r'): "set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", ASAP 2022-08-07T18:20:55.659Z,1659896455.659 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T18:20:57.893Z,1659896457.893 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-07T18:21:10.525Z,1659896470.525 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-07T18:21:10.526Z,1659896470.526 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-07T18:21:10.527Z,1659896470.527 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-07T18:21:10.528Z,1659896470.528 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-07T18:21:10.528Z,1659896470.528 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-07T18:21:25.944Z,1659896485.944 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-07T18:21:25.945Z,1659896485.945 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 40 meter 2022-08-07T18:21:25.946Z,1659896485.946 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 70 meter 2022-08-07T18:21:25.946Z,1659896485.946 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-07T18:21:25.947Z,1659896485.947 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -87.200960 degree 2022-08-07T18:21:33.042Z,1659896493.042 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-07T18:21:33.042Z,1659896493.042 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-07T18:21:33.043Z,1659896493.043 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-07T18:21:33.043Z,1659896493.043 [CommandExec](IMPORTANT): got command run 2022-08-07T18:21:33.046Z,1659896493.046 [CommandExec](IMPORTANT): Running 2022-08-07T18:21:33.422Z,1659896493.422 [MissionManager](IMPORTANT): Started mission Smear 2022-08-07T18:29:21.290Z,1659896961.290 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-87.200958 2022-08-07T18:34:00.333Z,1659897240.333 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-07T18:34:03.193Z,1659897243.193 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 40.410530 m , 4.102289 degC , 0.394400 ug/l . 2022-08-07T18:34:14.980Z,1659897254.980 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-07T18:34:15.134Z,1659897255.134 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-07T18:35:15.492Z,1659897315.492 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-07T18:35:16.282Z,1659897316.282 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1 2022-08-07T18:35:16.282Z,1659897316.282 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-07T18:35:16.709Z,1659897316.709 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-07T18:35:17.086Z,1659897317.086 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-07T18:35:17.500Z,1659897317.500 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-07T18:35:19.526Z,1659897319.526 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-07T18:35:53.857Z,1659897353.857 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@53,aRNAl_bac) 2022-08-07T18:35:56.683Z,1659897356.683 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2022-08-07T18:35:56.684Z,1659897356.684 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-07T18:35:57.102Z,1659897357.102 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-07T18:38:12.027Z,1659897492.027 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2022-08-07T18:38:12.473Z,1659897492.473 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-07T18:54:32.552Z,1659898472.552 [BPC1](FAULT): Failed to receive data from 7 sticks prior to timeout. Missing stick IDs are: 1, 2, 4, 6, 8, 16, 42. 2022-08-07T19:35:34.675Z,1659900934.675 [BPC1](FAULT): Failed to receive data from 8 sticks prior to timeout. Missing stick IDs are: 1, 2, 4, 5, 6, 8, 16, 42. 2022-08-07T20:13:36.074Z,1659903216.074 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1683.1) 2022-08-07T20:13:36.478Z,1659903216.478 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1683.1) 2022-08-07T20:13:38.504Z,1659903218.504 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-07T20:13:38.914Z,1659903218.914 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,aRNAl_bac) 2022-08-07T20:15:13.849Z,1659903313.849 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@53,aRNAl_bac) 2022-08-07T20:15:21.119Z,1659903321.119 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-07T20:15:21.532Z,1659903321.532 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2022-08-07T20:20:21.299Z,1659903621.299 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2022-08-07T20:20:22.519Z,1659903622.519 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-07T20:23:11.511Z,1659903791.511 [CBIT](FAULT): Humidity exceeds 150% of running average: Humidity:42 %. Average:27.719999 2022-08-07T20:26:51.636Z,1659904011.636 [BuoyancyServo](FAULT): Overload Error 2022-08-07T20:28:19.258Z,1659904099.258 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-08-07T20:28:49.240Z,1659904129.240 [BuoyancyServo](FAULT): Overload Error 2022-08-07T20:37:25.283Z,1659904645.283 [MissionManager](IMPORTANT): Started mission Default 2022-08-07T20:43:21.521Z,1659905001.521 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.930518 min 2022-08-07T20:45:03.083Z,1659905103.083 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9854, MT Status=1, MTMSN=783 2022-08-07T20:45:03.837Z,1659905103.837 [CommandExec](IMPORTANT): got command schedule asap "load Science/sci2.tl;set sci2.MissionTimeout 18 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 43.915495 degree;set sci2.Lon1 -86.70935 degree;set sci2.Speed 1.0 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.YoYoMinAltitude 15 m" lyxb 1 2.000000 2022-08-07T20:45:03.837Z,1659905103.837 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=lyxb 2022-08-07T20:45:03.838Z,1659905103.838 [CommandExec](IMPORTANT): Scheduled #20 (#1 of 2 with id='lyxb'): "load Science/sci2.tl;set sci2.MissionTimeout 18 h;set sci2.NeedCommsTime 60 min;set sci2.Lat1 43.915495 degree;set sci2.Lon1 -86.70935 degree;set sci2.Speed 1.0 m/s;set sci2.YoYoMaxDepth 50 m;set sci2.YoYoMinAltitude 15 m", ASAP 2022-08-07T20:45:24.394Z,1659905124.394 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9855, MT Status=1, MTMSN=784 2022-08-07T20:45:25.080Z,1659905125.080 [CommandExec](IMPORTANT): got command schedule asap "set sci2.MinAltitude 7 m;set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run" lyxb 2 2.000000 2022-08-07T20:45:25.080Z,1659905125.080 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=lyxb 2022-08-07T20:45:25.081Z,1659905125.081 [CommandExec](IMPORTANT): Scheduled #21 (#2 of 2 with id='lyxb'): "set sci2.MinAltitude 7 m;set sci2.MaxDepth 60 m;set sci2.MinOffshore 2 km;run", ASAP 2022-08-07T20:45:25.213Z,1659905125.213 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl 2022-08-07T20:45:25.214Z,1659905125.214 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx 2022-08-07T20:45:27.056Z,1659905127.056 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl 2022-08-07T20:45:39.378Z,1659905139.378 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 18 hour 2022-08-07T20:45:39.379Z,1659905139.379 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 60 minute 2022-08-07T20:45:39.379Z,1659905139.379 [CommandExec](IMPORTANT): got command set sci2.Lat1 43.915495 degree 2022-08-07T20:45:39.380Z,1659905139.380 [CommandExec](IMPORTANT): got command set sci2.Lon1 -86.70935 degree 2022-08-07T20:45:39.381Z,1659905139.381 [CommandExec](IMPORTANT): got command set sci2.Speed 1.0 meter_per_second 2022-08-07T20:45:39.382Z,1659905139.382 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 50 meter 2022-08-07T20:45:39.382Z,1659905139.382 [CommandExec](IMPORTANT): got command set sci2.YoYoMinAltitude 15 meter 2022-08-07T20:45:44.413Z,1659905144.413 [CommandExec](IMPORTANT): got command set sci2.MinAltitude 7 meter 2022-08-07T20:45:44.413Z,1659905144.413 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 60 meter 2022-08-07T20:45:44.414Z,1659905144.414 [CommandExec](IMPORTANT): got command set sci2.MinOffshore 2 kilometer 2022-08-07T20:45:44.414Z,1659905144.414 [CommandExec](IMPORTANT): got command run 2022-08-07T20:45:44.488Z,1659905144.488 [CommandExec](IMPORTANT): Running 2022-08-07T20:45:44.617Z,1659905144.617 [MissionManager](IMPORTANT): Started mission sci2 2022-08-07T20:49:28.205Z,1659905368.205 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 43.915497,-86.709351 2022-08-07T21:08:52.300Z,1659906532.300 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. 2022-08-07T21:44:39.364Z,1659908679.364 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-08-07T22:41:23.913Z,1659912083.913 [BPC1](FAULT): Failed to receive data from 9 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 6, 7, 8, 16, 42, 43. 2022-08-08T00:01:02.120Z,1659916862.120 [CBIT](IMPORTANT): Beginning ground fault scan 2022-08-08T00:01:13.094Z,1659916873.094 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002623 CHAN A1 (24V): -0.001106 CHAN A2 (12V): -0.000545 CHAN A3 (5V): -0.000360 CHAN B0 (3.3V): 0.000156 CHAN B1 (3.15aV): 0.000136 CHAN B2 (3.15bV): 0.000283 CHAN B3 (GND): 0.001470 OPEN: 0.007587 Full Scale: +/- 1 mA 2022-08-08T01:57:01.086Z,1659923821.086 [BPC1](FAULT): Failed to receive data from 7 sticks prior to timeout. Missing stick IDs are: 1, 4, 6, 7, 8, 16, 42. 2022-08-08T06:01:13.156Z,1659938473.156 [CBIT](IMPORTANT): Beginning ground fault scan 2022-08-08T06:01:24.042Z,1659938484.042 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003050 CHAN A1 (24V): -0.001268 CHAN A2 (12V): -0.000304 CHAN A3 (5V): -0.000513 CHAN B0 (3.3V): 0.000196 CHAN B1 (3.15aV): 0.000175 CHAN B2 (3.15bV): 0.027777 CHAN B3 (GND): -0.000108 OPEN: 0.007581 Full Scale: +/- 1 mA 2022-08-08T07:37:55.116Z,1659944275.116 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2022-08-08T08:34:01.056Z,1659947641.056 [ElevatorServo](FAULT): Reading outside of valid range:0.000000 2022-08-08T08:34:01.748Z,1659947641.748 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-08-08T11:39:58.452Z,1659958798.452 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 43.915497,-86.709351 2022-08-08T11:40:01.301Z,1659958801.301 [MissionManager](IMPORTANT): Started mission Default 2022-08-08T11:44:07.069Z,1659959047.069 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T114409 2022-08-08T11:44:14.650Z,1659959054.650 [CommandExec](IMPORTANT): got command schedule clear scheduleresume 2022-08-08T11:44:14.651Z,1659959054.651 [CommandExec](IMPORTANT): Cleared 0 scheduled commands. 2022-08-08T11:47:00.878Z,1659959220.878 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T114702 2022-08-08T11:47:08.360Z,1659959228.360 [CommandExec](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" n4qb 1 3.000000 2022-08-08T11:47:08.360Z,1659959228.360 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=n4qb 2022-08-08T11:47:08.361Z,1659959228.361 [CommandExec](IMPORTANT): Scheduled #22 (#1 of 3 with id='n4qb'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", ASAP 2022-08-08T11:47:09.838Z,1659959229.838 [DataOverHttps](IMPORTANT): SBD MOMSN=17096128, MTMSN=20220808T114711 2022-08-08T11:47:20.168Z,1659959240.168 [CommandExec](IMPORTANT): got command schedule asap "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree" n4qb 2 3.000000 2022-08-08T11:47:20.169Z,1659959240.169 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=n4qb 2022-08-08T11:47:20.170Z,1659959240.170 [CommandExec](IMPORTANT): Scheduled #23 (#2 of 3 with id='n4qb'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree", ASAP 2022-08-08T11:47:21.490Z,1659959241.490 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T114723 2022-08-08T11:47:28.607Z,1659959248.607 [CommandExec](IMPORTANT): got command schedule asap "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" n4qb 3 3.000000 2022-08-08T11:47:28.607Z,1659959248.607 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=n4qb 2022-08-08T11:47:28.608Z,1659959248.608 [CommandExec](IMPORTANT): Scheduled #24 (#3 of 3 with id='n4qb'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", ASAP 2022-08-08T11:47:28.745Z,1659959248.745 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T11:47:31.539Z,1659959251.539 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T11:47:47.130Z,1659959267.130 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-08T11:47:47.131Z,1659959267.131 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-08T11:47:47.132Z,1659959267.132 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-08T11:47:47.132Z,1659959267.132 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-08T11:47:47.133Z,1659959267.133 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-08T11:48:12.043Z,1659959292.043 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-08T11:48:12.043Z,1659959292.043 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 5 meter 2022-08-08T11:48:12.044Z,1659959292.044 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 15 meter 2022-08-08T11:48:12.045Z,1659959292.045 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-08T11:48:12.046Z,1659959292.046 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -86.52055 degree 2022-08-08T11:48:12.046Z,1659959292.046 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-08T11:48:17.499Z,1659959297.499 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-08T11:48:17.499Z,1659959297.499 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-08T11:48:17.500Z,1659959297.500 [CommandExec](IMPORTANT): got command run 2022-08-08T11:48:17.521Z,1659959297.521 [CommandExec](IMPORTANT): Running 2022-08-08T11:48:17.615Z,1659959297.615 [MissionManager](IMPORTANT): Started mission Smear 2022-08-08T11:48:39.274Z,1659959319.274 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T114841 2022-08-08T11:48:47.941Z,1659959327.941 [CommandExec](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" n4t1 1 3.000000 2022-08-08T11:48:47.941Z,1659959327.941 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=n4t1 2022-08-08T11:48:47.942Z,1659959327.942 [CommandExec](IMPORTANT): Scheduled #25 (#1 of 3 with id='n4t1'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-08T11:48:49.101Z,1659959329.101 [DataOverHttps](IMPORTANT): SBD MOMSN=17096142, MTMSN=20220808T114851 2022-08-08T11:48:57.831Z,1659959337.831 [CommandExec](IMPORTANT): got command schedule "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 30 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree" n4t1 2 3.000000 2022-08-08T11:48:57.836Z,1659959337.836 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=n4t1 2022-08-08T11:48:57.837Z,1659959337.837 [CommandExec](IMPORTANT): Scheduled #26 (#2 of 3 with id='n4t1'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 30 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-08T11:48:59.033Z,1659959339.033 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T114900 2022-08-08T11:49:09.610Z,1659959349.610 [CommandExec](IMPORTANT): got command schedule "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" n4t1 3 3.000000 2022-08-08T11:49:09.610Z,1659959349.610 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=n4t1 2022-08-08T11:49:09.611Z,1659959349.611 [CommandExec](IMPORTANT): Scheduled #27 (#3 of 3 with id='n4t1'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-08T11:49:56.599Z,1659959396.599 [DataOverHttps](IMPORTANT): SBD MOMSN=17096153, MTMSN=20220808T114958 2022-08-08T11:50:07.700Z,1659959407.700 [CommandExec](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min" n4v0 1 3.000000 2022-08-08T11:50:07.701Z,1659959407.701 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=n4v0 2022-08-08T11:50:07.702Z,1659959407.702 [CommandExec](IMPORTANT): Scheduled #28 (#1 of 3 with id='n4v0'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.MaxWaitNoFiring 30 min", AFTER MISSION 2022-08-08T11:50:08.809Z,1659959408.809 [DataOverHttps](IMPORTANT): SBD MTMSN=20220808T115010 2022-08-08T11:50:18.340Z,1659959418.340 [CommandExec](IMPORTANT): got command schedule "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree" n4v0 2 3.000000 2022-08-08T11:50:18.341Z,1659959418.341 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=n4v0 2022-08-08T11:50:18.341Z,1659959418.341 [CommandExec](IMPORTANT): Scheduled #29 (#2 of 3 with id='n4v0'): "set Smear.ReachEndWaypoint 0 bool;set Smear.SamplingYoYoMinDepth 40 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 43.615626 degree;set Smear.SamplingEndLongitude -86.52055 degree;set Smear.YoYoUpPitch 12 degree", AFTER MISSION 2022-08-08T11:50:19.664Z,1659959419.664 [DataOverHttps](IMPORTANT): SBD MOMSN=17096156, MTMSN=20220808T115021 2022-08-08T11:50:31.330Z,1659959431.330 [CommandExec](IMPORTANT): got command schedule "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" n4v0 3 3.000000 2022-08-08T11:50:31.331Z,1659959431.331 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=n4v0 2022-08-08T11:50:31.332Z,1659959431.332 [CommandExec](IMPORTANT): Scheduled #30 (#3 of 3 with id='n4v0'): "set Smear.YoYoDownPitch -12 degree;set Smear.NeedCommsTimeVeryLong 8 h;run", AFTER MISSION 2022-08-08T11:55:27.862Z,1659959727.862 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-86.520554 2022-08-08T11:56:52.735Z,1659959812.735 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-08T11:56:55.581Z,1659959815.581 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 5.359167 m , 22.292413 degC , 0.255200 ug/l . 2022-08-08T11:57:07.384Z,1659959827.384 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-08T11:57:07.689Z,1659959827.689 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-08T11:58:07.318Z,1659959887.318 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-08T11:58:08.139Z,1659959888.139 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -1 2022-08-08T11:58:08.139Z,1659959888.139 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-08T11:58:08.623Z,1659959888.623 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-08T11:58:08.942Z,1659959888.942 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-08T11:58:09.358Z,1659959889.358 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-08T11:58:11.374Z,1659959891.374 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-08T11:58:45.721Z,1659959925.721 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@52,aRNAl_bac) 2022-08-08T11:58:48.527Z,1659959928.527 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2022-08-08T11:58:48.528Z,1659959928.528 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-08T11:58:48.930Z,1659959928.930 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-08T12:01:03.119Z,1659960063.119 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2022-08-08T12:01:03.478Z,1659960063.478 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-08T12:01:24.196Z,1659960084.196 [CBIT](IMPORTANT): Beginning ground fault scan 2022-08-08T12:01:35.574Z,1659960095.574 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003336 CHAN A1 (24V): -0.001533 CHAN A2 (12V): -0.000327 CHAN A3 (5V): 0.000317 CHAN B0 (3.3V): 0.000016 CHAN B1 (3.15aV): 0.004423 CHAN B2 (3.15bV): -0.001480 CHAN B3 (GND): 0.001305 OPEN: 0.007692 Full Scale: +/- 1 mA 2022-08-08T12:27:05.013Z,1659961625.013 [BPC1](FAULT): Failed to receive data from 8 sticks prior to timeout. Missing stick IDs are: 3, 4, 6, 7, 8, 16, 42, 43. 2022-08-08T13:05:41.168Z,1659963941.168 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1186.9) 2022-08-08T13:05:41.558Z,1659963941.558 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1186.9) 2022-08-08T13:05:43.635Z,1659963943.635 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-08T13:05:44.001Z,1659963944.001 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,aRNAl_bac) 2022-08-08T13:07:18.938Z,1659964038.938 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@52,aRNAl_bac) 2022-08-08T13:07:26.213Z,1659964046.213 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-08T13:07:26.607Z,1659964046.607 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2022-08-08T13:08:07.007Z,1659964087.007 [BPC1](FAULT): Failed to receive data from 8 sticks prior to timeout. Missing stick IDs are: 3, 4, 6, 7, 8, 16, 42, 43. 2022-08-08T13:12:26.387Z,1659964346.387 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2022-08-08T13:12:27.598Z,1659964347.598 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-08T13:17:47.051Z,1659964667.051 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T13:17:50.743Z,1659964670.743 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T13:17:50.912Z,1659964670.912 [MissionManager](IMPORTANT): Started mission Default 2022-08-08T13:18:03.177Z,1659964683.177 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-08T13:18:03.178Z,1659964683.178 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-08T13:18:03.178Z,1659964683.178 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-08T13:18:03.179Z,1659964683.179 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-08T13:18:03.180Z,1659964683.180 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-08T13:18:18.669Z,1659964698.669 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-08T13:18:18.670Z,1659964698.670 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 15 meter 2022-08-08T13:18:18.671Z,1659964698.671 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 30 meter 2022-08-08T13:18:18.671Z,1659964698.671 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-08T13:18:18.672Z,1659964698.672 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -86.52055 degree 2022-08-08T13:18:18.673Z,1659964698.673 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-08T13:18:23.553Z,1659964703.553 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-08T13:18:23.554Z,1659964703.554 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-08T13:18:23.554Z,1659964703.554 [CommandExec](IMPORTANT): got command run 2022-08-08T13:18:23.557Z,1659964703.557 [CommandExec](IMPORTANT): Running 2022-08-08T13:18:23.861Z,1659964703.861 [MissionManager](IMPORTANT): Started mission Smear 2022-08-08T13:22:45.714Z,1659964965.714 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-86.520554 2022-08-08T13:25:02.757Z,1659965102.757 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-08T13:25:05.581Z,1659965105.581 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 15.294336 m , 13.376886 degC , 0.371200 ug/l . 2022-08-08T13:25:17.392Z,1659965117.392 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-08T13:25:17.528Z,1659965117.528 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-08T13:26:17.122Z,1659965177.122 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-08T13:26:17.923Z,1659965177.923 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -1 2022-08-08T13:26:17.925Z,1659965177.925 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-08T13:26:18.348Z,1659965178.348 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-08T13:26:18.788Z,1659965178.788 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-08T13:26:19.134Z,1659965179.134 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-08T13:26:21.152Z,1659965181.152 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-08T13:26:55.477Z,1659965215.477 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac) 2022-08-08T13:26:57.560Z,1659965217.560 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2022-08-08T13:26:57.915Z,1659965217.915 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-08T13:26:58.327Z,1659965218.327 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-08T13:29:13.646Z,1659965353.646 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2022-08-08T13:29:14.055Z,1659965354.055 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-08T13:41:30.158Z,1659966090.158 [DVL_micro](FAULT): Failed to initialize within timeout. 2022-08-08T13:45:23.191Z,1659966323.191 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2022-08-08T13:57:48.285Z,1659967068.285 [DVL_micro](FAULT): Failed to initialize within timeout. 2022-08-08T14:02:40.796Z,1659967360.796 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@606.3) 2022-08-08T14:02:41.178Z,1659967361.178 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@606.3) 2022-08-08T14:02:43.215Z,1659967363.215 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-08T14:02:43.618Z,1659967363.618 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac) 2022-08-08T14:04:18.543Z,1659967458.543 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@51,aRNAl_bac) 2022-08-08T14:04:25.823Z,1659967465.823 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-08T14:04:26.219Z,1659967466.219 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2022-08-08T14:09:25.994Z,1659967765.994 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2022-08-08T14:09:27.213Z,1659967767.213 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-08T14:18:18.940Z,1659968298.940 [Smear:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout 2022-08-08T14:23:44.256Z,1659968624.256 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-08-08T14:30:42.445Z,1659969042.445 [Smear:NeedComms:E:A_Timeout:A](IMPORTANT): second GPS update timeout 2022-08-08T14:30:44.236Z,1659969044.236 [CommandExec](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T14:30:46.745Z,1659969046.745 [CommandExec](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2022-08-08T14:30:46.877Z,1659969046.877 [MissionManager](IMPORTANT): Started mission Default 2022-08-08T14:30:57.969Z,1659969057.969 [CommandExec](IMPORTANT): got command set Smear.CartridgeType -1 count 2022-08-08T14:30:57.970Z,1659969057.970 [CommandExec](IMPORTANT): got command set Smear.MissionTimeout 8 hour 2022-08-08T14:30:57.970Z,1659969057.970 [CommandExec](IMPORTANT): got command set Smear.WaitDuration 15 minute 2022-08-08T14:30:57.971Z,1659969057.971 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45 minute 2022-08-08T14:30:57.971Z,1659969057.971 [CommandExec](IMPORTANT): got command set Smear.MaxWaitNoFiring 30 minute 2022-08-08T14:31:12.893Z,1659969072.893 [CommandExec](IMPORTANT): got command set Smear.ReachEndWaypoint 0 bool 2022-08-08T14:31:12.894Z,1659969072.894 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 40 meter 2022-08-08T14:31:12.894Z,1659969072.894 [CommandExec](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 70 meter 2022-08-08T14:31:12.895Z,1659969072.895 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLatitude 43.615626 degree 2022-08-08T14:31:12.896Z,1659969072.896 [CommandExec](IMPORTANT): got command set Smear.SamplingEndLongitude -86.52055 degree 2022-08-08T14:31:12.897Z,1659969072.897 [CommandExec](IMPORTANT): got command set Smear.YoYoUpPitch 12 degree 2022-08-08T14:31:17.768Z,1659969077.768 [CommandExec](IMPORTANT): got command set Smear.YoYoDownPitch -12 degree 2022-08-08T14:31:17.769Z,1659969077.769 [CommandExec](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8 hour 2022-08-08T14:31:17.769Z,1659969077.769 [CommandExec](IMPORTANT): got command run 2022-08-08T14:31:17.772Z,1659969077.772 [CommandExec](IMPORTANT): Running 2022-08-08T14:31:17.975Z,1659969077.975 [MissionManager](IMPORTANT): Started mission Smear 2022-08-08T14:36:48.705Z,1659969408.705 [Smear:SamplingToWaypoint:E.Waypoint](IMPORTANT): Navigating to waypoint: 43.615627,-86.520554 2022-08-08T14:41:51.370Z,1659969711.370 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2022-08-08T14:41:54.215Z,1659969714.215 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 40.459671 m , 4.263300 degC , 0.707600 ug/l . 2022-08-08T14:42:06.029Z,1659969726.029 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-08-08T14:42:06.144Z,1659969726.144 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-08-08T14:43:06.579Z,1659969786.579 [ESPComponent](IMPORTANT): ESP has connected as client 2022-08-08T14:43:07.389Z,1659969787.389 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -1 2022-08-08T14:43:07.390Z,1659969787.390 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-08-08T14:43:07.814Z,1659969787.814 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2022-08-08T14:43:08.177Z,1659969788.177 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2022-08-08T14:43:08.580Z,1659969788.580 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-08-08T14:43:10.601Z,1659969790.601 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2022-08-08T14:43:44.954Z,1659969824.954 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@50,aRNAl_bac) 2022-08-08T14:43:47.781Z,1659969827.781 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2022-08-08T14:43:47.782Z,1659969827.782 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=175.399994 issuedCmd_=Cmd.startFiltering 2022-08-08T14:43:48.187Z,1659969828.187 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-08-08T14:46:03.947Z,1659969963.947 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2022-08-08T14:46:04.350Z,1659969964.350 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-08-08T15:11:18.162Z,1659971478.162 [BPC1](FAULT): Failed to receive data from 9 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 6, 7, 8, 16, 42, 43. 2022-08-08T15:22:26.504Z,1659972146.504 [ElevatorServo](FAULT): Reading outside of valid range:0.000000 2022-08-08T15:22:27.168Z,1659972147.168 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-08-08T15:52:20.118Z,1659973940.118 [BPC1](FAULT): Failed to receive data from 10 sticks prior to timeout. Missing stick IDs are: 1, 3, 4, 5, 6, 7, 8, 16, 42, 43. 2022-08-08T16:49:05.111Z,1659977345.111 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@2000.0) 2022-08-08T16:49:05.503Z,1659977345.503 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@2000.0) 2022-08-08T16:49:07.551Z,1659977347.551 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-08-08T16:49:07.926Z,1659977347.926 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,aRNAl_bac) 2022-08-08T16:50:42.876Z,1659977442.876 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@50,aRNAl_bac) 2022-08-08T16:50:50.138Z,1659977450.138 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-08-08T16:50:50.553Z,1659977450.553 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2022-08-08T16:55:50.316Z,1659977750.316 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2022-08-08T16:55:51.138Z,1659977751.138 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Hardware Overcurrent Shutdown. 2022-08-08T16:55:51.557Z,1659977751.557 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2022-08-08T16:59:24.096Z,1659977964.096 [BuoyancyServo](FAULT): Overload Error 2022-08-08T16:59:55.616Z,1659977995.616 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:00:14.176Z,1659978014.176 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:00:35.212Z,1659978035.212 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:00:54.212Z,1659978054.212 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:01:24.088Z,1659978084.088 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:01:43.069Z,1659978103.069 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:03:58.452Z,1659978238.452 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:04:14.118Z,1659978254.118 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-08-08T17:04:17.008Z,1659978257.008 [BuoyancyServo](FAULT): Overload Error 2022-08-08T17:06:10.511Z,1659978370.511 [Smear:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout 2022-08-08T17:07:45.236Z,1659978465.236 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9986, MT Status=1, MTMSN=785 2022-08-08T17:07:46.207Z,1659978466.207 [CommandExec](IMPORTANT): got command configSet VerticalControl.massDefault 0.5 centimeter persist 2022-08-08T17:09:26.930Z,1659978566.930 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=9987, MT Status=1, MTMSN=786 2022-08-08T17:09:27.896Z,1659978567.896 [CommandExec](IMPORTANT): got command configSet VerticalControl.massDefault -15 millimeter persist 2022-08-08T17:19:15.991Z,1659979155.991 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-08-08T17:23:30.929Z,1659979410.929 [MissionManager](IMPORTANT): Started mission Default 2022-08-08T17:33:15.528Z,1659979995.528 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 9.736436 min 2022-08-08T17:41:59.580Z,1659980519.580 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 18.470829 min 2022-08-08T17:43:44.723Z,1659980624.723 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=10008, MT Status=1, MTMSN=787 2022-08-08T17:43:46.809Z,1659980626.809 [CommandExec](IMPORTANT): got command restart logs