2024-09-22T18:34:37.873Z,1727030077.873 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (2 messages): @10:34:35.63 Selecting Cartridge 29 @11:27:49.98 Sampled 1000.0ml 2024-09-22T18:34:37.876Z,1727030077.876 [ESPComponent](IMPORTANT): [sample #7] ESP Cartridge 29: S_FILTERING: start: 2024-09-22T17:37:42.818Z end: 2024-09-22T18:27:52.257Z Sampled: 1000.0ml 2024-09-22T19:19:43.766Z,1727032783.766 [CBIT](CRITICAL): Hardware Fault in component: RudderServo 2024-09-22T17:18:05.049Z,1727025485.049 [CommandExec](IMPORTANT): got command restart logs 2024-09-22T17:20:07.928Z,1727025607.928 [RudderServo](FAULT): Overload Error 2024-09-22T17:20:31.401Z,1727025631.401 [RudderServo](FAULT): Overload Error 2024-09-22T17:28:24.834Z,1727026104.834 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.033659 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 7.000000 count 2024-09-22T17:28:55.926Z,1727026135.926 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T17:32:15.974Z,1727026335.974 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.826920 m , settling for 1.000000 min 2024-09-22T17:33:28.727Z,1727026408.727 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T17:33:28.856Z,1727026408.856 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T17:34:22.022Z,1727026462.022 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T17:34:22.838Z,1727026462.838 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -1 2024-09-22T17:34:22.838Z,1727026462.838 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T17:34:23.281Z,1727026463.281 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.522003 m in 9.590387 degC water with 0.025960 ug/l chlorophyll fluorescence. 2024-09-22T17:34:23.637Z,1727026463.637 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T17:34:24.040Z,1727026464.040 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T17:34:26.061Z,1727026466.061 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T17:34:57.572Z,1727026497.572 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@29,aRNAl_bac) 2024-09-22T17:35:00.406Z,1727026500.406 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T17:35:00.408Z,1727026500.408 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T17:35:00.806Z,1727026500.806 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T17:37:42.818Z,1727026662.818 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T17:37:43.216Z,1727026663.216 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T17:52:27.287Z,1727027547.287 [RudderServo](FAULT): Overload Error 2024-09-22T18:02:23.560Z,1727028143.560 [RudderServo](FAULT): Overload Error 2024-09-22T18:06:25.147Z,1727028385.147 [RudderServo](FAULT): Overload Error 2024-09-22T18:09:57.589Z,1727028597.589 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T18:27:51.845Z,1727029671.845 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T18:27:52.258Z,1727029672.258 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T18:27:54.269Z,1727029674.269 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T18:27:54.674Z,1727029674.674 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@29,aRNAl_bac) 2024-09-22T18:28:52.923Z,1727029732.923 [RudderServo](FAULT): Overload Error 2024-09-22T18:29:29.612Z,1727029769.612 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@29,aRNAl_bac) 2024-09-22T18:29:37.349Z,1727029777.349 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T18:29:37.694Z,1727029777.694 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2024-09-22T18:34:37.473Z,1727030077.473 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2024-09-22T18:34:39.860Z,1727030079.860 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T18:34:39.956Z,1727030079.956 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 7.000000 count 2024-09-22T18:34:40.328Z,1727030080.328 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T18:34:40.733Z,1727030080.733 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T18:34:41.132Z,1727030081.132 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T18:34:41.537Z,1727030081.537 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.209442 m 2024-09-22T18:34:41.992Z,1727030081.992 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.639399 degC 2024-09-22T18:34:42.364Z,1727030082.364 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = nan ug/l 2024-09-22T18:34:43.161Z,1727030083.161 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 7.000000 count completed. 2024-09-22T18:34:47.624Z,1727030087.624 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T18:35:01.556Z,1727030101.556 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-22T18:39:22.368Z,1727030362.368 [RudderServo](FAULT): Overload Error 2024-09-22T18:43:23.183Z,1727030603.183 [RudderServo](FAULT): Overload Error 2024-09-22T18:50:59.608Z,1727031059.608 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T19:09:23.007Z,1727032163.007 [RudderServo](FAULT): Overload Error 2024-09-22T19:09:29.487Z,1727032169.487 [RudderServo](FAULT): Overload Error 2024-09-22T19:14:27.711Z,1727032467.711 [RudderServo](FAULT): Overload Error 2024-09-22T19:16:33.768Z,1727032593.768 [RudderServo](FAULT): Overload Error 2024-09-22T19:19:30.299Z,1727032770.299 [RudderServo](FAULT): Overload Error 2024-09-22T19:19:36.792Z,1727032776.792 [RudderServo](FAULT): Overload Error 2024-09-22T19:19:43.667Z,1727032783.667 [RudderServo](FAULT): Overload Error 2024-09-22T19:19:44.076Z,1727032784.076 [CommandExec](FAULT): Scheduling is paused 2024-09-22T19:19:44.953Z,1727032784.953 [MissionManager](IMPORTANT): Started mission Default 2024-09-22T19:32:01.760Z,1727033521.760 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T19:40:07.938Z,1727034007.938 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.376196 min 2024-09-22T19:46:12.078Z,1727034372.078 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 26.445213 min 2024-09-22T19:47:03.374Z,1727034423.374 [DataOverHttps](IMPORTANT): SBD MOMSN=23217760, MTMSN=20240922T194702 2024-09-22T19:47:10.690Z,1727034430.690 [CommandExec](IMPORTANT): got command schedule resume 2024-09-22T19:47:10.690Z,1727034430.690 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-22T19:49:21.173Z,1727034561.173 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T194920 2024-09-22T19:49:28.648Z,1727034568.648 [CommandExec](IMPORTANT): got command schedule 20240922T210000 "load Science/circle_sample.xml;set CircleSample.NumSamplers 31 count;set CircleSample.MissionTimeout 70 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4khpk 1 5.000000 2024-09-22T19:49:28.649Z,1727034568.649 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=4khpk 2024-09-22T19:49:28.650Z,1727034568.650 [CommandExec](IMPORTANT): Scheduled #6 (#1 of 5 with id='4khpk'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 31 count;set CircleSample.MissionTimeout 70 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", AT TIME: 2024-09-22T21:00:00Z 2024-09-22T19:49:29.622Z,1727034569.622 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T194928 2024-09-22T19:49:37.012Z,1727034577.012 [CommandExec](IMPORTANT): got command schedule 20240922T210000 "set CircleSample.UseOtherLRAUVSignals 0 bool;set CircleSample.TrackAcousticTarget 1 bool;set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min" 4khpk 2 5.000000 2024-09-22T19:49:37.013Z,1727034577.013 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=4khpk 2024-09-22T19:49:37.014Z,1727034577.014 [CommandExec](IMPORTANT): Scheduled #7 (#2 of 5 with id='4khpk'): "set CircleSample.UseOtherLRAUVSignals 0 bool;set CircleSample.TrackAcousticTarget 1 bool;set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min", AT TIME: 2024-09-22T21:00:00Z 2024-09-22T19:49:37.970Z,1727034577.970 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T194936 2024-09-22T19:49:45.445Z,1727034585.445 [CommandExec](IMPORTANT): got command schedule 20240922T210000 "set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m;set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m" 4khpk 3 5.000000 2024-09-22T19:49:45.445Z,1727034585.445 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=4khpk 2024-09-22T19:49:45.447Z,1727034585.447 [CommandExec](IMPORTANT): Scheduled #8 (#3 of 5 with id='4khpk'): "set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m;set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m", AT TIME: 2024-09-22T21:00:00Z 2024-09-22T19:49:46.361Z,1727034586.361 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T194945 2024-09-22T19:49:55.705Z,1727034595.705 [CommandExec](IMPORTANT): got command schedule 20240922T210000 "set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree;set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 60 s" 4khpk 4 5.000000 2024-09-22T19:49:55.706Z,1727034595.706 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=4khpk 2024-09-22T19:49:55.707Z,1727034595.707 [CommandExec](IMPORTANT): Scheduled #9 (#4 of 5 with id='4khpk'): "set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree;set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 60 s", AT TIME: 2024-09-22T21:00:00Z 2024-09-22T19:49:56.673Z,1727034596.673 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T194955 2024-09-22T19:50:05.800Z,1727034605.800 [CommandExec](IMPORTANT): got command schedule 20240922T210000 "set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4khpk 5 5.000000 2024-09-22T19:50:05.800Z,1727034605.800 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=4khpk 2024-09-22T19:50:05.802Z,1727034605.802 [CommandExec](IMPORTANT): Scheduled #10 (#5 of 5 with id='4khpk'): "set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-22T21:00:00Z 2024-09-22T19:50:06.754Z,1727034606.754 [DataOverHttps](IMPORTANT): SBD MTMSN=20240922T195005 2024-09-22T19:50:14.140Z,1727034614.140 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-22T19:50:16.486Z,1727034616.486 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-22T19:50:23.642Z,1727034623.642 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2024-09-22T19:50:23.642Z,1727034623.642 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-22T19:50:23.643Z,1727034623.643 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-22T19:50:23.669Z,1727034623.669 [CommandExec](IMPORTANT): got command run 2024-09-22T19:50:23.671Z,1727034623.671 [CommandExec](IMPORTANT): Running 2024-09-22T19:50:23.962Z,1727034623.962 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-22T19:54:02.592Z,1727034842.592 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-22T20:10:46.616Z,1727035846.616 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-09-22T20:13:03.629Z,1727035983.629 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T20:37:19.658Z,1727037439.658 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.712467,-122.186897 2024-09-22T20:42:39.633Z,1727037759.633 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-09-22T20:54:05.660Z,1727038445.660 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T21:00:00.082Z,1727038800.082 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-22T21:00:07.204Z,1727038807.204 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-22T21:00:07.291Z,1727038807.291 [MissionManager](IMPORTANT): Started mission Default 2024-09-22T21:00:19.230Z,1727038819.230 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 31 count 2024-09-22T21:00:19.231Z,1727038819.231 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 70 hour 2024-09-22T21:00:19.231Z,1727038819.231 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-22T21:00:19.232Z,1727038819.232 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-22T21:00:38.237Z,1727038838.237 [CommandExec](IMPORTANT): got command set CircleSample.UseOtherLRAUVSignals 0 bool 2024-09-22T21:00:38.238Z,1727038838.238 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 1 bool 2024-09-22T21:00:38.238Z,1727038838.238 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-22T21:00:38.239Z,1727038838.239 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-22T21:00:38.240Z,1727038838.240 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-22T21:00:51.154Z,1727038851.154 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-22T21:00:51.154Z,1727038851.154 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-22T21:00:51.155Z,1727038851.155 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-22T21:00:51.156Z,1727038851.156 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-22T21:00:51.156Z,1727038851.156 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-22T21:01:08.374Z,1727038868.374 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-22T21:01:08.382Z,1727038868.382 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-22T21:01:08.383Z,1727038868.383 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-22T21:01:08.385Z,1727038868.385 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-22T21:01:08.386Z,1727038868.386 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 60 second 2024-09-22T22:16:55.898Z,1727043415.898 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (2 messages): @14:16:24.88 Selecting Cartridge 28 @15:10:07.64 Sampled 1000.0ml 2024-09-22T22:16:55.903Z,1727043415.903 [ESPComponent](IMPORTANT): [sample #8] ESP Cartridge 28: S_FILTERING: start: 2024-09-22T21:19:31.289Z end: 2024-09-22T22:10:10.261Z Sampled: 1000.0ml 2024-09-23T00:13:26.082Z,1727050406.082 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (2 messages): @16:14:10.98 Selecting Cartridge 27 @17:06:37.60 Sampled 1000.0ml 2024-09-23T00:13:26.084Z,1727050406.084 [ESPComponent](IMPORTANT): [sample #9] ESP Cartridge 27: S_FILTERING: start: 2024-09-22T23:17:18.498Z end: 2024-09-23T00:06:40.501Z Sampled: 1000.0ml 2024-09-23T02:21:51.246Z,1727058111.246 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (2 messages): @18:18:09.64 Selecting Cartridge 26 @19:15:03.24 Sampled 1000.0ml 2024-09-23T02:21:51.254Z,1727058111.254 [ESPComponent](IMPORTANT): [sample #10] ESP Cartridge 26: S_FILTERING: start: 2024-09-23T01:21:16.598Z end: 2024-09-23T02:15:05.628Z Sampled: 1000.0ml 2024-09-22T21:01:18.718Z,1727038878.718 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-22T21:01:18.718Z,1727038878.718 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-22T21:01:18.719Z,1727038878.719 [CommandExec](IMPORTANT): got command run 2024-09-22T21:01:18.721Z,1727038878.721 [CommandExec](IMPORTANT): Running 2024-09-22T21:01:18.877Z,1727038878.877 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-22T21:01:18.996Z,1727038878.996 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-22T21:01:20.883Z,1727038880.883 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-22T21:01:22.097Z,1727038882.097 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-22T21:01:24.146Z,1727038884.146 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T21:11:23.706Z,1727039483.706 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.034236 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-22T21:14:04.494Z,1727039644.494 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.029236 m , settling for 1.000000 min 2024-09-22T21:15:17.272Z,1727039717.272 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T21:15:17.398Z,1727039717.398 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T21:16:11.308Z,1727039771.308 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T21:16:12.121Z,1727039772.121 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2024-09-22T21:16:12.122Z,1727039772.122 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T21:16:12.586Z,1727039772.586 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.243225 m in 9.483515 degC water with 0.016520 ug/l chlorophyll fluorescence. 2024-09-22T21:16:12.922Z,1727039772.922 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T21:16:13.383Z,1727039773.383 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T21:16:15.346Z,1727039775.346 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T21:16:46.862Z,1727039806.862 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@28,aRNAl_bac) 2024-09-22T21:16:49.689Z,1727039809.689 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T21:16:49.690Z,1727039809.690 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T21:16:50.106Z,1727039810.106 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T21:19:31.290Z,1727039971.290 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T21:19:31.692Z,1727039971.692 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T21:20:27.900Z,1727040027.900 [RudderServo](FAULT): Overload Error 2024-09-22T21:30:55.347Z,1727040655.347 [RudderServo](FAULT): Overload Error 2024-09-22T21:32:54.544Z,1727040774.544 [RudderServo](FAULT): Overload Error 2024-09-22T21:35:08.616Z,1727040908.616 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T22:09:46.116Z,1727042986.116 [RudderServo](FAULT): Overload Error 2024-09-22T22:10:09.858Z,1727043009.858 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0) 2024-09-22T22:10:10.261Z,1727043010.261 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-22T22:10:12.282Z,1727043012.282 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-22T22:10:12.690Z,1727043012.690 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@28,aRNAl_bac) 2024-09-22T22:11:47.626Z,1727043107.626 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@28,aRNAl_bac) 2024-09-22T22:11:55.307Z,1727043115.307 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-22T22:11:55.708Z,1727043115.708 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2024-09-22T22:14:11.588Z,1727043251.588 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-09-22T22:16:10.245Z,1727043370.245 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T22:16:55.502Z,1727043415.502 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2024-09-22T22:16:58.027Z,1727043418.027 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2024-09-22T22:16:58.445Z,1727043418.445 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-22T22:16:58.811Z,1727043418.811 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-22T22:16:59.239Z,1727043419.239 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-22T22:16:59.646Z,1727043419.646 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 151.071259 m 2024-09-22T22:17:00.026Z,1727043420.026 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.533808 degC 2024-09-22T22:17:00.438Z,1727043420.438 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.000000 ug/l 2024-09-22T22:17:01.228Z,1727043421.228 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-09-22T22:17:05.710Z,1727043425.710 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-22T22:42:03.072Z,1727044923.072 [RudderServo](FAULT): Overload Error 2024-09-22T22:57:12.089Z,1727045832.089 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-22T23:10:03.924Z,1727046603.924 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-22T23:10:14.808Z,1727046614.808 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003255 CHAN A1 (24V): -0.001373 CHAN A2 (12V): 0.000564 CHAN A3 (5V): 0.000192 CHAN B0 (3.3V): 0.000916 CHAN B1 (3.15aV): 0.000098 CHAN B2 (3.15bV): 0.000109 CHAN B3 (GND): 0.000379 OPEN: 0.004871 Full Scale: +/- 1 mA 2024-09-22T23:11:26.187Z,1727046686.187 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027832 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-09-22T23:11:51.260Z,1727046711.260 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.895218 m , settling for 1.000000 min 2024-09-22T23:13:04.072Z,1727046784.072 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-22T23:13:04.181Z,1727046784.181 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-22T23:13:57.701Z,1727046837.701 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-22T23:13:58.517Z,1727046838.517 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2024-09-22T23:13:58.517Z,1727046838.517 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-22T23:13:58.938Z,1727046838.938 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.471329 m in 9.480768 degC water with 0.022420 ug/l chlorophyll fluorescence. 2024-09-22T23:13:59.317Z,1727046839.317 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-22T23:13:59.722Z,1727046839.722 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-22T23:14:01.741Z,1727046841.741 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-22T23:14:33.262Z,1727046873.262 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@27,aRNAl_bac) 2024-09-22T23:14:36.087Z,1727046876.087 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2024-09-22T23:14:36.088Z,1727046876.088 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-22T23:14:36.554Z,1727046876.554 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-22T23:17:18.499Z,1727047038.499 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2024-09-22T23:17:18.902Z,1727047038.902 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-22T23:38:14.232Z,1727048294.232 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T00:06:40.078Z,1727050000.078 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T00:06:40.502Z,1727050000.502 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T00:06:42.485Z,1727050002.485 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T00:06:42.890Z,1727050002.890 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@27,aRNAl_bac) 2024-09-23T00:08:17.831Z,1727050097.831 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@27,aRNAl_bac) 2024-09-23T00:08:25.503Z,1727050105.503 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T00:08:25.910Z,1727050105.910 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2024-09-23T00:13:25.678Z,1727050405.678 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2024-09-23T00:13:28.145Z,1727050408.145 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2024-09-23T00:13:28.553Z,1727050408.553 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T00:13:28.934Z,1727050408.934 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T00:13:29.342Z,1727050409.342 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T00:13:29.765Z,1727050409.765 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.963364 m 2024-09-23T00:13:30.145Z,1727050410.145 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.321314 degC 2024-09-23T00:13:30.559Z,1727050410.559 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.000000 ug/l 2024-09-23T00:13:31.426Z,1727050411.426 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-09-23T00:13:35.849Z,1727050415.849 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T00:19:16.356Z,1727050756.356 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T00:20:06.643Z,1727050806.643 [ThrusterHE](FAULT): Could not get status:serial timeout 2024-09-23T00:24:44.472Z,1727051084.472 [RudderServo](FAULT): Overload Error 2024-09-23T01:00:18.388Z,1727053218.388 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T01:02:43.108Z,1727053363.108 [RudderServo](FAULT): Overload Error 2024-09-23T01:11:28.669Z,1727053888.669 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027425 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2024-09-23T01:15:49.664Z,1727054149.664 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.182251 m , settling for 1.000000 min 2024-09-23T01:17:02.516Z,1727054222.516 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T01:17:02.628Z,1727054222.628 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T01:17:55.794Z,1727054275.794 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T01:17:56.613Z,1727054276.613 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2024-09-23T01:17:56.614Z,1727054276.614 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T01:17:57.088Z,1727054277.088 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.356735 m in 9.303003 degC water with 0.033040 ug/l chlorophyll fluorescence. 2024-09-23T01:17:57.418Z,1727054277.418 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T01:17:57.834Z,1727054277.834 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T01:17:59.904Z,1727054279.904 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T01:18:31.358Z,1727054311.358 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@26,aRNAl_bac) 2024-09-23T01:18:34.181Z,1727054314.181 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T01:18:34.182Z,1727054314.182 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T01:18:34.591Z,1727054314.591 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T01:21:16.598Z,1727054476.598 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T01:21:17.002Z,1727054477.002 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T01:33:39.672Z,1727055219.672 [RudderServo](FAULT): Overload Error 2024-09-23T01:35:43.299Z,1727055343.299 [RudderServo](FAULT): Overload Error 2024-09-23T01:41:20.179Z,1727055680.179 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T02:02:26.892Z,1727056946.892 [RudderServo](FAULT): Overload Error 2024-09-23T02:04:25.340Z,1727057065.340 [RudderServo](FAULT): Overload Error 2024-09-23T02:06:20.055Z,1727057180.055 [RudderServo](FAULT): Overload Error 2024-09-23T02:15:05.221Z,1727057705.221 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T02:15:05.629Z,1727057705.629 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T02:15:07.660Z,1727057707.660 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T02:15:08.061Z,1727057708.061 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@26,aRNAl_bac) 2024-09-23T02:16:42.989Z,1727057802.989 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@26,aRNAl_bac) 2024-09-23T02:16:50.663Z,1727057810.663 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T02:16:51.074Z,1727057811.074 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2024-09-23T02:21:50.846Z,1727058110.846 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2024-09-23T02:21:53.308Z,1727058113.308 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2024-09-23T02:21:53.693Z,1727058113.693 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T02:21:54.100Z,1727058114.100 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T02:21:54.520Z,1727058114.520 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T02:21:54.907Z,1727058114.907 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.819885 m 2024-09-23T02:21:55.313Z,1727058115.313 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.348291 degC 2024-09-23T02:21:55.725Z,1727058115.725 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.035400 ug/l 2024-09-23T02:21:56.514Z,1727058116.514 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2024-09-23T02:22:00.990Z,1727058120.990 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T02:22:21.965Z,1727058141.965 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T02:24:45.055Z,1727058285.055 [RudderServo](FAULT): Overload Error 2024-09-23T02:29:51.315Z,1727058591.315 [RudderServo](FAULT): Overload Error 2024-09-23T02:40:32.859Z,1727059232.859 [RudderServo](FAULT): Overload Error 2024-09-23T02:49:02.684Z,1727059742.684 [CommandExec](IMPORTANT): got command 2024-09-23T02:49:02.684Z,1727059742.684 [CommandExec](FAULT): Incomplete syntax. Try: help 2024-09-23T03:01:07.608Z,1727060467.608 [CommandExec](IMPORTANT): got command stop 2024-09-23T03:01:07.608Z,1727060467.608 [CommandExec](IMPORTANT): Scheduling is paused 2024-09-23T03:01:08.377Z,1727060468.377 [MissionManager](IMPORTANT): Started mission Default 2024-09-23T03:03:24.072Z,1727060604.072 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T03:16:56.182Z,1727061416.182 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 15.789746 min 2024-09-23T03:21:46.891Z,1727061706.891 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032145 2024-09-23T03:21:54.368Z,1727061714.368 [CommandExec](IMPORTANT): got command schedule resume 2024-09-23T03:21:54.368Z,1727061714.368 [CommandExec](IMPORTANT): Scheduling is resumed 2024-09-23T03:24:52.860Z,1727061892.860 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032451 2024-09-23T03:25:00.163Z,1727061900.163 [CommandExec](IMPORTANT): got command schedule 20240923T040000 "load Science/circle_sample.xml;set CircleSample.NumSamplers 28 count;set CircleSample.MissionTimeout 60 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool" 4l2su 1 5.000000 2024-09-23T03:25:00.164Z,1727061900.164 [CommandExec](IMPORTANT): Scheduling command #1 of 5 with id=4l2su 2024-09-23T03:25:00.165Z,1727061900.165 [CommandExec](IMPORTANT): Scheduled #11 (#1 of 5 with id='4l2su'): "load Science/circle_sample.xml;set CircleSample.NumSamplers 28 count;set CircleSample.MissionTimeout 60 h;set CircleSample.AcousticTrackingTimeout 0.5 h;set CircleSample.NeedCommsMissionStart 0 bool", AT TIME: 2024-09-23T04:00:00Z 2024-09-23T03:25:01.128Z,1727061901.128 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032500 2024-09-23T03:25:08.708Z,1727061908.708 [CommandExec](IMPORTANT): got command schedule 20240923T040000 "set CircleSample.UseOtherLRAUVSignals 0 bool;set CircleSample.TrackAcousticTarget 1 bool;set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min" 4l2su 2 5.000000 2024-09-23T03:25:08.708Z,1727061908.708 [CommandExec](IMPORTANT): Scheduling command #2 of 5 with id=4l2su 2024-09-23T03:25:08.709Z,1727061908.709 [CommandExec](IMPORTANT): Scheduled #12 (#2 of 5 with id='4l2su'): "set CircleSample.UseOtherLRAUVSignals 0 bool;set CircleSample.TrackAcousticTarget 1 bool;set CircleSample.AcousticTargetDepth 0 m;set CircleSample.SendSampleResultToWG 1 bool;set CircleSample.WaitBeforeFiring 0 min", AT TIME: 2024-09-23T04:00:00Z 2024-09-23T03:25:09.644Z,1727061909.644 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032509 2024-09-23T03:25:17.139Z,1727061917.139 [CommandExec](IMPORTANT): got command schedule 20240923T040000 "set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m;set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m" 4l2su 3 5.000000 2024-09-23T03:25:17.140Z,1727061917.140 [CommandExec](IMPORTANT): Scheduling command #3 of 5 with id=4l2su 2024-09-23T03:25:17.141Z,1727061917.141 [CommandExec](IMPORTANT): Scheduled #13 (#3 of 5 with id='4l2su'): "set CircleSample.MaxWaitMustFire 120 min;set CircleSample.DepthCircle 150 m;set CircleSample.CircleYoYoMinDepth 10 m;set CircleSample.CircleYoYoMaxDepth 280 m;set CircleSample.MaxDepth 290 m", AT TIME: 2024-09-23T04:00:00Z 2024-09-23T03:25:18.044Z,1727061918.044 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032517 2024-09-23T03:25:25.368Z,1727061925.368 [CommandExec](IMPORTANT): got command schedule 20240923T040000 "set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree;set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 60 s" 4l2su 4 5.000000 2024-09-23T03:25:25.368Z,1727061925.368 [CommandExec](IMPORTANT): Scheduling command #4 of 5 with id=4l2su 2024-09-23T03:25:25.369Z,1727061925.369 [CommandExec](IMPORTANT): Scheduled #14 (#4 of 5 with id='4l2su'): "set CircleSample.CenterLatitude 36.712468 degree;set CircleSample.CenterLongitude -122.1869 degree;set CircleSample.Settle1 1 min;set CircleSample.IntervalRestartLogs 12 h;set CircleSample.TrackingUpdatePeriod 60 s", AT TIME: 2024-09-23T04:00:00Z 2024-09-23T03:25:26.296Z,1727061926.296 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T032525 2024-09-23T03:25:30.034Z,1727061930.034 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 24.354038 min 2024-09-23T03:25:33.829Z,1727061933.829 [CommandExec](IMPORTANT): got command schedule 20240923T040000 "set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run" 4l2su 5 5.000000 2024-09-23T03:25:33.830Z,1727061933.830 [CommandExec](IMPORTANT): Scheduling command #5 of 5 with id=4l2su 2024-09-23T03:25:33.831Z,1727061933.831 [CommandExec](IMPORTANT): Scheduled #15 (#5 of 5 with id='4l2su'): "set CircleSample.NumberOfStartingFixesToIgnore 1 count;set CircleSample:NeedComms.CommsTimeout 15 min;run", AT TIME: 2024-09-23T04:00:00Z 2024-09-23T03:32:12.848Z,1727062332.848 [DataOverHttps](IMPORTANT): SBD MTMSN=20240923T033211 2024-09-23T03:32:20.307Z,1727062340.307 [CommandExec](IMPORTANT): got command load ./Missions/Transport/keepstation.tl 2024-09-23T03:32:22.008Z,1727062342.008 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/keepstation.tl id=keepstation 2024-09-23T03:32:28.607Z,1727062348.607 [CommandExec](IMPORTANT): got command set keepstation.NeedCommsTime 10 minute 2024-09-23T03:32:28.607Z,1727062348.607 [CommandExec](IMPORTANT): got command set keepstation.Latitude 36.712468 degree 2024-09-23T03:32:28.608Z,1727062348.608 [CommandExec](IMPORTANT): got command set keepstation.Longitude -122.1869 degree 2024-09-23T03:32:28.609Z,1727062348.609 [CommandExec](IMPORTANT): got command run 2024-09-23T03:32:28.611Z,1727062348.611 [CommandExec](IMPORTANT): Running 2024-09-23T03:32:28.751Z,1727062348.751 [MissionManager](IMPORTANT): Started mission keepstation 2024-09-23T03:37:27.289Z,1727062647.289 [keepstation:TransitToStation:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.712467,-122.186897 2024-09-23T03:44:26.218Z,1727063066.218 [BPC1](FAULT): Failed to receive data from 16 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T05:17:38.696Z,1727068658.696 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (2 messages): @21:16:25.87 Selecting Cartridge 25 @22:10:50.64 Sampled 1000.0ml 2024-09-23T05:17:38.698Z,1727068658.698 [ESPComponent](IMPORTANT): [sample #11] ESP Cartridge 25: S_FILTERING: start: 2024-09-23T04:19:32.621Z end: 2024-09-23T05:10:53.058Z Sampled: 1000.0ml 2024-09-23T07:18:24.925Z,1727075904.925 [ESPComponent](IMPORTANT): [sample #12] ESP log summary report (2 messages): @23:17:20.62 Selecting Cartridge 24 @00:11:37.24 Sampled 1000.0ml 2024-09-23T07:18:24.926Z,1727075904.926 [ESPComponent](IMPORTANT): [sample #12] ESP Cartridge 24: S_FILTERING: start: 2024-09-23T06:20:27.044Z end: 2024-09-23T07:11:39.304Z Sampled: 1000.0ml 2024-09-23T09:20:18.370Z,1727083218.370 [ESPComponent](IMPORTANT): [sample #13] ESP log summary report (2 messages): @01:17:54.07 Selecting Cartridge 23 @02:13:29.91 Sampled 1000.0ml 2024-09-23T09:20:18.371Z,1727083218.371 [ESPComponent](IMPORTANT): [sample #13] ESP Cartridge 23: S_FILTERING: start: 2024-09-23T08:21:01.850Z end: 2024-09-23T09:13:32.345Z Sampled: 1000.0ml 2024-09-23T11:21:59.132Z,1727090519.132 [ESPComponent](IMPORTANT): [sample #14] ESP log summary report (2 messages): @03:19:01.39 Selecting Cartridge 22 @04:15:11.51 Sampled 1000.0ml 2024-09-23T11:21:59.134Z,1727090519.134 [ESPComponent](IMPORTANT): [sample #14] ESP Cartridge 22: S_FILTERING: start: 2024-09-23T10:22:08.621Z end: 2024-09-23T11:15:13.496Z Sampled: 1000.0ml 2024-09-23T13:20:33.613Z,1727097633.613 [ESPComponent](IMPORTANT): [sample #15] ESP log summary report (2 messages): @05:16:46.78 Selecting Cartridge 21 @06:13:45.88 Sampled 1000.0ml 2024-09-23T13:20:33.614Z,1727097633.614 [ESPComponent](IMPORTANT): [sample #15] ESP Cartridge 21: S_FILTERING: start: 2024-09-23T12:19:53.301Z end: 2024-09-23T13:13:47.995Z Sampled: 1000.0ml 2024-09-23T15:19:46.316Z,1727104786.316 [ESPComponent](IMPORTANT): [sample #16] ESP log summary report (2 messages): @07:18:59.00 Selecting Cartridge 20 @08:12:58.16 Sampled 1000.0ml 2024-09-23T15:19:46.318Z,1727104786.318 [ESPComponent](IMPORTANT): [sample #16] ESP Cartridge 20: S_FILTERING: start: 2024-09-23T14:22:06.208Z end: 2024-09-23T15:13:00.520Z Sampled: 1000.0ml 2024-09-23T04:00:00.145Z,1727064000.145 [CommandExec](IMPORTANT): got command load ./Missions/Science/circle_sample.xml 2024-09-23T04:00:07.642Z,1727064007.642 [CommandExec](IMPORTANT): Loaded ./Missions/Science/circle_sample.xml id=CircleSample 2024-09-23T04:00:07.793Z,1727064007.793 [MissionManager](IMPORTANT): Started mission Default 2024-09-23T04:00:20.645Z,1727064020.645 [CommandExec](IMPORTANT): got command set CircleSample.NumSamplers 28 count 2024-09-23T04:00:20.646Z,1727064020.646 [CommandExec](IMPORTANT): got command set CircleSample.MissionTimeout 60 hour 2024-09-23T04:00:20.647Z,1727064020.647 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTrackingTimeout 0.5 hour 2024-09-23T04:00:20.647Z,1727064020.647 [CommandExec](IMPORTANT): got command set CircleSample.NeedCommsMissionStart 0 bool 2024-09-23T04:00:37.913Z,1727064037.913 [CommandExec](IMPORTANT): got command set CircleSample.UseOtherLRAUVSignals 0 bool 2024-09-23T04:00:37.920Z,1727064037.920 [CommandExec](IMPORTANT): got command set CircleSample.TrackAcousticTarget 1 bool 2024-09-23T04:00:37.921Z,1727064037.921 [CommandExec](IMPORTANT): got command set CircleSample.AcousticTargetDepth 0 meter 2024-09-23T04:00:37.921Z,1727064037.921 [CommandExec](IMPORTANT): got command set CircleSample.SendSampleResultToWG 1 bool 2024-09-23T04:00:37.922Z,1727064037.922 [CommandExec](IMPORTANT): got command set CircleSample.WaitBeforeFiring 0 minute 2024-09-23T04:00:53.382Z,1727064053.382 [CommandExec](IMPORTANT): got command set CircleSample.MaxWaitMustFire 120 minute 2024-09-23T04:00:53.383Z,1727064053.383 [CommandExec](IMPORTANT): got command set CircleSample.DepthCircle 150 meter 2024-09-23T04:00:53.383Z,1727064053.383 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMinDepth 10 meter 2024-09-23T04:00:53.384Z,1727064053.384 [CommandExec](IMPORTANT): got command set CircleSample.CircleYoYoMaxDepth 280 meter 2024-09-23T04:00:53.385Z,1727064053.385 [CommandExec](IMPORTANT): got command set CircleSample.MaxDepth 290 meter 2024-09-23T04:01:07.742Z,1727064067.742 [CommandExec](IMPORTANT): got command set CircleSample.CenterLatitude 36.712468 degree 2024-09-23T04:01:07.743Z,1727064067.743 [CommandExec](IMPORTANT): got command set CircleSample.CenterLongitude -122.1869 degree 2024-09-23T04:01:07.744Z,1727064067.744 [CommandExec](IMPORTANT): got command set CircleSample.Settle1 1 minute 2024-09-23T04:01:07.745Z,1727064067.745 [CommandExec](IMPORTANT): got command set CircleSample.IntervalRestartLogs 12 hour 2024-09-23T04:01:07.745Z,1727064067.745 [CommandExec](IMPORTANT): got command set CircleSample.TrackingUpdatePeriod 60 second 2024-09-23T04:01:15.472Z,1727064075.472 [CommandExec](IMPORTANT): got command set CircleSample.NumberOfStartingFixesToIgnore 1 count 2024-09-23T04:01:15.472Z,1727064075.472 [CommandExec](IMPORTANT): got command set CircleSample:NeedComms.CommsTimeout 15 minute 2024-09-23T04:01:15.473Z,1727064075.473 [CommandExec](IMPORTANT): got command run 2024-09-23T04:01:15.475Z,1727064075.475 [CommandExec](IMPORTANT): Running 2024-09-23T04:01:15.719Z,1727064075.719 [MissionManager](IMPORTANT): Started mission CircleSample 2024-09-23T04:01:15.769Z,1727064075.769 [CircleSample:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 0.50 hours. 2024-09-23T04:01:17.767Z,1727064077.767 [CircleSample:MissionStartNoComms:A](IMPORTANT): Mission start no comms. 2024-09-23T04:01:18.986Z,1727064078.986 [CircleSample:CircleAndSample:B](IMPORTANT): Circle around location 36.712467 arcdeg , -122.186897 arcdeg , at radius = 20.000000 m and depth = 150.000000 m . 2024-09-23T04:01:20.984Z,1727064080.984 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T04:06:06.615Z,1727064366.615 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2024-09-23T04:11:20.178Z,1727064680.178 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 10.027189 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 1.000000 count 2024-09-23T04:14:05.422Z,1727064845.422 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.984756 m , settling for 1.000000 min 2024-09-23T04:15:18.206Z,1727064918.206 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T04:15:18.339Z,1727064918.339 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T04:16:12.252Z,1727064972.252 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T04:16:13.060Z,1727064973.060 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2024-09-23T04:16:13.061Z,1727064973.061 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T04:16:13.504Z,1727064973.504 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.557190 m in 9.341211 degC water with 0.016520 ug/l chlorophyll fluorescence. 2024-09-23T04:16:13.863Z,1727064973.863 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T04:16:14.271Z,1727064974.271 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T04:16:16.297Z,1727064976.297 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T04:16:47.809Z,1727065007.809 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@25,aRNAl_bac) 2024-09-23T04:16:50.187Z,1727065010.187 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T04:16:50.562Z,1727065010.562 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T04:16:50.972Z,1727065010.972 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T04:19:32.622Z,1727065172.622 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T04:19:33.015Z,1727065173.015 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T04:25:28.118Z,1727065528.118 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T04:26:39.302Z,1727065599.302 [RudderServo](FAULT): Overload Error 2024-09-23T04:40:59.918Z,1727066459.918 [ThrusterHE](FAULT): Could not get status:serial timeout 2024-09-23T04:46:02.654Z,1727066762.654 [RudderServo](FAULT): Overload Error 2024-09-23T04:47:57.791Z,1727066877.791 [RudderServo](FAULT): Overload Error 2024-09-23T05:06:29.662Z,1727067989.662 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T05:10:15.007Z,1727068215.007 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-23T05:10:25.753Z,1727068225.753 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003522 CHAN A1 (24V): -0.001053 CHAN A2 (12V): -0.000010 CHAN A3 (5V): 0.000166 CHAN B0 (3.3V): 0.000419 CHAN B1 (3.15aV): 0.000354 CHAN B2 (3.15bV): 0.000483 CHAN B3 (GND): 0.000946 OPEN: 0.004993 Full Scale: +/- 1 mA 2024-09-23T05:10:52.660Z,1727068252.660 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T05:10:53.059Z,1727068253.059 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T05:10:55.086Z,1727068255.086 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T05:10:55.500Z,1727068255.500 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@25,aRNAl_bac) 2024-09-23T05:12:30.436Z,1727068350.436 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@25,aRNAl_bac) 2024-09-23T05:12:38.117Z,1727068358.117 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T05:12:38.524Z,1727068358.524 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2024-09-23T05:17:38.308Z,1727068658.308 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2024-09-23T05:17:40.748Z,1727068660.748 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 1.000000 count 2024-09-23T05:17:41.166Z,1727068661.166 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T05:17:41.541Z,1727068661.541 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T05:17:41.950Z,1727068661.950 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T05:17:42.366Z,1727068662.366 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.687332 m 2024-09-23T05:17:42.749Z,1727068662.749 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.310999 degC 2024-09-23T05:17:43.157Z,1727068663.157 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.047200 ug/l 2024-09-23T05:17:43.973Z,1727068663.973 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 1.000000 count completed. 2024-09-23T05:17:48.474Z,1727068668.474 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T05:35:48.498Z,1727069748.498 [RudderServo](FAULT): Overload Error 2024-09-23T05:35:54.970Z,1727069754.970 [RudderServo](FAULT): Overload Error 2024-09-23T05:38:05.446Z,1727069885.446 [RudderServo](FAULT): Overload Error 2024-09-23T05:39:58.570Z,1727069998.570 [RudderServo](FAULT): Overload Error 2024-09-23T05:43:12.962Z,1727070192.962 [RudderServo](FAULT): Overload Error 2024-09-23T05:43:20.294Z,1727070200.294 [RudderServo](FAULT): Overload Error 2024-09-23T05:45:11.266Z,1727070311.266 [RudderServo](FAULT): Overload Error 2024-09-23T05:46:08.242Z,1727070368.242 [RudderServo](FAULT): Overload Error 2024-09-23T05:47:31.800Z,1727070451.800 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T05:48:16.306Z,1727070496.306 [RudderServo](FAULT): Overload Error 2024-09-23T06:11:22.804Z,1727071882.804 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030672 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 2.000000 count 2024-09-23T06:14:52.493Z,1727072092.493 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.138763 m , settling for 1.000000 min 2024-09-23T06:16:05.290Z,1727072165.290 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T06:16:05.424Z,1727072165.424 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T06:17:06.601Z,1727072226.601 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T06:17:07.407Z,1727072227.407 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -1 2024-09-23T06:17:07.408Z,1727072227.408 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T06:17:07.841Z,1727072227.841 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 148.916550 m in 9.338984 degC water with 0.041300 ug/l chlorophyll fluorescence. 2024-09-23T06:17:08.212Z,1727072228.212 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T06:17:08.629Z,1727072228.629 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T06:17:10.766Z,1727072230.766 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T06:17:42.205Z,1727072262.205 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@24,aRNAl_bac) 2024-09-23T06:17:44.221Z,1727072264.221 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T06:17:44.638Z,1727072264.638 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T06:17:45.077Z,1727072265.077 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T06:20:27.045Z,1727072427.045 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T06:20:27.446Z,1727072427.446 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T06:26:02.031Z,1727072762.031 [RudderServo](FAULT): Overload Error 2024-09-23T06:27:39.398Z,1727072859.398 [RudderServo](FAULT): Overload Error 2024-09-23T06:28:33.884Z,1727072913.884 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T06:42:10.539Z,1727073730.539 [RudderServo](FAULT): Overload Error 2024-09-23T07:09:35.684Z,1727075375.684 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T07:11:38.916Z,1727075498.916 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T07:11:39.304Z,1727075499.304 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T07:11:41.341Z,1727075501.341 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T07:11:41.727Z,1727075501.727 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@24,aRNAl_bac) 2024-09-23T07:13:16.669Z,1727075596.669 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@24,aRNAl_bac) 2024-09-23T07:13:24.355Z,1727075604.355 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T07:13:24.754Z,1727075604.754 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2024-09-23T07:13:26.839Z,1727075606.839 [RudderServo](FAULT): Overload Error 2024-09-23T07:18:24.516Z,1727075904.516 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2024-09-23T07:18:26.978Z,1727075906.978 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 2.000000 count 2024-09-23T07:18:27.372Z,1727075907.372 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T07:18:27.772Z,1727075907.772 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T07:18:28.189Z,1727075908.189 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T07:18:28.595Z,1727075908.595 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.833359 m 2024-09-23T07:18:28.984Z,1727075908.984 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.317896 degC 2024-09-23T07:18:29.404Z,1727075909.404 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.035400 ug/l 2024-09-23T07:18:30.208Z,1727075910.208 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 2.000000 count completed. 2024-09-23T07:18:34.652Z,1727075914.652 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T07:27:36.966Z,1727076456.966 [RudderServo](FAULT): Overload Error 2024-09-23T07:45:24.926Z,1727077524.926 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2024-09-23T07:50:37.535Z,1727077837.535 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T08:00:25.826Z,1727078425.826 [RudderServo](FAULT): Overload Error 2024-09-23T08:11:25.557Z,1727079085.557 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032267 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 3.000000 count 2024-09-23T08:15:34.965Z,1727079334.965 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.717575 m , settling for 1.000000 min 2024-09-23T08:16:47.782Z,1727079407.782 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T08:16:47.903Z,1727079407.903 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T08:17:40.699Z,1727079460.699 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T08:17:41.487Z,1727079461.487 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -1 2024-09-23T08:17:41.487Z,1727079461.487 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T08:17:41.939Z,1727079461.939 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.301468 m in 9.301233 degC water with 0.024780 ug/l chlorophyll fluorescence. 2024-09-23T08:17:42.268Z,1727079462.268 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T08:17:42.672Z,1727079462.672 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T08:17:44.691Z,1727079464.691 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T08:18:16.204Z,1727079496.204 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@23,aRNAl_bac) 2024-09-23T08:18:19.033Z,1727079499.033 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T08:18:19.035Z,1727079499.035 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T08:18:19.463Z,1727079499.463 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T08:21:01.851Z,1727079661.851 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T08:21:02.253Z,1727079662.253 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T08:31:39.378Z,1727080299.378 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T09:08:01.090Z,1727082481.090 [RudderServo](FAULT): Overload Error 2024-09-23T09:12:41.434Z,1727082761.434 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T09:13:31.944Z,1727082811.944 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T09:13:32.346Z,1727082812.346 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T09:13:34.357Z,1727082814.357 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T09:13:34.759Z,1727082814.759 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@23,aRNAl_bac) 2024-09-23T09:15:09.769Z,1727082909.769 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@23,aRNAl_bac) 2024-09-23T09:15:17.436Z,1727082917.436 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T09:15:17.809Z,1727082917.809 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2024-09-23T09:20:17.965Z,1727083217.965 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2024-09-23T09:20:20.430Z,1727083220.430 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 3.000000 count 2024-09-23T09:20:20.813Z,1727083220.813 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T09:20:21.228Z,1727083221.228 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T09:20:21.639Z,1727083221.639 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T09:20:22.020Z,1727083222.020 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.273315 m 2024-09-23T09:20:22.432Z,1727083222.432 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.191614 degC 2024-09-23T09:20:22.835Z,1727083222.835 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.047200 ug/l 2024-09-23T09:20:23.645Z,1727083223.645 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 3.000000 count completed. 2024-09-23T09:20:28.121Z,1727083228.121 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T09:20:30.270Z,1727083230.270 [DATMMP](FAULT): Buffer send receipt timeout failure. 2024-09-23T09:46:13.891Z,1727084773.891 [RudderServo](FAULT): Overload Error 2024-09-23T09:46:56.734Z,1727084816.734 [RudderServo](FAULT): Overload Error 2024-09-23T09:53:43.518Z,1727085223.518 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T10:11:28.198Z,1727086288.198 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030233 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 4.000000 count 2024-09-23T10:16:41.729Z,1727086601.729 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.220566 m , settling for 1.000000 min 2024-09-23T10:17:54.522Z,1727086674.522 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T10:17:54.665Z,1727086674.665 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T10:17:59.031Z,1727086679.031 [RudderServo](FAULT): Overload Error 2024-09-23T10:18:47.846Z,1727086727.846 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T10:18:48.709Z,1727086728.709 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -1 2024-09-23T10:18:48.710Z,1727086728.710 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T10:18:49.087Z,1727086729.087 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.481201 m in 9.194910 degC water with 0.028320 ug/l chlorophyll fluorescence. 2024-09-23T10:18:49.460Z,1727086729.460 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T10:18:49.860Z,1727086729.860 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T10:18:51.880Z,1727086731.880 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T10:19:23.385Z,1727086763.385 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@22,aRNAl_bac) 2024-09-23T10:19:25.395Z,1727086765.395 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T10:19:25.805Z,1727086765.805 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T10:19:26.206Z,1727086766.206 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T10:20:07.474Z,1727086807.474 [RudderServo](FAULT): Overload Error 2024-09-23T10:22:08.623Z,1727086928.623 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T10:22:09.029Z,1727086929.029 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T10:34:45.332Z,1727087685.332 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T10:35:01.654Z,1727087701.654 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2024-09-23T10:44:03.338Z,1727088243.338 [RudderServo](FAULT): Overload Error 2024-09-23T11:10:25.906Z,1727089825.906 [CBIT](IMPORTANT): Beginning ground fault scan 2024-09-23T11:10:36.845Z,1727089836.845 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003539 CHAN A1 (24V): -0.001299 CHAN A2 (12V): 0.000588 CHAN A3 (5V): 0.000095 CHAN B0 (3.3V): 0.000775 CHAN B1 (3.15aV): -0.000012 CHAN B2 (3.15bV): 0.000029 CHAN B3 (GND): 0.000257 OPEN: 0.004999 Full Scale: +/- 1 mA 2024-09-23T11:15:13.091Z,1727090113.091 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T11:15:13.497Z,1727090113.497 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T11:15:15.522Z,1727090115.522 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T11:15:15.920Z,1727090115.920 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@22,aRNAl_bac) 2024-09-23T11:15:47.488Z,1727090147.488 [BPC1](FAULT): Battery stick #3 (s/n: 00BF) reported TERMINATE_CHARGE_ALARM. Status code: 0x40C0. 2024-09-23T11:15:47.494Z,1727090147.494 [BPC1](FAULT): Failed to receive data from 13 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T11:16:50.867Z,1727090210.867 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@22,aRNAl_bac) 2024-09-23T11:16:58.545Z,1727090218.545 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T11:16:58.952Z,1727090218.952 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED 2024-09-23T11:21:58.720Z,1727090518.720 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally. 2024-09-23T11:22:01.173Z,1727090521.173 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 4.000000 count 2024-09-23T11:22:01.584Z,1727090521.584 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T11:22:02.006Z,1727090522.006 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T11:22:02.377Z,1727090522.377 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T11:22:02.798Z,1727090522.798 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.628433 m 2024-09-23T11:22:03.199Z,1727090523.199 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.143915 degC 2024-09-23T11:22:03.587Z,1727090523.587 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.047200 ug/l 2024-09-23T11:22:04.403Z,1727090524.403 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 4.000000 count completed. 2024-09-23T11:22:08.902Z,1727090528.902 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T11:56:49.526Z,1727092609.526 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T12:11:30.710Z,1727093490.710 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028190 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 5.000000 count 2024-09-23T12:14:26.891Z,1727093666.891 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.184219 m , settling for 1.000000 min 2024-09-23T12:15:39.675Z,1727093739.675 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T12:15:39.798Z,1727093739.798 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T12:16:32.895Z,1727093792.895 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T12:16:33.709Z,1727093793.709 [ESPComponent](IMPORTANT): [sample #15] Setting cartridge argument to -1 2024-09-23T12:16:33.709Z,1727093793.709 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T12:16:34.141Z,1727093794.141 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.716507 m in 9.085443 degC water with 0.034220 ug/l chlorophyll fluorescence. 2024-09-23T12:16:34.513Z,1727093794.513 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T12:16:34.915Z,1727093794.915 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T12:16:36.940Z,1727093796.940 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T12:17:08.464Z,1727093828.464 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PRIMING (@21,aRNAl_bac) 2024-09-23T12:17:11.276Z,1727093831.276 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T12:17:11.279Z,1727093831.279 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T12:17:11.695Z,1727093831.695 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T12:19:53.302Z,1727093993.302 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T12:19:53.686Z,1727093993.686 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T12:37:51.712Z,1727095071.712 [BPC1](FAULT): Failed to receive data from 14 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T12:38:05.082Z,1727095085.082 [RudderServo](FAULT): Overload Error 2024-09-23T13:13:47.584Z,1727097227.584 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T13:13:47.996Z,1727097227.996 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T13:13:50.009Z,1727097230.009 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T13:13:50.409Z,1727097230.409 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PROCESSED (@21,aRNAl_bac) 2024-09-23T13:15:25.351Z,1727097325.351 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPING (@21,aRNAl_bac) 2024-09-23T13:15:33.028Z,1727097333.028 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T13:15:33.437Z,1727097333.437 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPED 2024-09-23T13:18:53.839Z,1727097533.839 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T13:20:33.203Z,1727097633.203 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence completed normally. 2024-09-23T13:20:35.656Z,1727097635.656 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 5.000000 count 2024-09-23T13:20:36.098Z,1727097636.098 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T13:20:36.479Z,1727097636.479 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T13:20:36.863Z,1727097636.863 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T13:20:37.268Z,1727097637.268 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 150.135376 m 2024-09-23T13:20:37.687Z,1727097637.687 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.201990 degC 2024-09-23T13:20:38.087Z,1727097638.087 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.035400 ug/l 2024-09-23T13:20:38.881Z,1727097638.881 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 5.000000 count completed. 2024-09-23T13:20:43.392Z,1727097643.392 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T13:27:28.678Z,1727098048.678 [RudderServo](FAULT): Overload Error 2024-09-23T13:28:20.018Z,1727098100.018 [RudderServo](FAULT): Overload Error 2024-09-23T13:33:32.873Z,1727098412.873 [RudderServo](FAULT): Overload Error 2024-09-23T13:56:24.862Z,1727099784.862 [RudderServo](FAULT): Overload Error 2024-09-23T13:59:55.304Z,1727099995.304 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T14:03:57.774Z,1727100237.774 [RudderServo](FAULT): Overload Error 2024-09-23T14:07:23.846Z,1727100443.846 [RudderServo](FAULT): Overload Error 2024-09-23T14:10:59.182Z,1727100659.182 [RudderServo](FAULT): Overload Error 2024-09-23T14:11:33.485Z,1727100693.485 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032715 min ) is longer than MaxWaitFirstSample (= 10.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 6.000000 count 2024-09-23T14:11:44.043Z,1727100704.043 [RudderServo](FAULT): Overload Error 2024-09-23T14:16:38.927Z,1727100998.927 [CircleSample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.825272 m , settling for 1.000000 min 2024-09-23T14:17:51.774Z,1727101071.774 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2024-09-23T14:17:51.883Z,1727101071.883 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2024-09-23T14:18:45.830Z,1727101125.830 [ESPComponent](IMPORTANT): ESP has connected as client 2024-09-23T14:18:46.653Z,1727101126.653 [ESPComponent](IMPORTANT): [sample #16] Setting cartridge argument to -1 2024-09-23T14:18:46.653Z,1727101126.653 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2024-09-23T14:18:47.059Z,1727101127.059 [CircleSample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.115921 m in 9.113672 degC water with 0.028320 ug/l chlorophyll fluorescence. 2024-09-23T14:18:47.444Z,1727101127.444 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_LOG 2024-09-23T14:18:47.848Z,1727101127.848 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_STATUS 2024-09-23T14:18:49.864Z,1727101129.864 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_LOADING_CARTRIDGE 2024-09-23T14:19:21.381Z,1727101161.381 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PRIMING (@20,aRNAl_bac) 2024-09-23T14:19:23.801Z,1727101163.801 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PRIMING (@100.0) 2024-09-23T14:19:23.802Z,1727101163.802 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=94.169998 issuedCmd_=Cmd.startFiltering 2024-09-23T14:19:24.209Z,1727101164.209 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2024-09-23T14:22:06.209Z,1727101326.209 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_FILTERING (@0.0) 2024-09-23T14:22:06.632Z,1727101326.632 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2024-09-23T14:40:57.011Z,1727102457.011 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T14:52:58.750Z,1727103178.750 [ThrusterHE](FAULT): getSpeed error serial timeout 2024-09-23T14:52:58.750Z,1727103178.750 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2024-09-23T15:13:00.117Z,1727104380.117 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PAUSED (@1000.0) 2024-09-23T15:13:00.520Z,1727104380.520 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PROCESSING (@1000.0) 2024-09-23T15:13:02.547Z,1727104382.547 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2024-09-23T15:13:02.948Z,1727104382.948 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,aRNAl_bac) 2024-09-23T15:14:37.880Z,1727104477.880 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPING (@20,aRNAl_bac) 2024-09-23T15:14:45.569Z,1727104485.569 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2024-09-23T15:14:45.965Z,1727104485.965 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPED 2024-09-23T15:19:45.908Z,1727104785.908 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence completed normally. 2024-09-23T15:19:48.332Z,1727104788.332 [CircleSample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 6.000000 count 2024-09-23T15:19:48.742Z,1727104788.742 [CircleSample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 2024-09-23T15:19:49.152Z,1727104789.152 [CircleSample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 2024-09-23T15:19:49.544Z,1727104789.544 [CircleSample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 2024-09-23T15:19:49.994Z,1727104789.994 [CircleSample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 149.691360 m 2024-09-23T15:19:50.368Z,1727104790.368 [CircleSample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.106409 degC 2024-09-23T15:19:50.777Z,1727104790.777 [CircleSample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.011800 ug/l 2024-09-23T15:19:51.564Z,1727104791.564 [CircleSample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 6.000000 count completed. 2024-09-23T15:19:56.069Z,1727104796.069 [CircleSample:SetSampleOptions:E](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 1.000000 min . 2024-09-23T15:21:58.826Z,1727104918.826 [BPC1](FAULT): Failed to receive data from 15 sticks prior to timeout. Will not retry this scan cycle. Missing stick IDs are: 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 26, 34. 2024-09-23T16:01:16.240Z,1727107276.240 [CircleSample:CycleLogs:A:B](IMPORTANT): Restarting logs.