2021-08-16T12:38:49.907Z,1629117529.907 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2021-08-16T12:38:53.781Z,1629117533.781 [SBIT](IMPORTANT): git: 2021-08-04 2021-08-16T12:38:53.782Z,1629117533.782 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2021-08-16T12:39:41.457Z,1629117581.457 [SBIT](IMPORTANT): Beginning Startup BIT 2021-08-16T12:39:41.461Z,1629117581.461 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-16T12:39:52.401Z,1629117592.401 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.269223 CHAN A1 (24V): 0.349392 CHAN A2 (12V): 0.148244 CHAN A3 (5V): 0.055564 CHAN B0 (3.3V): 0.036701 CHAN B1 (3.15aV): 0.036590 CHAN B2 (3.15bV): 0.035497 CHAN B3 (GND): -0.001847 OPEN: 0.016277 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-16T12:40:21.462Z,1629117621.462 [SBIT](FAULT): Control surface position failure. 2021-08-16T12:40:35.182Z,1629117635.182 [SBIT](FAULT): Control surface position failure. 2021-08-16T12:40:35.605Z,1629117635.605 [SBIT](CRITICAL): SBIT FAILED 2021-08-16T12:40:35.605Z,1629117635.605 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2021-08-16T12:40:35.606Z,1629117635.606 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count; 2021-08-16T12:40:35.606Z,1629117635.606 [SBIT](IMPORTANT): BackseatComponent.loadAtStartup=1 bool; 2021-08-16T12:40:35.606Z,1629117635.606 [SBIT](IMPORTANT): BackseatComponent.simulateHardware=1 bool; 2021-08-16T12:40:35.606Z,1629117635.606 [SBIT](IMPORTANT): BuoyancyServo.limitLo=1750 count; 2021-08-16T12:40:35.606Z,1629117635.606 [SBIT](IMPORTANT): CBIT.abortDepth=15 meter; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): CBIT.stopDepth=10 meter; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): CTD_Seabird.minSalinityBound=0.01 practical_salinity_unit; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): DAT.loadAtStartup=0 bool; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=1 bool; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): DeadReckonUsingMultipleVelocitySources.loadAtStartup=0 bool; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): DeadReckonUsingSpeedCalculator.loadAtStartup=1 bool; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): Depth_Keller.offset=3.49 decibar; 2021-08-16T12:40:35.607Z,1629117635.607 [SBIT](IMPORTANT): ESPComponent.processCompleteTimeout=120 minute; 2021-08-16T12:40:35.608Z,1629117635.608 [SBIT](IMPORTANT): Express linearApproximation CTD_Seabird.bin_median_sea_water_salinity 0.010000 practical_salinity_unit; 2021-08-16T12:40:35.608Z,1629117635.608 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2021-08-16T12:40:35.608Z,1629117635.608 [SBIT](IMPORTANT): Express linearApproximation WetLabsBB2FL.bin_median_mass_concentration_of_chlorophyll_in_sea_water 0.200000 microgram_per_liter; 2021-08-16T12:40:35.608Z,1629117635.608 [SBIT](IMPORTANT): Express linearApproximation height_above_sea_floor 1.000000 meter; 2021-08-16T12:40:35.608Z,1629117635.608 [SBIT](IMPORTANT): HorizontalControl.rudLimit=10 degree; 2021-08-16T12:40:35.609Z,1629117635.609 [SBIT](IMPORTANT): RudderServo.loadAtStartup=0 bool; 2021-08-16T12:40:35.609Z,1629117635.609 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=438.149052 cubic_centimeter; 2021-08-16T12:40:35.609Z,1629117635.609 [SBIT](IMPORTANT): VerticalControl.massDefault=6.905350 millimeter; 2021-08-16T12:40:35.610Z,1629117635.610 [CommandLine](FAULT): Scheduling is paused 2021-08-16T12:40:36.003Z,1629117636.003 [MissionManager](IMPORTANT): Started mission Startup 2021-08-16T12:42:05.573Z,1629117725.573 [MissionManager](IMPORTANT): Started mission Default 2021-08-16T12:42:08.791Z,1629117728.791 [DataOverHttps](IMPORTANT): Could not stat file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:42:08.792Z,1629117728.792 [DataOverHttps](FAULT): Could not open file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:42:17.014Z,1629117737.014 [DataOverHttps](IMPORTANT): Could not stat file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:42:17.015Z,1629117737.015 [DataOverHttps](FAULT): Could not open file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:42:52.819Z,1629117772.819 [DataOverHttps](IMPORTANT): Could not stat file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:42:52.819Z,1629117772.819 [DataOverHttps](FAULT): Could not open file Logs/20210816T123843/Courier0000.lzma 2021-08-16T12:44:21.885Z,1629117861.885 [DataOverHttps](IMPORTANT): SBD MOMSN=15888872, MTMSN=20210816T124419 2021-08-16T12:44:34.690Z,1629117874.690 [CommandLine](IMPORTANT): got command schedule "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ESPCartridgeType_3 -4 count" 2uu1m 1 2.000000 2021-08-16T12:44:34.690Z,1629117874.690 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2uu1m 2021-08-16T12:44:34.691Z,1629117874.691 [CommandLine](IMPORTANT): Scheduled #1 (#1 of 2 with id='2uu1m'): "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ESPCartridgeType_3 -4 count" 2021-08-16T12:44:34.692Z,1629117874.692 [CommandLine](FAULT): Scheduling was paused by an error at 2021-08-16T12:40:35Z 2021-08-16T12:44:35.799Z,1629117875.799 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T124433 2021-08-16T12:44:43.605Z,1629117883.605 [CommandLine](IMPORTANT): got command schedule "set esp_sample_at_depth.ApproachSpeed 1 m/s;set esp_sample_at_depth.ApproachDepthRate 0.2 m/s;set esp_sample_at_depth.ApproachPitchLimit 12 degree;set esp_sample_at_depth.MinAltitude 0.5 m;set esp_sample_at_depth.MaxDepth 4.8 m;run" 2uu1m 2 2.000000 2021-08-16T12:44:43.606Z,1629117883.606 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2uu1m 2021-08-16T12:44:43.607Z,1629117883.607 [CommandLine](IMPORTANT): Scheduled #2 (#2 of 2 with id='2uu1m'): "set esp_sample_at_depth.ApproachSpeed 1 m/s;set esp_sample_at_depth.ApproachDepthRate 0.2 m/s;set esp_sample_at_depth.ApproachPitchLimit 12 degree;set esp_sample_at_depth.MinAltitude 0.5 m;set esp_sample_at_depth.MaxDepth 4.8 m;run" 2021-08-16T12:44:43.607Z,1629117883.607 [CommandLine](FAULT): Scheduling was paused by an error at 2021-08-16T12:40:35Z 2021-08-16T12:46:27.711Z,1629117987.711 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T124625 2021-08-16T12:46:36.189Z,1629117996.189 [CommandLine](IMPORTANT): got command stop 2021-08-16T12:46:36.189Z,1629117996.189 [CommandLine](IMPORTANT): Scheduling is paused for 2 commands 2021-08-16T12:46:37.135Z,1629117997.135 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T124635 2021-08-16T12:46:45.236Z,1629118005.236 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ESPCartridgeType_3 -4 count" 2uu5j 1 2.000000 2021-08-16T12:46:45.236Z,1629118005.236 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2uu5j 2021-08-16T12:46:45.237Z,1629118005.237 [CommandLine](IMPORTANT): Scheduled #3 (#1 of 2 with id='2uu5j'): "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ESPCartridgeType_3 -4 count" ASAP 2021-08-16T12:46:46.183Z,1629118006.183 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T124644 2021-08-16T12:46:54.546Z,1629118014.546 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_depth.ApproachSpeed 1 m/s;set esp_sample_at_depth.ApproachDepthRate 0.2 m/s;set esp_sample_at_depth.ApproachPitchLimit 12 degree;set esp_sample_at_depth.MinAltitude 0.5 m;set esp_sample_at_depth.MaxDepth 4.8 m;run" 2uu5j 2 2.000000 2021-08-16T12:46:54.547Z,1629118014.547 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2uu5j 2021-08-16T12:46:54.548Z,1629118014.548 [CommandLine](IMPORTANT): Scheduled #4 (#2 of 2 with id='2uu5j'): "set esp_sample_at_depth.ApproachSpeed 1 m/s;set esp_sample_at_depth.ApproachDepthRate 0.2 m/s;set esp_sample_at_depth.ApproachPitchLimit 12 degree;set esp_sample_at_depth.MinAltitude 0.5 m;set esp_sample_at_depth.MaxDepth 4.8 m;run" ASAP 2021-08-16T12:46:54.777Z,1629118014.777 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T12:46:57.755Z,1629118017.755 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T12:47:11.031Z,1629118031.031 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MissionTimeout 5.000000 hour 2021-08-16T12:47:11.032Z,1629118031.032 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.TargetDepth 2.000000 meter 2021-08-16T12:47:11.045Z,1629118031.045 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ESPCartridgeType_2 -18.000000 count 2021-08-16T12:47:11.046Z,1629118031.046 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ESPCartridgeType_3 -4.000000 count 2021-08-16T12:47:26.373Z,1629118046.373 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachSpeed 1.000000 meter_per_second 2021-08-16T12:47:26.374Z,1629118046.374 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachDepthRate 0.200000 meter_per_second 2021-08-16T12:47:26.375Z,1629118046.375 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachPitchLimit 12.000000 degree 2021-08-16T12:47:26.376Z,1629118046.376 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MinAltitude 0.500000 meter 2021-08-16T12:47:26.377Z,1629118046.377 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MaxDepth 4.800000 meter 2021-08-16T12:47:26.377Z,1629118046.377 [CommandLine](IMPORTANT): got command run 2021-08-16T12:47:26.389Z,1629118046.389 [CommandLine](IMPORTANT): Running 2021-08-16T12:47:26.539Z,1629118046.539 [MissionManager](IMPORTANT): Started mission esp_sample_at_depth 2021-08-16T12:52:55.164Z,1629118375.164 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 1.275935 m , settling for 3.000000 min 2021-08-16T12:55:56.181Z,1629118556.181 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-16T12:55:57.757Z,1629118557.757 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::' 2021-08-16T12:56:07.893Z,1629118567.893 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-16T12:56:08.170Z,1629118568.170 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-16T12:56:47.066Z,1629118607.066 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-16T12:56:47.988Z,1629118607.988 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -4 2021-08-16T12:56:48.015Z,1629118608.015 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-16T12:56:48.295Z,1629118608.295 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 1.889790 m in 24.637995 degC water with 8.443640 ug/l chlorophyll fluorescence. 2021-08-16T12:56:49.083Z,1629118609.083 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-16T12:56:49.492Z,1629118609.492 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-16T12:56:51.108Z,1629118611.108 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-16T12:57:30.706Z,1629118650.706 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@12,archiveHiBiomass_bac) 2021-08-16T12:57:32.316Z,1629118652.316 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2021-08-16T12:57:32.770Z,1629118652.770 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2021-08-16T12:57:33.122Z,1629118653.122 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-16T13:00:22.399Z,1629118822.399 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2021-08-16T13:00:22.808Z,1629118822.808 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-16T13:03:14.531Z,1629118994.531 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@42.3) 2021-08-16T13:03:14.915Z,1629118994.915 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@42.3) 2021-08-16T13:03:16.940Z,1629118996.940 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-16T13:03:17.349Z,1629118997.349 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@12,archiveHiBiomass_bac) 2021-08-16T13:16:04.571Z,1629119764.571 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@12,archiveHiBiomass_bac) 2021-08-16T13:16:13.865Z,1629119773.865 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-16T13:16:14.263Z,1629119774.263 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2021-08-16T13:21:14.055Z,1629120074.055 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2021-08-16T13:21:14.462Z,1629120074.462 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (3 messages): @08:57:01.30 Selecting Cartridge 12 @09:03:11.30 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @09:03:11.50 Sampled 42.3ml 2021-08-16T13:21:15.275Z,1629120075.275 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 2.024570 m , settling for 3.000000 min 2021-08-16T13:24:16.266Z,1629120256.266 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-16T13:24:28.074Z,1629120268.074 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-16T13:24:28.204Z,1629120268.204 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-16T13:25:07.156Z,1629120307.156 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-16T13:25:07.995Z,1629120307.995 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -18 2021-08-16T13:25:07.996Z,1629120307.996 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-16T13:25:08.402Z,1629120308.402 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 1.935581 m in 24.636988 degC water with 9.196480 ug/l chlorophyll fluorescence. 2021-08-16T13:25:08.775Z,1629120308.775 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-16T13:25:09.180Z,1629120309.180 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-16T13:25:11.221Z,1629120311.221 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-16T13:25:51.596Z,1629120351.596 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@9,microSPR_phyto) 2021-08-16T13:25:53.228Z,1629120353.228 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2021-08-16T13:25:53.616Z,1629120353.616 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=13.740000 issuedCmd_=Cmd.startFiltering 2021-08-16T13:25:54.012Z,1629120354.012 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-16T13:28:46.134Z,1629120526.134 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2021-08-16T13:28:46.556Z,1629120526.556 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-16T13:32:46.524Z,1629120766.524 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@75.1) 2021-08-16T13:32:46.928Z,1629120766.928 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@75.1) 2021-08-16T13:32:48.948Z,1629120768.948 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2021-08-16T13:32:49.358Z,1629120769.358 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@9,microSPR_phyto) 2021-08-16T14:25:44.504Z,1629123944.504 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@9,microSPR_phyto) 2021-08-16T14:25:55.418Z,1629123955.418 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-16T14:25:55.842Z,1629123955.842 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2021-08-16T14:30:55.595Z,1629124255.595 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2021-08-16T14:30:55.984Z,1629124255.984 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (2 messages): @09:25:21.55 Selecting Cartridge 9 @09:32:44.00 Sampled 75.1ml 2021-08-16T14:30:56.823Z,1629124256.823 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 2.275987 m , settling for 3.000000 min 2021-08-16T14:33:57.814Z,1629124437.814 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-16T14:34:09.622Z,1629124449.622 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-16T14:34:09.751Z,1629124449.751 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-16T14:34:49.095Z,1629124489.095 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-16T14:34:49.911Z,1629124489.911 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -4 2021-08-16T14:34:49.912Z,1629124489.912 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-16T14:34:50.365Z,1629124490.365 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 1.558023 m in 24.618921 degC water with 8.382160 ug/l chlorophyll fluorescence. 2021-08-16T14:34:50.718Z,1629124490.718 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-16T14:34:51.116Z,1629124491.116 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-16T14:34:53.137Z,1629124493.137 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-16T14:35:32.735Z,1629124532.735 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@10,archiveHiBiomass_bac) 2021-08-16T14:35:34.361Z,1629124534.361 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2021-08-16T14:35:34.759Z,1629124534.759 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2021-08-16T14:35:35.178Z,1629124535.178 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-16T14:38:26.452Z,1629124706.452 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2021-08-16T14:38:26.881Z,1629124706.881 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-16T14:42:27.237Z,1629124947.237 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@75.8) 2021-08-16T14:42:27.648Z,1629124947.648 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@75.8) 2021-08-16T14:42:29.665Z,1629124949.665 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-16T14:42:30.065Z,1629124950.065 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@10,archiveHiBiomass_bac) 2021-08-16T14:55:17.300Z,1629125717.300 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@10,archiveHiBiomass_bac) 2021-08-16T14:55:26.614Z,1629125726.614 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-16T14:55:26.992Z,1629125726.992 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2021-08-16T15:00:26.771Z,1629126026.771 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2021-08-16T15:00:27.187Z,1629126027.187 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (3 messages): @10:35:03.85 Selecting Cartridge 10 @10:42:24.25 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @10:42:24.45 Sampled 75.8ml 2021-08-16T15:00:28.002Z,1629126028.002 [MissionManager](IMPORTANT): Started mission Default 2021-08-16T15:02:22.722Z,1629126142.722 [BPC1](FAULT): Battery stick #6 (s/n: 00CA) reported TERMINATE_DISCHARGE_ALARM. Status code: 0x4AD0. 2021-08-16T15:08:52.478Z,1629126532.478 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.400202 min 2021-08-16T15:09:14.451Z,1629126554.451 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T150912 2021-08-16T15:09:22.398Z,1629126562.398 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" 2v0ra 1 2.000000 2021-08-16T15:09:22.399Z,1629126562.399 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2v0ra 2021-08-16T15:09:22.400Z,1629126562.400 [CommandLine](IMPORTANT): Scheduled #5 (#1 of 2 with id='2v0ra'): "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" ASAP 2021-08-16T15:09:23.564Z,1629126563.564 [DataOverHttps](IMPORTANT): SBD MOMSN=15889301, MTMSN=20210816T150921 2021-08-16T15:09:32.242Z,1629126572.242 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" 2v0ra 2 2.000000 2021-08-16T15:09:32.243Z,1629126572.243 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2v0ra 2021-08-16T15:09:32.244Z,1629126572.244 [CommandLine](IMPORTANT): Scheduled #6 (#2 of 2 with id='2v0ra'): "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" ASAP 2021-08-16T15:09:32.464Z,1629126572.464 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T15:09:34.730Z,1629126574.730 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T15:09:53.751Z,1629126593.751 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MissionTimeout 3.500000 hour 2021-08-16T15:09:53.752Z,1629126593.752 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.TargetDepth 2.000000 meter 2021-08-16T15:09:53.753Z,1629126593.753 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ESPCartridgeType_2 -18.000000 count 2021-08-16T15:09:53.753Z,1629126593.753 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachSpeed 0.000000 meter_per_second 2021-08-16T15:10:17.279Z,1629126617.279 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MinAltitude 1.000000 meter 2021-08-16T15:10:17.280Z,1629126617.280 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MaxDepth 4.800000 meter 2021-08-16T15:10:17.280Z,1629126617.280 [CommandLine](IMPORTANT): got command set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3.000000 meter 2021-08-16T15:10:17.281Z,1629126617.281 [CommandLine](IMPORTANT): got command run 2021-08-16T15:10:17.298Z,1629126617.298 [CommandLine](IMPORTANT): Running 2021-08-16T15:10:17.509Z,1629126617.509 [MissionManager](IMPORTANT): Started mission esp_sample_at_depth 2021-08-16T15:11:43.227Z,1629126703.227 [DataOverHttps](IMPORTANT): SBD MOMSN=15889356, MTMSN=20210816T151141 2021-08-16T15:11:55.194Z,1629126715.194 [CommandLine](IMPORTANT): got command stop 2021-08-16T15:11:56.032Z,1629126716.032 [MissionManager](IMPORTANT): Started mission Default 2021-08-16T15:15:10.038Z,1629126910.038 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T151508 2021-08-16T15:15:17.937Z,1629126917.937 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-16T15:15:18.261Z,1629126918.261 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-16UTC151518_onESP 2021-08-16T15:18:07.989Z,1629127087.989 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.192406 min 2021-08-16T15:18:25.778Z,1629127105.778 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T151823 2021-08-16T15:18:35.074Z,1629127115.074 [CommandLine](IMPORTANT): got command ! check 2021-08-16T15:18:35.365Z,1629127115.365 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Mon Aug 16 15:15:53 UTC 2021 -- ssh esp@10.89.11.3 upsync Uploading real real.* *.spr* *.pcr* *.tif* CTD-*.hex .././messages in /var/log/esp to ESPhonu@ESPshore:/home/ftp/honu/esp shutting ESP off 2021-08-16T15:24:55.257Z,1629127495.257 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.980133 min 2021-08-16T15:26:19.330Z,1629127579.330 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T152617 2021-08-16T15:26:31.681Z,1629127591.681 [CommandLine](IMPORTANT): got command configSet IBIT.batteryVoltageThreshold 12.500000 volt persist 2021-08-16T15:28:15.243Z,1629127695.243 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T152813 2021-08-16T15:28:27.300Z,1629127707.300 [CommandLine](IMPORTANT): got command configSet IBIT.batteryCapacityThreshold 40.000000 ampere_hour persist 2021-08-16T15:30:29.582Z,1629127829.582 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 18.553009 min 2021-08-16T15:36:24.082Z,1629128184.082 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 24.461759 min 2021-08-16T15:36:25.963Z,1629128185.963 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T153623 2021-08-16T15:36:36.933Z,1629128196.933 [CommandLine](IMPORTANT): got command schedule clear 2021-08-16T15:36:36.934Z,1629128196.934 [CommandLine](IMPORTANT): #Cleared scheduled commands:2 2021-08-16T15:36:36.934Z,1629128196.934 [CommandLine](IMPORTANT): got command schedule resume 2021-08-16T15:36:36.934Z,1629128196.934 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-16T15:39:04.142Z,1629128344.142 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T153902 2021-08-16T15:39:12.162Z,1629128352.162 [CommandLine](IMPORTANT): got command schedule 20210816T161900 "stop" 2021-08-16T15:39:12.177Z,1629128352.177 [CommandLine](IMPORTANT): Scheduled #7: "stop" at 2021-08-16T16:19:00Z 2021-08-16T15:39:19.362Z,1629128359.362 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:20.174Z,1629128360.174 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:20.983Z,1629128360.983 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:21.786Z,1629128361.786 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:22.600Z,1629128362.600 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:23.407Z,1629128363.407 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:24.218Z,1629128364.218 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:25.018Z,1629128365.018 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:25.826Z,1629128365.826 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:26.638Z,1629128366.638 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:27.442Z,1629128367.442 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:28.251Z,1629128368.251 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:29.058Z,1629128369.058 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:29.867Z,1629128369.867 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:30.679Z,1629128370.679 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:31.488Z,1629128371.488 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:32.295Z,1629128372.295 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:33.098Z,1629128373.098 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:33.908Z,1629128373.908 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:34.714Z,1629128374.714 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:35.530Z,1629128375.530 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:36.331Z,1629128376.331 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:37.139Z,1629128377.139 [DVL_micro](FAULT): Failed to initialize within timeout. 2021-08-16T15:39:37.140Z,1629128377.140 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:37.946Z,1629128377.946 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:38.763Z,1629128378.763 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:39.562Z,1629128379.562 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:40.371Z,1629128380.371 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:41.186Z,1629128381.186 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:41.986Z,1629128381.986 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:42.799Z,1629128382.799 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:43.620Z,1629128383.620 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:44.415Z,1629128384.415 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:45.218Z,1629128385.218 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:46.027Z,1629128386.027 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:46.835Z,1629128386.835 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:47.658Z,1629128387.658 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:48.455Z,1629128388.455 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:49.262Z,1629128389.262 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:50.071Z,1629128390.071 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:50.879Z,1629128390.879 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:51.682Z,1629128391.682 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:52.490Z,1629128392.490 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:53.299Z,1629128393.299 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:54.106Z,1629128394.106 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:54.931Z,1629128394.931 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:55.746Z,1629128395.746 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:56.134Z,1629128396.134 [DVL_micro](FAULT): Failed to initialize within timeout. 2021-08-16T15:39:56.203Z,1629128396.203 [CBIT](CRITICAL): Communications Fault in component: DVL_micro 2021-08-16T15:39:56.614Z,1629128396.614 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:56.632Z,1629128396.632 [CommandLine](FAULT): Scheduling is paused for 1 commands 2021-08-16T15:39:57.346Z,1629128397.346 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:58.194Z,1629128398.194 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:39:58.970Z,1629128398.970 [Power24vConverter](FAULT): LCB fault: Software Overcurrent. 2021-08-16T15:42:10.855Z,1629128530.855 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T154208 2021-08-16T15:42:19.194Z,1629128539.194 [CommandLine](IMPORTANT): got command schedule asap "load Maintenance/ballast_and_trim.xml;set ballast_and_trim.MissionTimeout 1 h;set ballast_and_trim.Depth1 2.5 m;set ballast_and_trim.ApproachSpeed 0 m/s;set ballast_and_trim.ApproachDepthRate 0.05 m/s" 2v2a1 1 2.000000 2021-08-16T15:42:19.195Z,1629128539.195 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2v2a1 2021-08-16T15:42:19.196Z,1629128539.196 [CommandLine](IMPORTANT): Scheduled #8 (#1 of 2 with id='2v2a1'): "load Maintenance/ballast_and_trim.xml;set ballast_and_trim.MissionTimeout 1 h;set ballast_and_trim.Depth1 2.5 m;set ballast_and_trim.ApproachSpeed 0 m/s;set ballast_and_trim.ApproachDepthRate 0.05 m/s" ASAP 2021-08-16T15:42:20.156Z,1629128540.156 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T154218 2021-08-16T15:42:31.102Z,1629128551.102 [CommandLine](IMPORTANT): got command schedule asap "set ballast_and_trim.EstimationTimeout 3 h;set ballast_and_trim.MinEstimationTime 175 min;set ballast_and_trim.MinAltitude 2 m;set ballast_and_trim.MaxDepth 4 m;set ballast_and_trim.MinOffshore 1 km;run" 2v2a1 2 2.000000 2021-08-16T15:42:31.102Z,1629128551.102 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2v2a1 2021-08-16T15:42:31.103Z,1629128551.103 [CommandLine](IMPORTANT): Scheduled #9 (#2 of 2 with id='2v2a1'): "set ballast_and_trim.EstimationTimeout 3 h;set ballast_and_trim.MinEstimationTime 175 min;set ballast_and_trim.MinAltitude 2 m;set ballast_and_trim.MaxDepth 4 m;set ballast_and_trim.MinOffshore 1 km;run" ASAP 2021-08-16T15:42:31.325Z,1629128551.325 [CommandLine](IMPORTANT): got command load ./Missions/Maintenance/ballast_and_trim.xml 2021-08-16T15:42:34.918Z,1629128554.918 [CommandLine](IMPORTANT): Loaded ./Missions/Maintenance/ballast_and_trim.xml 2021-08-16T15:42:37.113Z,1629128557.113 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.678512 min 2021-08-16T15:42:52.442Z,1629128572.442 [CommandLine](IMPORTANT): got command set ballast_and_trim.MissionTimeout 1.000000 hour 2021-08-16T15:42:52.443Z,1629128572.443 [CommandLine](IMPORTANT): got command set ballast_and_trim.Depth1 2.500000 meter 2021-08-16T15:42:52.444Z,1629128572.444 [CommandLine](IMPORTANT): got command set ballast_and_trim.ApproachSpeed 0.000000 meter_per_second 2021-08-16T15:42:52.445Z,1629128572.445 [CommandLine](IMPORTANT): got command set ballast_and_trim.ApproachDepthRate 0.050000 meter_per_second 2021-08-16T15:43:11.594Z,1629128591.594 [DVL_micro](FAULT): DVL failed to acquire valid data within timeout. 2021-08-16T15:43:12.848Z,1629128592.848 [CommandLine](IMPORTANT): got command set ballast_and_trim.EstimationTimeout 3.000000 hour 2021-08-16T15:43:12.849Z,1629128592.849 [CommandLine](IMPORTANT): got command set ballast_and_trim.MinEstimationTime 175.000000 minute 2021-08-16T15:43:12.850Z,1629128592.850 [CommandLine](IMPORTANT): got command set ballast_and_trim.MinAltitude 2.000000 meter 2021-08-16T15:43:12.850Z,1629128592.850 [CommandLine](IMPORTANT): got command set ballast_and_trim.MaxDepth 4.000000 meter 2021-08-16T15:43:12.851Z,1629128592.851 [CommandLine](IMPORTANT): got command set ballast_and_trim.MinOffshore 1.000000 kilometer 2021-08-16T15:43:12.851Z,1629128592.851 [CommandLine](IMPORTANT): got command run 2021-08-16T15:43:12.859Z,1629128592.859 [CommandLine](IMPORTANT): Running 2021-08-16T15:43:13.186Z,1629128593.186 [MissionManager](IMPORTANT): Started mission ballast_and_trim 2021-08-16T15:46:07.422Z,1629128767.422 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T154605 2021-08-16T15:46:15.957Z,1629128775.957 [CommandLine](IMPORTANT): got command schedule 20210816T161900 "schedule resume" 2021-08-16T15:46:15.958Z,1629128775.958 [CommandLine](IMPORTANT): Scheduled #10: "schedule resume" at 2021-08-16T16:19:00Z 2021-08-16T15:46:15.958Z,1629128775.958 [CommandLine](FAULT): Scheduling was paused by an error at 2021-08-16T15:39:56Z 2021-08-16T15:46:52.866Z,1629128812.866 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T154650 2021-08-16T15:47:01.393Z,1629128821.393 [CommandLine](IMPORTANT): got command schedule resume 2021-08-16T15:47:01.394Z,1629128821.394 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-16T16:19:00.057Z,1629130740.057 [CommandLine](IMPORTANT): got command stop 2021-08-16T16:19:00.058Z,1629130740.058 [CommandLine](IMPORTANT): Scheduling is paused for 2 commands 2021-08-16T16:19:00.265Z,1629130740.265 [CommandLine](FAULT): Ignoring time-scheduled command: schedule resume 2021-08-16T16:19:00.542Z,1629130740.542 [MissionManager](IMPORTANT): Started mission Default 2021-08-16T16:25:51.190Z,1629131151.190 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.837186 min 2021-08-16T16:28:55.823Z,1629131335.823 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T162853 2021-08-16T16:29:03.873Z,1629131343.873 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" 2v4g4 1 2.000000 2021-08-16T16:29:03.874Z,1629131343.874 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2v4g4 2021-08-16T16:29:03.876Z,1629131343.876 [CommandLine](IMPORTANT): Scheduled #11 (#1 of 2 with id='2v4g4'): "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" ASAP 2021-08-16T16:29:04.775Z,1629131344.775 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T162902 2021-08-16T16:29:12.710Z,1629131352.710 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" 2v4g4 2 2.000000 2021-08-16T16:29:12.711Z,1629131352.711 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2v4g4 2021-08-16T16:29:12.712Z,1629131352.712 [CommandLine](IMPORTANT): Scheduled #12 (#2 of 2 with id='2v4g4'): "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" ASAP 2021-08-16T16:29:12.790Z,1629131352.790 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T16:29:14.684Z,1629131354.684 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_depth.xml 2021-08-16T16:29:28.589Z,1629131368.589 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MissionTimeout 3.500000 hour 2021-08-16T16:29:28.590Z,1629131368.590 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.TargetDepth 2.000000 meter 2021-08-16T16:29:28.591Z,1629131368.591 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ESPCartridgeType_2 -18.000000 count 2021-08-16T16:29:28.592Z,1629131368.592 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachSpeed 0.000000 meter_per_second 2021-08-16T16:29:39.110Z,1629131379.110 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MinAltitude 1.000000 meter 2021-08-16T16:29:39.111Z,1629131379.111 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MaxDepth 4.800000 meter 2021-08-16T16:29:39.112Z,1629131379.112 [CommandLine](IMPORTANT): got command set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3.000000 meter 2021-08-16T16:29:39.112Z,1629131379.112 [CommandLine](IMPORTANT): got command run 2021-08-16T16:29:39.114Z,1629131379.114 [CommandLine](IMPORTANT): Running 2021-08-16T16:29:39.200Z,1629131379.200 [MissionManager](IMPORTANT): Started mission esp_sample_at_depth 2021-08-16T16:43:27.866Z,1629132207.866 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 1.944653 m , settling for 3.000000 min 2021-08-16T16:46:28.867Z,1629132388.867 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-16T16:46:40.657Z,1629132400.657 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-16T16:46:40.752Z,1629132400.752 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-16T16:47:18.531Z,1629132438.531 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-16T16:47:19.344Z,1629132439.344 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -4 2021-08-16T16:47:19.345Z,1629132439.345 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-16T16:47:19.814Z,1629132439.814 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 1.371836 m in 24.599237 degC water with 7.318440 ug/l chlorophyll fluorescence. 2021-08-16T16:47:20.184Z,1629132440.184 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-16T16:47:20.563Z,1629132440.563 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-16T16:47:22.575Z,1629132442.575 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-16T16:48:02.168Z,1629132482.168 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@8,archiveHiBiomass_bac) 2021-08-16T16:48:03.830Z,1629132483.830 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2021-08-16T16:48:04.207Z,1629132484.207 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2021-08-16T16:48:04.588Z,1629132484.588 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-16T16:50:53.876Z,1629132653.876 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2021-08-16T16:50:54.275Z,1629132654.275 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-16T16:52:00.946Z,1629132720.946 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@16.5) 2021-08-16T16:52:01.336Z,1629132721.336 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@16.5) 2021-08-16T16:52:03.360Z,1629132723.360 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-16T16:52:03.761Z,1629132723.761 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@8,archiveHiBiomass_bac) 2021-08-16T17:04:51.014Z,1629133491.014 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@8,archiveHiBiomass_bac) 2021-08-16T17:05:00.290Z,1629133500.290 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-16T17:05:00.688Z,1629133500.688 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2021-08-16T17:10:00.463Z,1629133800.463 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2021-08-16T17:10:00.872Z,1629133800.872 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (3 messages): @12:47:33.47 Selecting Cartridge 8 @12:51:58.20 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @12:51:58.40 Sampled 16.5ml 2021-08-16T17:10:01.728Z,1629133801.728 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 1.404235 m , settling for 3.000000 min 2021-08-16T17:13:02.717Z,1629133982.717 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-16T17:13:14.509Z,1629133994.509 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-16T17:13:14.659Z,1629133994.659 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-16T17:13:52.838Z,1629134032.838 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-16T17:13:53.650Z,1629134033.650 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -18 2021-08-16T17:13:53.651Z,1629134033.651 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-16T17:13:54.077Z,1629134034.077 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.208597 m in 24.593287 degC water with 8.154800 ug/l chlorophyll fluorescence. 2021-08-16T17:13:54.443Z,1629134034.443 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-16T17:13:54.864Z,1629134034.864 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-16T17:13:56.872Z,1629134036.872 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-16T17:14:37.265Z,1629134077.265 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@7,microSPR_phyto) 2021-08-16T17:14:38.885Z,1629134078.885 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2021-08-16T17:14:39.291Z,1629134079.291 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=13.740000 issuedCmd_=Cmd.startFiltering 2021-08-16T17:14:39.693Z,1629134079.693 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-16T17:17:29.802Z,1629134249.802 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2021-08-16T17:17:30.190Z,1629134250.190 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-16T17:21:37.032Z,1629134497.032 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@75.1) 2021-08-16T17:21:37.436Z,1629134497.436 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@75.1) 2021-08-16T17:21:39.468Z,1629134499.468 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2021-08-16T17:21:39.868Z,1629134499.868 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@7,microSPR_phyto) 2021-08-16T18:14:16.061Z,1629137656.061 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@7,microSPR_phyto) 2021-08-16T18:14:27.354Z,1629137667.354 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-16T18:14:27.760Z,1629137667.760 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2021-08-16T18:19:27.536Z,1629137967.536 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2021-08-16T18:19:27.947Z,1629137967.947 [ESPComponent](IMPORTANT): [sample #5] ESP log summary report (2 messages): @13:14:07.82 Selecting Cartridge 7 @13:21:35.31 Sampled 75.1ml 2021-08-16T18:19:29.203Z,1629137969.203 [MissionManager](IMPORTANT): Started mission Default 2021-08-16T18:19:29.437Z,1629137969.437 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-16T18:26:09.047Z,1629138369.047 [DataOverHttps](IMPORTANT): SBD MTMSN=20210816T182606 2021-08-16T18:26:16.989Z,1629138376.989 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-16T18:26:17.607Z,1629138377.607 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-16UTC182617_onESP 2021-08-16T18:27:54.335Z,1629138474.335 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.411624 min 2021-08-16T18:34:13.591Z,1629138853.591 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.732891 min