2019-07-11T18:29:19.596Z,1562869759.596 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg
2019-07-11T18:29:19.683Z,1562869759.683 [ESPComponent](IMPORTANT): ESPCOMPONENT_MANUAL_ESP INDICATED.
2019-07-11T18:29:19.905Z,1562869759.905 [StratificationFrontDetector](IMPORTANT): threshold set to: 0.999994 degC
2019-07-11T18:29:20.009Z,1562869760.009 [CommandLine](IMPORTANT): got command load ./Missions/Maintenance/sample.xml
2019-07-11T18:29:20.014Z,1562869760.014 [CommandLine](IMPORTANT): Loaded ./Missions/Maintenance/sample.xml
2019-07-11T18:29:20.027Z,1562869760.027 [SBIT](IMPORTANT): git: 2019-07-01-A
2019-07-11T18:29:20.108Z,1562869760.108 [CommandLine](IMPORTANT): got command set sample.Depth 1.000000 centimeter
2019-07-11T18:29:20.108Z,1562869760.108 [CommandLine](IMPORTANT): got command set sample.WaitBeforeSample 3.000000 second
2019-07-11T18:29:20.108Z,1562869760.108 [CommandLine](IMPORTANT): got command set sample:SampleAtDepth.TargetDepth 1.000000 centimeter
2019-07-11T18:29:20.108Z,1562869760.108 [CommandLine](IMPORTANT): got command run
2019-07-11T18:29:20.108Z,1562869760.108 [CommandLine](IMPORTANT): Running 
2019-07-11T18:29:40.474Z,1562869780.474 [SBIT](IMPORTANT): Beginning Startup BIT
2019-07-11T18:29:56.728Z,1562869796.728 [CommandLine](IMPORTANT): got command load ./Missions/Maintenance/sample.xml
2019-07-11T18:29:56.729Z,1562869796.729 [CommandLine](IMPORTANT): Loaded ./Missions/Maintenance/sample.xml
2019-07-11T18:29:58.287Z,1562869798.287 [CommandLine](IMPORTANT): got command set sample.Depth 1.000000 centimeter
2019-07-11T18:29:58.287Z,1562869798.287 [CommandLine](IMPORTANT): got command set sample.WaitBeforeSample 3.000000 second
2019-07-11T18:29:58.287Z,1562869798.287 [CommandLine](IMPORTANT): got command set sample:SampleAtDepth.TargetDepth 1.000000 centimeter
2019-07-11T18:29:58.287Z,1562869798.287 [CommandLine](IMPORTANT): got command run
2019-07-11T18:29:58.287Z,1562869798.287 [CommandLine](IMPORTANT): Running 
2019-07-11T18:30:34.082Z,1562869834.082 [SBIT](IMPORTANT): SBIT PASSED
2019-07-11T18:30:34.170Z,1562869834.170 [CommandLine](IMPORTANT): got command configSet list
2019-07-11T18:30:34.170Z,1562869834.170 [CommandLine](IMPORTANT): Listing configuration overrides from Data/persisted.cfg
2019-07-11T18:30:34.170Z,1562869834.170 [CommandLine](IMPORTANT): DataOverHttps.loadAtStartup=1 bool;

2019-07-11T18:30:34.472Z,1562869834.472 [MissionManager](IMPORTANT): Started mission Startup
2019-07-11T19:26:35.805Z,1562873195.805 [ESPComponent](CRITICAL): [sample #4] ESP sampling sequence terminated with error: Unexpected ESP state ESP_STATE_PROCESS_FAILED. Expecting one of: ESP_STATE_PROCESSED, ESP_STATE_PROCESSING
2019-07-11T18:30:50.075Z,1562869850.075 [MissionManager](IMPORTANT): Started mission sample
2019-07-11T18:33:01.295Z,1562869981.295 [sample:SampleAtDepth:SampleWrapper:B](IMPORTANT): At 0.011936 m , settling for 0.050000 min 
2019-07-11T18:33:06.896Z,1562869986.896 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Fail|Retry|Cmd::|Sampler::'
2019-07-11T18:33:07.296Z,1562869987.296 [ESPComponent](IMPORTANT): powerUpESP IGNORED because manualEsp_
2019-07-11T18:33:07.296Z,1562869987.296 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=30.0000)
2019-07-11T18:33:09.296Z,1562869989.296 [ESPComponent](IMPORTANT): ESP has connected as client
2019-07-11T18:33:10.096Z,1562869990.096 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -1
2019-07-11T18:33:10.096Z,1562869990.096 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2019-07-11T18:33:10.497Z,1562869990.497 [sample:SampleAtDepth:SampleWrapper:SampleESP:DoESPSample:B](IMPORTANT): ESP sampling at 0.101235 m in nan degC water with nan ug/l chlorophyll fluorescence.
2019-07-11T18:33:11.296Z,1562869991.296 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG
2019-07-11T18:33:11.696Z,1562869991.696 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS
2019-07-11T18:33:13.297Z,1562869993.297 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE
2019-07-11T18:33:48.902Z,1562870028.902 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac)
2019-07-11T18:33:50.503Z,1562870030.503 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@5.0)
2019-07-11T18:33:51.302Z,1562870031.302 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0)
2019-07-11T18:34:21.708Z,1562870061.708 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0)
2019-07-11T18:34:22.108Z,1562870062.108 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2019-07-11T18:34:37.710Z,1562870077.710 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@10.0)
2019-07-11T18:34:38.110Z,1562870078.110 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@10.0)
2019-07-11T18:34:40.511Z,1562870080.511 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac)
2019-07-11T18:36:16.126Z,1562870176.126 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@51,aRNAl_bac)
2019-07-11T18:36:28.527Z,1562870188.527 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally.
2019-07-11T18:36:28.928Z,1562870188.928 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (2 messages):
@11:33:16.06 Selecting Cartridge 51
@11:34:36.34 Sampled  10.0ml

2019-07-11T18:36:28.928Z,1562870188.928 [ESPComponent](IMPORTANT): powerDownESP IGNORED because manualEsp_
2019-07-11T18:36:31.328Z,1562870191.328 [ESPComponent](IMPORTANT): powerUpESP IGNORED because manualEsp_
2019-07-11T18:36:31.328Z,1562870191.328 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=30.0000)
2019-07-11T18:36:42.930Z,1562870202.930 [ESPComponent](IMPORTANT): ESP has connected as client
2019-07-11T18:36:43.730Z,1562870203.730 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -16
2019-07-11T18:36:43.730Z,1562870203.730 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2019-07-11T18:36:44.131Z,1562870204.131 [sample:SampleAtDepth:SampleWrapper:SampleESP:DoESPSample:B](IMPORTANT): ESP sampling at 0.273031 m in nan degC water with nan ug/l chlorophyll fluorescence.
2019-07-11T18:36:44.530Z,1562870204.530 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG
2019-07-11T18:36:44.930Z,1562870204.930 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS
2019-07-11T18:36:46.931Z,1562870206.931 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE
2019-07-11T18:37:22.936Z,1562870242.936 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@60,microSPR_phyto)
2019-07-11T18:37:24.537Z,1562870244.537 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@5.0)
2019-07-11T18:37:25.337Z,1562870245.337 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0)
2019-07-11T18:37:55.741Z,1562870275.741 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0)
2019-07-11T18:37:56.141Z,1562870276.141 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2019-07-11T18:38:11.743Z,1562870291.743 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@10.0)
2019-07-11T18:38:12.144Z,1562870292.144 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@10.0)
2019-07-11T18:38:14.545Z,1562870294.545 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@60,microSPR_phyto)
2019-07-11T19:15:02.495Z,1562872502.495 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@60,microSPR_phyto)
2019-07-11T19:15:33.299Z,1562872533.299 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally.
2019-07-11T19:15:33.699Z,1562872533.699 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (2 messages):
@11:36:49.68 Selecting Cartridge 60
@11:38:10.38 Sampled  10.0ml

2019-07-11T19:15:33.699Z,1562872533.699 [ESPComponent](IMPORTANT): powerDownESP IGNORED because manualEsp_
2019-07-11T19:15:36.099Z,1562872536.099 [ESPComponent](IMPORTANT): powerUpESP IGNORED because manualEsp_
2019-07-11T19:15:36.099Z,1562872536.099 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=30.0000)
2019-07-11T19:15:39.300Z,1562872539.300 [ESPComponent](IMPORTANT): ESP has connected as client
2019-07-11T19:15:40.100Z,1562872540.100 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1
2019-07-11T19:15:40.100Z,1562872540.100 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2019-07-11T19:15:40.501Z,1562872540.501 [sample:SampleAtDepth:SampleWrapper:SampleESP:DoESPSample:B](IMPORTANT): ESP sampling at 0.002872 m in nan degC water with nan ug/l chlorophyll fluorescence.
2019-07-11T19:15:40.900Z,1562872540.900 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2019-07-11T19:15:41.301Z,1562872541.301 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2019-07-11T19:15:43.301Z,1562872543.301 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2019-07-11T19:16:18.906Z,1562872578.906 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@49,aRNAl_bac)
2019-07-11T19:16:20.507Z,1562872580.507 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@5.0)
2019-07-11T19:16:21.307Z,1562872581.307 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0)
2019-07-11T19:16:51.712Z,1562872611.712 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2019-07-11T19:16:52.112Z,1562872612.112 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2019-07-11T19:17:07.714Z,1562872627.714 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@10.0)
2019-07-11T19:17:08.115Z,1562872628.115 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@10.0)
2019-07-11T19:17:10.515Z,1562872630.515 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,aRNAl_bac)
2019-07-11T19:18:46.129Z,1562872726.129 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@49,aRNAl_bac)
2019-07-11T19:18:58.531Z,1562872738.531 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2019-07-11T19:18:58.932Z,1562872738.932 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (2 messages):
@12:15:46.05 Selecting Cartridge 49
@12:17:06.34 Sampled  10.0ml

2019-07-11T19:18:58.932Z,1562872738.932 [ESPComponent](IMPORTANT): powerDownESP IGNORED because manualEsp_
2019-07-11T19:19:01.332Z,1562872741.332 [ESPComponent](IMPORTANT): powerUpESP IGNORED because manualEsp_
2019-07-11T19:19:01.332Z,1562872741.332 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=30.0000)
2019-07-11T19:19:08.933Z,1562872748.933 [ESPComponent](IMPORTANT): ESP has connected as client
2019-07-11T19:19:09.733Z,1562872749.733 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -16
2019-07-11T19:19:09.733Z,1562872749.733 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2019-07-11T19:19:10.134Z,1562872750.134 [sample:SampleAtDepth:SampleWrapper:SampleESP:DoESPSample:B](IMPORTANT): ESP sampling at 0.017771 m in nan degC water with nan ug/l chlorophyll fluorescence.
2019-07-11T19:19:10.534Z,1562872750.534 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2019-07-11T19:19:10.934Z,1562872750.934 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2019-07-11T19:19:12.934Z,1562872752.934 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2019-07-11T19:19:48.939Z,1562872788.939 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@58,microSPR_phyto)
2019-07-11T19:19:50.540Z,1562872790.540 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@5.0)
2019-07-11T19:19:51.340Z,1562872791.340 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@5.0)
2019-07-11T19:20:21.744Z,1562872821.744 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2019-07-11T19:20:22.144Z,1562872822.144 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2019-07-11T19:20:37.748Z,1562872837.748 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@10.0)
2019-07-11T19:20:38.147Z,1562872838.147 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@10.0)
2019-07-11T19:20:40.548Z,1562872840.548 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@58,microSPR_phyto)
2019-07-11T19:26:33.804Z,1562873193.804 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@58,microSPR_phyto)
2019-07-11T19:26:35.816Z,1562873195.816 [CommandLine](FAULT): Scheduling is paused
2019-07-11T19:26:36.205Z,1562873196.205 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (3 messages):
@12:19:15.69 Selecting Cartridge 58
@12:20:36.37 Sampled  10.0ml
ThreadError in PROCESSING -- killed thread

2019-07-11T19:26:36.205Z,1562873196.205 [ESPComponent](IMPORTANT): powerDownESP IGNORED because manualEsp_
2019-07-11T19:26:36.605Z,1562873196.605 [MissionManager](IMPORTANT): Started mission Default
2019-07-11T19:26:37.805Z,1562873197.805 [ESPComponent](IMPORTANT): [sample #4] No ESP summary messages.
2019-07-11T19:26:37.805Z,1562873197.805 [ESPComponent](IMPORTANT): powerDownESP IGNORED because manualEsp_
2019-07-11T19:26:56.751Z,1562873216.751 [CommandLine](IMPORTANT): got command exit
2019-07-11T19:26:58.608Z,1562873218.608 [ESPComponent](FAULT): Failed to power down
2019-07-11T19:26:58.608Z,1562873218.608 [ESPComponent](FAULT): Failed to power down secondary