Hi Roberto,

> On Nov 3, 2016, at 2:13 PM, Roberto Gwiazda <rgwiazda@mbari.org> wrote:
> 
> Wet labs and O2 data after 2/18 (1st deployment) are not published in SSDS. Were they not collected?
> Roberto

Actually SIAM acquired and logged sparse data samples (one sample every 8-9 minutes) for Aanderra and WETLabs between 2/18 and 2/29. After 2/29, no more Aanderra and WETLabs samples were logged. 

A detailed explanation follows. First, note that SIAM triggers a WETLabs sample when it receives a WH-1200 sample, and triggers an Aanderra sample when it receives a WETLabs sample. We used this scheme to reduce sample "jitter".  The drawback of this setup is that if SIAM doesn't receive WH-1200 samples for any reason, then Aanderra  and WETLabs samples are not acquired (beyond initial Aanderra and WETLabs samples acquired when SIAM starts, i.e. after a BIN reboot).

I've examined the SIAM logs and there are two reasons for the data loss:

1. The BIN began rebooting every 8-9 minutes on 2/18, and the reboot occurred each time when SIAM attempted to start the WH-600 driver. This behavior lasted until 2/29 when I reconfigured SIAM via Wave Glider acoustic link. Thus SIAM never got to the WH-1200 driver (supposed to start after WH-600), and so never triggered Aanderaa and WETLabs sampling. This explains why we see sparse Aanderra and WETLabs samples during this period; SIAM acquires a single sample on driver startup for those instruments, then waits for a WH-1200 sample, which never arrives because the WH-1200 driver never gets started.

2. On 2/29 we visited the BIN with the Wave Glider, and learned that the continuous BIN reboots that started on 2/18 were caused by startup of the WH-600 driver. I therefore attempted to modify the SIAM configuration via the acoustic link, to remove the problematic WH-600 driver. However in doing so I also removed the WH-1200 driver (unbeknownst to me). So the WH-1200 SIAM driver never started after 2/29, and no Aanderra or WETLabs samples ever triggered. Note that for unknown reasons the BIN's acoustic modem stopped responding on 2/29, so we never were able to check status after the final reboot and so never recognized the SIAM configuration error.

The positive news from this experience is that we've determined the reason for the BIN reboots which begain on 2/18 (and during BIN-2 deployment in August and September). The reboots begin when a SIAM log file reaches 2 GB size (maximum for a 32-bit system) - reaching this size limit causes SIAM to crash and results in a watchdog timeout after two minutes, which leads to BIN reboot. Our solution is to rename SIAM log files before they reach the 2 GB liimit, which will cause SIAM to open a new log file - this will prevent the reboot.

Regards,
Tom