The RH hot cell lab active sump high level sensor was tested as part of the yearly scheduled inspection.  High water level was simulated by tilting the sensor sideways with a long pole sample scoop.  The Main Control Room and Hua Yang from Nuclear Medicine were notified before the test.  The alarm sounded as expected, and a message was received in the main control room.  This is a latching signal, so a Control Room operator will go to the MHESA RCR lab to reset the alarm.  The water level was approximately 12" below the high level sensor, so the sump may need to be sampled and emptied soon.

The Remote Handling Hot Cell Lab Active Sump has been released to city sewer following the procedure in Document-64834.  The water was close to the sump high limit switch (approximately 5800L).  Before release the water was sampled, tested, and approved for release by RPG.  The tritium analysis form has been filled out and returned to RPG,.

It took approximately 5 hours to empty the sump with all valves fully open and the water flowing through the 10" Cuno filter.

 A call was received from the MCR at approximately 21:30 on Monday June 27th to report that active drains in the east end of the Meson Hall were backing up and overflowing.  I came in and found the leak coming from a broken spray nozzle in a sink in the rad waste area.  The leak was stopped at which point there were puddles between 1-2m diameter around the active sump top flange, the drain directly south of the warm cell water windows, and the drain in the counting room south of the clean room.  There was lots of water on the floor in the rad waste area as well.  No water was found in the TR-13 area.  A swipe was taken from the puddle around the active sump top flange, which was not above background level.  Robin Swanson (ops) contacted Joe Mildenberger who gave permission for the sump to be pumped out after a water sample was taken.  Robin took a water sample and left it for analysis by RPG, the sump pump was started and ran overnight.  At approximately 8am the next morning the sump was almost empty and the pump was stopped.  Floor swipes were taken where there had been puddles which were now dry - no contamination found.  The areas in the hot cell lab which had been wet were mopped.

 The high level sensor for the Meson Hall Hot Cell Lab Active Sump was tested today and worked as expected.  When the sensor is tilted (as occurs when the water level rises above it) an audible alarm sounds in the hot cell lab, and the Safety Panel in the MCR also alarms with "MHESA B1 RCR1 RM 6 WARN" displaying NOT OK.  This alarm sounds every 3 minutes in the MCR until the PLC outside of the RCR lab is reset.  When the sensor was tested on the RCR lab PLC there was a message indicating that the MH Hot Cell Lab Active Sump is full.

It is still unclear why the operators did not receive this alarm when the sump filled and overflowed last night (or possibly they did receive it, but it was ignored or silenced).  The operators from that shift are currently off until this Saturday, so they will be contacted next week to try to learn more.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

Further tests were done on the active sump high level float switch today.  The switch was triggered 1 time by filling the sump, and approximately 8 times by tilting the sensor manually.  A local alarm sounded in the hot cell lab each time, but only one time in the RCR Lab and MCR (this occurred on the final trial).  Fault report #9099 has been submitted to Controls Group.  Although the existing float sensor did cause a local alarm when the sump was filled, it's operation did not seem ideal:  it only triggered when completely horizontal, which wasn't until water was several inches above the base of the switch;  in addition it appeared that it may be possible for the switch to become stuck under the horizontal portion of the sump (possibly even in an untriggered orientation).  For these reasons a replacement level switch will also be investigated.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

To address the deficiencies identified in E-Logs #204 and #205 the old active sump high level switch was replaced with Omega LV164SS model (316SS stem and float, 4' length, standard mount) using adaptor flange TRH1316 to mount to the top flange of the sump.  A new electrical box was also installed (circuit diagram attached in PDF format).  The circuit was designed by Don Jackson and wired by Travis Cave.  Connection to the RCR Lab PLC was done by Dave Morris.

The switch was tested using the following procedure:

1.  The switch was triggered by submerging the base in water until the float lifted.  This was confirmed to be equivalent to disconnecting the CPC-4 connector for the switch from the new electrical box
2.  Switch connector disconnected:  Local alarm sounds in HC Lab, red "Sump High" LED light illuminates, green "Sump Good" LED turns off, RCR PLC reports high level (cannot be reset), main control room "MHESA B1 RCR1 RM 6 WARN" displays NOT OK  (this will sound every 3 minutes until the RCR PLC high level alarm is reset)
3. "Alarm Silence" button pressed on new electrical box:  Local alarm stops, LEDs on electrical box do not change, RCR PLC still cannot be reset
4. Switch connector reconnected:  "Sump High" LED turns off, "Sump Good" LED turns on, RCR PLC can be reset
5. Power to the electrical box removed by switching off the breaker:  All LEDs turn off, no alarm, RCR PLC reports high level

Pump out of water from the MHESA Lab active sump to the HC Lab active sump was tested while observing the HC Lab sump.  The water entering the HC lab sump did not splash on the new switch excessively, and did not cause any false alarms.

The new system will continue to be tested annually according to the calibration schedule.

 On March 13th in the morning the Meson Hall Hot Cell Lab active sump reached the high level sensor.  The system responded as expected:  high level panel warning light illuminated, siren sounded, and MCR notified via "B1 RCR1 RM6 WARN".  A 1L sample was delivered to RPG for analysis.  After approval, the sump was pumped out over the following two days.  The procedure in Document #64834 was followed for sampling and pump-out.

 The RH Lab active sump level reached the high limit yesterday due to an overflow of the ultrasonic cleaner in the Rad Waste Area.  The sump was sampled by RPG, and after analysis the water was approved for release.  The water was pumped out today.  Local and MCR alarms functioned as expected.

 The RH Lab Active Sump reached the high level sensor at some point overnight before the April 8th work day.  Maico Dalla Valle noticed the alarm in the morning and notified RPG and myself.  I also received a notification by e-mail from Vicky Hannemayer who had been contacted by Ops because of the RCR alarm.

After sampling and testing by RPG the sump contents were released under work permit C2021-04-08-5 as per the procedure in Document-64834.  The approximate volume of the full sump is 7270L.

On Aug. 18, 2022, Operations informed RH group that an alarm activated for the full status of the MHESA B1 RCR1 Lab holding tank. Additionally another alarm was reported in the RH control room regarding the RH sump full (Southwest Meson Hall Extension - Remote Handling area). However, there is an alarm signal for the RH sump that was seen as OK at the MHESA Lab HMI.

Upon examining the RH active sump it was clear that the water level was in fact quite high, and the red warning light on the local control box was on.

After sampling and testing by RPG the sump contents were released under work permit C2022-08-19-3 as per the procedure in Document-64834. The approximate volume of the full sump is 7270L.

When the water level was below the warning threshold (i.e. red warning light came off, and green light indicating "sump good" came on), a request was made to the control room to reset the alarm. Operations cleared the alarm at the RCR Lab controls and retired the signal defeat that was in place in the DCR (“RCRL warn” signal was defeated).

Edit (2022-08-23): BL1A holding tank water level was checked - it is empty.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

The flow rate sensor used to monitor the drain rate from BL1A holding tank to active sump, located in the active sump area, was repaired. The paddle wheel, o-ring, and shaft were replaced. This part was significantly worn prior to replacement.

It is recommended to change the filter adjacent to it as it is suspected that this filter is partially clogged, resulting in low flow rate when draining the tank.

Furthermore, it is also recommended to change solenoid valve SV1 as it is suspected that it does not work.

The CUNO filters were changed 2023-01-17 for:

• BL1A Holding Tank drain output, adjacent to the pump
• RH Active Sump, adjacent to the V2 valve

Both of these filters are in-line with the drainage lines from the BL1A holding tank to the active sump. It was suspected that they have not been changed in many years, if ever. Prior to changing the filters, the maximum flow rate measured from the Proteus paddle wheel sensor during draining was around 0.85 gpm (the draining procedure document indicates it should be between 0.9 to 2.5 gpm). After replacing both filters, which each showed significant clogging of sediment, the flow rate significantly increased and the maximum of 2.5 gpm was achievable. Note that when the drain pump was initially turned on, it took some time to reach the desired flow rate due to air in the water lines - the flow rate was quite low until the lines were once again filled with water.

It is suggested that these filters should be replaced every ~3-5 years, depending on usage, or otherwise as-needed (it is possible to see through the clear walls and determine how dirty they are). This MRO item will be added to the checklist of routine T1/T2 related tasks performed to prevent flow rate issues in the future during draining.

Filters were bagged and discarded as active waste.

Attached photos show old filter in bucket, and newly replaced filter, for the RH Active Sump.

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

The level sensor for the active sump in the RH lab was tested.
The level sensor is working properly and is operational.
 

During a replacement of the RCR1 active sump (mechanical services group), approximately 600L of non-active grey water was released into the RH active sump. At this time, this should be all of the water in the RH sump (approximately 600L total).

The active sump high level sensor was tested and correctly alarmed locally. However the alarm in the MHESA RCR lab did not make any sound and no notification was recieved at the main control room. After some investigation, it is believed that the RCR lab alarm 'had already produced a notification at the control room and so no new notification was produced by testing the remote handling sump. Alternatively, the RCR sump alarm may have been silenced in the control room and so the notification was not seen. 

We will look to modify the system so that our sump's alarm does not piggy-back off the RCR sump alarm when notifying the control room.

blah blah blah....