On Friday August 8th Ron Kuramoto observed that the Meson Hall air amplifier cycle speed had increased.  Cycle time measurements were taken on Monday August 11th.  The lower amplifier was found to be cycling 17 times per minute, and the upper air amplifier 16 times per minute (approximately 4s cycle period).  The cycle period for both amplifiers was measured in June 2012 and found to be approximately 25s.  Note that a cycle is defined as the time between air exhaust events (this may actually only be 1/2 of a full cycle of the amplifier mechanism).  Both amplifiers were delivering air pressure approximately 134psi during the test.

The upper air amplifier was rebuilt in 2012 and has not been used since.  This test shows that the amplifiers are performing normally and that the increased cycle speed is due to a leak elsewhere in the system.  The location of the air leak will be investigated further on maintenance days or during the September mini shutdown.  The exhaust pipes of the amplifier assembly were warm to the touch due to the increased cycle speed, but not excessively so, and the current cycle speed is not thought to be of immediate concern.

The lower amplifier was left running and the exhaust pressure was turned down to 125psi to slightly reduce the cycle speed (it is not thought to be necessary to have air pressure above 120psi)

 The following work on the T2-MK1 target was completed sporadically over the last few months:

Sept 22:
- T2-MK1 target moved from storage pit to hot cell (new camera view to check flask doors and latch engagement works great)
- Rubber pad from storage pit standoffs was stuck under flange - removed and cleaned base of flange
- Preliminary leak check: target positions and  delta seals no response, left bellows (facing ladder) had tiny response up to 3.0E-8 with recovery in ~5s, right bellows no response
- Removed plugs at Position #1 and #3

Oct 14:
- Installed 12mm Be target #106 at Position 1 (confirmed with Yuri that it may be useful to have a short Be target on the T2 ladder)
- Installed 5cm target #301 at Position 3
- Leak checked target positions: no response, only mild delayed drift
- Position 4 plugs removed

Nov 10:
- Installed 5cm target #302 at Position 4
- Repeated leak check: no response at all target positions
- Took vertical and horizontal measurements for all new targets
- Position limit switches and potentiometer confirmed working ok

Nov 24:
- Moved to Storage Pit Hole #4
 

During a replacement of the RCR1 active sump (mechanical services group), approximately 600L of non-active grey water was released into the RH active sump. At this time, this should be all of the water in the RH sump (approximately 600L total).

Further tests were done on the active sump high level float switch today.  The switch was triggered 1 time by filling the sump, and approximately 8 times by tilting the sensor manually.  A local alarm sounded in the hot cell lab each time, but only one time in the RCR Lab and MCR (this occurred on the final trial).  Fault report #9099 has been submitted to Controls Group.  Although the existing float sensor did cause a local alarm when the sump was filled, it's operation did not seem ideal:  it only triggered when completely horizontal, which wasn't until water was several inches above the base of the switch;  in addition it appeared that it may be possible for the switch to become stuck under the horizontal portion of the sump (possibly even in an untriggered orientation).  For these reasons a replacement level switch will also be investigated.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

The active sump high level sensor was tested and correctly alarmed locally. However the alarm in the MHESA RCR lab did not make any sound and no notification was recieved at the main control room. After some investigation, it is believed that the RCR lab alarm 'had already produced a notification at the control room and so no new notification was produced by testing the remote handling sump. Alternatively, the RCR sump alarm may have been silenced in the control room and so the notification was not seen. 

We will look to modify the system so that our sump's alarm does not piggy-back off the RCR sump alarm when notifying the control room.

On Aug. 18, 2022, Operations informed RH group that an alarm activated for the full status of the MHESA B1 RCR1 Lab holding tank. Additionally another alarm was reported in the RH control room regarding the RH sump full (Southwest Meson Hall Extension - Remote Handling area). However, there is an alarm signal for the RH sump that was seen as OK at the MHESA Lab HMI.

Upon examining the RH active sump it was clear that the water level was in fact quite high, and the red warning light on the local control box was on.

After sampling and testing by RPG the sump contents were released under work permit C2022-08-19-3 as per the procedure in Document-64834. The approximate volume of the full sump is 7270L.

When the water level was below the warning threshold (i.e. red warning light came off, and green light indicating "sump good" came on), a request was made to the control room to reset the alarm. Operations cleared the alarm at the RCR Lab controls and retired the signal defeat that was in place in the DCR (“RCRL warn” signal was defeated).

Edit (2022-08-23): BL1A holding tank water level was checked - it is empty.

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

The active item discovered on January 9, 2015 outside the BL1A bootbox has been moved to the Meson Hall east hot cell for storage.  The item was analyzed by RPG and the radiation was found to be coming from activated metal contained by dried paint in the bottom of a plastic container.  The solidified paint puck was removed from the container which is not active.  The item is approximately 20microSv/hr at 0.5m, and 1.3mSv/hr on contact.

The following items were moved today from the Warm Cell to the Rad Waste Pit for long term storage:

1. A 55 gallon drum containing various active components from the East Hot Cell.  The drum was labelled on the body and lid with the identification number RWC-55-01-2019.

2. The old M9 Q1Q2 service stand, which was removed from the beamline in the 2019 winter shutdown

Photos taken during the move are attached, as well as a copy of the loading record form for the drum.  The form has also been given to RPG.

A second drum (RWC-55-02-2019) remains in the Warm Cell, and will later be moved to the cyclotron vault tunnel for storage.  It was resurveyed after the first drum was removed and deemed too low-level for storage in the pit (300uSv/hr at 0.5m)

Summary: Clarification of T2 water temperature readouts

Detail: This e-log entry by Doug Preddy is being made to clarify the T2 water package temperature read outs. The temperature read outs on the T2 water package have been switched. The read out for inlet temperature is actually the outlet temperature and the read out for outlet temperature is actually the inlet temperature. On Sunday Oct 19 2014 at 19:45, the T2 water outlet Temp (T2) started to climb giving warning messages at 20:05. Operators tried to decrease the BL1A current, but the temp. continued to climb and eventually tripped the beam off. Operators defined BL1A OFF, and continued to inject to BL2A only. They tried to contact I. Earle, but were unable to. They left message with D. Preddy. The T2 temp. stayed at 45 deg., operators tried to reset the water package and start it, but it tripped almost right away, and increased the temp. by 0.5 deg. Beam was left off. On Monday Oct. 20 Ron Kuramoto found a replacement thermister. It was wired in to the BOP over the south catwalk. The T2 outlet temperature immediately dropped to ambiant. From this test it was concluded that the installed thermistor on the T2 outlet water pipe had failed. The wiring was restored to the original configuration. It was decided to switch T2 Water Inlet Temp and Outlet Temp sensors at the T2 electronics rack to provide a trip on the Inlet Water temperature instead of the outlet Water Temperature. As the T2 water package runs in a closed loop, the outlet temperature signal has been swapped with the inlet temperature by Doug P. and Tony T. This will provide thermal protection (trip at 45C) in addition to the T2 Target high limit bit (trip at 60C). The water flow to this circuit is also monitored and will trip if the flow drops. This action was approved in a meeting with Doug Preddy, Ron Kuramoto, Grant Minor, Yuri Bylinski, Lia Merminga, and Bob Laxdal present on Monday Oct 20th. There is a second, redundant temperature sensor on the outlet of the target water which was installed recently as a safety upgrade to protect the heat exchanger and meet CSA pressure vessel code, see dwg TBP0902. There is a controller for this sensor with a hard-trip switch for the cyclotron, and a temperature read-back display on the Meson Hall mezzanine racks. This read-back should be monitored by operations once per shift while beam is on target to ensure that outlet water temperature is stable and at an expected value. Sensor cabling swapped at Meson Hall, south mezzanine, bay 1, B.O.P. #3. Sensor T1, cable #12102 moved to TB1, terminals 3 & 4. Sensor T2, cable #12105 moved to TB1, terminals 1 & 2. Defeat #7131 removed. Defeat# 7133 Fault report #7722   Will look into replacing the faulty Thermister in the 2015 major shutdown.

On 14Oct2014, the 1AT2 Collimator B water outlet temperature rose to 100^o C. 

On Oct 7^th, during the Mini Shutdown, the T2 cooling package was uncovered to investigate the Cu-ALCW water leak dripping into the 1A Tunnel.

The Target shield block was lifted and found that top of the monolith was dry.  Uncovering the east side of the cooling package directly above the vertical services on the south wall revealed a pin hole leak at a 1/2” 45^o copper elbow on the return line circuit of the T2 cooling package.

The cooling package was turned off followed by valving off the Cu-ALCW service lines leading up to the cooling package to let the Tunnel floor dry over night.

On Oct 8^th, the Tunnel floor was dry and Cu-ALCW service lines feeding the cooling package were drained from the Tunnel by placing a 5 gallon pail under the drain lines filling it about 1/3 full until it stopped.

Mel Good was brought in to replace the faulty elbow.  The Cu-ALCW was then turned back on with no obvious leaks.  At 16:00 approximately 1.5 hours after the repair, I went into the 1A tunnel to inspect for water leaks at the isolation valves and saw water dripping down the vertical piping on the south wall again.   

Another leak was found on the 90^o elbow just ahead of the replaced 45^o elbow. There appeared to 2-3 fine mists spraying out towards the adjacent copper elbow (look at the end of the cable tie on the following photos).

The Cu-ALCW lines leading up to cooling package were again valved off to let the Tunnel floor dry up over night.

On Oct 9^th, the Cu-ALCW lines leading up to the cooling package were again drained followed by replacing the faulty 90^o elbow. The elbow was fitted with new pieces to improve the configuration. We noticed that old copper pieces were worn thin. These damaged parts measured 20,000 c/m on the 44-2 and 40 c/m on the RM-14 monitors and were taken to the RH H/cell lab for inspection.

The T2 cooling package turned on ok about 1.5 hours after repairing the 2^nd water leak.

Mel Good received 0.03mSv over the two day repair. 

I received 0.10 mSv over three days, picking up 0.05mSv on Oct 7^th in the Tunnel where the fields were 200-300uSv/Hr by the isolation valves.  The next the day levels dropped to 40uSv/Hr.  There was no contamination found on the Tunnel floor, however 80,000 c/m on the RM14  and 50,000 c/m on the 44-2 monitors were found on the swipes taken at the bottom edge of the concrete shield blocks.

NOTE: the Cu-ALCW service lines leading up to the T2 cooling package feeds the following 4 devices; Collimator A, Collimator B, the T2 monolith cooling and the Target water heat exchanger.

On Sep. 3^rd during the 6 hr maintenance day, Doug Preddy & I went into the 1A tunnel under “controlled access” to investigate a water leak that triggered the Water Detection Sensors.  It was also noticed that the Cu-ALCW tank level was losing water.

Water was found to be leaking down the south wall chase where the T2 water package goes up the wall. Drip rate was ~1 drip every 15 seconds.

In the late afternoon of Sep. 9^th;

Day one of the 1.5 day maintenance, the tunnel was surveyed by Fiona H. see RPG for field map & floor swipe results. 

Doug, John McKinnon & I set up a plastic sheet with a funnel connected to a 8ft. lg. poly-flow tubing to direct leaking water from the T2 into the south gutter.  This set up was left over night for the floor to dry up and resume the water leak investigation. See photos.

On Sep. 10^th;

@ 08:30 about 70% of the 1A tunnel floor had dried up and the diverted water into the south side gutter had travelled to the T1 area.

The T2 cooling package was then turned off followed by valving off the Cu-ALCW feeding the T2 cooling package. A few more strips of red tuck tape were added to secure the temporary plastic diversion sheet.

@ 11:30 the x-strip graph (T/W  MS 230) of the Cu-ALCW tank level showed that the leak had stopped and leveled off.

@ 13:15 the 1A tunnel floor had dried up with no water seen dripping from the vertical chase.  Doug also mentioned that the “water sensors” that were activated reflected this. The Cu-ALCW was then valved back on to the T2 cooling package and within 3 minutes water began dripping from the south wall vertical service chase which now appeared to about  1 drip/sec.

Fiona took a water sample (20 ml vial) from the drips and also found ~ 1000 c/m (44-2 monitor) on the tacky mat just on the inside of 1A tunnel door.  The 1A Tunnel and South TNF were then locked-up by Derek H., Violeta T. & Doug.   The T2 cooling package was turned back on ok.  Will try to repair this water leak during the Oct. mini shutdown.

 A 5lb roll of 347SS sheet was received today.  This alloy is to be used for the target windows of the new T1 and T2 beryllium target cassettes that will be manufactured in the near future.  The material was ordered from Alloys International (PO# TR206407).  A mill spec report was included with the material, a scanned copy is attached in PDF format.  The roll will be stored in the meson hall RH hot cell lab office on the shelving above the work bench.

The 4" Marman flange clamp (IRH0001) labeled "Clamp A",  as used in the testing detailed in Document-114623 Release 1, has been disassembled.  The parts were used to build a TRH1257 clamp which uses the new TRH1259 jaw with 0.480" jaw root width.  This new clamp has been labeled "TRH1257-#01", and will be tested with Helicoflex H-303654 seals in the near future.

The T2 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

No leaks or abnormalities observed - system running normally.

Secondary flow sensor B1A:T2CS:FGSEC:RDFLOW on the T2 cooling station suddenly became very noisy at around 11 pm, 01 September 2024.

The sensor reading would fluctuate from 0-150 gpm which is beyond typical noise levels for these sensors (~2gpm).

The noise spontaneously ended at around 4 am, 05 September 2024.

The noise likely originated from some stuck debris preventing the paddle wheel from turning normally that got spontaneously dislodged.

Alternatively, changes in the environment temperature could have broken some electrical contact that recovered when the temperature cooled over the weekend. We will make a note to look into this sensor in the coming shutdown and perform preventative replacement of components.