Defeated T1 and T2 cooling package vacuum interlocks at 1-C panel on South mezzanine in Meson Hall to allow cooling packages to continue running after beamlines are vented.

Notified operations.

Replaced all o-rings and filters on the T1 cooling package.  Replaced resin can.  Refilled package and restarted. 
Can inlet resistivity at 9.8MΩ *cm, outlet at 11.5MΩ*cm after approximately 3.5 hours.

(Collimator A filter and modified filter housing to be installed at a later date)

While checking operation of the M9 and M20 beam blockers earlier this week, a strange noise was noticed coming from the T2 cooling package area.  It is a squealing sound that starts when M9BB, M20BB, or T2 profile monitor are actuated and continues for approximately 30 seconds, then stops.  The source was eventually narrowed down to somewhere on the air supply line coming from the air amplifiers on the south mezanine.  When the air amplifiers are adjusted below ~95psi, the noise stops (the standard setting is 110psi).  A 20psi Nupro check valve on the line was suspected to be causing the noise.  It was removed, rebuilt, and reinstalled, but there was no improvement.

A few of the blocks directly east of the T2 cooling package must be removed so we can have access to the lines in that area.  We will continue our investigation next week.

 The following work was performed on the T1 cooling package:

- New filter housing installed for Col. A filter.  The new housing has a drain valve to prevent water spills during future filter changes.
- Ball valve installed upstream of Col. A filter to allow proper isolation of filter during filter changes.

The following work was performed on the T2 cooling package:

- After further investigation it was confirmed that the air-line noise discovered last week was being caused by a fault check valve.  A new valve will be ordered and installed.
- The M20BB air supply Asco valve was rebuilt to fix an air leak.
- The backing plate, paddle-wheel, shaft, and o-ring for Q2 Proteus meter were replaced.  (fixed fluctuating read-out)
- Replaced 90 micron Nupro filter on the demin. line to address low flow issue (flow increased from 1.15gpm to 2.50gpm)
- Replaced Inlet Pressure Transducer (P1) in BL1A tunnel  (fixed fluctuating read-out)

 

The following work was performed at the T2 monolith:

- Signals were checked to confirm proper micro-switch actuation for M9 and M20 beam blockers (all ok)
* When M20BB is actuated a slight creaking or squeaking noise is heard during one part of the travel.  The main shaft and air cylinder shafts were lubricated, but there was no change.   (this is not thought to be a serious issue, and won't be addressed at this time)
- Replaced o-rings on the upper 1-way metering valve for the profile monitor air supply (fixed air leak)
- Replaced all sections of pneumatic hose for top and bottom supply lines to the profile monitor (old line was cracking)
- Checked actuation of profile monitor by powering 4-way Asco valve at cooling package (traveled smoothly)
* Control room was unable actuate the profile monitor remotely, Brian Minato will be notified of this issue
* The "monitor out" display does not show in the control room when the profile monitor is in the out position.  The micro switches were checked and seem fine.

The T1 expansion tank warning level sensor had tripped and the level was approximately 1cm from the base of the tank.  No trip warning was observed when Tom Lyth checked the panels last week.  The T1 cooling package and target monolith were inspected and no water was observed.  Two puddles of water were found in the BL1A tunnel around the T1 package area: one directly below the 1AQ9 Cable Interlock Tray, and another approximately 1.5m to the west.  However, water was observed in the same area when the tunnel was accessed in January.  The T1 volume is currently vented, so there is no indication from the vacuum system as to whether or not water has leaked into the beamline.

The tank was refilled, and the level had not changed after 1 hour.  I will continue to monitor the level.

 The water level in the T1 cooling package expansion tank was checked again today approximately 24 hours after it was refilled.  The level had dropped about 5cm, equating to roughly 3.5L of water lost.

The T1 package was inspected thoroughly and no water was found.  A leak on the target in the beamline is suspected.  The T1 package has been turned off and the water lines were removed from the target.  Tomorrow we will request vacuum group to pump down on the T1 volume which should indicate whether or not there is water in the beam line.

 Vacuum group pumped down on the T1 & T2 volumes (because there is no cold trap on the T1 pump) for approximately 1 hour this morning.  A significant amount of water was found in the base of the cold trap which confirms our suspicions of a water leak on the T1 target.

Tomorrow we will transfer the T1 target from the beam line to the hot cell for inspection, leak checking, and repair.

 Today I noticed a "NOT OK" signal from T2 water expansion tank warning level sensor.  I checked the water level in the tank and confirmed that the water level in the tank had indeed dropped.  The water package is currently not connected to the target so it is not possible for this water to enter the beam line.  

My initial suspicion is that the female Hansen fittings are leaking.  If this is the case, the water will be leaking onto the top of the T2 monolith.  The water lines were disconnected from the target on April 12, 2012 and at that time the expansion tank level was checked and confirmed to be full.   Approximately 7 liters of water has been lost since that time.  If the leak rate is steady this equates to 166mL per day.

I will investigate further on Monday morning.  If the cause of the leak is not obvious, a thorough investigation will have to wait until the M20 front end is covered as radiation levels are high around the T2 monolith and cooling package areas.  The current water level in the expansion tank has been recorded, and will be checked again Monday morning in order to see if the level is still decreasing and at what rate.

 The T2 cooling package, the top of the T2 monolith, and the Hansen water connections which connect the package to the target were inspected and no evidence of a water leak was found.  Kim wipes had been stuffed inside the female Hansen fittings after they were disconnected on April 12th, and these were also completely dry.  I checked with control room and there are no log entries regarding anyone starting the cooling package during the relevant dates.  I also contacted Graham Waters (who sometimes works on the water package controls) and he has not turned on the package since it was shut off on April 12th.

The water level in the expansion tank has not changed since last inspection on May 24th.  I will continue to check the water level daily.

 The T2 water package was reconnected to the target and started after being shut-off since April 12, 2012.  The M9 and M20 groups do not anticipate having to vent the T2 volume, so the package will be left running in preparation for BL1A start-up in early July.  There was no change in T2 vacuum levels during start-up, and therefore no water leaks into the vacuum volume.

The T2 expansion tank water loss observed on May 24 (Meson Hall E-Log #33) seems to have been a one time event, possibly caused by someone starting the T2 water package while the target was disconnected.  The level of the tank has been checked regularly since the water loss, and there have been no further drops in the level.

Mike Mouat enabled recording for T1 and T2 expansion tank warning and trip level sensors on XTPAGE/XSTRIP.

XSTRIP output is in discreet integers as follows:
2 = Both sensors OK
1 = Warning level NOT OK
0 = Trip level NOT OK

Mike is now working on having a warning message show up in the control room when the sensors read NOT OK.

 The target water outlet flow transducers on the T1 and T2 cooling packages were serviced (new shaft, paddle wheel, & o-ring) to address a noisy signal.  The service corrected the noisy signal.  Additionally, it appears the worn parts were causing the transducers to read high, as both dropped approximately 1 gallon/minute after the service.  Both transducers would occasionally trip the high flow warning (8.5gpm), and should now no longer do so.

A plot of the water outlet flows before and after the service is attached.  Cyclotron fault report #6086 was returned.

 Mike Mouat and Juan Pon have modified the control system so that the control room will receive a message when the warning or trip level sensors for the T1 & T2 expansion tanks read "NOT OK".  The message also displays the current water height in the tank.  This will help with earlier detection of water leaks from the cooling package or from the target ladder in the vacuum volume when the vacuum interlock has been defeated.

All sensors were checked by depressing the level sensor float and the correct warnings came up on the message reader.

 The following tasks were performed during the 2012 September Shutdown:

- Both targets moved to position 0 in preparation for testing of profile monitors (portable ladder control box was used for controlling ladder motors)
- T2 monitor OK; T1 monitor would not hit 'IN' limit
- The T2 branch of the amplified air system was closed, now the T1 monitor hits the 'IN' limit properly  (conclusion: a known leak in T2 section of amplified air is causing a drop in pressure of the system which prevents the T1 profile monitor from actuating properly)

- Both cooling packages were inspected and photographed (nothing abnormal observed)
- Both cooling packages drained, and water samples delivered to RPG
- Expansion tank level sensors tested at both packages: All OK
- All Hansen quick connect o-rings replaced at T1 package
- 90 micron filter on demin line replaced at T1 (caused ~0.2gpm increase of Q2 flow)
- Both packages refilled and restarted (operating normally)
- Q2 flow on both packages adjusted to ~1.45gpm

- The cause of the amplified air leak was discovered to be a cracked air dryer located in the BL1A service tunnel (photo attached)
- The air dryer was replaced which stopped the leak, resulting in a significant decrease in air amplifier cycle period and an increase in air amplifier output pressure from ~110 to 120psi

- Movement of the T1 ladder caused a spike in beam line vacuum;  After further testing it was concluded that the T1 target was leaking water
- T1-MK2 target was transferred from the beam line to the hot cell; T1-MK1 target was transferred from the storage pit to the beam line
- T1-MK1 profile and protect monitor electronics checked by Bill Rawnsley: Both OK
- Initially a leak was suspected at T1, but it was later discovered that the volume was just pumping down slowly due to water in the beam line from the T1-MK2 leak
- T1 profile monitor was not actuating properly;  It was found to be caused by a short to ground on one of the profile monitor limit switch wires.  The faulty wire was replaced by Probes Group
- T1 and T2 profile monitors tested.  Both actuating properly.
- New target ladder information sheet delivered to ops

- The leak in T1-MK2 was found to be coming from the center of the beam spot on the entry side of the position 3 target (photo attached, but not great quality).  This is a 12mm Be target, and has been running in beam position since initial start-up this spring.

Friday Jan 25:
- Both sides of heat exchanger drained, disconnected, and capped
- Heat exchanger transported to warm cell (field from H.E. approximately 15µSv/hr at 0.5m)

Monday Jan 28:
- Both sides of heat exchanger flushed with city water
- Heat exchanger lifted and tilted to drain as much water as possible 

- The new tube array was picked up from Kaltech on Feb 8th (observed vessel holding 60psi on hydro test at their shop)
- Vacuum leak checked at TRIUMF: would not pump down
- Applied 20psi air, and found leaks on both tube sheets using Snoop

- Dropped off tube array at Kaltech on Feb 13th for repairs
- To fix the welds on the manifold side it was necessary to cut open the manifold, then weld on a cap after re-doing the welds
- Calculations were performed to determine the stress on the welds of the new cap.  The weld was found to have a factor of safety of 27.4 (notes attached in PDF format)

- Repaired tube array was pressurized to 65psi with air at Kaltech and placed in a water bath: no bubbles.  Also checked welds with Snoop: no leaks
- Transported tube array to TRIUMF on Feb 20th
- Helium leak checked assembly, no leaks

- Old heat exchanger assembly was disassembled: old o-rings were stiff, but not cracking
- Small active metal chunks and flakes were found in the tube side outlet manifold (~1mSv/hr on contact)
- The old tube array was measured to be 8µSv/hr at 0.5m, and 500µSv/hr on contact at specific locations, which indicates that active chunks of metal may be stuck at certain places in some tubes
- Shell side of old tube array had  brownish/orange buildup, tube side appeared fairly clean (see photo below)

- The new tube assembly was inserted into the H.E. shell (required some filing on baffle welds)
- The welds between the baffle strips and large tube sheet were found to interfere with the flange on the shell as shown below (this was an oversight on the drawings, not machine shop error)

- The tube sheet face was skimmed on the lathe by Maico and Dan Wright to remove the interfering weld material
- Repeated the vacuum leak check after machining: no leaks
- The modified tube array fit properly within the shell as shown below

- All parts were cleaned using Windex, followed by methanol, with special attention to sealing surfaces
- The heat exchanger was reassembled using new o-rings, new fasteners, and NeverSeez compound on all fasteners
- A static water test at city supply pressure (~70psi) was performed on the vessel, and a leak was found coming from the tube stub seal (#319 o-ring)
- An o-ring calculation was performed which indicated that the current configuration only compressed the o-ring 15.4%
- 1.50" OD, 1.080" ID shims were machined from 6061 aluminum at 0.017", 0.022", and 0.029" thicknesses as shown
- The addition of a 0.029" shim was calculated to increase the o-ring compression to 27.7%

- The 0.029" thick shim was installed as shown below

- The vessel was filled with water and pressurized to city supply pressure (~70psi): no leaks
- The pressure was increased to 150psi (1.5X the working pressure) using a Reed hydrostatic test pump supplied by Beamlines Group
- No water leaks were observed
- The vessel was valved off while under pressure downstream of the pump hose
- The following pressure drop was observed over the course of a weekend:
     Mar 1st @ 2:35pm:  151psi
     Mar 1st @ 4:30pm:  142psi
     Mar 1st @ 5:30pm:  139psi
     Mar 4th @ 9:00am:  70psi

- This indicates a slow leak either through an o-ring seal on the vessel, the valve used for pressure testing, or one of the fittings used for pressure testing
- The vessel was drained and pressurized with air to house pressure (~100psi)
- Snoop was applied around all o-ring seals: no leaks found
- The leak check with Snoop indicates that the leak must have been through a valve, on a pipe fitting, or of too low a rate to be of any concern

- The heat exchanger was tipped on its side to allow the shell side to be fully filled with water
- After filling with city water, a ball valve and solenoid valve were installed on the shell side ports to prevent water leaking during transport (see photo below)

- The heat exchanger was bagged and transported back to the T2 cooling package
- Testing of the heat exchanger will be done when the T2 target is reinstalled and the CuALCW system is back up (in approximately 2 weeks time)

 The T1 and T2 expansion tanks were filled on Monday April 8th.  There was no change in the level when inspected today (April 12th).  Visual inspection of the packages also found no leaks.

Installation, wiring, and commissioning of the high temperature limit switches for the T1 and T2 cooling packages was completed today as part of SAS project SASP0120.  The relay output from the switches was wired in series with the Central Safety System signals for T1 and T2 (cable number 13250 and 12250 respectively) in break-out panel #2 on the Meson Hall south mezzanine.  There were already 3 other circuits wired in series with each signal for various other interlock requirements.

After wiring, commissioning of the switches was performed according to the following procedure:

- With the limit switches in a non-tripped state, it was confirmed that the safety signals were reaching the CSS in the main control room as expected (had to jumper T2 circuit to achieve this as the T2 target is not currently installed)
- The limit switch was tripped by adjusting the set point on the switch, it was confirmed that the signal was lost
- The set point was changed to a non-trip level, and the switch trip cleared, confirmed that the signal in MCR returned
- The enable signal from each cooling package was removed by shutting off the cooling package (for T2 just removed jumper), confirmed that safety signal lost as expected
- Reversed this condition and confirmed that the signal returned
- Checked that the thermocouples are performing properly, and the switches actually trip on high temperature by directly applying heat to the thermocouple area on the cooling package using a heat gun;  the switches tripped as expected when the set-points were exceeded

The results were discussed with John Drozdoff (Safety Systems group leader) who approved the method.

The final task remaining for SASP0120 is to route the PRV outlets to the 1A tunnel active drain using flexible hosing.

Yesterday (April 18th) the T1 and T2 cooling package PRV outlet lines were routed to the active drain in the 1A tunnel.  All items for the safety and standards compliance upgrade for T1 and T2 as specified in Document-68861 are now complete.

 The T1 cooling package inlet pressure (P1) has been dropping since approximately May 5, 2013 from the regular operating pressure of approximately 75psi to the present value of 47psi.  An increase in the rate of decrease started around August 5th as well as increased noise in the signal.  Plots attached below.  No corresponding change in the outlet pressure, and no change in the water flow rate (also no change in inlet and oulet temperatures) leads to the conclusion that the transducer is failing or gas is trapped in the lines leading to the transducers in the BL1A tunnel.

Jamie Cessford and Ron Kuramoto are aware of the situation.  The issue will be addressed during the September mini-shutdown.