On Oct 7^th, during the Mini Shutdown, the T2 cooling package was uncovered to investigate the Cu-ALCW water leak dripping into the 1A Tunnel.

The Target shield block was lifted and found that top of the monolith was dry.  Uncovering the east side of the cooling package directly above the vertical services on the south wall revealed a pin hole leak at a 1/2” 45^o copper elbow on the return line circuit of the T2 cooling package.

The cooling package was turned off followed by valving off the Cu-ALCW service lines leading up to the cooling package to let the Tunnel floor dry over night.

On Oct 8^th, the Tunnel floor was dry and Cu-ALCW service lines feeding the cooling package were drained from the Tunnel by placing a 5 gallon pail under the drain lines filling it about 1/3 full until it stopped.

Mel Good was brought in to replace the faulty elbow.  The Cu-ALCW was then turned back on with no obvious leaks.  At 16:00 approximately 1.5 hours after the repair, I went into the 1A tunnel to inspect for water leaks at the isolation valves and saw water dripping down the vertical piping on the south wall again.   

Another leak was found on the 90^o elbow just ahead of the replaced 45^o elbow. There appeared to 2-3 fine mists spraying out towards the adjacent copper elbow (look at the end of the cable tie on the following photos).

The Cu-ALCW lines leading up to cooling package were again valved off to let the Tunnel floor dry up over night.

On Oct 9^th, the Cu-ALCW lines leading up to the cooling package were again drained followed by replacing the faulty 90^o elbow. The elbow was fitted with new pieces to improve the configuration. We noticed that old copper pieces were worn thin. These damaged parts measured 20,000 c/m on the 44-2 and 40 c/m on the RM-14 monitors and were taken to the RH H/cell lab for inspection.

The T2 cooling package turned on ok about 1.5 hours after repairing the 2^nd water leak.

Mel Good received 0.03mSv over the two day repair. 

I received 0.10 mSv over three days, picking up 0.05mSv on Oct 7^th in the Tunnel where the fields were 200-300uSv/Hr by the isolation valves.  The next the day levels dropped to 40uSv/Hr.  There was no contamination found on the Tunnel floor, however 80,000 c/m on the RM14  and 50,000 c/m on the 44-2 monitors were found on the swipes taken at the bottom edge of the concrete shield blocks.

NOTE: the Cu-ALCW service lines leading up to the T2 cooling package feeds the following 4 devices; Collimator A, Collimator B, the T2 monolith cooling and the Target water heat exchanger.

On 14Oct2014, the 1AT2 Collimator B water outlet temperature rose to 100^o C. 

Summary: Clarification of T2 water temperature readouts

Detail: This e-log entry by Doug Preddy is being made to clarify the T2 water package temperature read outs. The temperature read outs on the T2 water package have been switched. The read out for inlet temperature is actually the outlet temperature and the read out for outlet temperature is actually the inlet temperature. On Sunday Oct 19 2014 at 19:45, the T2 water outlet Temp (T2) started to climb giving warning messages at 20:05. Operators tried to decrease the BL1A current, but the temp. continued to climb and eventually tripped the beam off. Operators defined BL1A OFF, and continued to inject to BL2A only. They tried to contact I. Earle, but were unable to. They left message with D. Preddy. The T2 temp. stayed at 45 deg., operators tried to reset the water package and start it, but it tripped almost right away, and increased the temp. by 0.5 deg. Beam was left off. On Monday Oct. 20 Ron Kuramoto found a replacement thermister. It was wired in to the BOP over the south catwalk. The T2 outlet temperature immediately dropped to ambiant. From this test it was concluded that the installed thermistor on the T2 outlet water pipe had failed. The wiring was restored to the original configuration. It was decided to switch T2 Water Inlet Temp and Outlet Temp sensors at the T2 electronics rack to provide a trip on the Inlet Water temperature instead of the outlet Water Temperature. As the T2 water package runs in a closed loop, the outlet temperature signal has been swapped with the inlet temperature by Doug P. and Tony T. This will provide thermal protection (trip at 45C) in addition to the T2 Target high limit bit (trip at 60C). The water flow to this circuit is also monitored and will trip if the flow drops. This action was approved in a meeting with Doug Preddy, Ron Kuramoto, Grant Minor, Yuri Bylinski, Lia Merminga, and Bob Laxdal present on Monday Oct 20th. There is a second, redundant temperature sensor on the outlet of the target water which was installed recently as a safety upgrade to protect the heat exchanger and meet CSA pressure vessel code, see dwg TBP0902. There is a controller for this sensor with a hard-trip switch for the cyclotron, and a temperature read-back display on the Meson Hall mezzanine racks. This read-back should be monitored by operations once per shift while beam is on target to ensure that outlet water temperature is stable and at an expected value. Sensor cabling swapped at Meson Hall, south mezzanine, bay 1, B.O.P. #3. Sensor T1, cable #12102 moved to TB1, terminals 3 & 4. Sensor T2, cable #12105 moved to TB1, terminals 1 & 2. Defeat #7131 removed. Defeat# 7133 Fault report #7722   Will look into replacing the faulty Thermister in the 2015 major shutdown.

 - On Nov 13 at ~1am the BL1A vacuum went bad.  The T1 and T2 volumes were isolated and pumped on separately, the results indicated a water leak from the T1 target.  The level of the expansion tank was slightly below normal at this time, but not enough to be conclusive.  The leak was confirmed at ~9am by starting the T1 cooling package which caused a dramatic vacuum spike.  Approximately 1" was lost from the expansion tank when the package was started, equal to approximately 2.5L of water.

- The T1-MK1 target was removed from the beam line and transported to the hot cell.  A mound of ice was visible on the target lower plate while it was being lifted.  A radiation field of 160mSv/hr was measured at 0.5m from the target during removal.  Upon inspection in the hot cell, a small but clear hole was visible on the entry window of the position 3 target cassette just slightly above and to the right of center if facing downstream.  No water was visible on the profile or protect monitors.  Approximately 1" of water was visible in the base of the T1 monolith.  The monolith was left open to air overnight, and approximately 0.5" of water was visible in the morning on Nov 14.

- Vacuum group was consulted, and given the relatively small amount of water remaining they agreed to pump the water out of the beam line rather than have Remote Handling group try to remove the water before target installation.  The T1-MK2 target was transported from Hole #3 in the storage pit to the beam line.  A field of 5mSv/hr was measured at 0.5m from the target.  All services were attached.

- An electrical check was performed on the profile and protect monitors by David Cameron from the 1A mezzanine.  Both devices appeared normal.  Plant Group then began to replace the blocks over T1.  Graham Waters updated the control software so that the correct target ladder potentiometer values are referenced.  An updated target information sheet was prepared and delivered to Operations (PDF File Attached)

While the 1A blocks over the T1 area were removed last week the opportunity was used to investigate an air leak on the amplified air system discovered on August 11, 2014.

The leak was found to be coming from a device related to the septum polarity switch located behind (south of) the septum power supply on the 1A blocks.  This device is not in use, so the amplified air to the device was disconnected by plugging the relevant branch of a brass 3/8" Swagelok T fitting in the area.  This stopped the leak, and the air amplifier is cycling at usual speed after being restarted.

 Between Nov 14 ~ Dec 9 the following activities took place:

- The leaky 10mm Be target at pos'n 3 was removed and plugged
- The loose protect monitor electrical conduit was secured in place using an aluminum shim
- David Cameron performed an electrical check on the protect and profile monitors and got the expected response from both
- The target was flushed with water, then air
- The target was pumped down using a cold trap to capture remaining moisture
- The target reached a minimum pressure of 80mTorr on the leak detector Hastings gauge
- Helium leak check revealed a leak at both of the position 1 Swagelok caps (up to 80 on 50x scale on left side, and 100 on 100x scale on right side when facing ladder with 1s spray of 5psi helium)
- The leak rate did not improve with tightening of the caps
- The caps were removed, ladder side threads inspected (no obvious damage observed), and new caps installed
- Helium leak check repeated: Pos'n 1 left side now leak tight, right side leak rate was worse (could not completely open throttle valve on leak detector)
- Torque was increased on the cap with no improvement in leak rate
- The right side cap was removed and when trying a new cap it would not spin freely
- The ladder side threads were inspected more carefully with the Nikon level:  a small dent across the first thread at approximately 2 o'clock position was seen as well as some material build-up or possible galling seen at the 1st and 2nd threads around the 4 o'clock position.
- The threads were filed to improve their profile which allowed a new cap to be installed with light resistance (still would not spin freely)
- Leak check was performed with a minimum pressure of 70mTorr reached (throttle valve fully open, roughing closed), with large response still at the right side pos'n 1 cap, No change with moderate tightening

It is suspected that the ladder side fitting sealing face is damaged causing a poor seal.  When a cap was installed and torqued this may have caused the galling or material build up observed at the 4 o'clock position (but not the dent on the 1st thread at the 2 o'clock position)

Various experts on-site will be consulted about how to best proceed before doing further work.  One option is to perform a static water test.  If the target is water leak tight then it may be used in the beam line.  If repair efforts are not successful and the target is not water leak tight then the target ladder will have to be replaced which is approximately a 2 week job.

The T2 cooling package was visually inspected today after blocks were removed.  With the cooling package off the following leaks were observed:

- Collimator A return line ball valve: ~ 2 drops / second
- Collimator B return line ball valve: ~ 1 drop / 3 seconds

The cooling package was then started and the following leaks were observed:

- T2 heat exchanger secondary side, north end ball valve (CuALCW):  ~ 1 drop / sec
- Target water return line ball valve above reservoir: very slow leak, wet but no drips visible

These leaks explain the water found in the tunnel, and the slow drop in T2 expansion tank level.  However it is a little mysterious why four valves all failed around the same time on two different water systems..

The copper active supply to the T2 area will be valved off to stop the leaks, and the valves will be serviced or replaced during this shutdown.

Both defeats applied to T2 cooling package thermistors were removed today (Col. B interlock was jumpered, and the target water inlet and outlet signals had been swapped).  It was confirmed that all thermistors are now wired correctly.  The readback of both target water thermistors is currently unstable.  Inspection of the Lemo connectors showed that the internal plastic parts of all connectors were damaged or missing.  If the T1/T2 controls upgrade proceeds this shutdown, then all thermistors will be replaced with thermocouples, so after a decision is made on that project (sometime in Feb 2015) these thermistors will either be repaired or replaced.

- Both 3/4" ball valves on the T1 cooling package target water circuit were rebuilt, leak tested with house air pressure, and reinstalled

- The 3/4" ball valve on the T1 heat exchanger secondary side was replaced with a new unit

- The 3/4" needle valve on the T1 heat exchanger secondary side was rebuilt, leak tested with house air pressure, and reinstalled

- The T1 cooling package was refilled and restarted using a bypass across the inlet/outlet Hansen fittings

- The package was inspected while running and no leaks were visible (video survey taken with file stored on Isaac's PC)

- All T1 Cooling Package MRO activities are now complete

 The T1-MK1 target is currently in the hot cell for repair of a leak at position 1.  See E-Log #129 for details on previous work.

- Pos'n 1 right nut removed (a little stiff, but no excessive force required)
- Exterior threads cleaned w/ Scotchbrite disc on Dremel tool
- Inside sealing face inspected w/ Nikon level.  Possible brownish material on sealing face at 10 o'clock and 12/1 o'clock positions
- Threads inspected w/ Nikon level: small dent at 1st thread 2 o'clock position as seen before and small amount of material build-up or galling at 1st & 2nd thread 4 o'clock position.  Otherwise threads looked clean and straight
- Constructed Dremel attachment using ~1/2" thick, ~4" long piece of Scotchbrite folded over a 1/16" diameter steel rod, secured with zap-straps and shaped w/ scissors to fit inside the ladder port (see photo)
- Cleaned and polished the sealing surface w/ new Scotchbrite Dremel tool
- Inspected inside sealing surface: brown coloured material no longer visible.  A clear scratch/dent is visible at 10 o'clock position.
- Blew out hole and new nut with compressed air
- Installed nut (went on much easier than before which indicates that the Scotchbrite disc thread cleaning method is effective)
- Leak check performed:  pumped down to ~70mTorr on Hastings gauge w/ throttle valve fully open and roughing valve closed
- Sizeable leak at Pos'n 1 right side nut, as before.  No improvement with tightening

The next step will be to perform a static water test on the cooling lines to see if the plug is water leak tight.  If it is, no further action is necessary.  If not, either the ladder must be replaced, or other methods to achieve a seal investigated.
 

This is a continuation of work detailed in E-Log #129 and #140.

- Static water test performed on the target cooling lines:
     - Test setup assembled and leak checked by pressurising with house air, submerging in water and checking for bubbles (see photos)
     - System filled, pressurized, and valved off on Friday Feb 13 at 5:30pm,  Gauge read 60psi
     - Checked Monday Feb 16 @ 9:00am.  Gauge read 56psi
     - Checked Monday Feb 16 @ 5:00pm.  Gauge read 56psi
     - Checked Tuesday Feb 17 @ 9:30am.  Gauge read 55.5psi  (0.5psi / 24hrs rate)

- Checked results with Dimo and he requested a helium leak check be performed with a newer model leak detector to quantify leak rate:
     - Using Varian 979 leak detector, pumped ~1.5hrs w/ cold trap to remove water
     - TP press: 0.0*10^-4 torr,  Base leak rate ~7.0*10^-9 atm cc / sec
     - Tested all normal leak check locations and did general helium flood with no response at all  (1.5psi helium pressure)
     - Over ~ 20 minute period the base leak rate gradually rose to 8.8*10^-9 atm cc / sec, at one point spiked to high 10^-9 level, then returned to base line
     - Reviewed results with Dimo, he requested test be repeated after the target sits vented overnight

- Target vented and left overnight, leak check repeated Feb 18 starting 9am:
     - 0.0*10^-4 torr TP pressure after ~5 mins
     - 10^-9 atm cc / sec leak rate level after ~20 mins
     - All locations sprayed with 1.5psi helium, general flood in ladder area --> no response (Dimo and Edi present)
     - Target deemed OK for use in beam line by Dimo, he requested the cooling lines be filled with helium after installation in b/l while leak checking T1 volume

- Target will be transported to the storage pit tomorrow (Feb 19) and is ready for use as the spare T1 target

The target had an obvious helium leak when first tested (E-Log #129 and #140), which could not be found when helium leak checking again after the static water test.  Some possible explanations are:  water remaining in the lines blocked the leak path,  water remaining in the lines froze when vacuum was pulled blocking the leak path, having water in the lines caused some corrosion or other deposit which plugged the leak.

The helium leak at plugged position 1 right side measured before the static water test is the same, or slightly less than the leak rate when the target was first removed from the beam line.  There were no T1 vacuum issues during the running period, therefore no issues are expected when this target is put in service again.

The following work was done today at the T2 cooling package:

- Remainder of #4 Hansen o-rings replaced  (see E-log from Jan 30, 2015)
- Installed the following ball valves which have been rebuilt & tested with 100psi air: target outlet 3/4" (x2), resin can outlet filter isolation 1/2" (x2), heat exchanger secondary side outlet 3/4" (x2), collimator A outlet 3/4", collimator B outlet 3/8"
- Installed the following ball valve which has been tightened & tested with 100psi air (no rebuild kit available): resin supply side 1/2"
- Disassembled the target water inlet thermistor probe assembly to check compatibility with thermocouple probes purchased for T1/T2 controls upgrade
- Misplaced a fitting that the probe inserts through, plugged thermistor port as a temporary measure (the outlet thermistor still functions as machine protection)
- Filled cooling package and opened CuALCW supply and return valves to T2 area in BL1A tunnel
- Started package, inspected for leaks, tightened various Swagelok fittings to stop minor leaks
- Tightened valve stem packing compression nut on Col.A/M8BB shut-off valve and Col B. shut-off valve on cooling package front panel to stop drip leaks from the valve stems
- Inspected again for leaks while cooling package running:  none observed

The T2 cooling package is ready for a interlock response timing test which will be performed tomorrow (part of T1/T2 controls upgrade).
Before the T2 cooling package is ready for operation, the target water inlet thermistor must be reinstalled, and the thermistor connectors must be replaced (if the thermistors are not replaced with thermocouples as part of the controls upgrade)

Total dose:

Isaac - 0.08mSv
Keith - 0.01mSv
 

The Inlet (T1), Outlet (T2), Col.A (T6), and Col. B (T7) thermistors at the T2 cooling package all gave intermittent faulty readings at various times between Oct - Dec 2014.  It is suspected that this was caused by a water leak in the area causing the terminals to get wet.  Inspection of the sensors revealed that they are in poor shape and that the readouts sometimes fluctuate when the probe or connectors are handled.  All four thermistor probes have been replaced.  The plastic interior of the Lemo connectors were found to be crumbling, and were replaced with crimp connections.  All T2 thermistors now appear to be functioning normally.  The T2 cooling package was started (with a bypass instead of the target assembly).  Flow through Collimators A and B has also been started.  The T2 package was inspected while running and no leaks were found.

Cyclotron Fault #7722 has been returned.

The Meson Hall RH Lab Warm Cell was used by Beamlines Group for hydrostatic testing of the TNF heat exchanger.  The location was chosen because of access to a water supply, active drain, and the heat exchanger has removable contamination.  The work took place on April 30th and May 1, 2015.  New plastic sheet was placed over the 6 foot block in the center of the warm cell which was used as a work area.  The plastic was removed after completion of the job.  The floor around the block was swiped and 100cpm contamination was found.

See attached photos.

At 9:15am this morning BL1A tripped off due to loss of the watchdog signals for the T1/T2 control system.  Graham Waters investigated and the problem was found to be a corrupted initialization file.  The error was corrected, and BL1A was started by 11:30am.
 

The following tasks were completed over the past 2 weeks:

- Aquarium Technology Ltd. (UK) was contacted regarding best method for replacing window seal.  A detailed procedure was provided (attached below)
- The warm cell windows were determined to be acrylic, not glass.  Estimated weight: 450lb per panel

- All water in the warm cell windows was released to grassy area outside the lab after approval from Gord Wood (OH&S) and Joe Mildenberger (RPG)
- Warm cell was cleaned up and blocks were configured to make fields as low as possible in the work area around the windows (~2micro Sv/hr max field around windows).  No contamination found on warm cell floor.
- The old seal around each window was cut mechanically with a thin blade
- The panel spacer rods were removed  (most were constructed of SS and Al, however a couple were mild steel and were badly corroded)
- The window panels were separated from the frame using a rubber mallet and moved to the center of the tank (the panels were not removed from the tanks for this repair)
- The old seal material was removed from the panels and the aluminum frame using WD40 or mineral spirits to help break it down (this was a very tedious and unpleasant task)
- Pitting due to corrosion was found in various places on the aluminum frame.  It appeared to be worst where mild steel spacer rods were used, and also worse at weld locations (see attached photo).  Max depth of pits ~1/8"
- The corrosion was cleaned as much as possible using a wire brush then rags with mineral spirits
- A final clean of sealing surfaces was done with isopropyl alcohol
- A deep pit (~1/4" deep) was found on the inside of the inside of the aluminum frame on the west tank, south window (see attached photo)
- This pit, as well as some other smaller ones on the same side were filled with Dow Corning 791 sealant material

- Rubber stand-offs (Digi-Key #SJ5009-0-ND) were installed on the frame side in the center of the sealing faces (3 top, 3 bottom)
- The panels were moved back into place resting on 3/8" thick aluminum spacers (see attached photo).  These replaced spacers of the same size and material that were slightly corroded

The following tasks remain to be completed:

- Install panel spacer rods (mild steel rods to be replaced with SS or Al)
- Apply sealant to all panels from outside the tank as detailed in the procedure using 20oz pneumatic applicator caulk gun (PO: TR206556)
- Install corrosion inhibiting pads inside tank.  Will use 6 pads per tank, (4x McMaster 3609K2 & 2x McMaster 3590K2 per tank)
- Perform final clean of inside of windows and tank
- After 14 days cure time fill tanks with city water and allow to sit to check for leaks
- If leak tight, use water in tanks to back-flush the sand filter
- Refill tanks and start pump with filter in normal mode
- Occasionally check for water leaks and check status of pitting inside tank over the coming months/years
- Also monitor corrosion inhibiting pads and replace or change materials if necessary

Summarized Seal Replacement Procedure provided by Aquarium Technology Ltd:

1) Cut out the viewing panel entirely and mechanically clean all sealant traces from both the panel and the frame bearing faces of the tank structure

2) Degrease all sealing surfaces with something like pure petroleum spirit (mineral spirit) which won't harm the acrylic

3) Reinstall the clean viewing panel spacing it off the frame bearing face by around 10mm using self adhesive rubber stand-offs  (depending on panel flatness likely no more than 3 needed along the top and bottom)

4) Brace the panel in place, mask off the frame and panel, and then inject Dow Corning 791 sealant from the outside.  Ensure that the sealant goes in under pressure, fully wets both surfaces, and completely fills the void.

5) Allow the sealant to cure for 10-14 days before filling the tanks

Since the previous E-Log the following tasks have been completed:

- Panel spacer rods installed at 4 locations in each tank (two top, two bottom at approximately 1/3 and 2/3 the way across)
- Spacer rods expanded only slightly to begin compressing the rubber window spacers while still leaving at least a 8mm gap between window and frame
- Window and frame around seal area masked off with masking tape
- Sealant was applied from the outside using a plastic cone tip with a copper tube attachment to allow sealant injection to the bottom of the sealant cavity.  The process involved having one person inside the tank to make sure the sealant cavity was completely filled with no voids. A total of 42 of the 20oz. sausages were required.  Two spare unopened sausages remain.

The sealant will now be left for 2 weeks to cure before water is added to the tank.

The warm cell water window tanks were filled on September 17 after 2 weeks of curing time.  A water leak was found coming from the seals of a plastic pipe joining the two tanks near the top.  This was repaired using rubber gaskets and quick cure caulking.  The tank was refilled on September 18 and no longer leaked from that location.  The sand filter was backflushed with the water exhausted to the grass area outside the hot cell lab until the water ran clear.  The pump was then started with the water circulating normally through the filter.  Small water leaks were observed coming from 3 of the 4 plastic ball valves on the filter loop piping.  These leaks stopped after approximately 3 days of continuous running, and are still leak tight now.

During this repair job it was observed that the support wheels for the warm cell manipulators are crumbling and should be replaced before the next time the manipulators are used.

 The stepper motor that drives the target ladder on T2-MK2 has failed.  This is the cause of the difficulty moving the target ladder yesterday afternoon (see previous e-log).  This was confirmed this morning by attempting to move the T1-MK1 target ladder in the storage pit using the portable control box - the ladder moved with no issues.

The motor is scheduled for replacement during the 2016 winter shutdown unless it is needed more urgently.  Operations have been informed that T2 ladder movement will not be possible until this motor is replaced.

On Monday Oct 5th a decision was made to uncover and remove the T2 Protect Monitor due to faulty readings.  The monitor had been behaving strangely since the September shutdown, and had been getting worse over time.

- Blocks were removed over T2 starting Monday afternoon.  It is necessary to remove the narrow 12' block and two 6' blocks west of the T2 plug block in order to fit the flask frame in for protect monitor removal

- The monitor was moved to the hot cell by 7pm on Monday.  Pierre was operating the crane and there were no issues with the move.  (monitor measured 15mSv/hr at 0.5m)

Oct 6:

- Monitor inspected: a thin piece of foil was found jammed into the entrance side of the monitor (see photos) & a heat or burn mark was seen on the exit side above and to the south of the plate gap.  It is suspected that this material traveled down the beamline at high speed when there was a vacuum burst during the September shutdown.  In light of this information it was decided to remove the T2 target and T1 target as well (see following e-logs)

- Monitor elevation measured: 1835.8mm (april 2015 measurement was 1836.1 --> ok)

- Old monitor cassette removed, new one installed.  Elevation checked: 1832.2mm - did not match, removed, decontaminated, and returned to Probes for adjustment

- Adjusted monitor installed, elevation checked: 1835.8mm --> ok

- Electrical check done by Probes group at hot cell: found left plate shorted to ground

- Cassette removed, decontaminated, and repaired by Probes Group

- Cassette reinstalled, and electrical check repeated --> ok

Oct 7:

- Monitor elevation re-checked: 1836.2mm --> ok

- T2 protect returned to beamline, bolts installed, cables connected

Oct 8:

- Vacuum pumped on T2 volume after T2-MK2 target returned (see next e-log):  Only reached 2 Torr and a leak was heard coming from T2 protect flange.

- T2 protect was unbolted and lifted ~1in for cleaning of the flange base and o-ring

- The T2 volume then pumped down normally

- A helium leak check of all flanges on the T2 monolith was done by Vacuum Group --> no leaks found