Summary: Clarification of T2 water temperature readouts

Detail: This e-log entry by Doug Preddy is being made to clarify the T2 water package temperature read outs. The temperature read outs on the T2 water package have been switched. The read out for inlet temperature is actually the outlet temperature and the read out for outlet temperature is actually the inlet temperature. On Sunday Oct 19 2014 at 19:45, the T2 water outlet Temp (T2) started to climb giving warning messages at 20:05. Operators tried to decrease the BL1A current, but the temp. continued to climb and eventually tripped the beam off. Operators defined BL1A OFF, and continued to inject to BL2A only. They tried to contact I. Earle, but were unable to. They left message with D. Preddy. The T2 temp. stayed at 45 deg., operators tried to reset the water package and start it, but it tripped almost right away, and increased the temp. by 0.5 deg. Beam was left off. On Monday Oct. 20 Ron Kuramoto found a replacement thermister. It was wired in to the BOP over the south catwalk. The T2 outlet temperature immediately dropped to ambiant. From this test it was concluded that the installed thermistor on the T2 outlet water pipe had failed. The wiring was restored to the original configuration. It was decided to switch T2 Water Inlet Temp and Outlet Temp sensors at the T2 electronics rack to provide a trip on the Inlet Water temperature instead of the outlet Water Temperature. As the T2 water package runs in a closed loop, the outlet temperature signal has been swapped with the inlet temperature by Doug P. and Tony T. This will provide thermal protection (trip at 45C) in addition to the T2 Target high limit bit (trip at 60C). The water flow to this circuit is also monitored and will trip if the flow drops. This action was approved in a meeting with Doug Preddy, Ron Kuramoto, Grant Minor, Yuri Bylinski, Lia Merminga, and Bob Laxdal present on Monday Oct 20th. There is a second, redundant temperature sensor on the outlet of the target water which was installed recently as a safety upgrade to protect the heat exchanger and meet CSA pressure vessel code, see dwg TBP0902. There is a controller for this sensor with a hard-trip switch for the cyclotron, and a temperature read-back display on the Meson Hall mezzanine racks. This read-back should be monitored by operations once per shift while beam is on target to ensure that outlet water temperature is stable and at an expected value. Sensor cabling swapped at Meson Hall, south mezzanine, bay 1, B.O.P. #3. Sensor T1, cable #12102 moved to TB1, terminals 3 & 4. Sensor T2, cable #12105 moved to TB1, terminals 1 & 2. Defeat #7131 removed. Defeat# 7133 Fault report #7722   Will look into replacing the faulty Thermister in the 2015 major shutdown.

On 14Oct2014, the 1AT2 Collimator B water outlet temperature rose to 100^o C. 

On Oct 7^th, during the Mini Shutdown, the T2 cooling package was uncovered to investigate the Cu-ALCW water leak dripping into the 1A Tunnel.

The Target shield block was lifted and found that top of the monolith was dry.  Uncovering the east side of the cooling package directly above the vertical services on the south wall revealed a pin hole leak at a 1/2” 45^o copper elbow on the return line circuit of the T2 cooling package.

The cooling package was turned off followed by valving off the Cu-ALCW service lines leading up to the cooling package to let the Tunnel floor dry over night.

On Oct 8^th, the Tunnel floor was dry and Cu-ALCW service lines feeding the cooling package were drained from the Tunnel by placing a 5 gallon pail under the drain lines filling it about 1/3 full until it stopped.

Mel Good was brought in to replace the faulty elbow.  The Cu-ALCW was then turned back on with no obvious leaks.  At 16:00 approximately 1.5 hours after the repair, I went into the 1A tunnel to inspect for water leaks at the isolation valves and saw water dripping down the vertical piping on the south wall again.   

Another leak was found on the 90^o elbow just ahead of the replaced 45^o elbow. There appeared to 2-3 fine mists spraying out towards the adjacent copper elbow (look at the end of the cable tie on the following photos).

The Cu-ALCW lines leading up to cooling package were again valved off to let the Tunnel floor dry up over night.

On Oct 9^th, the Cu-ALCW lines leading up to the cooling package were again drained followed by replacing the faulty 90^o elbow. The elbow was fitted with new pieces to improve the configuration. We noticed that old copper pieces were worn thin. These damaged parts measured 20,000 c/m on the 44-2 and 40 c/m on the RM-14 monitors and were taken to the RH H/cell lab for inspection.

The T2 cooling package turned on ok about 1.5 hours after repairing the 2^nd water leak.

Mel Good received 0.03mSv over the two day repair. 

I received 0.10 mSv over three days, picking up 0.05mSv on Oct 7^th in the Tunnel where the fields were 200-300uSv/Hr by the isolation valves.  The next the day levels dropped to 40uSv/Hr.  There was no contamination found on the Tunnel floor, however 80,000 c/m on the RM14  and 50,000 c/m on the 44-2 monitors were found on the swipes taken at the bottom edge of the concrete shield blocks.

NOTE: the Cu-ALCW service lines leading up to the T2 cooling package feeds the following 4 devices; Collimator A, Collimator B, the T2 monolith cooling and the Target water heat exchanger.

On Sep. 3^rd during the 6 hr maintenance day, Doug Preddy & I went into the 1A tunnel under “controlled access” to investigate a water leak that triggered the Water Detection Sensors.  It was also noticed that the Cu-ALCW tank level was losing water.

Water was found to be leaking down the south wall chase where the T2 water package goes up the wall. Drip rate was ~1 drip every 15 seconds.

In the late afternoon of Sep. 9^th;

Day one of the 1.5 day maintenance, the tunnel was surveyed by Fiona H. see RPG for field map & floor swipe results. 

Doug, John McKinnon & I set up a plastic sheet with a funnel connected to a 8ft. lg. poly-flow tubing to direct leaking water from the T2 into the south gutter.  This set up was left over night for the floor to dry up and resume the water leak investigation. See photos.

On Sep. 10^th;

@ 08:30 about 70% of the 1A tunnel floor had dried up and the diverted water into the south side gutter had travelled to the T1 area.

The T2 cooling package was then turned off followed by valving off the Cu-ALCW feeding the T2 cooling package. A few more strips of red tuck tape were added to secure the temporary plastic diversion sheet.

@ 11:30 the x-strip graph (T/W  MS 230) of the Cu-ALCW tank level showed that the leak had stopped and leveled off.

@ 13:15 the 1A tunnel floor had dried up with no water seen dripping from the vertical chase.  Doug also mentioned that the “water sensors” that were activated reflected this. The Cu-ALCW was then valved back on to the T2 cooling package and within 3 minutes water began dripping from the south wall vertical service chase which now appeared to about  1 drip/sec.

Fiona took a water sample (20 ml vial) from the drips and also found ~ 1000 c/m (44-2 monitor) on the tacky mat just on the inside of 1A tunnel door.  The 1A Tunnel and South TNF were then locked-up by Derek H., Violeta T. & Doug.   The T2 cooling package was turned back on ok.  Will try to repair this water leak during the Oct. mini shutdown.

 During the maintenance day today, amplified air to the T1 area was valved off in an attempt to pin point the location of the air leak.  The air amplifier cycle rate returned to normal with the T1 line valved off, indicating the leak is somewhere on the T1 profile monitor air cylinders, the T1 septum polarity switch, or on the supply lines to these devices.  Because these devices will not be used in the near future, the line will be left valved off.  Eric Chapman has applied a defeat tag to the valve and made an entry on the Cyclotron Logbook.

The cause of the leak will be investigated further during the 2014 Fall Mini-Shutdown

On Friday August 8th Ron Kuramoto observed that the Meson Hall air amplifier cycle speed had increased.  Cycle time measurements were taken on Monday August 11th.  The lower amplifier was found to be cycling 17 times per minute, and the upper air amplifier 16 times per minute (approximately 4s cycle period).  The cycle period for both amplifiers was measured in June 2012 and found to be approximately 25s.  Note that a cycle is defined as the time between air exhaust events (this may actually only be 1/2 of a full cycle of the amplifier mechanism).  Both amplifiers were delivering air pressure approximately 134psi during the test.

The upper air amplifier was rebuilt in 2012 and has not been used since.  This test shows that the amplifiers are performing normally and that the increased cycle speed is due to a leak elsewhere in the system.  The location of the air leak will be investigated further on maintenance days or during the September mini shutdown.  The exhaust pipes of the amplifier assembly were warm to the touch due to the increased cycle speed, but not excessively so, and the current cycle speed is not thought to be of immediate concern.

The lower amplifier was left running and the exhaust pressure was turned down to 125psi to slightly reduce the cycle speed (it is not thought to be necessary to have air pressure above 120psi)

The T1-P2 (outlet) pressure transducer was replaced to address an erroneous signal.  The XTPAGE value now reads as expected.

The T2-P1 (inlet) transducer was also replaced to address a noisy (and worsening) signal.  Replacing the transducer seems to have had no effect on the XTPAGE values.  A possible controls issue is suspected and Jamie Cessford will be notified so that the matter is directed to the appropriate person.

No dose was incurred during the procedure.

The T2 target was moved from Position 4 to Position 0 at 11:15am so the profile monitor could be used during the development shift.

The T2 cooling package tripped regularly throughout the day on high flow while in position 0.

At 10:45pm the T2 target was returned to the previous running position: approximately 2mm above the nominal position 4 elevation (pot ratio fluctuating slightly around 0.799) so that beam could be run overnight.

The T2 target was exercised to position 0 around 6:15pm so that the T2 profile monitor could be used by operations.  The T2 water package was re-enabled when the target was in position 0, however this resulted in some trips of the cooling package.

Operations called around 7:45 to request return of the target to about 2mm above position 4.

The target position was re-set so that the position reading fluctuated roughly between 2.7 and 1.6 mm above position 4 (hovering roughly around 2.0 mm), and so the potentiometer reading was hovering roughly between 0.801 and 0.798.

The cooling package was re-started, and called operations to confirm the system was OK.

The T2 protect monitor was returned to the beam line using the shielded flask.  The alignment pin did not align with the hole in the vacuum flange, but could be aligned by hand by rotating the monitor assembly while it was hanging from the flange.  The protect monitor vacuum flange was bolted down and all services connected.  The T2 protect and profile monitors were checked by Dave Cameron (both OK).  Vacuum was pumped down and the cooling package started without incident.  After installation of the cap block the M20 beam blocker and the T2 profile monitor were exercised (both OK).  The updated target info, elevations, and beam spot measurements were delivered to the control room operators.

While the monolith was exposed the T2-MK2 target assembly lift hat was rotated slightly to match the orientation of T2 Protect and Col. A.  Hopefully this will improve the alignment for the next time the target is lifted with the flask.

When beam was started in mid-April, it was observed by Operations Group that with the beam centered vertically within the protect monitor, the profile monitor showed the beam spot to be approximately 6mm high.  The protect monitor was changed during the shut-down due to a failure in November 2013, so it was suspected that the new protect monitor was not aligned properly.  The beam spot appeared to be dangerously close to the top weld of the target window so it was decided the target assembly would be removed for inspection to visually confirm the suspected misalignment before increasing beam current.  Since start-up, a maximum beam current of approximately 30 micro-amps had been used on the 10cm beryllium target in position 4 which previously had not seen beam.

On Monday April 28th, the T2-MK2 target was transported from the beam line to the hot cell.   The target assembly was measured to be 298mSv/hr at 0.5m.  A clear beam spot was visible on the entrance and exit windows of the target approximately 6mm high of vertical center (photos attached).  Target and profile monitor elevation measurements were conducted, and confirmed those previously taken on the target by Tom Lyth.  This exercise confirmed that the position 4 target and profile monitor were aligned properly, and that misalignment of the protect monitor was the cause of the beam being high on the target.  It was then decided that the protect monitor assembly would be removed and a new monitor cassette installed to remedy the problem.

On Wednesday April 30th the T2-MK2 target was returned to the beam line and the protect monitor assembly was transported to the hot cell.  A maximum field of 15mSv/hr at 0.5m from the protect monitor was measured.  Elevation of the monitor cassette was checked, and was found to be 6.8mm above the elevation of the position 4 target.  The monitor cassette was removed from the assembly.  The cassette was approximately 100μSv/hr at 0.5m and 2mSv/hr on contact, with 200cpm measured from a swipe of the outside surfaces (not the inside plates).  The cassette was returned to Probes Group for possible use in the future.  A new monitor assembly was supplied by Scott Kellog on Friday morning with the 6.8mm adjustment.   The monitor was installed on the protect monitor assembly.  Elevation was checked and found to be within 0.5mm of the target elevation.  After installation an electrical check was performed by Dave Cameron and Mike Russel.

The protect monitor assembly is ready for installation in the beam line on Monday morning.

The T2-MK2 target (currently installed in the beamline) was moved from position 0 to position 4 (10cm Be) using the portable control box.
The T2 cooling package was started after moving the ladder.

There was no change in the BL1A vacuum.  The T2 expansion tank water level dropped slightly, as expected (around 1/16"), and was refilled to ~1/4" above the high level sensor top nut.

30 micro-amps of beam was put down BL1A and the T2 target and cooling system appear to be functioning normally.

 The T1 cooling package was started around 2pm today.  There was no change to the T1 volume vacuum during startup.

The package was inspected for water leaks: None found.

The demin loop flow (Q2) was adjusted to ~1.5gpm.

The target out water resistivity has reached 12 mega-ohm-centimeters after approximately 2.5 hours, indicating that the resin does not need to be changed.

T2 Profile Monitor actuation was tested today from the control room and all is working properly.  The position (IN / OUT) also displays correctly.
To my knowledge nobody has looked at this system, so it is a mystery why it suddenly started working.  This e-log will be updated if any explanation is found.

Actuation of the M20 beam blocker and T1 Profile Monitor were also tested and are working as expected.

The T2 monolith may now be covered.

Update:  A blown fuse in the control racks was replaced yesterday by Dave Cameron, which explains the change.

The resin can was removed from the T2 cooling package and transported to the warm cell.  A replacement can with fresh resin was then installed.

A 12' sling and the standard four bridle was used to prevent the crane hook block from hitting the mezzanine railing on the south side.

The T2 package was started and inspected for leaks - none found.

Although T2 resin can replacement was scheduled for the 2015 shutdown, a decline in conductivity (with beam off) was observed since the cooling package was started in mid-march.
(See attached XSTRIP plot showing gradual decline and then increase after resin can swap)

The T2 resin can will now be scheduled for replacement every 2 years (rather than 3).  It is still unknown whether the T1 resin can will also need replacing every 2 years.

The T2 profile monitor cannot be actuated from the control room.  When the solenoid valve that supplies air to the air cylinders is powered directly, the monitor moves in normally, therefore a controls issues is suspected.

Additionally the monitor position shows neither IN nor OUT.  This does not change when the monitor is at either limit.  Proper actuation of the target micro-switches has been confirmed.  The T2-MK2 target is currently installed, but this was also an issue in Nov/Dec 2013 with the T2-MK1 target installed.

Fault report #7276 has been filed, targeted at Cyclotron Controls Group.

The T2 P1 pressure transducer (Omega PX315-100GI) and T2 Q1 Proteus flow meter (0150SSTF3) have been replaced with new units to address issues with noisy signals.

Water detecting strips were installed around the T1 and T2 areas today with Doug Preddy and contractors from SMT Research.  Four sensors were installed at various locations on each cooling package.  Four sensors were installed on the top surface of the T2 monolith.  In addition, four sensors were routed for future installation at the T1 monolith (it is currently covered by blocks and requires cleaning and painting before sensor installation).

See the attached PDF for sensor ID numbers and placement specifics.

Photos are attached of the sensors installed around the T2 monolith and cooling package.

Bill Rawnsley completed an electrical check of the T2 profile monitor today from the Meson Hall mezzanine - All OK.  

The T2 cooling package was refilled and started.  There was no change to the T2 vacuum when starting the package.  While running, the package was visually inspected for water leaks - no leaks found.  

The lower air amplifier has also been started and is operating normally (the o-ring seal of the moisture trap on the amplifier inlet required cleaning and lubricating to seal properly).

David Cameron and Mike Russel completed an electrical check of the T2 protect monitor last week - All OK.

On the control panel on the mezzanine the "monitor in" lamp is currently illuminated which is incorrect (the monitor is currently out, and proper actuation of the micro switches was checked).  This problem started last year and did not seem to cause issues during 2013 running period.  

Operators were unable to move the T2 profile monitor to the in position from the control room.  This may be caused by a separate interlock that is not satisfied, or may be a new controls issue.  To confirm that the T2 target assembly is not at fault, power was supplied directly to the profile monitor air cylinder solenoid valve and the monitor traveled into position normally.  This will be investigated further before blocks are replaced.

After the monitor assembly was moved to the hot cell, electrical checks were performed by Bill Rawnsley which showed the same results as when tested in the beam line.  A new monitor was installed (supplied by Probes Group), the tests were repeated, with all results appearing normal.  The monitor assembly was transported to the beam line this morning and installed at the T2 monolith.  All cables were attached and the flange was bolted down.

Bill was notified by e-mail and it was suggested that he perform one final electrical check from the mezzanine before the monolith is covered.