The T1 cooling package inlet pressure (P1) has been dropping since approximately May 5, 2013 from the regular operating pressure of approximately 75psi to the present value of 47psi.  An increase in the rate of decrease started around August 5th as well as increased noise in the signal.  Plots attached below.  No corresponding change in the outlet pressure, and no change in the water flow rate (also no change in inlet and oulet temperatures) leads to the conclusion that the transducer is failing or gas is trapped in the lines leading to the transducers in the BL1A tunnel.

Jamie Cessford and Ron Kuramoto are aware of the situation.  The issue will be addressed during the September mini-shutdown.

 On September 9th in the morning Ron Kuramoto noticed that the T2 cooling package expansion tank level was reading NOT OK for the warning level.  Looking at XSTRIP showed that the warning level tripped at approximately 1:50am on September 8th.  The tank was refilled at 4pm on September 9th to the usual level (1/2" above the nut on the trip level float).  The expansion tank level will be checked tomorrow and periodically throughout the week to determine if there is a water leak, and if so what the leak rate is.  Note that there has been no significant change in BL1A vacuum, so a water leak into the vacuum volume is not suspected.

 The BL1A tunnel was accessed today to assess a faulty reading from the T1 cooling package inlet pressure transducer (T1P1).  The line was first bled to remove gas bubbles which did not change the reading.  The transducer was replaced with a new unit (OMEGA PX315-100GI).  The pressure reading returned to the usual value of approximately 74psi (faulty reading had been steadily decreasing over the last few months and was at approximately 36psi).

The supply lines for all T1 and T2 pressure transducers in the BL1A tunnel were bled, and the expansion tanks topped up.

This transducer was last replaced in May 2010.

 The T2 cooling package was uncovered this morning and a leak was found coming from the Swagelok ball valve on the 3/4" target water outlet line.  The water was coming from the stem of the valve, indicating that water was leaking through the valve packing, rather than at the fittings for the valve.  The leak rate was approximately 1 drop every 5 seconds and a small puddle was visible on the top of the water reservoir.  The water package was drained (samples were taken and given to RPG).  After draining, the valve was replaced with a spare unit.  The cooling package was refilled and restarted.  The package was inspected while running and no visible water leaks were found.  The flow rate through the resin can was adjusted from approximately 1.7 to 1.5 gallons per minute.

 Complete overhaul of all manipulators in the Remote Handling Hot Cell Lab in the Meson Hall was performed by Peter Dudley from Central Research Laboratories between October 28 - November 8th.  Overhaul included replacing all tapes and cables, disassembling, cleaning, lubricating, and rebuilding the master and slave wrist assemblies, adjusting motion locks, and other repairs/maintenance as required.  All 6 manipulator arms in the lab are now fully functional.  Pete reported that some cables from the manipulators in the west side hot cell were showing heavy wear, and cables from the other arms showed mild to moderate wear.

The highest amount of contamination was 1600cpm from a wet wipe on the inner slave boom on the west manipulator in the west hot cell.  Pete received a total dose of 0.10mSv as measured from DRDs worn during this period.

The temporary boot box area in the lab has been disassembled.  No contamination was found on the lab floor, tools, or manipulator removal cart.

T2 target water leak was identified earlier this morning.

Isaac Earle, Ron Kuramoto and Keith Ng co-ordinated Meson Hall blocks removal and target flask transport.

Target assembly was removed from B/L approximately 5pm.

Vacuum "suck tube" (capacity ~ 10L) for removing water in B/L was prepared by Isaac Earle.

Approximately 5 L of water was seen in the bottom of the T2 target ladder vacuum tube enclosure, referenced by the ladder assembly locating spigot (see attached photo).

Most of the water was removed by Grant Minor, Ron Kuramoto, and Keith Ng using the suck tube, job finishing around 8pm November 22nd.

About 1/8" water remains in the bottom of the vacuum tube enclosure, which will have to be left to evaporation overnight and vacuum pumps (see attached photo).

Full suck tube was tagged and left on the north-east side of the Meson Hall blocks near TNF.

Spare target replacement, pump down and block replacement to happen tomorrow, Isaac Earle and Grant Minor to co-ordinate w/ support from Plant Group and Vacuum Group.

- T2-MK1 target transported from the storage area by Isaac and Grant and installed at the T2 monolith.  All services were connected.  Storage pit covered.

- Edi began pumping approximately 10am, and was not able to pump below ~20 Torr.  Dimo came in to assist.

- The vacuum interlock was bypassed and the cooling package started to check for water to vacuum leaks:  No change in vacuum.

- Methanol was sprayed around the flange area to look for leaks, and a leak was found on the north edge.  A piece of black rubber was visible and it appeared that the o-ring was out of position and pinched by the flange.

- The beamline was vented, the target lifted approximately 1" w/o using the flask. The rubber was found to be a pad from the storage pit stand-offs that had become stuck on the underside of the T2-MK1 target flange.  This was removed.  The o-ring was in the proper position and undamaged.

- The target was lowered, and pumping restarted.  Only  around 4 Torr was achieved.

- The beam line was vented once more, and the target lifted approximately 6".  Both flanges and the o-ring were cleaned and inspected.  A small amount of dirt or grease was found on the wypall.

- The target was lowered and pumping restarted.  The beam line pumped down at the normal rate.  At approximately 200 mTorr a helium leak check was performed.  The top of the T2 monolith was flooded with helium and there was no response from the leak detector.

- It was observed that the "MONITOR IN" lamp was illuminated on the T2 control panel.  Isaac confirmed that the monitor was fully out and tested the limit switches, no change in lamp status.  Confirmed with control room that they have no warnings or interlocks from T2 that would prevent them from running beam.  It's possible that this lamp has always been illuminated even with the monitor out.

- Plant group began covering the T2 area at approximately 6pm

- The M20 beam blocker was tested after installing cap block, and actuated properly.

- The T2 target ladder was moved from position 0 to position 3 using the portable control box

- T2 expansion tank was topped up to 1/2" above the nut on the warning level sensor

- At 6:30pm 1ACG4 reads 150 mTorr and dropping slowly

 The T2-MK2 target in the hot cell was flushed with water and then purged with air so that stagnant water is not left sitting in the target.  Replacement of the failed 10cm Be cassette will take place in January 2014.

 The leaking 10cm Be target was removed from position 3 on the T2-MK2 target ladder and the water ports were blanked-off with 3/8" Swagelok plugs.  A 10cm Be target remains in position 4.  The target was pumped down using the cold trap and a helium leak check was performed.  All target locations on the ladder were found to be leak tight.  The delta seal locations and fittings above the vacuum flange were also leak tight, however small leaks were discovered at the weld at the lower end of each bellows, details below.

Using the old leak detector in the Meson Hall HC Lab, at baseline level 18 on 10X scale with the target oriented as when installed in the beamline, using 1/2 second application of helium at ~3psi:
East side bellows: response up to 90 on 10X scale, returned to baseline in ~5s
West side bellows: response up to 30 on 50X scale, returned to baseline in ~5s

Based on previous helium leaks of this size it is expected that the bellows are still water leak tight.  Replacement of the bellows will not be considered unless a water leak is found.

After an electrical check is performed on the profile monitor by Probes Group, this target will be returned to the beam line and pumped down to check for water leaks.  If water leak tight, the T2-MK2 target will be used for operation in 2014.
 

Photos are attached of the failed 10cm Be target, as well as the T2-MK2 target assembly with leak locations indicated.

 An electrical check was performed on the T2-MK2 profile monitor by Bill Rawnsley.  All signals produced the expected response, and Bill approved the monitor for use.  Bill has saved screenshots of the signal profile.

The target ladder position micro-switches and profile monitor in/out limit switches were also tested: all OK.

The T2-MK2 target is now ready to be transported to the beam line.

 A water leak developed this morning from a city water supply line at a hose connection near the T2 expansion tank.  Water was observed to be dripping off the mezzanine onto the plastic sheet covering the T2 cooling package.  The leak has been stopped and the leaky fitting is being repaired by Carston Frasden.

The T2 cooling package was inspected: small amount of water on the top of the plastic sheet, no water visible below the plastic, and all package components were dry.
The T2 expansion tank level was checked: no change.

 The T2 Protect Monitor was transported from the beam line to the hot cell this morning using the transport flask.  The move went smoothly.  The only difficulty was locating the yellow flask alignment frame to the T2 monolith as the T2 protect holes on the yellow frame are tighter than for other devices.  It was necessary to remove four screws that secure the top flange of the T2 protect monitor assembly to the flange on the monolith.  These screws are stored on the top of the monolith in a labelled bag.  The T2 protect o-ring was in good condition, but was replaced anyways, and a blank-off was installed over the hole so that vacuum can be pulled on the T2 volume.  A maximum field of 15mSv/hr was measured at 0.5m from the monitor while lifting.

Probes Group may now begin testing / repair of the monitor.  Bill Rawnsley has been notified.

Photos attached.

Update (March 19, 2014):  Bill Rawnsley thinks that the damage was caused by mis-steered beam (not from water).  He points to the evidence of heating in the attached photo.

After the monitor assembly was moved to the hot cell, electrical checks were performed by Bill Rawnsley which showed the same results as when tested in the beam line.  A new monitor was installed (supplied by Probes Group), the tests were repeated, with all results appearing normal.  The monitor assembly was transported to the beam line this morning and installed at the T2 monolith.  All cables were attached and the flange was bolted down.

Bill was notified by e-mail and it was suggested that he perform one final electrical check from the mezzanine before the monolith is covered.

The T2 profile monitor cannot be actuated from the control room.  When the solenoid valve that supplies air to the air cylinders is powered directly, the monitor moves in normally, therefore a controls issues is suspected.

Additionally the monitor position shows neither IN nor OUT.  This does not change when the monitor is at either limit.  Proper actuation of the target micro-switches has been confirmed.  The T2-MK2 target is currently installed, but this was also an issue in Nov/Dec 2013 with the T2-MK1 target installed.

Fault report #7276 has been filed, targeted at Cyclotron Controls Group.

When beam was started in mid-April, it was observed by Operations Group that with the beam centered vertically within the protect monitor, the profile monitor showed the beam spot to be approximately 6mm high.  The protect monitor was changed during the shut-down due to a failure in November 2013, so it was suspected that the new protect monitor was not aligned properly.  The beam spot appeared to be dangerously close to the top weld of the target window so it was decided the target assembly would be removed for inspection to visually confirm the suspected misalignment before increasing beam current.  Since start-up, a maximum beam current of approximately 30 micro-amps had been used on the 10cm beryllium target in position 4 which previously had not seen beam.

On Monday April 28th, the T2-MK2 target was transported from the beam line to the hot cell.   The target assembly was measured to be 298mSv/hr at 0.5m.  A clear beam spot was visible on the entrance and exit windows of the target approximately 6mm high of vertical center (photos attached).  Target and profile monitor elevation measurements were conducted, and confirmed those previously taken on the target by Tom Lyth.  This exercise confirmed that the position 4 target and profile monitor were aligned properly, and that misalignment of the protect monitor was the cause of the beam being high on the target.  It was then decided that the protect monitor assembly would be removed and a new monitor cassette installed to remedy the problem.

On Wednesday April 30th the T2-MK2 target was returned to the beam line and the protect monitor assembly was transported to the hot cell.  A maximum field of 15mSv/hr at 0.5m from the protect monitor was measured.  Elevation of the monitor cassette was checked, and was found to be 6.8mm above the elevation of the position 4 target.  The monitor cassette was removed from the assembly.  The cassette was approximately 100μSv/hr at 0.5m and 2mSv/hr on contact, with 200cpm measured from a swipe of the outside surfaces (not the inside plates).  The cassette was returned to Probes Group for possible use in the future.  A new monitor assembly was supplied by Scott Kellog on Friday morning with the 6.8mm adjustment.   The monitor was installed on the protect monitor assembly.  Elevation was checked and found to be within 0.5mm of the target elevation.  After installation an electrical check was performed by Dave Cameron and Mike Russel.

The protect monitor assembly is ready for installation in the beam line on Monday morning.

The T2 protect monitor was returned to the beam line using the shielded flask.  The alignment pin did not align with the hole in the vacuum flange, but could be aligned by hand by rotating the monitor assembly while it was hanging from the flange.  The protect monitor vacuum flange was bolted down and all services connected.  The T2 protect and profile monitors were checked by Dave Cameron (both OK).  Vacuum was pumped down and the cooling package started without incident.  After installation of the cap block the M20 beam blocker and the T2 profile monitor were exercised (both OK).  The updated target info, elevations, and beam spot measurements were delivered to the control room operators.

While the monolith was exposed the T2-MK2 target assembly lift hat was rotated slightly to match the orientation of T2 Protect and Col. A.  Hopefully this will improve the alignment for the next time the target is lifted with the flask.

The T1-P2 (outlet) pressure transducer was replaced to address an erroneous signal.  The XTPAGE value now reads as expected.

The T2-P1 (inlet) transducer was also replaced to address a noisy (and worsening) signal.  Replacing the transducer seems to have had no effect on the XTPAGE values.  A possible controls issue is suspected and Jamie Cessford will be notified so that the matter is directed to the appropriate person.

No dose was incurred during the procedure.

On Friday August 8th Ron Kuramoto observed that the Meson Hall air amplifier cycle speed had increased.  Cycle time measurements were taken on Monday August 11th.  The lower amplifier was found to be cycling 17 times per minute, and the upper air amplifier 16 times per minute (approximately 4s cycle period).  The cycle period for both amplifiers was measured in June 2012 and found to be approximately 25s.  Note that a cycle is defined as the time between air exhaust events (this may actually only be 1/2 of a full cycle of the amplifier mechanism).  Both amplifiers were delivering air pressure approximately 134psi during the test.

The upper air amplifier was rebuilt in 2012 and has not been used since.  This test shows that the amplifiers are performing normally and that the increased cycle speed is due to a leak elsewhere in the system.  The location of the air leak will be investigated further on maintenance days or during the September mini shutdown.  The exhaust pipes of the amplifier assembly were warm to the touch due to the increased cycle speed, but not excessively so, and the current cycle speed is not thought to be of immediate concern.

The lower amplifier was left running and the exhaust pressure was turned down to 125psi to slightly reduce the cycle speed (it is not thought to be necessary to have air pressure above 120psi)

 During the maintenance day today, amplified air to the T1 area was valved off in an attempt to pin point the location of the air leak.  The air amplifier cycle rate returned to normal with the T1 line valved off, indicating the leak is somewhere on the T1 profile monitor air cylinders, the T1 septum polarity switch, or on the supply lines to these devices.  Because these devices will not be used in the near future, the line will be left valved off.  Eric Chapman has applied a defeat tag to the valve and made an entry on the Cyclotron Logbook.

The cause of the leak will be investigated further during the 2014 Fall Mini-Shutdown

On Sep. 3^rd during the 6 hr maintenance day, Doug Preddy & I went into the 1A tunnel under “controlled access” to investigate a water leak that triggered the Water Detection Sensors.  It was also noticed that the Cu-ALCW tank level was losing water.

Water was found to be leaking down the south wall chase where the T2 water package goes up the wall. Drip rate was ~1 drip every 15 seconds.

In the late afternoon of Sep. 9^th;

Day one of the 1.5 day maintenance, the tunnel was surveyed by Fiona H. see RPG for field map & floor swipe results. 

Doug, John McKinnon & I set up a plastic sheet with a funnel connected to a 8ft. lg. poly-flow tubing to direct leaking water from the T2 into the south gutter.  This set up was left over night for the floor to dry up and resume the water leak investigation. See photos.

On Sep. 10^th;

@ 08:30 about 70% of the 1A tunnel floor had dried up and the diverted water into the south side gutter had travelled to the T1 area.

The T2 cooling package was then turned off followed by valving off the Cu-ALCW feeding the T2 cooling package. A few more strips of red tuck tape were added to secure the temporary plastic diversion sheet.

@ 11:30 the x-strip graph (T/W  MS 230) of the Cu-ALCW tank level showed that the leak had stopped and leveled off.

@ 13:15 the 1A tunnel floor had dried up with no water seen dripping from the vertical chase.  Doug also mentioned that the “water sensors” that were activated reflected this. The Cu-ALCW was then valved back on to the T2 cooling package and within 3 minutes water began dripping from the south wall vertical service chase which now appeared to about  1 drip/sec.

Fiona took a water sample (20 ml vial) from the drips and also found ~ 1000 c/m (44-2 monitor) on the tacky mat just on the inside of 1A tunnel door.  The 1A Tunnel and South TNF were then locked-up by Derek H., Violeta T. & Doug.   The T2 cooling package was turned back on ok.  Will try to repair this water leak during the Oct. mini shutdown.