On September 9th in the morning Ron Kuramoto noticed that the T2 cooling package expansion tank level was reading NOT OK for the warning level.  Looking at XSTRIP showed that the warning level tripped at approximately 1:50am on September 8th.  The tank was refilled at 4pm on September 9th to the usual level (1/2" above the nut on the trip level float).  The expansion tank level will be checked tomorrow and periodically throughout the week to determine if there is a water leak, and if so what the leak rate is.  Note that there has been no significant change in BL1A vacuum, so a water leak into the vacuum volume is not suspected.

 The BL1A tunnel was accessed today to assess a faulty reading from the T1 cooling package inlet pressure transducer (T1P1).  The line was first bled to remove gas bubbles which did not change the reading.  The transducer was replaced with a new unit (OMEGA PX315-100GI).  The pressure reading returned to the usual value of approximately 74psi (faulty reading had been steadily decreasing over the last few months and was at approximately 36psi).

The supply lines for all T1 and T2 pressure transducers in the BL1A tunnel were bled, and the expansion tanks topped up.

This transducer was last replaced in May 2010.

 The T2 cooling package was uncovered this morning and a leak was found coming from the Swagelok ball valve on the 3/4" target water outlet line.  The water was coming from the stem of the valve, indicating that water was leaking through the valve packing, rather than at the fittings for the valve.  The leak rate was approximately 1 drop every 5 seconds and a small puddle was visible on the top of the water reservoir.  The water package was drained (samples were taken and given to RPG).  After draining, the valve was replaced with a spare unit.  The cooling package was refilled and restarted.  The package was inspected while running and no visible water leaks were found.  The flow rate through the resin can was adjusted from approximately 1.7 to 1.5 gallons per minute.

 An end-to-end test was performed on the new T1 and T2 high temperature safety switches yesterday during the maintenance day.  The switch set points were lowered below the current temperature and it was confirmed that all warning messages and XTPAGE status changes occurred as expected.

 In preparation for Edi to vent the BL1A vacuum volume, the T1 and T2 cooling packages were both shut off.  They will remain off until vacuum is restored in BL1A in a few weeks time.  This afternoon, water will be purged from the T1 and T2 targets using compressed air so that stagnant water is not sitting in the targets for an extended period of time.

The air amplifier was also shut off and will remain off for the duration of the shutdown.  The lower of the two amplifiers was operating during the last running period.

Water was drained from the T1 and T2 cooling packages, samples taken, and delivered to RPG.

All filters on the T1 and T2 cooling packages were changed except for the T1 demin. line filter which is hard to access and could not be properly drained.  This filter will be changed next shutdown when the area is more accessible while the resin cans are being changed.

The field from each filter was measured after removal:

T1 Target Water Filter:  180μSv/hr @ 0.5m, 8.26mSv/hr on contact
T1 Collimator A Filter:   0.86μSv/hr @ 0.5m, 0.61μSv/hr on contact  (same as background)
T2 Target Water Filter:   185μSv/hr @ 0.5m, 10.5mSv/hr on contact
T2 Demineralizer Water Filter:   2.10μSv/hr @ 0.5m, 30.0μSv/hr on contact

9000cpm were measured on Isaac's coveralls after completing this work, most likely from laying on the concrete blocks next to the cooling packages to access the filters.  Keith's coveralls had 300cpm.  Both pairs were bagged and taken to active laundry.  The filter cartridges on Isaac's respirator had 700cpm.  RPG was notified and Danka will do a gamma spec on the respirator filters.

The T1 and T2 cooling package water will be released after approval from RPG.

Bill Rawnsley completed an electrical check of the T2 profile monitor today from the Meson Hall mezzanine - All OK.  

The T2 cooling package was refilled and started.  There was no change to the T2 vacuum when starting the package.  While running, the package was visually inspected for water leaks - no leaks found.  

The lower air amplifier has also been started and is operating normally (the o-ring seal of the moisture trap on the amplifier inlet required cleaning and lubricating to seal properly).

David Cameron and Mike Russel completed an electrical check of the T2 protect monitor last week - All OK.

On the control panel on the mezzanine the "monitor in" lamp is currently illuminated which is incorrect (the monitor is currently out, and proper actuation of the micro switches was checked).  This problem started last year and did not seem to cause issues during 2013 running period.  

Operators were unable to move the T2 profile monitor to the in position from the control room.  This may be caused by a separate interlock that is not satisfied, or may be a new controls issue.  To confirm that the T2 target assembly is not at fault, power was supplied directly to the profile monitor air cylinder solenoid valve and the monitor traveled into position normally.  This will be investigated further before blocks are replaced.

The T2 P1 pressure transducer (Omega PX315-100GI) and T2 Q1 Proteus flow meter (0150SSTF3) have been replaced with new units to address issues with noisy signals.

The resin can was removed from the T2 cooling package and transported to the warm cell.  A replacement can with fresh resin was then installed.

A 12' sling and the standard four bridle was used to prevent the crane hook block from hitting the mezzanine railing on the south side.

The T2 package was started and inspected for leaks - none found.

Although T2 resin can replacement was scheduled for the 2015 shutdown, a decline in conductivity (with beam off) was observed since the cooling package was started in mid-march.
(See attached XSTRIP plot showing gradual decline and then increase after resin can swap)

The T2 resin can will now be scheduled for replacement every 2 years (rather than 3).  It is still unknown whether the T1 resin can will also need replacing every 2 years.

 The T1 cooling package was started around 2pm today.  There was no change to the T1 volume vacuum during startup.

The package was inspected for water leaks: None found.

The demin loop flow (Q2) was adjusted to ~1.5gpm.

The target out water resistivity has reached 12 mega-ohm-centimeters after approximately 2.5 hours, indicating that the resin does not need to be changed.

The T1-P2 (outlet) pressure transducer was replaced to address an erroneous signal.  The XTPAGE value now reads as expected.

The T2-P1 (inlet) transducer was also replaced to address a noisy (and worsening) signal.  Replacing the transducer seems to have had no effect on the XTPAGE values.  A possible controls issue is suspected and Jamie Cessford will be notified so that the matter is directed to the appropriate person.

No dose was incurred during the procedure.

On 14Oct2014, the 1AT2 Collimator B water outlet temperature rose to 100^o C. 

Summary: Clarification of T2 water temperature readouts

Detail: This e-log entry by Doug Preddy is being made to clarify the T2 water package temperature read outs. The temperature read outs on the T2 water package have been switched. The read out for inlet temperature is actually the outlet temperature and the read out for outlet temperature is actually the inlet temperature. On Sunday Oct 19 2014 at 19:45, the T2 water outlet Temp (T2) started to climb giving warning messages at 20:05. Operators tried to decrease the BL1A current, but the temp. continued to climb and eventually tripped the beam off. Operators defined BL1A OFF, and continued to inject to BL2A only. They tried to contact I. Earle, but were unable to. They left message with D. Preddy. The T2 temp. stayed at 45 deg., operators tried to reset the water package and start it, but it tripped almost right away, and increased the temp. by 0.5 deg. Beam was left off. On Monday Oct. 20 Ron Kuramoto found a replacement thermister. It was wired in to the BOP over the south catwalk. The T2 outlet temperature immediately dropped to ambiant. From this test it was concluded that the installed thermistor on the T2 outlet water pipe had failed. The wiring was restored to the original configuration. It was decided to switch T2 Water Inlet Temp and Outlet Temp sensors at the T2 electronics rack to provide a trip on the Inlet Water temperature instead of the outlet Water Temperature. As the T2 water package runs in a closed loop, the outlet temperature signal has been swapped with the inlet temperature by Doug P. and Tony T. This will provide thermal protection (trip at 45C) in addition to the T2 Target high limit bit (trip at 60C). The water flow to this circuit is also monitored and will trip if the flow drops. This action was approved in a meeting with Doug Preddy, Ron Kuramoto, Grant Minor, Yuri Bylinski, Lia Merminga, and Bob Laxdal present on Monday Oct 20th. There is a second, redundant temperature sensor on the outlet of the target water which was installed recently as a safety upgrade to protect the heat exchanger and meet CSA pressure vessel code, see dwg TBP0902. There is a controller for this sensor with a hard-trip switch for the cyclotron, and a temperature read-back display on the Meson Hall mezzanine racks. This read-back should be monitored by operations once per shift while beam is on target to ensure that outlet water temperature is stable and at an expected value. Sensor cabling swapped at Meson Hall, south mezzanine, bay 1, B.O.P. #3. Sensor T1, cable #12102 moved to TB1, terminals 3 & 4. Sensor T2, cable #12105 moved to TB1, terminals 1 & 2. Defeat #7131 removed. Defeat# 7133 Fault report #7722   Will look into replacing the faulty Thermister in the 2015 major shutdown.

The following tasks have been performed at T1:

- T1 monolith painted using grey two part epoxy paint
- Video inspection taken while package running. Small leak from both valves on H.E. secondary side (video on Isaac's PC, too large to upload)
- Cooling package sampled and drained, drain valve closed
- All reservoir o-rings changed
- All Hansen fitting o-rings changed (4x #6 size, 2x #4 size)
- All three cuno filters and filter housing o-rings changed (used filters 2mSv/hr on contact, temporarily stored on blocks by T1 package)
- Failed T1 P1 (inlet) pressure transducer replaced
- Supply and return valves for T1 CuALCW closed in tunnel to allow leaking valves to be replaced

- New resin can installed at T1 (this note added oct 15 2015 - was previously undocumented)
 

The following tasks have been performed at T2:

- Video inspection taken while package running. Documented leaks mentioned in E-Log #133 (video on Isaac's PC, too large to upload)
- Cooling package sampled and drained, drain valve closed

RPG has analyzed the water samples and given approval for water from both packages to be released.

Both defeats applied to T2 cooling package thermistors were removed today (Col. B interlock was jumpered, and the target water inlet and outlet signals had been swapped).  It was confirmed that all thermistors are now wired correctly.  The readback of both target water thermistors is currently unstable.  Inspection of the Lemo connectors showed that the internal plastic parts of all connectors were damaged or missing.  If the T1/T2 controls upgrade proceeds this shutdown, then all thermistors will be replaced with thermocouples, so after a decision is made on that project (sometime in Feb 2015) these thermistors will either be repaired or replaced.

- Both 3/4" ball valves on the T1 cooling package target water circuit were rebuilt, leak tested with house air pressure, and reinstalled

- The 3/4" ball valve on the T1 heat exchanger secondary side was replaced with a new unit

- The 3/4" needle valve on the T1 heat exchanger secondary side was rebuilt, leak tested with house air pressure, and reinstalled

- The T1 cooling package was refilled and restarted using a bypass across the inlet/outlet Hansen fittings

- The package was inspected while running and no leaks were visible (video survey taken with file stored on Isaac's PC)

- All T1 Cooling Package MRO activities are now complete

- T2-MK2 target (currently in the T2 monolith) purged with air

- All cooling package reservoir o-rings changed
- Target water line #6 Hansen o-rings changed at both ends
- M8 BB / Col. A supply hose Hansen o-rings changed at both ends

- Target water and demin circuit Cuno filters changed
- Used filters double-bagged, labeled, and stored on 1A blocks north of boot box area

- The following Swagelok ball valves removed for service:  Tgt return (X2), Sec return (X2), Demin supply, Demin return (X2), Col A return, Col B return

*** Four #4 Hansen o-rings on package panel still require changing (size was not available from stores)

The following work was done today at the T2 cooling package:

- Remainder of #4 Hansen o-rings replaced  (see E-log from Jan 30, 2015)
- Installed the following ball valves which have been rebuilt & tested with 100psi air: target outlet 3/4" (x2), resin can outlet filter isolation 1/2" (x2), heat exchanger secondary side outlet 3/4" (x2), collimator A outlet 3/4", collimator B outlet 3/8"
- Installed the following ball valve which has been tightened & tested with 100psi air (no rebuild kit available): resin supply side 1/2"
- Disassembled the target water inlet thermistor probe assembly to check compatibility with thermocouple probes purchased for T1/T2 controls upgrade
- Misplaced a fitting that the probe inserts through, plugged thermistor port as a temporary measure (the outlet thermistor still functions as machine protection)
- Filled cooling package and opened CuALCW supply and return valves to T2 area in BL1A tunnel
- Started package, inspected for leaks, tightened various Swagelok fittings to stop minor leaks
- Tightened valve stem packing compression nut on Col.A/M8BB shut-off valve and Col B. shut-off valve on cooling package front panel to stop drip leaks from the valve stems
- Inspected again for leaks while cooling package running:  none observed

The T2 cooling package is ready for a interlock response timing test which will be performed tomorrow (part of T1/T2 controls upgrade).
Before the T2 cooling package is ready for operation, the target water inlet thermistor must be reinstalled, and the thermistor connectors must be replaced (if the thermistors are not replaced with thermocouples as part of the controls upgrade)

Total dose:

Isaac - 0.08mSv
Keith - 0.01mSv
 

The Inlet (T1), Outlet (T2), Col.A (T6), and Col. B (T7) thermistors at the T2 cooling package all gave intermittent faulty readings at various times between Oct - Dec 2014.  It is suspected that this was caused by a water leak in the area causing the terminals to get wet.  Inspection of the sensors revealed that they are in poor shape and that the readouts sometimes fluctuate when the probe or connectors are handled.  All four thermistor probes have been replaced.  The plastic interior of the Lemo connectors were found to be crumbling, and were replaced with crimp connections.  All T2 thermistors now appear to be functioning normally.  The T2 cooling package was started (with a bypass instead of the target assembly).  Flow through Collimators A and B has also been started.  The T2 package was inspected while running and no leaks were found.

Cyclotron Fault #7722 has been returned.

Both cooling packages are running steadily with the targets in Position 0.  

The demin flow (Q2) for both was tuned to approximately 1.5USgpm.  

A defeat was applied to T2Q1 flow to prevent high level nuisance trips.  This will remain in effect for the 2015 running period.

T2P1 Sensor (water inlet pressure) has a noisy signal.  This was not improved by replacing the sensor.  It is not expected to cause issues during operation and will be addressed (if still present) after the 2016 upgrade to PLC/EPICS control.

The expansion tank for both packages was topped up to approximately 1/4" above the top nut of the warning level sensor.