The following work was done today at the T2 cooling package:

- Remainder of #4 Hansen o-rings replaced  (see E-log from Jan 30, 2015)
- Installed the following ball valves which have been rebuilt & tested with 100psi air: target outlet 3/4" (x2), resin can outlet filter isolation 1/2" (x2), heat exchanger secondary side outlet 3/4" (x2), collimator A outlet 3/4", collimator B outlet 3/8"
- Installed the following ball valve which has been tightened & tested with 100psi air (no rebuild kit available): resin supply side 1/2"
- Disassembled the target water inlet thermistor probe assembly to check compatibility with thermocouple probes purchased for T1/T2 controls upgrade
- Misplaced a fitting that the probe inserts through, plugged thermistor port as a temporary measure (the outlet thermistor still functions as machine protection)
- Filled cooling package and opened CuALCW supply and return valves to T2 area in BL1A tunnel
- Started package, inspected for leaks, tightened various Swagelok fittings to stop minor leaks
- Tightened valve stem packing compression nut on Col.A/M8BB shut-off valve and Col B. shut-off valve on cooling package front panel to stop drip leaks from the valve stems
- Inspected again for leaks while cooling package running:  none observed

The T2 cooling package is ready for a interlock response timing test which will be performed tomorrow (part of T1/T2 controls upgrade).
Before the T2 cooling package is ready for operation, the target water inlet thermistor must be reinstalled, and the thermistor connectors must be replaced (if the thermistors are not replaced with thermocouples as part of the controls upgrade)

Total dose:

Isaac - 0.08mSv
Keith - 0.01mSv
 

The Inlet (T1), Outlet (T2), Col.A (T6), and Col. B (T7) thermistors at the T2 cooling package all gave intermittent faulty readings at various times between Oct - Dec 2014.  It is suspected that this was caused by a water leak in the area causing the terminals to get wet.  Inspection of the sensors revealed that they are in poor shape and that the readouts sometimes fluctuate when the probe or connectors are handled.  All four thermistor probes have been replaced.  The plastic interior of the Lemo connectors were found to be crumbling, and were replaced with crimp connections.  All T2 thermistors now appear to be functioning normally.  The T2 cooling package was started (with a bypass instead of the target assembly).  Flow through Collimators A and B has also been started.  The T2 package was inspected while running and no leaks were found.

Cyclotron Fault #7722 has been returned.

Both cooling packages are running steadily with the targets in Position 0.  

The demin flow (Q2) for both was tuned to approximately 1.5USgpm.  

A defeat was applied to T2Q1 flow to prevent high level nuisance trips.  This will remain in effect for the 2015 running period.

T2P1 Sensor (water inlet pressure) has a noisy signal.  This was not improved by replacing the sensor.  It is not expected to cause issues during operation and will be addressed (if still present) after the 2016 upgrade to PLC/EPICS control.

The expansion tank for both packages was topped up to approximately 1/4" above the top nut of the warning level sensor.

- All T1 and T2 values on XTPAGE were stripped back to April 1, 2015:  All systems appear to be functioning normally, nothing unusual observed

- Expansion tank levels checked visually:  T1 level at 1/2" above top sensor nut, T2 level at 1/4" below top sensor nut --> filled to 1/2" above

- Lower air amplifier in use; cool to touch; approximately 25s between each cycle (when air is exhausted from device)

 The T1 and T2 cooling packages were drained to the BL1A holding tank on Jan 8 2016.  The volumes were 100L and 80L respectively.  Before starting there was 350L of water in the holding tank from the 500MeV system which Steve Sapriken drained in September 2015.  The T1 and T2 systems were sampled and delivered to RPG.  After receiving permission from RPG to release all 3 portions of water it was pumped into the city sewer on Jan 18, 2016.

The following work was performed at the T1 cooling package and target between January 18-21, 2016:

- Cuno water filters removed (3), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, and Collimator A solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 each)
- Target side #6 Hansen QD fittings replaced because internal seal had failed and fittings were leaking when disconnected (2)
- Profile monitor copper air supply line leak was repaired
- All air QD fittings (6 total) for profile monitor actuation were replaced, the air restricting valves were tuned, and the flexible hose assemblies were checked for gross air leaks
- "T8" temp sensor was found to be located on the CuALCW common return line, not on the Col A supply line as labeled on XTPAGE.  Position was swapped

Total dose for this work was 0.17mSv for Isaac Earle, and 0.10mSv for Keith Ng

The following work was performed at the T2 cooling package and target between Jan 25 - Feb 3, 2016:

- Cuno water filters removed (2), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, Collimator A solenoid, and Collimator B solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 sensors total)
- Target side #6 Hansen QD fittings replaced
- All air QD fittings (6 total) for profile monitor actuation were replaced, some of the flexible hosing was replaced (cracking), and the flexible hose assemblies were checked for gross air leaks
- Top surface of the cooling package resin can support platform was painted (was rusty)
- Resin can swapped (previous resin swap was during Shutdown 2014)
- Target water check valve was removed and tested (working as expected), then reinstalled

 The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

The following work was completed on the T1 cooling package:
- water drained to BL1A holding tank
- all resevoir o-rings changed
- all cuno filters changed:
     M15 permanent magnet filter < 5uSv/hr at 0.5m, dark grey colour
     Target inlet filter 210uSv/hr at 0.5m, light brown colour
     Resin filter, <5uSv/hr, looks clean and new
- package re-filled and started
- demin circuit flow tuned to 1.0gpm
- #6 female Hansen fittings at target end of flexible hose swapped for previously used SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion on the inside)
 

The T2 cooling package was also drained to the 1A holding tank.
After approval from RPG the holding tank water was released.

Note: the appearance of the used cuno filters is consistent with previous years.

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 The following work has been completed on the T2 cooling package:

- T2 system drained to South TNF holding tank (completed in Jan 2017, water already released)
- Target water and demin. circuit Cuno filters changed
- Both female Hansen fittings at target end of flexible hose replaced with new SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion inside)
- Small leak on pipe thread fitting near target water supply solenoid repaired
- Reservoir o-rings replaced
- System filled and re-started, no leaks observed, no drop in water level over 48 hour run period
- Demin. circuit flow rate tuned to 1.0gpm

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

 The following work was performed on the T1 and T2 cooling packages during the 2018 shutdown:

- T1 and T2 cooling packages both sampled and drained to South TNF holding tank
- Water released after testing and approval from RPG

- T1 high active and M15 filters changed (demin. filter to be changed next year)
- Replaced all T1 Proteus paddle wheels, shafts, and o-rings
- Replaced T1 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T1 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T1 package filled and started, no change to T1 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

- T2 high active filter changed (demin. filter to be changed next year)
- Replaced all T2 Proteus paddle wheels, shafts, and o-rings
- Replaced T2 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T2 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T2 resin can changed;  old resin can flushed with water then air in Warm Cell and left to decay
- T2 package filled and started, no change to T2 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

The cooling packages are functioning normally and are ready for the running period.

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.