Replacement of the power supply fans for the T1 Power Supply, T2 Power Supply, as well as an additional power supply in rack 1A was completed this morning.  The boxes were blown out with air to remove dirt and dust.  The inside of all boxes were inspected and no evidence of melting or burning was found. 

After installation the output voltage was measured for each power supply using a multimiter on first the DC setting, then on the AC setting to check for fluctuation (as recommended by Erwin Klassen).  Results below: 

T1 Power Supply:  24.2V DC, 0.001V AC 
T2 Power Supply: 23.6V DC, 0.002V AC 
Other Power Supply: 24.2V DC, 0.004V AC 

The fans are all running properly, the power supplies are cool to the touch, and the T1 and T2 cooling packages are up and running. 

The following work was completed between Jan 16 - Feb 19, 2019:

• T1 and T2 cooling packages drained to South TNF holding tank, water sampled and tested, released after approval from RPG
• T1 resin can swapped
• All Cuno filters changed at T1 (3 total)
• T1 inlet (T1CSPGIN) and outlet (T1CSPGOUT) pressure gauges replaced (archived data on EPICS indicated both starting to fail)
• T2 target water Cuno filter replaced
• All T1 and T2 solenoid valves changed to new models without power management circuit which will hopefully solve the problem of frequent failures (SASP0670)
  Parker brass body valve 73218BN4UN00 with solenoid C222C2 solenoid used for T1CSSVSEC and T2CSSVSEC
  STC stainless steel body valve 2S160-1/2-2-D with solenoid 2W200C-2-D used for T1CSSVTGT, T1SVCOL, T2CSSVTGT, T2SVCOLA, T2SVCOLB
  Necessary changes to PLC BOP wiring made by Tony Tateyama
  All new valves confirmed operational and leak tight
  Cyclotron Fault #11955 returned
• T1 and T2 profile monitor air cyclinder flow control valves replaced with new models tuned to ~1/8 turn open.  Part is Rego F125B (see PO #3044305)
• T1-MK2 profile monitor air cylinders replaced (both were leaking through the top seal)
• T1 and T2 profile monitor actuation checked - both move in and out slowly and smoothly
• M20 BB actuation checked - movement is slow and smooth
• T1 and T2 ladder movement checked - both ok
• T1-MK2 male Hansen fittings replaced as well as female Hansens which connect to them (these are the last remaining Hansen fittings to be replaced at T1/T2)
• Both cooling packages filled and started, now running smoothly.  Inspected for leaks (none found).  No change to 1A vacuum when packages started
   

Note 1: The T1 package water pump is slightly noisy when running (it has been this way since the replacement pump was installed last year) - Will investigate further at a later date

Note 2: In the week of Feb 11-15 Vacuum Group pressurized the T1 target cooling circuit with 30psi helium to try to find the vacuum load on the T1 volume.  This was done twice with the beamline vented for a few hours in between in case water had frozen at a leak location. The conclusion after testing is that the T1 cooling water is not the source of the vacuum load at T1.  Helium was also sprayed all around the top of the T1 monolith (T1 target and M15 permanent magnet) - there was no response on the leak detector

My total dose for this work was 0.15mSv

Both cooling packages are running steadily with the targets in Position 0.  

The demin flow (Q2) for both was tuned to approximately 1.5USgpm.  

A defeat was applied to T2Q1 flow to prevent high level nuisance trips.  This will remain in effect for the 2015 running period.

T2P1 Sensor (water inlet pressure) has a noisy signal.  This was not improved by replacing the sensor.  It is not expected to cause issues during operation and will be addressed (if still present) after the 2016 upgrade to PLC/EPICS control.

The expansion tank for both packages was topped up to approximately 1/4" above the top nut of the warning level sensor.

Upon uncovering the T1 and T2 areas to prepare for regular cooling package maintenance, it was discovered that multiple ball valves were leaking. Primarily the Swagelok SS-45S12 parts, but some SS-45S8 parts have failed as well. There were small puddles of water underneath many of the valves. It appears that the leak is due to a compromised seal. It is suspected that these parts failed from radiation exposure. They have PTFE packing material which is highly susceptible to decomposition under irradiation, and as far as we can tell they have not really been repaired/replaced (aside from a written note on one saying the seal was changed in 1997).

It is recommended to add the regular repair/replacement of these valves to the annual shutdown MRO checklist.

Once replaced, this e-log will be updated to document which ones were replaced.

On Jan. 13, the T1 and T2 cooling package water reservoirs were drained to the BL1A holding tank. Water samples were obtained for each system and provided to RPG for analysis.

Prior to starting draining, the BL1A holding tank water level was approximately 330L. After draining, the water level was approximately 550L (checked Jan. 16).

 BL1A was started yesterday and brought up to 10μA.  A problem was noticed with the T1 protect monitor (the right and bottom plates were not showing any response when the beam was swept across the monitor).  The protect monitor is new and was installed on the target assembly in February 2012, so this is the first time it has seen beam.  A 'blip' test was performed on the monitor in the hot cell and once again from the mezzanine after the target had been installed in the beam line.  Both tests showed that the monitor was performing normally, so an internal wiring mistake on the monitor was initially suspected and it was decided that the T1-MK1 target should be swapped with the T1-MK2 target.

Today the T1 monolith was uncovered, the T1-MK1 target was moved to the storage pit, and the T1-MK2 target was moved from the hot cell to the beam line.  The T1 volume pumped down normally after installation and a 'blip' test performed from the mezzanine showed that the protect monitor on the new target was functioning normally.  The potentiometer look-up tables were switched to the appropriate target (initially Graham Waters had difficulty with the control software, but was able to solve the problem by accessing the program remotely).  The T1 target is now ready for BL1A start-up.

This afternoon Bill Rawnsley discovered that there were two different protect monitor designs, one with a different internal wiring configuration than what has been used in the past.  If the protect monitor on the T1-MK1 target is the design with the different configuration, this could explain the symptoms observed yesterday.  If we had known this earlier, it may have been possible to solve the problem by installing a simple wire adapter at the mezzanine to switch the wires in the connector to the same configuration as the original design and the target swap would not have been necessary (saving time, dose, and the potential for damaging equipment due to handling).  This will be investigated further to determine if it is really the case.

The T1-MK2 target was transferred from the beam line to the hot cell.  Water was seen in the bottom of the target pipe confirming that the leak is from the target.  The lower cone of the target assembly was submerged in the water and came up dripping wet.  The profile monitor, protect monitor, and two of the target cassettes were also wet.  As it is not possible for those to be submerged, water was likely spraying from either the bellows flanges, or somewhere on the target ladder.  It is estimated that 15L - 20L of water leaked from the target in total.

Inspection, leak checking, and repair on the target will begin next week.  We will suck up as much of the water in the beam line as possible from the top of the monolith using a suction device designed for this purpose.  The remainder will have to be pumped out by vacuum group.

The T1-MK1 target protect and profile monitor electronics were 'blip' tested by Bill Rawnsley on April 5, 2013.  The response from the profile monitor appeared normal, however the protect monitor response was very weak.  After various debugging it was found that the 9 pin connector on the target on the vacuum side of the vacuum flange had become disconnected from the vacuum flange feed-through.  It is suspected that this happened while changing vacuum flange seals in the hot cell.  The connector was plugged in to the feed-through and reassembled.  The monitor was retested on April 8th and gave a normal response.

The lower air amplifier was started and actuation of the T1 profile monitor tested from the control room:  functioning normally.

Unused cables and hoses, concrete pieces, and various other garbage was removed from the top of the T1 monolith.  Spiro Wrap around the T1 profile monitor cable was replaced as it had degraded.

The T1 target is now ready for beam.

 Ron Kuramoto, Keith Ng, John McKinnon, and myself spent some time today investigating water and air lines related to the T1 Rotary Collimator (Col. A).  We discovered the following:

- There is a panel on the back (south) side of the T1 cooling package with 3 air lines for Col. A actuation, Col A cooling water supply & return, and the Col. Shield water cooling supply and return
- The air lines coming from the panel were decomissioned in 2003 by Tom Lyth, and at least 2 out of 3 were replaced with CuALCW water from the M13 header in the BL1A tunnel
- Motion / actuation of the rotary collimator was disabled in Oct 1994 and it is now used only in one position
- The two water lines coming from the M13 header (which connect to the actuation cylinder) will be disconnected, capped, and tagged.  The M13 header can then be completely removed
- It is suspected, but not yet 100% confirmed that Col. A and the Collimator Shield are still cooled by the lines passing through the panel at the back of the cooling package
- A flow test will be performed to confirm this, which will also confirm that the Q4 flow meter at the cooling package measures cooling water flow to Col. A

 The water level in the T1 cooling package expansion tank was checked again today approximately 24 hours after it was refilled.  The level had dropped about 5cm, equating to roughly 3.5L of water lost.

The T1 package was inspected thoroughly and no water was found.  A leak on the target in the beamline is suspected.  The T1 package has been turned off and the water lines were removed from the target.  Tomorrow we will request vacuum group to pump down on the T1 volume which should indicate whether or not there is water in the beam line.

 Vacuum group pumped down on the T1 & T2 volumes (because there is no cold trap on the T1 pump) for approximately 1 hour this morning.  A significant amount of water was found in the base of the cold trap which confirms our suspicions of a water leak on the T1 target.

Tomorrow we will transfer the T1 target from the beam line to the hot cell for inspection, leak checking, and repair.

 The BL1A tunnel was accessed today to assess a faulty reading from the T1 cooling package inlet pressure transducer (T1P1).  The line was first bled to remove gas bubbles which did not change the reading.  The transducer was replaced with a new unit (OMEGA PX315-100GI).  The pressure reading returned to the usual value of approximately 74psi (faulty reading had been steadily decreasing over the last few months and was at approximately 36psi).

The supply lines for all T1 and T2 pressure transducers in the BL1A tunnel were bled, and the expansion tanks topped up.

This transducer was last replaced in May 2010.

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

The following work was completed on the T1 cooling package:
- water drained to BL1A holding tank
- all resevoir o-rings changed
- all cuno filters changed:
     M15 permanent magnet filter < 5uSv/hr at 0.5m, dark grey colour
     Target inlet filter 210uSv/hr at 0.5m, light brown colour
     Resin filter, <5uSv/hr, looks clean and new
- package re-filled and started
- demin circuit flow tuned to 1.0gpm
- #6 female Hansen fittings at target end of flexible hose swapped for previously used SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion on the inside)
 

The T2 cooling package was also drained to the 1A holding tank.
After approval from RPG the holding tank water was released.

Note: the appearance of the used cuno filters is consistent with previous years.

The following work was performed at the T1 cooling package and target between January 18-21, 2016:

- Cuno water filters removed (3), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, and Collimator A solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 each)
- Target side #6 Hansen QD fittings replaced because internal seal had failed and fittings were leaking when disconnected (2)
- Profile monitor copper air supply line leak was repaired
- All air QD fittings (6 total) for profile monitor actuation were replaced, the air restricting valves were tuned, and the flexible hose assemblies were checked for gross air leaks
- "T8" temp sensor was found to be located on the CuALCW common return line, not on the Col A supply line as labeled on XTPAGE.  Position was swapped

Total dose for this work was 0.17mSv for Isaac Earle, and 0.10mSv for Keith Ng

 this Work was done on 09 Dec 2021.

Since Isaac Earle was leaving Triumf he was giving us (maico, chad, adam, and myself) a demonstration of how to remove targets, protect monitor, and wire scanner. During the removal it was noticed that one of the pins for the wire scanner was bent and it would not be possible to reattach without straightening the pins.

Luckily the fields from the ladder were low enough (374 uSv/h) and we were able to retrieve the wire scanner and manually bend the pins back. The wire scanner was reattached without incident, and now will need diagnostics to check on the target ladder to ensure all is operational.

 Graham Waters has activated the appropriate target ladder position look-up tables for the T1-MK1 target which was installed in the beam line yesterday.

On May 31, the low active water in BL1A was turned off briefly and upon re-starting, the flow to the T1 heat exchanger was very close to the reported warn limit (~6 gpm). 

To see if this low flow could be remedied, we entered the BL1A tunnels briefly on Jun 06 but found no flow regulator valves for the T1 heat exchanger. 

The low flow to the heat exchanger after the secondary water pump was re-started may be a normal characteristic of the system. Alternatively, this may have been caused by entrapped air in the heat exhanger after it was drained during 2023 shutdown to replace a leaky valve. 

The latter is grounded in the fact that the flow in T2 recovered immediately after the flow was re-started. T2 was drained through a drain line in the tunnels whereas T1 was drained at the cooling station through a filter/copper line. 

The difference in draining methods may have resulted in entrapped air and thus flow issues in T1 but not T2. 

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.