On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

The T1 and T2 cooling packages were inspected today while running.  No water leaks were observed at either package including the high-active loops and copper active. 

The T1 pump was fairly noisy (video was taken to record the sound).  This is the new pump installed in August 2018 (see E-Log #242).  Maico reports that it had a similar noise when started after installation in August, and they were unsure if it was a normal sound or not.  The pump it replaced ran quietly, as well as the pump currently running at T2.  Further investigation will be performed this shutdown to try and determine the cause of the noise and appropriate action.

Both packages have been turned off and pump contactor control signals disconnected in preparation for MRO work and power outage this weekend.

The following work was completed between Jan 16 - Feb 19, 2019:

• T1 and T2 cooling packages drained to South TNF holding tank, water sampled and tested, released after approval from RPG
• T1 resin can swapped
• All Cuno filters changed at T1 (3 total)
• T1 inlet (T1CSPGIN) and outlet (T1CSPGOUT) pressure gauges replaced (archived data on EPICS indicated both starting to fail)
• T2 target water Cuno filter replaced
• All T1 and T2 solenoid valves changed to new models without power management circuit which will hopefully solve the problem of frequent failures (SASP0670)
  Parker brass body valve 73218BN4UN00 with solenoid C222C2 solenoid used for T1CSSVSEC and T2CSSVSEC
  STC stainless steel body valve 2S160-1/2-2-D with solenoid 2W200C-2-D used for T1CSSVTGT, T1SVCOL, T2CSSVTGT, T2SVCOLA, T2SVCOLB
  Necessary changes to PLC BOP wiring made by Tony Tateyama
  All new valves confirmed operational and leak tight
  Cyclotron Fault #11955 returned
• T1 and T2 profile monitor air cyclinder flow control valves replaced with new models tuned to ~1/8 turn open.  Part is Rego F125B (see PO #3044305)
• T1-MK2 profile monitor air cylinders replaced (both were leaking through the top seal)
• T1 and T2 profile monitor actuation checked - both move in and out slowly and smoothly
• M20 BB actuation checked - movement is slow and smooth
• T1 and T2 ladder movement checked - both ok
• T1-MK2 male Hansen fittings replaced as well as female Hansens which connect to them (these are the last remaining Hansen fittings to be replaced at T1/T2)
• Both cooling packages filled and started, now running smoothly.  Inspected for leaks (none found).  No change to 1A vacuum when packages started
   

Note 1: The T1 package water pump is slightly noisy when running (it has been this way since the replacement pump was installed last year) - Will investigate further at a later date

Note 2: In the week of Feb 11-15 Vacuum Group pressurized the T1 target cooling circuit with 30psi helium to try to find the vacuum load on the T1 volume.  This was done twice with the beamline vented for a few hours in between in case water had frozen at a leak location. The conclusion after testing is that the T1 cooling water is not the source of the vacuum load at T1.  Helium was also sprayed all around the top of the T1 monolith (T1 target and M15 permanent magnet) - there was no response on the leak detector

My total dose for this work was 0.15mSv

 Annual MRO work on the T1 and T2 cooling packages is now complete.  The following tasks were performed:

• Water from both packages drained to the South TNF holding tank (samples taken)
• All Proteus paddle wheels, shafts, and o-rings replaced (5 each at T1 and T2)
• All Cuno filters changed
• Archived data for all sensor readbacks inspected.  Only faulty sensor was B1A:T1CS:FGDEM
• Both packages refilled and restarted; Operating smoothly; Inspected for leaks, none found; Demin flow at ~1.0gpm
• B1A:T1CS:FGDEM readback is fixed with new paddle wheel
• South TNF holding tank water released after approval from RPG
• Old filters added to plastic pail with other T1/T2 filters located directly north of the BL1A blocks boot-box area
• Active water from filter canisters dumped into the RH lab active sink

The following work was completed during the 2020 shutdown period for BL1A (Jan-Aug 2020):

• T1 and T2 EPICS data checked for anomalies.  None found, all sensors behaving as expected
• T1 and T2 cooling package water drained to storage tank, sampled, and released
• All cuno filters changed (3 for T1, 2 for T2)
• All T1 and T2 proteus paddle wheels changed
• T1 and T2 hansen fitting connections inspected: all good, no leaks
• Both packages refilled, restarted, demin flow tuned to 1gpm, and checked for leaks: very small leak from 3/4" line going into heat exchanger on T1 target loop (not new, tightening doesn't help) 

• Profile monitor air hoses inspected: ok
• Air amplifiers inspected: cycle time ok, no leaks
• M9 beam blocker overhaul completed including new air cylinders
• M9 and M20 BB actuation checked
• T1 and T2 profile monitor actuation checked
• T1 and T2 ladder movement checked
• XTPAGE 7L updated to indicate which targets spent
• Target ladder status sheet updated
• ASCO valves for M9BB, M20BB, T1 prof mon, T2 prof mon all serviced (with rebuild kit)

August 25, 2020 update:

- The cap block was replaced over T2
- Actuation of the M9-T2 and M20 beam blockers was tested.  Both moved fine and limits displayed correctly

The T2 Collimator A flow meter signal (B1A:T2:FGCOLA) began fluctuating and occasionally reading 0gpm on Sunday Oct 25 causing BL1A trips and machine down time.

On the maintenance day today (Tues Oct 27), the T2 package was uncovered and the T2 Col. A flow meter was opened.  No unusual deposits were observed as seen recently with other flowmeters for BL1A magnets on the CuALCW system.  Some minor wear was observed on the dowel pin shaft, and the paddle wheel didn't spin quite as freely as when new.  A small amount of reddish/brown film (rust?) was seen on the internal SS faces.  The faces were wiped clean and new dowel pin, paddle wheel and o-ring were installed.

The pin, paddlewheel, and o-ring were also replaced for B1A:T2:FGCOLB (no wear observed on old pin) and B1A:T2CS:FGTGT (some wear observed on pin, more than Col. A).  They were not replaced on B1A:T2CS:FGSEC or B1A:T2CS:FGDEM at this time, as neither of these has a noisy signal, and neither can cause trips in the control system.

After all paddlewheels were re-installed and the system restarted, the flowmeters behaved as expected.  The fluctuations that occurred on Sunday were not happening this morning prior to the change, so we're unsure at this point if this has solved the problem.  StripTool was used before and after the change to observe fluctuations in the readbacks, and no noticeable difference was observed.

During testing of the flow meters we observed that B1A:T2:SVCOLB seemed to not respond one of the times it was commanded to close, and one or two other times it seemed to close after a few seconds delay.

Note: All T1 and T2 paddlewheels, pins, and o-rings were changed earlier this year.  They're scheduled to be replaced every 2 years as per the T1/T2 maintenance plan.

With the assistance of Maico we changed the paddle wheel, o ring, and shaft in the Collimator A flow meter (Q4) at the T2 package.

Flow meter was wiped clean to ensure o ring surface was clean.

Everything was changed without incident, and ops confirmed that the flow rate is now normal.

The paddle wheels, o-rings, and shafts were changed at the T2 cooling package for the following sensors:

• B1A:T2:FGCOLA
• T2CS:FGTGT
• T2CS:FGDEM
• T2CS:FGSEC
• T2:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

Two of the sensors, FGCOLA and FGTGT, experienced anomalies during the previous beam run which resulted in faults (#14656 and #14501, respectively). The system is not yet running in order to confirm whether the replacement has resolved the faults.

The schematic which contains these sensors is TBP0902.

The paddle wheels, o-rings, and shafts were changed at the T1 cooling package for the following sensors:

• B1A:T1:FGCOLA
• T1CS:FGTGT
• T1CS:FGDEM
• T1CS:FGSEC
• T1:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

The schematic which contains these sensors is TBP0902.

The four o-rings on top of the T1 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

The four o-rings on top of the T2 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

All filters were replaced for the T1 cooling system: demineralizing resin circuit, target water system, and M15.

All filters were replaced for the T2 cooling system: demineralizing resin circuit and target water system.

The T1 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

 No leaks or abnormalities observed - system running normally.

NOTE: the needle valve for demin flow control adjustment was behaving strangely - it needed to be tapped when loosening to increase flow in order for results to be observed. Suggest replacement next shutdown (2023) - a calendar reminder was set to do this.

The T2 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

No leaks or abnormalities observed - system running normally.

 The T1/T2 water packages were drained into the B1AAWHT (280L), the water was sampled and tested by RPG, results included in elog.

Once given the all clear myself and Maico released the water from the B1AAWHT (300L). 

It was discovered that the solenoid valve SV1 which is located in the RH lab beside the sump, is broken and will need replacing, picture included in elog.

Having the solenoid broken caused the draining to go slower then normal.

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.