The following work was completed between Jan 16 - Feb 19, 2019:

• T1 and T2 cooling packages drained to South TNF holding tank, water sampled and tested, released after approval from RPG
• T1 resin can swapped
• All Cuno filters changed at T1 (3 total)
• T1 inlet (T1CSPGIN) and outlet (T1CSPGOUT) pressure gauges replaced (archived data on EPICS indicated both starting to fail)
• T2 target water Cuno filter replaced
• All T1 and T2 solenoid valves changed to new models without power management circuit which will hopefully solve the problem of frequent failures (SASP0670)
  Parker brass body valve 73218BN4UN00 with solenoid C222C2 solenoid used for T1CSSVSEC and T2CSSVSEC
  STC stainless steel body valve 2S160-1/2-2-D with solenoid 2W200C-2-D used for T1CSSVTGT, T1SVCOL, T2CSSVTGT, T2SVCOLA, T2SVCOLB
  Necessary changes to PLC BOP wiring made by Tony Tateyama
  All new valves confirmed operational and leak tight
  Cyclotron Fault #11955 returned
• T1 and T2 profile monitor air cyclinder flow control valves replaced with new models tuned to ~1/8 turn open.  Part is Rego F125B (see PO #3044305)
• T1-MK2 profile monitor air cylinders replaced (both were leaking through the top seal)
• T1 and T2 profile monitor actuation checked - both move in and out slowly and smoothly
• M20 BB actuation checked - movement is slow and smooth
• T1 and T2 ladder movement checked - both ok
• T1-MK2 male Hansen fittings replaced as well as female Hansens which connect to them (these are the last remaining Hansen fittings to be replaced at T1/T2)
• Both cooling packages filled and started, now running smoothly.  Inspected for leaks (none found).  No change to 1A vacuum when packages started
   

Note 1: The T1 package water pump is slightly noisy when running (it has been this way since the replacement pump was installed last year) - Will investigate further at a later date

Note 2: In the week of Feb 11-15 Vacuum Group pressurized the T1 target cooling circuit with 30psi helium to try to find the vacuum load on the T1 volume.  This was done twice with the beamline vented for a few hours in between in case water had frozen at a leak location. The conclusion after testing is that the T1 cooling water is not the source of the vacuum load at T1.  Helium was also sprayed all around the top of the T1 monolith (T1 target and M15 permanent magnet) - there was no response on the leak detector

My total dose for this work was 0.15mSv

 Annual MRO work on the T1 and T2 cooling packages is now complete.  The following tasks were performed:

• Water from both packages drained to the South TNF holding tank (samples taken)
• All Proteus paddle wheels, shafts, and o-rings replaced (5 each at T1 and T2)
• All Cuno filters changed
• Archived data for all sensor readbacks inspected.  Only faulty sensor was B1A:T1CS:FGDEM
• Both packages refilled and restarted; Operating smoothly; Inspected for leaks, none found; Demin flow at ~1.0gpm
• B1A:T1CS:FGDEM readback is fixed with new paddle wheel
• South TNF holding tank water released after approval from RPG
• Old filters added to plastic pail with other T1/T2 filters located directly north of the BL1A blocks boot-box area
• Active water from filter canisters dumped into the RH lab active sink

The following work was completed during the 2020 shutdown period for BL1A (Jan-Aug 2020):

• T1 and T2 EPICS data checked for anomalies.  None found, all sensors behaving as expected
• T1 and T2 cooling package water drained to storage tank, sampled, and released
• All cuno filters changed (3 for T1, 2 for T2)
• All T1 and T2 proteus paddle wheels changed
• T1 and T2 hansen fitting connections inspected: all good, no leaks
• Both packages refilled, restarted, demin flow tuned to 1gpm, and checked for leaks: very small leak from 3/4" line going into heat exchanger on T1 target loop (not new, tightening doesn't help) 

• Profile monitor air hoses inspected: ok
• Air amplifiers inspected: cycle time ok, no leaks
• M9 beam blocker overhaul completed including new air cylinders
• M9 and M20 BB actuation checked
• T1 and T2 profile monitor actuation checked
• T1 and T2 ladder movement checked
• XTPAGE 7L updated to indicate which targets spent
• Target ladder status sheet updated
• ASCO valves for M9BB, M20BB, T1 prof mon, T2 prof mon all serviced (with rebuild kit)

August 25, 2020 update:

- The cap block was replaced over T2
- Actuation of the M9-T2 and M20 beam blockers was tested.  Both moved fine and limits displayed correctly

The T2 Collimator A flow meter signal (B1A:T2:FGCOLA) began fluctuating and occasionally reading 0gpm on Sunday Oct 25 causing BL1A trips and machine down time.

On the maintenance day today (Tues Oct 27), the T2 package was uncovered and the T2 Col. A flow meter was opened.  No unusual deposits were observed as seen recently with other flowmeters for BL1A magnets on the CuALCW system.  Some minor wear was observed on the dowel pin shaft, and the paddle wheel didn't spin quite as freely as when new.  A small amount of reddish/brown film (rust?) was seen on the internal SS faces.  The faces were wiped clean and new dowel pin, paddle wheel and o-ring were installed.

The pin, paddlewheel, and o-ring were also replaced for B1A:T2:FGCOLB (no wear observed on old pin) and B1A:T2CS:FGTGT (some wear observed on pin, more than Col. A).  They were not replaced on B1A:T2CS:FGSEC or B1A:T2CS:FGDEM at this time, as neither of these has a noisy signal, and neither can cause trips in the control system.

After all paddlewheels were re-installed and the system restarted, the flowmeters behaved as expected.  The fluctuations that occurred on Sunday were not happening this morning prior to the change, so we're unsure at this point if this has solved the problem.  StripTool was used before and after the change to observe fluctuations in the readbacks, and no noticeable difference was observed.

During testing of the flow meters we observed that B1A:T2:SVCOLB seemed to not respond one of the times it was commanded to close, and one or two other times it seemed to close after a few seconds delay.

Note: All T1 and T2 paddlewheels, pins, and o-rings were changed earlier this year.  They're scheduled to be replaced every 2 years as per the T1/T2 maintenance plan.

With the assistance of Maico we changed the paddle wheel, o ring, and shaft in the Collimator A flow meter (Q4) at the T2 package.

Flow meter was wiped clean to ensure o ring surface was clean.

Everything was changed without incident, and ops confirmed that the flow rate is now normal.

The paddle wheels, o-rings, and shafts were changed at the T2 cooling package for the following sensors:

• B1A:T2:FGCOLA
• T2CS:FGTGT
• T2CS:FGDEM
• T2CS:FGSEC
• T2:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

Two of the sensors, FGCOLA and FGTGT, experienced anomalies during the previous beam run which resulted in faults (#14656 and #14501, respectively). The system is not yet running in order to confirm whether the replacement has resolved the faults.

The schematic which contains these sensors is TBP0902.

The paddle wheels, o-rings, and shafts were changed at the T1 cooling package for the following sensors:

• B1A:T1:FGCOLA
• T1CS:FGTGT
• T1CS:FGDEM
• T1CS:FGSEC
• T1:FGCOLB

No major issues were noted. Some residue was observed upon cleaning the sensors. No leaks observed when turning water isolation valves back on.

The schematic which contains these sensors is TBP0902.

The four o-rings on top of the T1 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

The four o-rings on top of the T2 cooling package water reservoir were changed as per the recommended 5 year replacement schedule.

All filters were replaced for the T1 cooling system: demineralizing resin circuit, target water system, and M15.

All filters were replaced for the T2 cooling system: demineralizing resin circuit and target water system.

The T1 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

 No leaks or abnormalities observed - system running normally.

NOTE: the needle valve for demin flow control adjustment was behaving strangely - it needed to be tapped when loosening to increase flow in order for results to be observed. Suggest replacement next shutdown (2023) - a calendar reminder was set to do this.

The T2 cooling system was refilled with water via the expansion tank until the tank level read approximately 38 cm.

The system pump was restarted. All sensors displayed acceptable nominal values. Demin flow was tuned to 1.0 gpm.

No leaks or abnormalities observed - system running normally.

 The T1/T2 water packages were drained into the B1AAWHT (280L), the water was sampled and tested by RPG, results included in elog.

Once given the all clear myself and Maico released the water from the B1AAWHT (300L). 

It was discovered that the solenoid valve SV1 which is located in the RH lab beside the sump, is broken and will need replacing, picture included in elog.

Having the solenoid broken caused the draining to go slower then normal.

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

On Jan. 13, the T1 and T2 cooling package water reservoirs were drained to the BL1A holding tank. Water samples were obtained for each system and provided to RPG for analysis.

Prior to starting draining, the BL1A holding tank water level was approximately 330L. After draining, the water level was approximately 550L (checked Jan. 16).

T1 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T1 (this list will be updated as work progresses):

• Drained water
• Hansen fittings o-rings inspected: OK
• Replaced demineralizing flow metering valve (part number SS-8BK*)... this part was replaced because the equivalent part on T2 exhibited potential signs of wear during 2022 and was replaced in 2022 mini-shutdown (October). There are two things to note about this part: 1) when loosening it, it makes an audible *click* at some point during loosening rather than a smooth continuous feeling like the new replacement part, and 2) it appears to have originally been installed backwards in terms of the flow rate direction arrow. It is now installed correctly.
• Demineralizing resin can - exchanged with fresh resin (2023-01-18... dose rate of old resin which is currently located in the warm cell is approximately 40 uSv/h on contact)
• Replaced filters:
  
  • Target water CUNO filter (it was relatively clean)
  • M15 CUNO filter (it was relatively clean)
  • Demineralizing 90 micron filter (it was very dirty with tiny pieces of debris)
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 321)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). See screenshot attached of T1 state at approximately 4:00pm on 2023-03-06 after the system had been running for a significant time.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

T2 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T2 (this list will be updated as work progresses):

• Drained water [2023-01-16]
• Replaced ball valve located between heat exchanger and target flow sensor FGTGT (part number SS-45S12)... this part, as well as numerous other ones, were noted to have been leaking. See e-log 308
• Replaced both CUNO filters (target water and demin. resin circuit filter) [2023-02-01]
• Replaced 90 micron demin. filter (the filter was very dirty - it was last changed in Feb. 2012 as indicated by written note) [2023-02-01]
• Inspected Hansen fittings (o-rings look OK) [2023-02-01]
• Replaced o-ring, shaft, and paddle wheel for flow sensor FGSEC (Q3). The old paddle wheel had significant wear in the shaft hole which resulted in sensor readback issues observed during 2022 operation. It is likely that this was caused by debris or a manufacturing defect which got worse over time. Photos show a comparison between old and new. [2023-02-01]
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 322)
• Changed M20 beam block o-ring seals, and performed vacuum check (see e-logs 319, 324, and 323)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). Note: target water conductivity sensor (CONDTGT) took a few hours to ramp up to appropriate value. See screenshot attached of T2 state at approximately 9:15am on 2023-03-07 after the system ran overnight.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.