The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

We tested the air amplifiers by actuating the beam blockers and profile monitors on the T1 and T2 target stations. 

The blockers and profile monitors were each actuated (brought up/out and down/in fully) > 5 times and vacuum remained stable throughout (see attached).

The cycle rate of the air amplifiers (time between 'puffs') under steady conditions (target devices kept in the out position) was measured to be between 60-90 seconds, with 90 seconds corresponding to when the 'flow control' ball valve is fully shut and ~60 seconds when the valve is opened (either partly or fully). 

When a target device is brought up, a cycle event will occur shortly after the device reaches the out position.

This cycle time is much longer than last reported in 2014 (see elog #119), by a factor of 2-3. 

The top amplifier sounds different from the bottom amplifier (cycles between being 'rattly' and 'quite' between cycles).

Seeing how the beam blockers and profile monitors were actuated smoothly with the air amplifiers, we will keep the system running off the lower air amplifier with the 'flow control' ball valve kept open (not shown).

At some point we will look into disassembling the top amplifier to see why it sounds differently between cycle events.

For reference, both amplifiers read 20psi at the regulator and 120 psi to the target station. 

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

 We replaced tha O-ring on M20 at T2 and brought the target station down to vacuum, which allowed the target water pump to be turned on. 

After turning the pump on, we inspected the replaced valves for any leaks and found none - the flow on the demineralizing line was adjusted to 1.2GPM (also done at T1) by turning the needle valve.

Will observe till tomorrow to ensure that the T2 system is stable before proceeding with last system check.

 We moved M20 back to the T2 monolith and re did the connections for air + limit switches + T2 profile monitor air.

Note: T2 profile monitor electronics sparked when we moved M20 out initially. (Update: Shengli Liu from probes group performed a test on the T2 profile monitor electronics on Mar24 and found that they are working properly)

M20 started leaking when we tried to bring it to vaccum - from the monolith o ring seal and not the main shaft seal - will look to correct on monday.

Verified through measurement that the replacement o ring was the correct size (#268 0.139" diameter compared to the 0.131-0.137" that was remved - likely shrunk over use).

Update (Mar 06, 2023): M20 was lifted ~5 in above the monolith flange and we identified that the leak was caused due to damage to the o ring (image attached).

The o ring likely damaged when M20 was brought into contact wit the monolith flange, lifted up again, and brought down for a final time before vaccum check.

The above procedure was done because the flask tends to stall close to the lowermost position and prevents unlatching unless lifted systems are lowered fully.

This will be noted in updated procedures.

 T1/T2 valves were not leaky still (only a few droplets) - pending T2 cooling system pump on for final leak verification.

I have filled the water package on T2 with DI water. The expansion tank is at 38.1cm. This may change when the pump is turned on.

I have refilled the water package on T1 with DI water. The Expansion Tank is at 37cm. This may change when pump is turned on.

3x 3/4" swagelok ball valves (2x at T2 and 1x at T1) were replaced after the water cooling line was shut off at the BL1A tunnels late yesterday. 

We were able to relieve pressure from the T2 water cooler + M8 colA/B (they are the same line) at the BL1A tunnel through a drain port but no such port exists for the T1 heat exchanger line in BL1A.

The T2 heat exchanger line was virtually dry at the replaced valves after pressure was relieved.

The T1 heat exchanger line was still pressurized but while looking for a suitable port to drain at the cooling package, one of the copper fittings at the 1/4" heat exchanger line broke off at the connection to a green valve. 

~1L of water dripped from this line and this was sufficient to relieve pressure from the T1 heat exchanger line - allowing for the 3/4" ball valve to be replaced. 

Until the green valve gets replaced, the T1 heat exchanger line must remain shut off in the BL1A tunnels as the broken copper line is not plugged currently.

Update (last picture): the green valve was replaced with an equivalent 1/4" swagelok ball valve. We also incorporated two 90 deg. bends in the copper tubing to help relieve any stresses that may develop.

The M20 beam blocker was moved from the T2 monolith to the meson hall hot cell with the remote handling flask (completed ~4pm).

This operation was performed in preparation for M20 maintenance (main shaft o-ring replacement + air cylinder re-lubrication).

RPG surveyors measured 20,000 CPM from swiping the surface of the beam blocker shaft, and measured 25 mSv/hr on contact with the beam blocker.

Fields on the hot cell roof were at 100 uSv/hr at 0.5 meters from the top of the M20 BB. This should reduce drastically with the introduction of lead blanket shielding (see attached pictures). 

5 and 3 Swagelok ball valves were replaced from the T1 and T2 cooling packages respectively.

These are compression fit valves, 1/2" or 3/4" size.

1 valve in T2 was replaced previously, bringing the total # of replaced valves in T2 to 4.

Remaining valves that need to be replaced:

• T1:
  • 1x heat exchanger valve was not replaced because the heat exchanger line was not drained prior to the operation.
• T2:
  • 2x valves for the heat exchanger line were not replaced because the heat exchanger line was not drained prior to the operation.
  • 1x M8 beam blocker cooling line was not replaced for the same reasion.

Remaining valves that may need to be replaced (verify after next leak check):

• T1:
  • ~4 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 2x panel mounted valves need to be inspected for leaks.
• T2:
  
  • ~3 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 3x panel mounted valves need to be inspected for leaks.
  • 1x valve from the resin flask outlet has threaded connections with plumbing tape - this valve may not be leaky.

Next week, will drain all water lines associated with the cooling package before replacing remaining valves - then perform leak check. 

 (attached hand written notes)

 Update: remaining valves replaced (see: https://elog.triumf.ca/TIS/RH-Meson+Hall/320)

A 0.75" SwageLok ball valve that was taken out from the T1 cooling package was inspected (SS-45S12).

All valves in the T1/T2 cooling packages likely leaked due to damage to plastic/rubber sealing components. 

SEE PICTURES ATTACHED.

We can see that some bits of white plastic, possibly a bushing, has disintegrated and is leaking out from the valve shaft.

Additionally, there is a considerable amount of dirt/grit in and around the ball.

The surface of the valve opening exibits visible signs of wear (new valve opening is perfectly smooth).

Finally, a red-colored material is found on some small stainless steel parts which is either buildup from contaminants in the line or worn silicon/rubber lining.

Hypothesis: 90 um filter which had not been replaced caused the water in the line to build up particulates and wear plastic/rubber parts in the valve, causing them to leak. 

From the previous test (ELOG 315), we know that the leak is not due to depressurization.

Water reservoir in T1 started draining at ~7.30 am, will leave till tomorrow or overnight to fully drain. 

(Preparation to replace all leaky valves). 

Drained water samples collected for RPG. 

Maico/Eric/Albert:

The expansion tank in T1 was filled to ~36cm in the morning.

We contacted the control room to test flowrates in the T1 cooling loop after new installation of resin in the demineralizing line.

Epics interlocks tripped immediately as pump was turned on (flow too low or pressure to high) due to new resin.

Turning on the pump repeatedly extends the delay before system trips, until eventually the pump can run continuously. 

Flowrate through demin line initially too high (~2.5 gpm), adjusted flow control valve to bring it to ~1.4 gpm (setpoint at the time ~0.4-1.5 gpm).

When the pump was turned on initially, saw physical pressure gauge (dial type) bottom out/make greater than 1 full rotation (>50 Psi). 

No connection between the physical dial to EPICS so no cause for immediate concern.

All valves are more leaky than when the pumps are off - leaks definitely not due to depressurization. 

Attached: photo of the likely 'faulty' dial pressure gauge. 

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

The CUNO filters were changed 2023-01-17 for:

• BL1A Holding Tank drain output, adjacent to the pump
• RH Active Sump, adjacent to the V2 valve

Both of these filters are in-line with the drainage lines from the BL1A holding tank to the active sump. It was suspected that they have not been changed in many years, if ever. Prior to changing the filters, the maximum flow rate measured from the Proteus paddle wheel sensor during draining was around 0.85 gpm (the draining procedure document indicates it should be between 0.9 to 2.5 gpm). After replacing both filters, which each showed significant clogging of sediment, the flow rate significantly increased and the maximum of 2.5 gpm was achievable. Note that when the drain pump was initially turned on, it took some time to reach the desired flow rate due to air in the water lines - the flow rate was quite low until the lines were once again filled with water.

It is suggested that these filters should be replaced every ~3-5 years, depending on usage, or otherwise as-needed (it is possible to see through the clear walls and determine how dirty they are). This MRO item will be added to the checklist of routine T1/T2 related tasks performed to prevent flow rate issues in the future during draining.

Filters were bagged and discarded as active waste.

Attached photos show old filter in bucket, and newly replaced filter, for the RH Active Sump.

The flow rate sensor used to monitor the drain rate from BL1A holding tank to active sump, located in the active sump area, was repaired. The paddle wheel, o-ring, and shaft were replaced. This part was significantly worn prior to replacement.

It is recommended to change the filter adjacent to it as it is suspected that this filter is partially clogged, resulting in low flow rate when draining the tank.

Furthermore, it is also recommended to change solenoid valve SV1 as it is suspected that it does not work.

T2 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T2 (this list will be updated as work progresses):

• Drained water [2023-01-16]
• Replaced ball valve located between heat exchanger and target flow sensor FGTGT (part number SS-45S12)... this part, as well as numerous other ones, were noted to have been leaking. See e-log 308
• Replaced both CUNO filters (target water and demin. resin circuit filter) [2023-02-01]
• Replaced 90 micron demin. filter (the filter was very dirty - it was last changed in Feb. 2012 as indicated by written note) [2023-02-01]
• Inspected Hansen fittings (o-rings look OK) [2023-02-01]
• Replaced o-ring, shaft, and paddle wheel for flow sensor FGSEC (Q3). The old paddle wheel had significant wear in the shaft hole which resulted in sensor readback issues observed during 2022 operation. It is likely that this was caused by debris or a manufacturing defect which got worse over time. Photos show a comparison between old and new. [2023-02-01]
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 322)
• Changed M20 beam block o-ring seals, and performed vacuum check (see e-logs 319, 324, and 323)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). Note: target water conductivity sensor (CONDTGT) took a few hours to ramp up to appropriate value. See screenshot attached of T2 state at approximately 9:15am on 2023-03-07 after the system ran overnight.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

T1 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T1 (this list will be updated as work progresses):

• Drained water
• Hansen fittings o-rings inspected: OK
• Replaced demineralizing flow metering valve (part number SS-8BK*)... this part was replaced because the equivalent part on T2 exhibited potential signs of wear during 2022 and was replaced in 2022 mini-shutdown (October). There are two things to note about this part: 1) when loosening it, it makes an audible *click* at some point during loosening rather than a smooth continuous feeling like the new replacement part, and 2) it appears to have originally been installed backwards in terms of the flow rate direction arrow. It is now installed correctly.
• Demineralizing resin can - exchanged with fresh resin (2023-01-18... dose rate of old resin which is currently located in the warm cell is approximately 40 uSv/h on contact)
• Replaced filters:
  
  • Target water CUNO filter (it was relatively clean)
  • M15 CUNO filter (it was relatively clean)
  • Demineralizing 90 micron filter (it was very dirty with tiny pieces of debris)
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 321)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). See screenshot attached of T1 state at approximately 4:00pm on 2023-03-06 after the system had been running for a significant time.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

On Jan. 13, the T1 and T2 cooling package water reservoirs were drained to the BL1A holding tank. Water samples were obtained for each system and provided to RPG for analysis.

Prior to starting draining, the BL1A holding tank water level was approximately 330L. After draining, the water level was approximately 550L (checked Jan. 16).

Upon uncovering the T1 and T2 areas to prepare for regular cooling package maintenance, it was discovered that multiple ball valves were leaking. Primarily the Swagelok SS-45S12 parts, but some SS-45S8 parts have failed as well. There were small puddles of water underneath many of the valves. It appears that the leak is due to a compromised seal. It is suspected that these parts failed from radiation exposure. They have PTFE packing material which is highly susceptible to decomposition under irradiation, and as far as we can tell they have not really been repaired/replaced (aside from a written note on one saying the seal was changed in 1997).

It is recommended to add the regular repair/replacement of these valves to the annual shutdown MRO checklist.

Once replaced, this e-log will be updated to document which ones were replaced.