5 and 3 Swagelok ball valves were replaced from the T1 and T2 cooling packages respectively.

These are compression fit valves, 1/2" or 3/4" size.

1 valve in T2 was replaced previously, bringing the total # of replaced valves in T2 to 4.

Remaining valves that need to be replaced:

• T1:
  • 1x heat exchanger valve was not replaced because the heat exchanger line was not drained prior to the operation.
• T2:
  • 2x valves for the heat exchanger line were not replaced because the heat exchanger line was not drained prior to the operation.
  • 1x M8 beam blocker cooling line was not replaced for the same reasion.

Remaining valves that may need to be replaced (verify after next leak check):

• T1:
  • ~4 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 2x panel mounted valves need to be inspected for leaks.
• T2:
  
  • ~3 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 3x panel mounted valves need to be inspected for leaks.
  • 1x valve from the resin flask outlet has threaded connections with plumbing tape - this valve may not be leaky.

Next week, will drain all water lines associated with the cooling package before replacing remaining valves - then perform leak check. 

 (attached hand written notes)

 Update: remaining valves replaced (see: https://elog.triumf.ca/TIS/RH-Meson+Hall/320)

A 0.75" SwageLok ball valve that was taken out from the T1 cooling package was inspected (SS-45S12).

All valves in the T1/T2 cooling packages likely leaked due to damage to plastic/rubber sealing components. 

SEE PICTURES ATTACHED.

We can see that some bits of white plastic, possibly a bushing, has disintegrated and is leaking out from the valve shaft.

Additionally, there is a considerable amount of dirt/grit in and around the ball.

The surface of the valve opening exibits visible signs of wear (new valve opening is perfectly smooth).

Finally, a red-colored material is found on some small stainless steel parts which is either buildup from contaminants in the line or worn silicon/rubber lining.

Hypothesis: 90 um filter which had not been replaced caused the water in the line to build up particulates and wear plastic/rubber parts in the valve, causing them to leak. 

From the previous test (ELOG 315), we know that the leak is not due to depressurization.

Water reservoir in T1 started draining at ~7.30 am, will leave till tomorrow or overnight to fully drain. 

(Preparation to replace all leaky valves). 

Drained water samples collected for RPG. 

Maico/Eric/Albert:

The expansion tank in T1 was filled to ~36cm in the morning.

We contacted the control room to test flowrates in the T1 cooling loop after new installation of resin in the demineralizing line.

Epics interlocks tripped immediately as pump was turned on (flow too low or pressure to high) due to new resin.

Turning on the pump repeatedly extends the delay before system trips, until eventually the pump can run continuously. 

Flowrate through demin line initially too high (~2.5 gpm), adjusted flow control valve to bring it to ~1.4 gpm (setpoint at the time ~0.4-1.5 gpm).

When the pump was turned on initially, saw physical pressure gauge (dial type) bottom out/make greater than 1 full rotation (>50 Psi). 

No connection between the physical dial to EPICS so no cause for immediate concern.

All valves are more leaky than when the pumps are off - leaks definitely not due to depressurization. 

Attached: photo of the likely 'faulty' dial pressure gauge. 

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

The CUNO filters were changed 2023-01-17 for:

• BL1A Holding Tank drain output, adjacent to the pump
• RH Active Sump, adjacent to the V2 valve

Both of these filters are in-line with the drainage lines from the BL1A holding tank to the active sump. It was suspected that they have not been changed in many years, if ever. Prior to changing the filters, the maximum flow rate measured from the Proteus paddle wheel sensor during draining was around 0.85 gpm (the draining procedure document indicates it should be between 0.9 to 2.5 gpm). After replacing both filters, which each showed significant clogging of sediment, the flow rate significantly increased and the maximum of 2.5 gpm was achievable. Note that when the drain pump was initially turned on, it took some time to reach the desired flow rate due to air in the water lines - the flow rate was quite low until the lines were once again filled with water.

It is suggested that these filters should be replaced every ~3-5 years, depending on usage, or otherwise as-needed (it is possible to see through the clear walls and determine how dirty they are). This MRO item will be added to the checklist of routine T1/T2 related tasks performed to prevent flow rate issues in the future during draining.

Filters were bagged and discarded as active waste.

Attached photos show old filter in bucket, and newly replaced filter, for the RH Active Sump.

The flow rate sensor used to monitor the drain rate from BL1A holding tank to active sump, located in the active sump area, was repaired. The paddle wheel, o-ring, and shaft were replaced. This part was significantly worn prior to replacement.

It is recommended to change the filter adjacent to it as it is suspected that this filter is partially clogged, resulting in low flow rate when draining the tank.

Furthermore, it is also recommended to change solenoid valve SV1 as it is suspected that it does not work.

T2 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T2 (this list will be updated as work progresses):

• Drained water [2023-01-16]
• Replaced ball valve located between heat exchanger and target flow sensor FGTGT (part number SS-45S12)... this part, as well as numerous other ones, were noted to have been leaking. See e-log 308
• Replaced both CUNO filters (target water and demin. resin circuit filter) [2023-02-01]
• Replaced 90 micron demin. filter (the filter was very dirty - it was last changed in Feb. 2012 as indicated by written note) [2023-02-01]
• Inspected Hansen fittings (o-rings look OK) [2023-02-01]
• Replaced o-ring, shaft, and paddle wheel for flow sensor FGSEC (Q3). The old paddle wheel had significant wear in the shaft hole which resulted in sensor readback issues observed during 2022 operation. It is likely that this was caused by debris or a manufacturing defect which got worse over time. Photos show a comparison between old and new. [2023-02-01]
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 322)
• Changed M20 beam block o-ring seals, and performed vacuum check (see e-logs 319, 324, and 323)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). Note: target water conductivity sensor (CONDTGT) took a few hours to ramp up to appropriate value. See screenshot attached of T2 state at approximately 9:15am on 2023-03-07 after the system ran overnight.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

T1 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T1 (this list will be updated as work progresses):

• Drained water
• Hansen fittings o-rings inspected: OK
• Replaced demineralizing flow metering valve (part number SS-8BK*)... this part was replaced because the equivalent part on T2 exhibited potential signs of wear during 2022 and was replaced in 2022 mini-shutdown (October). There are two things to note about this part: 1) when loosening it, it makes an audible *click* at some point during loosening rather than a smooth continuous feeling like the new replacement part, and 2) it appears to have originally been installed backwards in terms of the flow rate direction arrow. It is now installed correctly.
• Demineralizing resin can - exchanged with fresh resin (2023-01-18... dose rate of old resin which is currently located in the warm cell is approximately 40 uSv/h on contact)
• Replaced filters:
  
  • Target water CUNO filter (it was relatively clean)
  • M15 CUNO filter (it was relatively clean)
  • Demineralizing 90 micron filter (it was very dirty with tiny pieces of debris)
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 321)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). See screenshot attached of T1 state at approximately 4:00pm on 2023-03-06 after the system had been running for a significant time.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

On Jan. 13, the T1 and T2 cooling package water reservoirs were drained to the BL1A holding tank. Water samples were obtained for each system and provided to RPG for analysis.

Prior to starting draining, the BL1A holding tank water level was approximately 330L. After draining, the water level was approximately 550L (checked Jan. 16).

Upon uncovering the T1 and T2 areas to prepare for regular cooling package maintenance, it was discovered that multiple ball valves were leaking. Primarily the Swagelok SS-45S12 parts, but some SS-45S8 parts have failed as well. There were small puddles of water underneath many of the valves. It appears that the leak is due to a compromised seal. It is suspected that these parts failed from radiation exposure. They have PTFE packing material which is highly susceptible to decomposition under irradiation, and as far as we can tell they have not really been repaired/replaced (aside from a written note on one saying the seal was changed in 1997).

It is recommended to add the regular repair/replacement of these valves to the annual shutdown MRO checklist.

Once replaced, this e-log will be updated to document which ones were replaced.

It was reported on Dec. 20, 2022 that when Operations attempted to run the target ladder motor, it was unresponsive. A fault report was created: 15477.

During winter shutdown 2023, the T2 area was uncovered and the motor was inspected on 2023-01-11 by Adam Newsome and Maico Dalla Valle. The connectors and cabling appeared OK.

Upon removing the motor from its mounted position, it became clear that the coupling part between the motor and the gearbox had failed. The motor was running properly but not actually connected to the drive shaft and therefore nothing happened when ops tried to move the ladder. We confirmed that the motor runs properly in no-load condition when it is isolated from the rest of the assembly. We were able to move the rest of the drive shaft by hand and it does not appear to have any significant resistance, so it is suspected that this coupling failed due to radiation exposure over a long period of time rather than overload. The coupling part is plastic and designed intentionally to break under high loads (it is thinned out in the middle, where it snapped). Hopefully it did not actually see any high loads (i.e. there is no other root cause of this issue) and it simply degraded so much that it eventually broke.  

To resolve this issue, we will fabricate/purchase and install another one (Albert, Maico) and work with operations to test that all target ladder positions are reachable once it is installed. We will also confirm at that time that the limit switches and potentiometer readback function as expected.

It is recommended to consider the following changes to the drive system going forward to prevent this from happening in the future:

• Replace these plastic coupling parts with metal versions
• Replace the plastic coupling part that is thinned out to intentionally fail under high load with another method of system protection. The same end result could be achieved using a clutch or perhaps a fuse/circuit breaker which would effectively inhibit motion if the motor experienced overload. This would be a significant improvement over breakage because it would not require replacing the part (dose-intensive) and would be more easily recognizable.

For now, this drop-in replacement should suffice because it is not likely that the part will degrade so significantly within the next few years that this would happen again (unless there is a different root cause aside from radiation that we are not aware of!).

[update 2023-01-01] The broken coupler has been replaced by Albert Kong and Adam Newsome. The replacement part was tested - DCR operated the target ladder motor, moving it up and down through all of its positions multiple times. Everything mechanically and electrically appeared OK during motion. DCR confirmed they are able to reach each position and the proper readbacks are present - everything looks OK on their end. The issue is considered resolved and the fault is returned.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

The T2-MK2 target was transported from the east hot cell to the storage pit at position 3.

As was noted in the past, position 3 is difficult to align. The flask alignment pins prevent the flask from positioning in the correct location, and it is required to have the flask just above the pins to ensure that the pins did not restrict the flask movement.

It is recommended to install a camera system to assist with the target placement and retrieval, which will help reduce the amount of dose one receives during this operation.

The target elevations measurement and beam spot checks were completed on Sept. 15, 2022. This was done after a July 2022 target exchange for target positions 3 and 5.

Updated target elevation drawing can be found in DocuShare (Collection-33652)

Mechanical Services Group changed the pre-filter in the ventilation system on Aug. 29, 2022 as per work permit C2022-08-29-2.

It is recommended that Mechanical Services change the HEPA filter as soon as possible. These filters are on order with approximately 7 month lead time. 

Edit: as of 2023-05-03, the HEPA filter has been changed.

The water level inside the BL1A active water holding tank was checked today (Aug. 23, 2022). The tank is essentially empty at this time.

On Aug. 18, 2022, Operations informed RH group that an alarm activated for the full status of the MHESA B1 RCR1 Lab holding tank. Additionally another alarm was reported in the RH control room regarding the RH sump full (Southwest Meson Hall Extension - Remote Handling area). However, there is an alarm signal for the RH sump that was seen as OK at the MHESA Lab HMI.

Upon examining the RH active sump it was clear that the water level was in fact quite high, and the red warning light on the local control box was on.

After sampling and testing by RPG the sump contents were released under work permit C2022-08-19-3 as per the procedure in Document-64834. The approximate volume of the full sump is 7270L.

When the water level was below the warning threshold (i.e. red warning light came off, and green light indicating "sump good" came on), a request was made to the control room to reset the alarm. Operations cleared the alarm at the RCR Lab controls and retired the signal defeat that was in place in the DCR (“RCRL warn” signal was defeated).

Edit (2022-08-23): BL1A holding tank water level was checked - it is empty.