Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

Maico/Eric/Albert:

The expansion tank in T1 was filled to ~36cm in the morning.

We contacted the control room to test flowrates in the T1 cooling loop after new installation of resin in the demineralizing line.

Epics interlocks tripped immediately as pump was turned on (flow too low or pressure to high) due to new resin.

Turning on the pump repeatedly extends the delay before system trips, until eventually the pump can run continuously. 

Flowrate through demin line initially too high (~2.5 gpm), adjusted flow control valve to bring it to ~1.4 gpm (setpoint at the time ~0.4-1.5 gpm).

When the pump was turned on initially, saw physical pressure gauge (dial type) bottom out/make greater than 1 full rotation (>50 Psi). 

No connection between the physical dial to EPICS so no cause for immediate concern.

All valves are more leaky than when the pumps are off - leaks definitely not due to depressurization. 

Attached: photo of the likely 'faulty' dial pressure gauge. 

A 0.75" SwageLok ball valve that was taken out from the T1 cooling package was inspected (SS-45S12).

All valves in the T1/T2 cooling packages likely leaked due to damage to plastic/rubber sealing components. 

SEE PICTURES ATTACHED.

We can see that some bits of white plastic, possibly a bushing, has disintegrated and is leaking out from the valve shaft.

Additionally, there is a considerable amount of dirt/grit in and around the ball.

The surface of the valve opening exibits visible signs of wear (new valve opening is perfectly smooth).

Finally, a red-colored material is found on some small stainless steel parts which is either buildup from contaminants in the line or worn silicon/rubber lining.

Hypothesis: 90 um filter which had not been replaced caused the water in the line to build up particulates and wear plastic/rubber parts in the valve, causing them to leak. 

From the previous test (ELOG 315), we know that the leak is not due to depressurization.

 The T1 cooling package was started around 2pm today.  There was no change to the T1 volume vacuum during startup.

The package was inspected for water leaks: None found.

The demin loop flow (Q2) was adjusted to ~1.5gpm.

The target out water resistivity has reached 12 mega-ohm-centimeters after approximately 2.5 hours, indicating that the resin does not need to be changed.

T1 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T1 (this list will be updated as work progresses):

• Drained water
• Hansen fittings o-rings inspected: OK
• Replaced demineralizing flow metering valve (part number SS-8BK*)... this part was replaced because the equivalent part on T2 exhibited potential signs of wear during 2022 and was replaced in 2022 mini-shutdown (October). There are two things to note about this part: 1) when loosening it, it makes an audible *click* at some point during loosening rather than a smooth continuous feeling like the new replacement part, and 2) it appears to have originally been installed backwards in terms of the flow rate direction arrow. It is now installed correctly.
• Demineralizing resin can - exchanged with fresh resin (2023-01-18... dose rate of old resin which is currently located in the warm cell is approximately 40 uSv/h on contact)
• Replaced filters:
  
  • Target water CUNO filter (it was relatively clean)
  • M15 CUNO filter (it was relatively clean)
  • Demineralizing 90 micron filter (it was very dirty with tiny pieces of debris)
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 321)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). See screenshot attached of T1 state at approximately 4:00pm on 2023-03-06 after the system had been running for a significant time.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

 The T1 cooling package inlet pressure (P1) has been dropping since approximately May 5, 2013 from the regular operating pressure of approximately 75psi to the present value of 47psi.  An increase in the rate of decrease started around August 5th as well as increased noise in the signal.  Plots attached below.  No corresponding change in the outlet pressure, and no change in the water flow rate (also no change in inlet and oulet temperatures) leads to the conclusion that the transducer is failing or gas is trapped in the lines leading to the transducers in the BL1A tunnel.

Jamie Cessford and Ron Kuramoto are aware of the situation.  The issue will be addressed during the September mini-shutdown.

Water reservoir in T1 started draining at ~7.30 am, will leave till tomorrow or overnight to fully drain. 

(Preparation to replace all leaky valves). 

Drained water samples collected for RPG. 

- Both 3/4" ball valves on the T1 cooling package target water circuit were rebuilt, leak tested with house air pressure, and reinstalled

- The 3/4" ball valve on the T1 heat exchanger secondary side was replaced with a new unit

- The 3/4" needle valve on the T1 heat exchanger secondary side was rebuilt, leak tested with house air pressure, and reinstalled

- The T1 cooling package was refilled and restarted using a bypass across the inlet/outlet Hansen fittings

- The package was inspected while running and no leaks were visible (video survey taken with file stored on Isaac's PC)

- All T1 Cooling Package MRO activities are now complete

The T1 expansion tank warning level sensor had tripped and the level was approximately 1cm from the base of the tank.  No trip warning was observed when Tom Lyth checked the panels last week.  The T1 cooling package and target monolith were inspected and no water was observed.  Two puddles of water were found in the BL1A tunnel around the T1 package area: one directly below the 1AQ9 Cable Interlock Tray, and another approximately 1.5m to the west.  However, water was observed in the same area when the tunnel was accessed in January.  The T1 volume is currently vented, so there is no indication from the vacuum system as to whether or not water has leaked into the beamline.

The tank was refilled, and the level had not changed after 1 hour.  I will continue to monitor the level.

Installation, wiring, and commissioning of the high temperature limit switches for the T1 and T2 cooling packages was completed today as part of SAS project SASP0120.  The relay output from the switches was wired in series with the Central Safety System signals for T1 and T2 (cable number 13250 and 12250 respectively) in break-out panel #2 on the Meson Hall south mezzanine.  There were already 3 other circuits wired in series with each signal for various other interlock requirements.

After wiring, commissioning of the switches was performed according to the following procedure:

- With the limit switches in a non-tripped state, it was confirmed that the safety signals were reaching the CSS in the main control room as expected (had to jumper T2 circuit to achieve this as the T2 target is not currently installed)
- The limit switch was tripped by adjusting the set point on the switch, it was confirmed that the signal was lost
- The set point was changed to a non-trip level, and the switch trip cleared, confirmed that the signal in MCR returned
- The enable signal from each cooling package was removed by shutting off the cooling package (for T2 just removed jumper), confirmed that safety signal lost as expected
- Reversed this condition and confirmed that the signal returned
- Checked that the thermocouples are performing properly, and the switches actually trip on high temperature by directly applying heat to the thermocouple area on the cooling package using a heat gun;  the switches tripped as expected when the set-points were exceeded

The results were discussed with John Drozdoff (Safety Systems group leader) who approved the method.

The final task remaining for SASP0120 is to route the PRV outlets to the 1A tunnel active drain using flexible hosing.

 Annual MRO work on the T1 and T2 cooling packages is now complete.  The following tasks were performed:

• Water from both packages drained to the South TNF holding tank (samples taken)
• All Proteus paddle wheels, shafts, and o-rings replaced (5 each at T1 and T2)
• All Cuno filters changed
• Archived data for all sensor readbacks inspected.  Only faulty sensor was B1A:T1CS:FGDEM
• Both packages refilled and restarted; Operating smoothly; Inspected for leaks, none found; Demin flow at ~1.0gpm
• B1A:T1CS:FGDEM readback is fixed with new paddle wheel
• South TNF holding tank water released after approval from RPG
• Old filters added to plastic pail with other T1/T2 filters located directly north of the BL1A blocks boot-box area
• Active water from filter canisters dumped into the RH lab active sink

Grant moved the T1 and T2 targets from position 0 to position 4 this evening at around 9pm.

The position 4 selector button on the T1 rack was not working.  The manual control box had to be used with the adapter cable and military connectors for both T1 and T2 to achieve position 4 on both targets.

 The target water outlet flow transducers on the T1 and T2 cooling packages were serviced (new shaft, paddle wheel, & o-ring) to address a noisy signal.  The service corrected the noisy signal.  Additionally, it appears the worn parts were causing the transducers to read high, as both dropped approximately 1 gallon/minute after the service.  Both transducers would occasionally trip the high flow warning (8.5gpm), and should now no longer do so.

A plot of the water outlet flows before and after the service is attached.  Cyclotron fault report #6086 was returned.

- The T1-MK1 target was moved from the hot cell to storage pit hole #3 (ready as T1 spare)

- The T2-MK2 target was moved from the beam line to the hot cell for inspection and in preparation for T2/M9 work this shutdown
- The target measured up to 90mSv/hr at approximately 1m while being lifted
- Beam spots on the target will be measured and recorded to determine ideal offset (if any) for next running period
- A 10mSv/hr collimated field was measured coming from the T2 monolith hole
- An aluminum blank-off was installed over the hole followed by a stack of circular steel discs for shielding which reduced the max field in the area to 700uSv/hr

 The T1 and T2 targets were both moved from Position 0 to Position 4 at approximately 2:30am on Saturday October 10th using the portable control box.  Operations had tried to move the T1 target to position 4 using the old control system but for unknown reasons it would move to any position except position 4 and the portable control box had to be used.  The portable control box should continue to be used for T1 and T2 target moves until the new control system is installed in the 2016 shutdown.

 During the recent shutdown for the T2/M9 joint repair Graham Waters replaced the corrupted application code on the controller RAM disk.  The T1 target position control is now functioning properly.  T2 target position control is still not operational due to an electronics problem within the motor driver.   The T2 ladder position may still be moved using the portable control box.  The T2 target ladder was moved to position 3 (10cm Be) today in preparation for the upcoming running period.

An electrical check has been completed for the T1 & T2 Profile and Protect Monitors by Dave Cameron and Mike Russell.  They report that all results look normal.

 Diagnostics group completed their "blip" signal response tests on the T1 and T2 profile and protect monitors.  A small amount of leakage was observed on the T2 profile monitor, but was not thought to be a significant concern.  All other results were normal.

The T1 and T2 profile monitors were used during today's development shift.  Target ladder movement for T1 was done at the panel on the mezzanine, for T2 the portable control box was used.  Both monitors actuated properly and the scans did not show any "missing" wires.  The profiles at both targets were found to be too broad and were adjusted to span 2 wires horizontally and 6-7 wires vertically (Yi-Nong Rao).

Controls Group replaced the wiring and wire harness for the profile monitor limit switches on T2.  The monitor was actuated remotely and functioned properly.

The T1 profile monitor was also tested and actuated properly.  The T1 monolith is now ready to be covered so that blocks may be removed from the M20 area.