The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.