Maico/Eric/Albert:

The expansion tank in T1 was filled to ~36cm in the morning.

We contacted the control room to test flowrates in the T1 cooling loop after new installation of resin in the demineralizing line.

Epics interlocks tripped immediately as pump was turned on (flow too low or pressure to high) due to new resin.

Turning on the pump repeatedly extends the delay before system trips, until eventually the pump can run continuously. 

Flowrate through demin line initially too high (~2.5 gpm), adjusted flow control valve to bring it to ~1.4 gpm (setpoint at the time ~0.4-1.5 gpm).

When the pump was turned on initially, saw physical pressure gauge (dial type) bottom out/make greater than 1 full rotation (>50 Psi). 

No connection between the physical dial to EPICS so no cause for immediate concern.

All valves are more leaky than when the pumps are off - leaks definitely not due to depressurization. 

Attached: photo of the likely 'faulty' dial pressure gauge. 

Water reservoir in T1 started draining at ~7.30 am, will leave till tomorrow or overnight to fully drain. 

(Preparation to replace all leaky valves). 

Drained water samples collected for RPG. 

A 0.75" SwageLok ball valve that was taken out from the T1 cooling package was inspected (SS-45S12).

All valves in the T1/T2 cooling packages likely leaked due to damage to plastic/rubber sealing components. 

SEE PICTURES ATTACHED.

We can see that some bits of white plastic, possibly a bushing, has disintegrated and is leaking out from the valve shaft.

Additionally, there is a considerable amount of dirt/grit in and around the ball.

The surface of the valve opening exibits visible signs of wear (new valve opening is perfectly smooth).

Finally, a red-colored material is found on some small stainless steel parts which is either buildup from contaminants in the line or worn silicon/rubber lining.

Hypothesis: 90 um filter which had not been replaced caused the water in the line to build up particulates and wear plastic/rubber parts in the valve, causing them to leak. 

From the previous test (ELOG 315), we know that the leak is not due to depressurization.

3x 3/4" swagelok ball valves (2x at T2 and 1x at T1) were replaced after the water cooling line was shut off at the BL1A tunnels late yesterday. 

We were able to relieve pressure from the T2 water cooler + M8 colA/B (they are the same line) at the BL1A tunnel through a drain port but no such port exists for the T1 heat exchanger line in BL1A.

The T2 heat exchanger line was virtually dry at the replaced valves after pressure was relieved.

The T1 heat exchanger line was still pressurized but while looking for a suitable port to drain at the cooling package, one of the copper fittings at the 1/4" heat exchanger line broke off at the connection to a green valve. 

~1L of water dripped from this line and this was sufficient to relieve pressure from the T1 heat exchanger line - allowing for the 3/4" ball valve to be replaced. 

Until the green valve gets replaced, the T1 heat exchanger line must remain shut off in the BL1A tunnels as the broken copper line is not plugged currently.

Update (last picture): the green valve was replaced with an equivalent 1/4" swagelok ball valve. We also incorporated two 90 deg. bends in the copper tubing to help relieve any stresses that may develop.

I have refilled the water package on T1 with DI water. The Expansion Tank is at 37cm. This may change when pump is turned on.

I have filled the water package on T2 with DI water. The expansion tank is at 38.1cm. This may change when the pump is turned on.

 We replaced tha O-ring on M20 at T2 and brought the target station down to vacuum, which allowed the target water pump to be turned on. 

After turning the pump on, we inspected the replaced valves for any leaks and found none - the flow on the demineralizing line was adjusted to 1.2GPM (also done at T1) by turning the needle valve.

Will observe till tomorrow to ensure that the T2 system is stable before proceeding with last system check.

On May 31, the low active water in BL1A was turned off briefly and upon re-starting, the flow to the T1 heat exchanger was very close to the reported warn limit (~6 gpm). 

To see if this low flow could be remedied, we entered the BL1A tunnels briefly on Jun 06 but found no flow regulator valves for the T1 heat exchanger. 

The low flow to the heat exchanger after the secondary water pump was re-started may be a normal characteristic of the system. Alternatively, this may have been caused by entrapped air in the heat exhanger after it was drained during 2023 shutdown to replace a leaky valve. 

The latter is grounded in the fact that the flow in T2 recovered immediately after the flow was re-started. T2 was drained through a drain line in the tunnels whereas T1 was drained at the cooling station through a filter/copper line. 

The difference in draining methods may have resulted in entrapped air and thus flow issues in T1 but not T2. 

Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

 See Cyclotron fault 16553. 

"B1A:T2CS:FGTGT readback is toggling at the warn limit of 3.0 GPM and over the last week has started crossing over the trip threshold of 2.8 GPM, tripping off the water package. Initial Action Taken: 1A is scheduled to take beam on December 13."

Upon reviewing the flow trend over the past semester, the flowrate has been hovering around 3.0 GPM the whole time.

Meanwhile, the water temperatures in/out of the T2 target has remained stable between 24C to 31C with the warn and trip limits > 35C. 

It should therefore be safe to run the target cooling water at a lower flowrate.

As a temporary solution, the low trip limit was adjusted to 2.5 GPM. While the warn was kept at 3.0 GPM.

We will look at how flowrate through the target can be increased in the coming shutdown.

For reference, the T1 target flow trip was set to 2.0 GPM an warn was set to 2.3 GPM (see MH-RH ELOG 331).

UPDATE(June 18, 2024):after shutdown service and replacement of the proteus paddle wheels, flow through the target seems to have recovered and is stable at around 3.7-4 gpm.

Jan 10, 2024:

The cooling packages for T1/T2 was drained this morning. ~150 L from each of the cooling package reservoirs + expansion tank was drained to the BL1A holding tank. The holding tank held ~400 L initially and is therefore now filled to ~700 L (out of a 800 L capacity). We will likely drain the holding tank in the following morning at which point this ELOG will be updated.

---------------------------------------------------------------

UPDATE (Feb 01, 2024): After recieving instruction from RPG to drain the holding tank to the active sump (due to delays in completing Tritium sample tests), we mistakenly drained ~50L of the 1A holding tank water to the city sewage system (initially it held ~610L and after the draining it read ~580L). This error was caused by misinterpretation of the 1A tank draining systems/procedure. In fact, it is uncommon for RH to pump active water from the 1A holding tank into the sump. It is also unclrear whether a valve configuration to do so exists (if it does, no flow reading can be taken). 

However, because of this error, we learned that the plumbing to the city water drain is slightly blocked (see image).

Our plan of action moving forward will be to wait for the Tritium results and coordinate with RPG to determine the appropriate next steps.

 ---------------------------------------------------------------

UPDATE (May 02, 2024): The active sump level was inspected visually on April 29 and it was noticed that it was close to full. 1x 1L and 2x 20mL samples were then collected and passed to RPG for testing. When RPG completes their Tritium analysis, we will drain both the holding tank and the active sump. See this link.

 ---------------------------------------------------------------

UPDATE (Aug 06, 2024): The tritium analysis was finally completed for T1/T2, since the sump only received high active water from the removed T1 target ladder, tritium analysis for the sump water would have been covered by the T1/T2 analysis. Gamma spectroscopy was done for the active sump (here). The draining procedure will be recorded in a separate ELOG.

 The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

The resin flask on the T2 cooling package was exchanged today (new resin can prepped in the morning). 

The newly installed resin flask was placed slightly off-set from the marking on the platform to ensure that the braided hose will clear the blocks when replaced at the end of shutdown.

The spent resin can was dropped off in the hot cell, ready to be prepped for drying. 

See attached image for illustration of the rigging solution used, ~20 ft was covered by two sligs and a shackle to clear the MH mezzanine.

 The following tasks were completed in preparation of re-filling the T1 and T2 cooling packages:

1. All valves were returned to the 'open' configuration (note, compressed air to collimators at T1 is to remain closed - rotating collimator no longer in use || also note, odd handle configuration at T1 panel when all valves are open - see attached).
2. Leaks from valves were checked, two leakly valves found at T2 (Swagelok SS-44S6 and SS-45S12), the stem of these valves were re-tightened and leaks were remedied (see picures).
3. Spent CUNO filters in buckets were dealt with: water drained to MH HC active sink, and filters organized into a single 'dry' bucket.

The cooling packages will be filled to 38 cm at the expansion tank tomorrow, and the pumps turned on to check for leaks at pressure, at which point this ELOG will be updated.

UPDATE:

The cooling packages were filled on Jan 16 to ~38 cm, after which both pumps were turned on and a few more leaky valves were identified at the T2 station (none at T1).

We remedied the leaks by tightening their stems and will continue to monitor both systems for a few days.

Besides the leak, we found that the TGT:OUT pressure gage was incorrectly reading 0 psi. We will be entering the 1A tunnels to drain the 1A holding tank soon, at which point we will look to remedy this issue (either a valve was left closed, or an air column is trapped at the gage port).

UPDATE (30 Jan, 2024): TGT:OUT pressure transducer at T1 was successfully replaced and is now reading the correct pressure (see image, ~20 psi is about the same as readings from 2021 when the sensor was working properly).

 The recently removed resin can from the T2 target station was rinsed with city water and flushed with compressed air for 1 cycle. 

We will run it through a few more cycles before proceeding with drying under the fume extractor, at which point this ELOG will be updated. 

UPDATE (Feb 08, 2024): The T2 resin can underwent 3 more air cycles. Today it was moved under the fume extractor with the lid cracked open and will be left to dry till next shutdown. It is worth noting that some clumps of old resin was found in the secondary resin can which was previously under the fume extractor. This resin should be disposed of alongside the T2 can's resin once dried.

At ~8am on May 06, the expansion tank level sensor for the T2 cooling system suddenly became noisy. 

Cyclotron fault ref: 16915

Approximately 11:00 am today, the noisy sensor was replaced with a spare, upon which it was learned that the spare sensor is broken (registers 0 level and not detected on PC through USB adapter).

The old (noisy sensor) was then replaced at around 12.00 pm onto the expansion tank and it was found that the noise had subsided.

A possible explanation to the noise would be loose connections/grounding wire.

We will continue to monitor the sensor in the coming days, and order replacement sensors has been placed and we will be able to replace the sensor soon should it become noisy again.

Edit 2024-05-07 - A. Newsome: EPICS monitoring shows the sensor appears to be behaving normally since the aforementioned events. Most likely attributed to improper grounding. The fault will be closed. See attached screenshot.

Secondary flow sensor B1A:T2CS:FGSEC:RDFLOW on the T2 cooling station suddenly became very noisy at around 11 pm, 01 September 2024.

The sensor reading would fluctuate from 0-150 gpm which is beyond typical noise levels for these sensors (~2gpm).

The noise spontaneously ended at around 4 am, 05 September 2024.

The noise likely originated from some stuck debris preventing the paddle wheel from turning normally that got spontaneously dislodged.

Alternatively, changes in the environment temperature could have broken some electrical contact that recovered when the temperature cooled over the weekend. We will make a note to look into this sensor in the coming shutdown and perform preventative replacement of components.

The T1 and T2 cooling packages were drained starting at 14:36, by 14:48 both expansion tanks had dried, meaning the draining rate for both tanks were ~80L/hr, generally it will only take 2 hours to fully drain the system.

Tritium samples were collected after allowing the system to drain for ~5 minutes to clear out water in the drain lines and actually collect samples from the reservoirs. The samples were passed onto RPG for analysis.

The drain valves on the cooling packages and in the BL1A tunnel is left open.

At 9:15am this morning BL1A tripped off due to loss of the watchdog signals for the T1/T2 control system.  Graham Waters investigated and the problem was found to be a corrupted initialization file.  The error was corrected, and BL1A was started by 11:30am.