The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

T1-MK1 was moved from the target station to the hot cell roof. 

Radiation surveys report 140 mSv/hr at 0.5m for the bare T1-MK1 target ladder.

On the hot cell roof, fields are 220uSv/h at 0.5m.

T1-MK2 was subsequently transported from the storage pit to the target station. 

After the transport operation, the 1A vacuum volume was then pumped down, and reached approximatley 1/10 of the vacuum level prior to venting.

Update: a few hours after the vacuum volume started pumping down, vacuum levels stabilized to the level observed prior to venting (see image) - confirming that the o-ring seal on the target is good and that the transfer operation was a success. 

------------------------------------------ 

During the transport operation we noticed that the chain drive on the target transfer flask was slipping considerably more when powered through the power bar on the camera stand. 

This issue was remedied by powering the transfer flask through a dedicated plug, and will be noted in procedures for future transport operations.

Jan 10, 2024:

The cooling packages for T1/T2 was drained this morning. ~150 L from each of the cooling package reservoirs + expansion tank was drained to the BL1A holding tank. The holding tank held ~400 L initially and is therefore now filled to ~700 L (out of a 800 L capacity). We will likely drain the holding tank in the following morning at which point this ELOG will be updated.

---------------------------------------------------------------

UPDATE (Feb 01, 2024): After recieving instruction from RPG to drain the holding tank to the active sump (due to delays in completing Tritium sample tests), we mistakenly drained ~50L of the 1A holding tank water to the city sewage system (initially it held ~610L and after the draining it read ~580L). This error was caused by misinterpretation of the 1A tank draining systems/procedure. In fact, it is uncommon for RH to pump active water from the 1A holding tank into the sump. It is also unclrear whether a valve configuration to do so exists (if it does, no flow reading can be taken). 

However, because of this error, we learned that the plumbing to the city water drain is slightly blocked (see image).

Our plan of action moving forward will be to wait for the Tritium results and coordinate with RPG to determine the appropriate next steps.

 ---------------------------------------------------------------

UPDATE (May 02, 2024): The active sump level was inspected visually on April 29 and it was noticed that it was close to full. 1x 1L and 2x 20mL samples were then collected and passed to RPG for testing. When RPG completes their Tritium analysis, we will drain both the holding tank and the active sump. See this link.

 ---------------------------------------------------------------

UPDATE (Aug 06, 2024): The tritium analysis was finally completed for T1/T2, since the sump only received high active water from the removed T1 target ladder, tritium analysis for the sump water would have been covered by the T1/T2 analysis. Gamma spectroscopy was done for the active sump (here). The draining procedure will be recorded in a separate ELOG.

 See Cyclotron fault 16553. 

"B1A:T2CS:FGTGT readback is toggling at the warn limit of 3.0 GPM and over the last week has started crossing over the trip threshold of 2.8 GPM, tripping off the water package. Initial Action Taken: 1A is scheduled to take beam on December 13."

Upon reviewing the flow trend over the past semester, the flowrate has been hovering around 3.0 GPM the whole time.

Meanwhile, the water temperatures in/out of the T2 target has remained stable between 24C to 31C with the warn and trip limits > 35C. 

It should therefore be safe to run the target cooling water at a lower flowrate.

As a temporary solution, the low trip limit was adjusted to 2.5 GPM. While the warn was kept at 3.0 GPM.

We will look at how flowrate through the target can be increased in the coming shutdown.

For reference, the T1 target flow trip was set to 2.0 GPM an warn was set to 2.3 GPM (see MH-RH ELOG 331).

UPDATE(June 18, 2024):after shutdown service and replacement of the proteus paddle wheels, flow through the target seems to have recovered and is stable at around 3.7-4 gpm.

Today we re-lubricated the o-rings on the top air amplifier for T1/T2, and replaced the o-rings on the bottom air amplifier with new and also lubricated o-rings. 

We used Haskel lubricant #50866.

The bottom air amplifier spool assembly was noticably dirtier than the top (see attached). and we noticed some worn spots on the o-rings. This was not the case with the top air amplifier spool (it has not been used outside of testing since being fully serviced in 2012).

Replacing the o-rings did not result in any perceptible change to the cycle rate and sound of either air amplifier (~50 sec/cycle, same sounds made by the piston before and after servicing). 

We initially planned to also remove the sleeve on the amplifiers (see image) but could not do so easily. We will contact Haskel to get some input regarding how best to remove the sleeve.

The plan is to perform a full teardown of the lower air amplifier next shutdown (2024).

The level sensor for the active sump in the RH lab was tested.
The level sensor is working properly and is operational.
 

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

On May 31, the low active water in BL1A was turned off briefly and upon re-starting, the flow to the T1 heat exchanger was very close to the reported warn limit (~6 gpm). 

To see if this low flow could be remedied, we entered the BL1A tunnels briefly on Jun 06 but found no flow regulator valves for the T1 heat exchanger. 

The low flow to the heat exchanger after the secondary water pump was re-started may be a normal characteristic of the system. Alternatively, this may have been caused by entrapped air in the heat exhanger after it was drained during 2023 shutdown to replace a leaky valve. 

The latter is grounded in the fact that the flow in T2 recovered immediately after the flow was re-started. T2 was drained through a drain line in the tunnels whereas T1 was drained at the cooling station through a filter/copper line. 

The difference in draining methods may have resulted in entrapped air and thus flow issues in T1 but not T2. 

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

The HEPA filter in the nuclear ventilation system for the Meson Hall Hot Cell was changed and tested by mechanical services under work permit C2023-04-25-14.

They have given approval to operate again. The system is re-energized.

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

We tested the air amplifiers by actuating the beam blockers and profile monitors on the T1 and T2 target stations. 

The blockers and profile monitors were each actuated (brought up/out and down/in fully) > 5 times and vacuum remained stable throughout (see attached).

The cycle rate of the air amplifiers (time between 'puffs') under steady conditions (target devices kept in the out position) was measured to be between 60-90 seconds, with 90 seconds corresponding to when the 'flow control' ball valve is fully shut and ~60 seconds when the valve is opened (either partly or fully). 

When a target device is brought up, a cycle event will occur shortly after the device reaches the out position.

This cycle time is much longer than last reported in 2014 (see elog #119), by a factor of 2-3. 

The top amplifier sounds different from the bottom amplifier (cycles between being 'rattly' and 'quite' between cycles).

Seeing how the beam blockers and profile monitors were actuated smoothly with the air amplifiers, we will keep the system running off the lower air amplifier with the 'flow control' ball valve kept open (not shown).

At some point we will look into disassembling the top amplifier to see why it sounds differently between cycle events.

For reference, both amplifiers read 20psi at the regulator and 120 psi to the target station. 

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

 We replaced tha O-ring on M20 at T2 and brought the target station down to vacuum, which allowed the target water pump to be turned on. 

After turning the pump on, we inspected the replaced valves for any leaks and found none - the flow on the demineralizing line was adjusted to 1.2GPM (also done at T1) by turning the needle valve.

Will observe till tomorrow to ensure that the T2 system is stable before proceeding with last system check.

 We moved M20 back to the T2 monolith and re did the connections for air + limit switches + T2 profile monitor air.

Note: T2 profile monitor electronics sparked when we moved M20 out initially. (Update: Shengli Liu from probes group performed a test on the T2 profile monitor electronics on Mar24 and found that they are working properly)

M20 started leaking when we tried to bring it to vaccum - from the monolith o ring seal and not the main shaft seal - will look to correct on monday.

Verified through measurement that the replacement o ring was the correct size (#268 0.139" diameter compared to the 0.131-0.137" that was remved - likely shrunk over use).

Update (Mar 06, 2023): M20 was lifted ~5 in above the monolith flange and we identified that the leak was caused due to damage to the o ring (image attached).

The o ring likely damaged when M20 was brought into contact wit the monolith flange, lifted up again, and brought down for a final time before vaccum check.

The above procedure was done because the flask tends to stall close to the lowermost position and prevents unlatching unless lifted systems are lowered fully.

This will be noted in updated procedures.

 T1/T2 valves were not leaky still (only a few droplets) - pending T2 cooling system pump on for final leak verification.

I have filled the water package on T2 with DI water. The expansion tank is at 38.1cm. This may change when the pump is turned on.

I have refilled the water package on T1 with DI water. The Expansion Tank is at 37cm. This may change when pump is turned on.

3x 3/4" swagelok ball valves (2x at T2 and 1x at T1) were replaced after the water cooling line was shut off at the BL1A tunnels late yesterday. 

We were able to relieve pressure from the T2 water cooler + M8 colA/B (they are the same line) at the BL1A tunnel through a drain port but no such port exists for the T1 heat exchanger line in BL1A.

The T2 heat exchanger line was virtually dry at the replaced valves after pressure was relieved.

The T1 heat exchanger line was still pressurized but while looking for a suitable port to drain at the cooling package, one of the copper fittings at the 1/4" heat exchanger line broke off at the connection to a green valve. 

~1L of water dripped from this line and this was sufficient to relieve pressure from the T1 heat exchanger line - allowing for the 3/4" ball valve to be replaced. 

Until the green valve gets replaced, the T1 heat exchanger line must remain shut off in the BL1A tunnels as the broken copper line is not plugged currently.

Update (last picture): the green valve was replaced with an equivalent 1/4" swagelok ball valve. We also incorporated two 90 deg. bends in the copper tubing to help relieve any stresses that may develop.

The M20 beam blocker was moved from the T2 monolith to the meson hall hot cell with the remote handling flask (completed ~4pm).

This operation was performed in preparation for M20 maintenance (main shaft o-ring replacement + air cylinder re-lubrication).

RPG surveyors measured 20,000 CPM from swiping the surface of the beam blocker shaft, and measured 25 mSv/hr on contact with the beam blocker.

Fields on the hot cell roof were at 100 uSv/hr at 0.5 meters from the top of the M20 BB. This should reduce drastically with the introduction of lead blanket shielding (see attached pictures).