As of 2024-04-03, the BL1A Holding Tank water level is approximately 580L. It is recommended to drain the tank prior to the start of the operating year.

The lower amplifier for the T2 target station air supply was removed for teardown, inspection, and rebuild. 

The goal of this operation is to understand wear development in the air amplifier over ~13 years of operation, and potentially determine a recommeded service interval.

---------------------------------------------------------------

The T2 volume was vented for an unrelated maintenance operation during this time.

During testing, prior to removing the lower amplifier, both regulators were set to ~20 psi. 

The upper air amplifier resulted in ~125 psi at the outlet while the lower air amplifier resulted in ~120 psi at the outlet.

The upper air amplifier had more audible air flowing out from the muffler than the lower amplifier. 

'Scratching' sounds in both amplifiers were comparable.

The following cycle times were recorded with the M20 BB raised/out (min:sec):

    UPPER: 1:22 / 1:00 / 2:06

    LOWER: 2:30 / 1:46 / 2:12

The following times were recorded to raise the M20 BB (sec): 

    UPPER: 8.36 / 8.76

    LOWER: 8.56 / 9.10

These times will be compared against after completing the teardown and rebuild of the lower amplifier, at which point this ELOG will be updated.

-----------------------------------------------------------------

UPDATE (Feb 23, 2024):

The lower air amplifier was serviced (photos in 'S:\Albert Kong\Shutdown Files\2024\Feb20 T2 Lower Amplifier Cleanup'):

1. Full assembly cleaning.
2. Piston and barrel was cleaned and lubricated, o-rings and plastic components replaced (with new lubricated ones).
3. Pilot valve components replaced (except plug).
4. Check valves (4x) cleaned and plastic/rubber components replaced (with new lubricated ones). 
5. Muffler cleaned (blown out with compressed air).
6. Spool and sleeve assembly cleaned, o-rings replaced (with new lubricated ones), and rubber stopper replaced.
7. Clamping rods tightened to ~17 ft-lbs. 

Note: the piston o-rings were difficult to seat on the piston body/teflon ring. During assembly, we instead seated the o-ring in the barrel on the piston plates (see picture), which allowed the oring to be seated properly, before placing the piston body onto the piston rod.

After servicing, the amplifier was returned to the station, air connections reconnected, and tested. 

Note: it is recommended to do torque-ing of the clamping rods as a final step to simplify mounting of the amplilfier and re-doing connections to the rest of the compressed air system.

The first observation we made was how silently the lower amplifier now operates when cycled: only the exhaust sound from the muffler can be heard.

Note that the piston's motion can be heard when listening ~5cm away from the amplifier barrel. 

The following times were recorded to raise the M20 BB (sec):

    UPPER: ~8.5 

    LOWER: ~7.7s

The following cycle times were recorded with the M20 BB in the out/raised position (min:sec):

    UPPER: 1:05 / 1:04

    LOWER: 1.22 / 1:45

The outlet pressure from the amplifier registers 120 psi with the regulator set only to 15 psi (improvement from previous performance as well as the upper air amplifier's performance).

We will check in on the amplifier next week to see if it still operates silently and can actuate the beam blockers/profile monitor without issue, at which point this ELOG will be updated. 

-----------------------------------------------------------------

UPDATE (Feb 26, 2024):

The lower amplifier was inspected this morning, higher volume sound could be heard from the drum in concert with the motion of the piston, but still much quieter than before servicing.

Feb 1 2024

Hand Pump observed to be losing pressure at full extension of cylinder, seals likely blown. Due to Enerpac P84 being an obsolete model, a newer model hand pump was ordered.

Feb 15 2024

Hand pump (Enerpac P84 Ultima) installed at Indium Extruder Station, observed to work properly in both directions of flow, deemed operational. 

 The recently removed resin can from the T2 target station was rinsed with city water and flushed with compressed air for 1 cycle. 

We will run it through a few more cycles before proceeding with drying under the fume extractor, at which point this ELOG will be updated. 

UPDATE (Feb 08, 2024): The T2 resin can underwent 3 more air cycles. Today it was moved under the fume extractor with the lid cracked open and will be left to dry till next shutdown. It is worth noting that some clumps of old resin was found in the secondary resin can which was previously under the fume extractor. This resin should be disposed of alongside the T2 can's resin once dried.

The recently removed T1-MK1 target ladder was flushed with city water for ~30s at all target positions. It was then purged with compressed air at ~5 psi for ~1 minute at all target positions. We then left the target ladder in pos5 to fully extend the bellows and allow it to dry overnight.

This ELOG will be updated as we progress with work done to dry the target ladder and eventually exchange the spent targets.

-----------------------------------

UPDATE (Feb 02, 2024): in the morning, the target ladder was moved to pos4, and air was allowed to run through the morning, in the afternoon the ladder was moved to pos2 and air was ran. At the end of the day, the ladder was moved to pos 0 and air was ran through over the weekend.

-----------------------------------

UPDATE (Feb 05, 2024): the following procedures were completed.

1. the air connections on the target ladder was replaced with the leak test/vacuum connections as per instructions.
2. LN2 was taken from the ISAC-I facility and used to perform two pump-down cycles with the Agilent He leak detector + cold trap.
3. First pumpdown cycle: 3min45sec to go from roughing to fine pumping, left for a total pumpdown time of1hr20mins, a relatively small amount of ice formed on the cold trap cylinder (see image).
4. Second pumpdown cycle: 2min30sec to go from roughing to fine pumping, 4min30sec to reach 1e-3 Torr, 5min to reach < 1e-3 Torr. 
5. Due to good pumpdown in the second cycle, we proceeded with leak testing: Baseling 1.0e-10 Torr-L/sec | ~1 sec He applications at 3.5 Psi directed at fittings/welds/bellows. Worse case measured leak was 0.9e-7 Torr-l/sec in pos 13 of fig 12 of Document 46600 (see taken image for detailed measurements at various locations).

We will prepare an update to the procedure to better log the leak rate measurements and update this ELOG/the procedure. Note that ice removed from second pumpdown was actually comparable to the first (see image, ice knocked off during cold trap removal), but total pumpdown duration (including leak testing) was ~4 hours for the second pumpdown. 

 UPDATE (Feb 13, 2024): see attached record with new template.

 The following tasks were completed:

1. Yesterday (Jan 18), the T1/T2 cooling packages were inspected for leaks and some moisture was found underneath the CUNO filters behind the resin cans on both T1/T2. It was determined that the source of the leak was a loose fitting on the drain valves of the CUNO filter housing (see photo). Today we confirmed that no more moisture is found underneath the CUNO filter after these fittings were tightened. A note will be made to keep this in mind for future shutdown operations.
2. The shaft couplings on the T1/T2 target ladder drive system were inspected: no cracks were identified on the rubber couplings.
3. Actuation of the T1/T2 profile monitors were tested. Vacuum levels at T1/T2 remained stable throughout (see picture).
4. The pneumatic connections/hose/fittings for the T1/T2 profile monitors, and M9/M20 BB were inspected - therere were some air leaks out from the ASCO valve + pneumatic flow control valve on the T2 profile monitor when it was in the down/out position. The same leak out of the pneumatic valve by the profile monitor connection was found at T1. Given that both profile monitors actuated properly however, these leaks were determined to not be an issue.

Next week, we will continue work by actuating the M9/M20 BB at T2, at which point this ELOG will be updated.

UPDATE Jan 30, 2024: The M9 and M20 beam blockers were tested, no leaks through the tubing were observed, actuation was quick, ops confirmed no perceptible change in 1A vacuum levels during actuation and EPICS reads correct blocker position signals.

Note: the M20 blocker was actuated by ops, while the M9/T2 blocker was actuated using the ASU on the B2 level of the Meson Hall. The ASU did not require the area to be blocked off to actuate the T2 blocker.

------------------------------------

Note: Ops noted that the profile monitor interlock status has to be reset if a timeout is encountered (such as if the pump were to be turned off or a trip was encountered) before the profile monitor can be actuated (see picture).

Note: When inspecting the system, we tried to open the rotary collimator air supply and the handle on the pannel mounted valve broke off (like it did for the T2 valve last shutdown - see picture). 

UPDATE Jan 30, 2024: The handle to the rotary collimator air supply valve was replaced (see image).

Note: There was also some small leaks at the hose clamps at the T2 profile monitor, in the out position. This leak was however smaller than the leak through the valve. 

 The following tasks were completed in preparation of re-filling the T1 and T2 cooling packages:

1. All valves were returned to the 'open' configuration (note, compressed air to collimators at T1 is to remain closed - rotating collimator no longer in use || also note, odd handle configuration at T1 panel when all valves are open - see attached).
2. Leaks from valves were checked, two leakly valves found at T2 (Swagelok SS-44S6 and SS-45S12), the stem of these valves were re-tightened and leaks were remedied (see picures).
3. Spent CUNO filters in buckets were dealt with: water drained to MH HC active sink, and filters organized into a single 'dry' bucket.

The cooling packages will be filled to 38 cm at the expansion tank tomorrow, and the pumps turned on to check for leaks at pressure, at which point this ELOG will be updated.

UPDATE:

The cooling packages were filled on Jan 16 to ~38 cm, after which both pumps were turned on and a few more leaky valves were identified at the T2 station (none at T1).

We remedied the leaks by tightening their stems and will continue to monitor both systems for a few days.

Besides the leak, we found that the TGT:OUT pressure gage was incorrectly reading 0 psi. We will be entering the 1A tunnels to drain the 1A holding tank soon, at which point we will look to remedy this issue (either a valve was left closed, or an air column is trapped at the gage port).

UPDATE (30 Jan, 2024): TGT:OUT pressure transducer at T1 was successfully replaced and is now reading the correct pressure (see image, ~20 psi is about the same as readings from 2021 when the sensor was working properly).

The resin flask on the T2 cooling package was exchanged today (new resin can prepped in the morning). 

The newly installed resin flask was placed slightly off-set from the marking on the platform to ensure that the braided hose will clear the blocks when replaced at the end of shutdown.

The spent resin can was dropped off in the hot cell, ready to be prepped for drying. 

See attached image for illustration of the rigging solution used, ~20 ft was covered by two sligs and a shackle to clear the MH mezzanine.

 The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

T1-MK1 was moved from the target station to the hot cell roof. 

Radiation surveys report 140 mSv/hr at 0.5m for the bare T1-MK1 target ladder.

On the hot cell roof, fields are 220uSv/h at 0.5m.

T1-MK2 was subsequently transported from the storage pit to the target station. 

After the transport operation, the 1A vacuum volume was then pumped down, and reached approximatley 1/10 of the vacuum level prior to venting.

Update: a few hours after the vacuum volume started pumping down, vacuum levels stabilized to the level observed prior to venting (see image) - confirming that the o-ring seal on the target is good and that the transfer operation was a success. 

------------------------------------------ 

During the transport operation we noticed that the chain drive on the target transfer flask was slipping considerably more when powered through the power bar on the camera stand. 

This issue was remedied by powering the transfer flask through a dedicated plug, and will be noted in procedures for future transport operations.

Jan 10, 2024:

The cooling packages for T1/T2 was drained this morning. ~150 L from each of the cooling package reservoirs + expansion tank was drained to the active sump holding tank. The active sump holding tank held ~400 L initially and is therefore now filled to ~700 L (out of a 800 L capacity). We will likely drain the holding tank in the following morning at which point this ELOG will be updated.

---------------------------------------------------------------

UPDATE (Feb 01, 2024): After recieving instruction from RPG to drain the holding tank to the active sump (due to delays in completing Tritium sample tests), we mistakenly drained ~50L of the 1A holding tank water to the city sewage system (initially it held ~610L and after the draining it read ~580L). This error was caused by misinterpretation of the 1A tank draining systems/procedure. In fact, it is uncommon for RH to pump active water from the 1A holding tank into the sump. It is also unclrear whether a valve configuration to do so exists (if it does, no flow reading can be taken). 

However, because of this error, we learned that the plumbing to the city water drain is slightly blocked (see image).

Our plan of action moving forward will be to wait for the Tritium results and coordinate with RPG to determine the appropriate next steps.

 See Cyclotron fault 16553. 

"B1A:T2CS:FGTGT readback is toggling at the warn limit of 3.0 GPM and over the last week has started crossing over the trip threshold of 2.8 GPM, tripping off the water package. Initial Action Taken: 1A is scheduled to take beam on December 13."

Upon reviewing the flow trend over the past semester, the flowrate has been hovering around 3.0 GPM the whole time.

Meanwhile, the water temperatures in/out of the T2 target has remained stable between 24C to 31C with the warn and trip limits > 35C. 

It should therefore be safe to run the target cooling water at a lower flowrate.

As a temporary solution, the low trip limit was adjusted to 2.5 GPM. While the trip was kept at 3.0 GPM.

We will look at how flowrate through the target can be increased in the coming shutdown.

For reference, the T1 target flow trip was set to 2.0 GPM an warn was set to 2.3 GPM (see MH-RH ELOG 331).

Today we re-lubricated the o-rings on the top air amplifier for T1/T2, and replaced the o-rings on the bottom air amplifier with new and also lubricated o-rings. 

We used Haskel lubricant #50866.

The bottom air amplifier spool assembly was noticably dirtier than the top (see attached). and we noticed some worn spots on the o-rings. This was not the case with the top air amplifier spool (it has not been used outside of testing since being fully serviced in 2012).

Replacing the o-rings did not result in any perceptible change to the cycle rate and sound of either air amplifier (~50 sec/cycle, same sounds made by the piston before and after servicing). 

We initially planned to also remove the sleeve on the amplifiers (see image) but could not do so easily. We will contact Haskel to get some input regarding how best to remove the sleeve.

The plan is to perform a full teardown of the lower air amplifier next shutdown (2024).

The level sensor for the active sump in the RH lab was tested.
The level sensor is working properly and is operational.
 

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

On May 31, the low active water in BL1A was turned off briefly and upon re-starting, the flow to the T1 heat exchanger was very close to the reported warn limit (~6 gpm). 

To see if this low flow could be remedied, we entered the BL1A tunnels briefly on Jun 06 but found no flow regulator valves for the T1 heat exchanger. 

The low flow to the heat exchanger after the secondary water pump was re-started may be a normal characteristic of the system. Alternatively, this may have been caused by entrapped air in the heat exhanger after it was drained during 2023 shutdown to replace a leaky valve. 

The latter is grounded in the fact that the flow in T2 recovered immediately after the flow was re-started. T2 was drained through a drain line in the tunnels whereas T1 was drained at the cooling station through a filter/copper line. 

The difference in draining methods may have resulted in entrapped air and thus flow issues in T1 but not T2. 

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

The HEPA filter in the nuclear ventilation system for the Meson Hall Hot Cell was changed and tested by mechanical services under work permit C2023-04-25-14.

They have given approval to operate again. The system is re-energized.

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops.