The following work has been completed on the T2 cooling package:

- T2 system drained to South TNF holding tank (completed in Jan 2017, water already released)
- Target water and demin. circuit Cuno filters changed
- Both female Hansen fittings at target end of flexible hose replaced with new SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion inside)
- Small leak on pipe thread fitting near target water supply solenoid repaired
- Reservoir o-rings replaced
- System filled and re-started, no leaks observed, no drop in water level over 48 hour run period
- Demin. circuit flow rate tuned to 1.0gpm

At ~8am on May 06, the expansion tank level sensor for the T2 cooling system suddenly became noisy. 

Cyclotron fault ref: 16915

Approximately 11:00 am today, the noisy sensor was replaced with a spare, upon which it was learned that the spare sensor is broken (registers 0 level and not detected on PC through USB adapter).

The old (noisy sensor) was then replaced at around 12.00 pm onto the expansion tank and it was found that the noise had subsided.

A possible explanation to the noise would be loose connections/grounding wire.

We will continue to monitor the sensor in the coming days, and order replacement sensors has been placed and we will be able to replace the sensor soon should it become noisy again.

Edit 2024-05-07 - A. Newsome: EPICS monitoring shows the sensor appears to be behaving normally since the aforementioned events. Most likely attributed to improper grounding. The fault will be closed. See attached screenshot.

The T2 cooling package water was drained to the BL1A active sump, refilled, cycled, and drained again.  Samples of T2 cooling package active water was given to RPG, tested, and approved for release.

On January 21 at approximately 4pm the full volume of the sump (280L) was released to the city sewer.

 The T2 cooling package was uncovered this morning and a leak was found coming from the Swagelok ball valve on the 3/4" target water outlet line.  The water was coming from the stem of the valve, indicating that water was leaking through the valve packing, rather than at the fittings for the valve.  The leak rate was approximately 1 drop every 5 seconds and a small puddle was visible on the top of the water reservoir.  The water package was drained (samples were taken and given to RPG).  After draining, the valve was replaced with a spare unit.  The cooling package was refilled and restarted.  The package was inspected while running and no visible water leaks were found.  The flow rate through the resin can was adjusted from approximately 1.7 to 1.5 gallons per minute.

Both defeats applied to T2 cooling package thermistors were removed today (Col. B interlock was jumpered, and the target water inlet and outlet signals had been swapped).  It was confirmed that all thermistors are now wired correctly.  The readback of both target water thermistors is currently unstable.  Inspection of the Lemo connectors showed that the internal plastic parts of all connectors were damaged or missing.  If the T1/T2 controls upgrade proceeds this shutdown, then all thermistors will be replaced with thermocouples, so after a decision is made on that project (sometime in Feb 2015) these thermistors will either be repaired or replaced.

The resin can was removed from the T2 cooling package and transported to the warm cell.  A replacement can with fresh resin was then installed.

A 12' sling and the standard four bridle was used to prevent the crane hook block from hitting the mezzanine railing on the south side.

The T2 package was started and inspected for leaks - none found.

Although T2 resin can replacement was scheduled for the 2015 shutdown, a decline in conductivity (with beam off) was observed since the cooling package was started in mid-march.
(See attached XSTRIP plot showing gradual decline and then increase after resin can swap)

The T2 resin can will now be scheduled for replacement every 2 years (rather than 3).  It is still unknown whether the T1 resin can will also need replacing every 2 years.

The following work was done today at the T2 cooling package:

- Remainder of #4 Hansen o-rings replaced  (see E-log from Jan 30, 2015)
- Installed the following ball valves which have been rebuilt & tested with 100psi air: target outlet 3/4" (x2), resin can outlet filter isolation 1/2" (x2), heat exchanger secondary side outlet 3/4" (x2), collimator A outlet 3/4", collimator B outlet 3/8"
- Installed the following ball valve which has been tightened & tested with 100psi air (no rebuild kit available): resin supply side 1/2"
- Disassembled the target water inlet thermistor probe assembly to check compatibility with thermocouple probes purchased for T1/T2 controls upgrade
- Misplaced a fitting that the probe inserts through, plugged thermistor port as a temporary measure (the outlet thermistor still functions as machine protection)
- Filled cooling package and opened CuALCW supply and return valves to T2 area in BL1A tunnel
- Started package, inspected for leaks, tightened various Swagelok fittings to stop minor leaks
- Tightened valve stem packing compression nut on Col.A/M8BB shut-off valve and Col B. shut-off valve on cooling package front panel to stop drip leaks from the valve stems
- Inspected again for leaks while cooling package running:  none observed

The T2 cooling package is ready for a interlock response timing test which will be performed tomorrow (part of T1/T2 controls upgrade).
Before the T2 cooling package is ready for operation, the target water inlet thermistor must be reinstalled, and the thermistor connectors must be replaced (if the thermistors are not replaced with thermocouples as part of the controls upgrade)

Total dose:

Isaac - 0.08mSv
Keith - 0.01mSv
 

T2 Cooling Package MRO began Jan. 16, 2023.

The following maintenance tasks have been completed for T2 (this list will be updated as work progresses):

• Drained water [2023-01-16]
• Replaced ball valve located between heat exchanger and target flow sensor FGTGT (part number SS-45S12)... this part, as well as numerous other ones, were noted to have been leaking. See e-log 308
• Replaced both CUNO filters (target water and demin. resin circuit filter) [2023-02-01]
• Replaced 90 micron demin. filter (the filter was very dirty - it was last changed in Feb. 2012 as indicated by written note) [2023-02-01]
• Inspected Hansen fittings (o-rings look OK) [2023-02-01]
• Replaced o-ring, shaft, and paddle wheel for flow sensor FGSEC (Q3). The old paddle wheel had significant wear in the shaft hole which resulted in sensor readback issues observed during 2022 operation. It is likely that this was caused by debris or a manufacturing defect which got worse over time. Photos show a comparison between old and new. [2023-02-01]
• Replaced leaky valves (see e-log 318 and 320)
• Refilled expansion tank water (see e-log 322)
• Changed M20 beam block o-ring seals, and performed vacuum check (see e-logs 319, 324, and 323)
• Tuned demin flow rate to 1.1 gpm, turned on pump [2023-03-06], confirmed system functionality (all sensors OK, no water leaks observed). Note: target water conductivity sensor (CONDTGT) took a few hours to ramp up to appropriate value. See screenshot attached of T2 state at approximately 9:15am on 2023-03-07 after the system ran overnight.

As of now, the cooling system is ready for beam. Prior to beam-on, actuation tests of key items such as the target ladder and beam blocker need to be performed. It is expected that this will be done within the next 1-2 weeks.

Bill Rawnsley completed an electrical check of the T2 profile monitor today from the Meson Hall mezzanine - All OK.  

The T2 cooling package was refilled and started.  There was no change to the T2 vacuum when starting the package.  While running, the package was visually inspected for water leaks - no leaks found.  

The lower air amplifier has also been started and is operating normally (the o-ring seal of the moisture trap on the amplifier inlet required cleaning and lubricating to seal properly).

David Cameron and Mike Russel completed an electrical check of the T2 protect monitor last week - All OK.

On the control panel on the mezzanine the "monitor in" lamp is currently illuminated which is incorrect (the monitor is currently out, and proper actuation of the micro switches was checked).  This problem started last year and did not seem to cause issues during 2013 running period.  

Operators were unable to move the T2 profile monitor to the in position from the control room.  This may be caused by a separate interlock that is not satisfied, or may be a new controls issue.  To confirm that the T2 target assembly is not at fault, power was supplied directly to the profile monitor air cylinder solenoid valve and the monitor traveled into position normally.  This will be investigated further before blocks are replaced.

The T2 Collimator A flow meter signal (B1A:T2:FGCOLA) began fluctuating and occasionally reading 0gpm on Sunday Oct 25 causing BL1A trips and machine down time.

On the maintenance day today (Tues Oct 27), the T2 package was uncovered and the T2 Col. A flow meter was opened.  No unusual deposits were observed as seen recently with other flowmeters for BL1A magnets on the CuALCW system.  Some minor wear was observed on the dowel pin shaft, and the paddle wheel didn't spin quite as freely as when new.  A small amount of reddish/brown film (rust?) was seen on the internal SS faces.  The faces were wiped clean and new dowel pin, paddle wheel and o-ring were installed.

The pin, paddlewheel, and o-ring were also replaced for B1A:T2:FGCOLB (no wear observed on old pin) and B1A:T2CS:FGTGT (some wear observed on pin, more than Col. A).  They were not replaced on B1A:T2CS:FGSEC or B1A:T2CS:FGDEM at this time, as neither of these has a noisy signal, and neither can cause trips in the control system.

After all paddlewheels were re-installed and the system restarted, the flowmeters behaved as expected.  The fluctuations that occurred on Sunday were not happening this morning prior to the change, so we're unsure at this point if this has solved the problem.  StripTool was used before and after the change to observe fluctuations in the readbacks, and no noticeable difference was observed.

During testing of the flow meters we observed that B1A:T2:SVCOLB seemed to not respond one of the times it was commanded to close, and one or two other times it seemed to close after a few seconds delay.

Note: All T1 and T2 paddlewheels, pins, and o-rings were changed earlier this year.  They're scheduled to be replaced every 2 years as per the T1/T2 maintenance plan.

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

 The T1/T2 water packages were drained into the B1AAWHT (280L), the water was sampled and tested by RPG, results included in elog.

Once given the all clear myself and Maico released the water from the B1AAWHT (300L). 

It was discovered that the solenoid valve SV1 which is located in the RH lab beside the sump, is broken and will need replacing, picture included in elog.

Having the solenoid broken caused the draining to go slower then normal.

Jan 10, 2024:

The cooling packages for T1/T2 was drained this morning. ~150 L from each of the cooling package reservoirs + expansion tank was drained to the BL1A holding tank. The holding tank held ~400 L initially and is therefore now filled to ~700 L (out of a 800 L capacity). We will likely drain the holding tank in the following morning at which point this ELOG will be updated.

---------------------------------------------------------------

UPDATE (Feb 01, 2024): After recieving instruction from RPG to drain the holding tank to the active sump (due to delays in completing Tritium sample tests), we mistakenly drained ~50L of the 1A holding tank water to the city sewage system (initially it held ~610L and after the draining it read ~580L). This error was caused by misinterpretation of the 1A tank draining systems/procedure. In fact, it is uncommon for RH to pump active water from the 1A holding tank into the sump. It is also unclrear whether a valve configuration to do so exists (if it does, no flow reading can be taken). 

However, because of this error, we learned that the plumbing to the city water drain is slightly blocked (see image).

Our plan of action moving forward will be to wait for the Tritium results and coordinate with RPG to determine the appropriate next steps.

 ---------------------------------------------------------------

UPDATE (May 02, 2024): The active sump level was inspected visually on April 29 and it was noticed that it was close to full. 1x 1L and 2x 20mL samples were then collected and passed to RPG for testing. When RPG completes their Tritium analysis, we will drain both the holding tank and the active sump. See this link.

 ---------------------------------------------------------------

UPDATE (Aug 06, 2024): The tritium analysis was finally completed for T1/T2, since the sump only received high active water from the removed T1 target ladder, tritium analysis for the sump water would have been covered by the T1/T2 analysis. Gamma spectroscopy was done for the active sump (here). The draining procedure will be recorded in a separate ELOG.

The T1 and T2 cooling packages were drained starting at 14:36, by 14:48 both expansion tanks had dried, meaning the draining rate for both tanks were ~80L/hr, generally it will only take 2 hours to fully drain the system.

Tritium samples were collected after allowing the system to drain for ~5 minutes to clear out water in the drain lines and actually collect samples from the reservoirs. The samples were passed onto RPG for analysis.

The drain valves on the cooling packages and in the BL1A tunnel is left open.

 The following changes were made to the T1/T2 PLC control system:

- De-glitching system investigated and Tony discovered that it was programmed incorrectly:  the de-glitch time was observed only when a PV changed from bad state to good, not the opposite (meaning there was actually no de-glitching in place since system start-up in 2016!).  This was reversed and the following de-glitch times were programmed:  100ms for trip thresholds except 300ms for flow meter trips, and 2s de-glitch on all warning thresholds.  The new system was tested with the PLC in simulation mode and seems to be working correctly.  This should eliminate or significantly reduce nuisance warnings and trips from the system.

- PLC system code was modified so that the target version displays '???' when no target is installed instead of defaulting to 'MK1'  (all target positions registered as 'plugged' with no target installed, so there was no danger to the system, but this was still confusing to operators)

3x 3/4" swagelok ball valves (2x at T2 and 1x at T1) were replaced after the water cooling line was shut off at the BL1A tunnels late yesterday. 

We were able to relieve pressure from the T2 water cooler + M8 colA/B (they are the same line) at the BL1A tunnel through a drain port but no such port exists for the T1 heat exchanger line in BL1A.

The T2 heat exchanger line was virtually dry at the replaced valves after pressure was relieved.

The T1 heat exchanger line was still pressurized but while looking for a suitable port to drain at the cooling package, one of the copper fittings at the 1/4" heat exchanger line broke off at the connection to a green valve. 

~1L of water dripped from this line and this was sufficient to relieve pressure from the T1 heat exchanger line - allowing for the 3/4" ball valve to be replaced. 

Until the green valve gets replaced, the T1 heat exchanger line must remain shut off in the BL1A tunnels as the broken copper line is not plugged currently.

Update (last picture): the green valve was replaced with an equivalent 1/4" swagelok ball valve. We also incorporated two 90 deg. bends in the copper tubing to help relieve any stresses that may develop.

5 and 3 Swagelok ball valves were replaced from the T1 and T2 cooling packages respectively.

These are compression fit valves, 1/2" or 3/4" size.

1 valve in T2 was replaced previously, bringing the total # of replaced valves in T2 to 4.

Remaining valves that need to be replaced:

• T1:
  • 1x heat exchanger valve was not replaced because the heat exchanger line was not drained prior to the operation.
• T2:
  • 2x valves for the heat exchanger line were not replaced because the heat exchanger line was not drained prior to the operation.
  • 1x M8 beam blocker cooling line was not replaced for the same reasion.

Remaining valves that may need to be replaced (verify after next leak check):

• T1:
  • ~4 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 2x panel mounted valves need to be inspected for leaks.
• T2:
  
  • ~3 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 3x panel mounted valves need to be inspected for leaks.
  • 1x valve from the resin flask outlet has threaded connections with plumbing tape - this valve may not be leaky.

Next week, will drain all water lines associated with the cooling package before replacing remaining valves - then perform leak check. 

 (attached hand written notes)

 Update: remaining valves replaced (see: https://elog.triumf.ca/TIS/RH-Meson+Hall/320)

Isaac Earle has recently been managing the fabrication of some new Be target cassettes for the T1 / T2 Meson Hall target ladders (WO 41527).

The machine shop has outsourced the cassette tubes to an external shop for wire EDM cutting, as it produces a much more repeatable and consistent wall thickness without the need for delicate forming and welding.

However, the legacy end-cap design TBP1690 calls out a thin-wall formed frame to help brace the window in place and provide material to fusion weld the rim of the end cap to the cassette tube.  The legacy design frame is extremely thin compared to the tube wall thickness, and this mismatch is extremely difficult to EB weld reliably.  If a mistake is made, the entire EDM cassette tube is scrapped.

Bob Welbourn and the machine shop have come up with an improved design for the frame and window with slightly modified forming tools.

The window thickness .003" and tube wall thickness .010" remain the same.  The only change is to the thickness of the welding frame, which is now matched with the tube wall thickness (previously .003", now .010").

Photos of the old (right) and new (left) weld results are attached, as well as the new forming tools created by Bob Welbourn.

The shop is now able to make a confident and reliable weld for the end caps.

I am not sure that the increased frame thickness will have any affect on the beam entry into the target cassette, but my assumption is that this region is not critical.  Now, if the beam does enter that material at the edge of the cap / tube, and the temperature changes during operation, there will be a more secure weld there due to this design improvement.

The machine shop was waiting for the OK to finish welding the end caps into the frames using this new technique / design, and seeing as how the old option was likely to produce more leaking or scrap parts, I gave them the go-ahead to proceed.

Isaac will take care of the drawing and documentation updates when he returns from holiday.

Thanks and regards,

Grant

 The following tasks were completed:

1. Yesterday (Jan 18), the T1/T2 cooling packages were inspected for leaks and some moisture was found underneath the CUNO filters behind the resin cans on both T1/T2. It was determined that the source of the leak was a loose fitting on the drain valves of the CUNO filter housing (see photo). Today we confirmed that no more moisture is found underneath the CUNO filter after these fittings were tightened. A note will be made to keep this in mind for future shutdown operations.
2. The shaft couplings on the T1/T2 target ladder drive system were inspected: no cracks were identified on the rubber couplings.
3. Actuation of the T1/T2 profile monitors were tested. Vacuum levels at T1/T2 remained stable throughout (see picture).
4. The pneumatic connections/hose/fittings for the T1/T2 profile monitors, and M9/M20 BB were inspected - therere were some air leaks out from the ASCO valve + pneumatic flow control valve on the T2 profile monitor when it was in the down/out position. The same leak out of the pneumatic valve by the profile monitor connection was found at T1. Given that both profile monitors actuated properly however, these leaks were determined to not be an issue.

Next week, we will continue work by actuating the M9/M20 BB at T2, at which point this ELOG will be updated.

UPDATE Jan 30, 2024: The M9 and M20 beam blockers were tested, no leaks through the tubing were observed, actuation was quick, ops confirmed no perceptible change in 1A vacuum levels during actuation and EPICS reads correct blocker position signals.

Note: the M20 blocker was actuated by ops, while the M9/T2 blocker was actuated using the ASU on the B2 level of the Meson Hall. The ASU did not require the area to be blocked off to actuate the T2 blocker.

------------------------------------

Note: Ops noted that the profile monitor interlock status has to be reset if a timeout is encountered (such as if the pump were to be turned off or a trip was encountered) before the profile monitor can be actuated (see picture).

Note: When inspecting the system, we tried to open the rotary collimator air supply and the handle on the pannel mounted valve broke off (like it did for the T2 valve last shutdown - see picture). 

UPDATE Jan 30, 2024: The handle to the rotary collimator air supply valve was replaced (see image).

Note: There was also some small leaks at the hose clamps at the T2 profile monitor, in the out position. This leak was however smaller than the leak through the valve. 

 A new version of the T1/T2 PLC code has been uploaded to the PLC CPU.  The main change in the new version is a fix to a bug related to the target MK# which may have caused the previous CPU crashes.  The EPICS interface now needs to be updated so that no target information is displayed on the EPICS screens when there is no target installed, or jumpers for both MK1 and MK2 are detected.  This must be tested during the next maintenance day.  The appropriate tests are Section 6.4.1 Test #5 and #6 from Document #118467