Completed work at m9 beam line:

- Removed existing service stand and associated cable and water services. Legacy b1 water services still need to be removed for future magnet installation. Old service stand is currently in warm cell waiting for disposal.

- M9 Q345 and vacuum roughing line were cleaned by CMMS group.

- M9 B1 is disconnected but remains in beam line ready for removal and refurbishment for 2020 shut down. Relevant ports have been sealed off.

- New Q1 Q2 baseplate was test fit and did not fit with guide post bolted in place, interferes with T2 port blank off plate.

- 3d lidar scans of magnet space were done.

- Leica tracker data was taken of the holes in the poured in place concrete pad.


Photos from shutdown are on remote handling group folder, under "\\trwindata\groups\remote handling\Photos\2019\2019 m9 project"

After extensive testing over the last 2 weeks it was determined that the protect monitor on the T1-MK2 target had an issue with intermittent shorting between the high voltage line and ground (movement of the target ladder was causing the short to come and go).

On the May 21 maintenance day the T1-MK2 target was moved from the beamline to the hot cell (102mSv/hr @ 0.5m).  The T1-MK1 target was then moved from Storage Pit location #3 to the beam line (as in the past, alignment of the flask to the #3 position hole was challenging, and the flask had to be suspended above the alignment pins to achieve alignment with the hole - In the future this storage pit location should be avoided).

Cameras were mounted on the T1 alignment frame for remote viewing of alignment with the hole as well as flask door position and hook latch position.  The camera locations chosen provided useful views and did not interfere with the target or frame.  After procurement of additional mounting brackets the cameras will be permanently installed at these locations (already completed on the T2 frame).

The target moves were completed successfully, vacuum pump-down was normal, and electrical checks of the T1-MK1 profile monitor and protect monitor were normal.  When start-up of the cooling system was attempted problems were encountered with the control system.  Multiple channels on the digital output module were shorting, causing the power supply fuse in the PLC cabinet to blow.  This was eventually tracked down to two damaged cables (B1A:T1CS:SVSEC and B1A:T1:SVCOL) which somehow had been crushed during removal of shielding.  The cables were replaced and all cables secured as much as possible in the vertical chase leading up from T1.  After replacing the cables we discovered that the digital output for the SVSEC signal was stuck in 'drive' status.  The module was replaced and then the control system was working normally (approximately 11am on May 23rd).

The T1 cooling system was started (no change to 1ACG2 vacuum), ladder movement through all target positions was tested, and profile monitor actuation was tested.  Updated target info sheets were submitted to Ops (attached in PDF format) and XTPAGE 7L was updated.

Diagnosis and repair of the T1-MK2 target will take place in the hot cell over the coming weeks.

Cyclotron Fault #12266 was returned.

The following work was done to the T1-MK2 target in the hotcell between the dates of May 24, 2019 - Jan 30, 2020 (See E-Log #251 for further information):

• May 24: Flushed position 2 with water, purged all positions with air.  Left air running through position 0 over the weekend
• May 27: Switched to position 5, left air running to dry out bellows
• May 28: Started pump out with cold-trap;  Reached "fine test" (TP pressure < 10E-2 Torr) after ~10-15 minutes, left for 2 hours, removed ice build-up then restarted leak detector;  
  Down to 1.0E-3 Torr TP pressure and 1.0E-8 Torr*L/s after ~10min
  Leak checked with 5psi helium, ~0.5s spray: Response at lower delta seal right side (when facing ladder) up to 3.8E-6 Torr*L/s and drift responses from nearby locations
• June 27: Position 1 target removed - no issues, required light tapping with hamme
  Position 4 target removed - lots of hammer hits required
  Both target beam spots photographed.  Saw what looks like hole on Position 4 entrance window (see attached photos)
• July 8: Protect monitor inspected visually using Nikon level by Dave Cameron and Shengli Liu - Nothing abnormal observed
  Electrical check w/ multimeter: all OK
  Observed ground to HV plates while moving ladder from Position 5 to 0 - all OK
  Observed ground to HV plates while moving profile monitor IN then OUT - all OK
  Found intermittent short from HV to ground when shaking the target assembly or tapping the protect monitor cassette:  must replace
• July 18: Cleaned Position 1 and 4 Swagelok sealing surfaces on the target ladder using the Dremel tool with burgandy Scotchbrite
  Ready for new targets to be installed
  Plugs installed at Position 4
• July 24: Target #105 installed at Position 1
  Started leak detector pumpdown with cold trap;  Only pumped down to 4.0E-3 Torr;  Big helium response at lower right delta seal (10E-5 Torr*L/s range)
  Concluded that the leak got worse during the target replacement (likely from the hammering required to remove Position 4 target
  Tightened bolts for lower right delta seal then repeated leak check: same results, therefore must replace seals
  Received replacement protect monitor cassette from Probes Group and installed
• July 29: Probes group did blip test on new protect monitor - all OK
  We repeated the test for shorts with shaking and tapping on the target assembly and monitor - no shorts observed
  Concluded that protect monitor issues is now fixed
• Early Jan 2020: Lower cone and plate removed, ladder removed (old style w/o plate at top)
  Removed seal spacers, the one for the lower right delta seal was stuck, had to use a flat edge screwdriver as a wedge with aluminum hammer to dislodge
  Removed delta seals, observed some deposits or corrosion at lower right seal location
  Cleaned spacers using Scotchbrite on Dremel
  Used 3D printed jigs with glued on burgandy Scotchbrite to clean sealing surfaces (no air ratchet or dremel, polished with manipulators until a clean shiny surface was produced)
  Had to remove bolts for the ladder flanges to clean the surfaces, and when reinstalling one of them it cross threaded and couldnt be removed
  Bolted instead at different hole locations (still at opposite sides of flange) but also had to switch bolt locations on upper seals to do this because bolts will interfere if using same threaded hole
  Eventually got everything back together with new delta seals installed (TBP0454B, machine shop order #44138)
  Note: the upper delta seals were not disturbed - new bolts installed and torqued before old ones were removed
• Jan 22: Leak check, pumped with cold trap: down to 0.0E-10 Torr*L/s and 1.0E-3Torr TP pressure after 9.5 minutes
  Using 5 psi helium, 0.5s spray;  No response at any previous locations;  Very small drift response up to 0.5E-10 Torr*L/s
  Concluded that leak is fixed
• Jan 26: Base plate and cone reinstalled - no issues
• Jan 30: David Kyle and Shengli Liu re-tested the profile monitor and protect monitors - No issues
  Measured vertical and horizontal positions for all targets and protect monitor, updated measurements sheet (PDF attached)
  Ladder moved to Position 0
  Target assembly ready to be moved to storage pit
   

 A 4” Marmon aluminum blank-off flange and o-ring were replaced on the south port of the T1 target monolith vacuum vessel. This port was accessed from the 1A Service Tunnel were the repair took place.

See the attached pdf file “1AT1 Vacuum Leak Repair-Feb2020” for a summary of the repair.

The T2 Collimator A flow meter signal (B1A:T2:FGCOLA) began fluctuating and occasionally reading 0gpm on Sunday Oct 25 causing BL1A trips and machine down time.

On the maintenance day today (Tues Oct 27), the T2 package was uncovered and the T2 Col. A flow meter was opened.  No unusual deposits were observed as seen recently with other flowmeters for BL1A magnets on the CuALCW system.  Some minor wear was observed on the dowel pin shaft, and the paddle wheel didn't spin quite as freely as when new.  A small amount of reddish/brown film (rust?) was seen on the internal SS faces.  The faces were wiped clean and new dowel pin, paddle wheel and o-ring were installed.

The pin, paddlewheel, and o-ring were also replaced for B1A:T2:FGCOLB (no wear observed on old pin) and B1A:T2CS:FGTGT (some wear observed on pin, more than Col. A).  They were not replaced on B1A:T2CS:FGSEC or B1A:T2CS:FGDEM at this time, as neither of these has a noisy signal, and neither can cause trips in the control system.

After all paddlewheels were re-installed and the system restarted, the flowmeters behaved as expected.  The fluctuations that occurred on Sunday were not happening this morning prior to the change, so we're unsure at this point if this has solved the problem.  StripTool was used before and after the change to observe fluctuations in the readbacks, and no noticeable difference was observed.

During testing of the flow meters we observed that B1A:T2:SVCOLB seemed to not respond one of the times it was commanded to close, and one or two other times it seemed to close after a few seconds delay.

Note: All T1 and T2 paddlewheels, pins, and o-rings were changed earlier this year.  They're scheduled to be replaced every 2 years as per the T1/T2 maintenance plan.

Today myself and Adam Newsome started the leak detection steps for a target exchange and notice that the Helium line was not connected/broken from inside the hotcell.

A new line will need to be installed and the line attached to the helium wand.

This will need to be done when the target has been removed.

We will be installing a temporary line so that we can complete the target exchange first.

The temporary line will be put through the tool port as this seems the most practical and least likely to interfere with operations.

After will will have to schedule some time to install a permanent helium line with some newer flexible tubing. 

Also notice at the operator side that there is a small oil leak that appears to be coming from the hotcell window oil.

We have added an indicator(tape with pen marking) to the oil level to see how much we are loosing.

We will have to check the torques of the window bolts to ensure they are adequate.

If there is still an issue we may have to look at replacing the window seal.

Also Chad Fisher was consulted about the possible solutions 

The tubing for the helium wand inside the hot cell is broken and needs to be replaced. More tubing has been ordered and the job will be completed in the coming weeks.

It was reported today by Peter Bratt that the nuclear ventilation system for the hot cell is not functioning. At this time, the cause of the issue is unknown. Remote Handling and Electrical Services are investigating. This e-log will be updated when the issue is resolved.

Update: the issue was caused by worn belts which had been smoking. The belts must be replaced.

It was reported on Dec. 20, 2022 that when Operations attempted to run the target ladder motor, it was unresponsive. A fault report was created: 15477.

During winter shutdown 2023, the T2 area was uncovered and the motor was inspected on 2023-01-11 by Adam Newsome and Maico Dalla Valle. The connectors and cabling appeared OK.

Upon removing the motor from its mounted position, it became clear that the coupling part between the motor and the gearbox had failed. The motor was running properly but not actually connected to the drive shaft and therefore nothing happened when ops tried to move the ladder. We confirmed that the motor runs properly in no-load condition when it is isolated from the rest of the assembly. We were able to move the rest of the drive shaft by hand and it does not appear to have any significant resistance, so it is suspected that this coupling failed due to radiation exposure over a long period of time rather than overload. The coupling part is plastic and designed intentionally to break under high loads (it is thinned out in the middle, where it snapped). Hopefully it did not actually see any high loads (i.e. there is no other root cause of this issue) and it simply degraded so much that it eventually broke.  

To resolve this issue, we will fabricate/purchase and install another one (Albert, Maico) and work with operations to test that all target ladder positions are reachable once it is installed. We will also confirm at that time that the limit switches and potentiometer readback function as expected.

It is recommended to consider the following changes to the drive system going forward to prevent this from happening in the future:

• Replace these plastic coupling parts with metal versions
• Replace the plastic coupling part that is thinned out to intentionally fail under high load with another method of system protection. The same end result could be achieved using a clutch or perhaps a fuse/circuit breaker which would effectively inhibit motion if the motor experienced overload. This would be a significant improvement over breakage because it would not require replacing the part (dose-intensive) and would be more easily recognizable.

For now, this drop-in replacement should suffice because it is not likely that the part will degrade so significantly within the next few years that this would happen again (unless there is a different root cause aside from radiation that we are not aware of!).

[update 2023-01-01] The broken coupler has been replaced by Albert Kong and Adam Newsome. The replacement part was tested - DCR operated the target ladder motor, moving it up and down through all of its positions multiple times. Everything mechanically and electrically appeared OK during motion. DCR confirmed they are able to reach each position and the proper readbacks are present - everything looks OK on their end. The issue is considered resolved and the fault is returned.

The flow rate sensor used to monitor the drain rate from BL1A holding tank to active sump, located in the active sump area, was repaired. The paddle wheel, o-ring, and shaft were replaced. This part was significantly worn prior to replacement.

It is recommended to change the filter adjacent to it as it is suspected that this filter is partially clogged, resulting in low flow rate when draining the tank.

Furthermore, it is also recommended to change solenoid valve SV1 as it is suspected that it does not work.

It was noticed today (2023-01-17) that the valve V6, used in draining the BL1A Holding Tank to the Active Sump, was leaky. This valve should be replaced as soon as possible. It has been added to the MS Planner maintenance tasks list.

5 and 3 Swagelok ball valves were replaced from the T1 and T2 cooling packages respectively.

These are compression fit valves, 1/2" or 3/4" size.

1 valve in T2 was replaced previously, bringing the total # of replaced valves in T2 to 4.

Remaining valves that need to be replaced:

• T1:
  • 1x heat exchanger valve was not replaced because the heat exchanger line was not drained prior to the operation.
• T2:
  • 2x valves for the heat exchanger line were not replaced because the heat exchanger line was not drained prior to the operation.
  • 1x M8 beam blocker cooling line was not replaced for the same reasion.

Remaining valves that may need to be replaced (verify after next leak check):

• T1:
  • ~4 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 2x panel mounted valves need to be inspected for leaks.
• T2:
  
  • ~3 miscelaneous small valves (likely 3/8") were not replaced - want to check if they actually leak beforehand.
  • 3x panel mounted valves need to be inspected for leaks.
  • 1x valve from the resin flask outlet has threaded connections with plumbing tape - this valve may not be leaky.

Next week, will drain all water lines associated with the cooling package before replacing remaining valves - then perform leak check. 

 (attached hand written notes)

 Update: remaining valves replaced (see: https://elog.triumf.ca/TIS/RH-Meson+Hall/320)

 We replaced tha O-ring on M20 at T2 and brought the target station down to vacuum, which allowed the target water pump to be turned on. 

After turning the pump on, we inspected the replaced valves for any leaks and found none - the flow on the demineralizing line was adjusted to 1.2GPM (also done at T1) by turning the needle valve.

Will observe till tomorrow to ensure that the T2 system is stable before proceeding with last system check.

See Cyclotron Fault 15787. The target ladder motor failed to move to the commanded position during a test. The T1 area was uncovered and the motor assembly was inspected by M. Dalla Valle, A. Newsome, and A. Kong. It was determined that the rubber coupler which connects the motor shaft to the drive assembly had degraded and snapped. This is similar to an incident which recently occurred on T2. The coupler was replaced with an all-metal version. The system was tested by the DCR operators. They ran the target ladder to each extreme (position 5 to position 0) multiple times. The target ladder was also moved to a few positions which had targets installed, and the cooling package was energized to run water through and confirm functionality. All feedback sensors for the target ladder assembly and cooling package were observed to behave normally, as confirmed by the DCR. RH group confirmed visually that there were no observable issues. The system is now considered operational.

It is recommended to change all drive system coupler parts on both T1 and T2 during the next shutdown to prevent this issue from happening in the future. These parts should be inspected during the annual inspection of the T1/T2 assemblies as well.

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

Feb 1 2024

Hand Pump observed to be losing pressure at full extension of cylinder, seals likely blown. Due to Enerpac P84 being an obsolete model, a newer model hand pump was ordered.

Feb 15 2024

Hand pump (Enerpac P84 Ultima) installed at Indium Extruder Station, observed to work properly in both directions of flow, deemed operational. 

At ~8am on May 06, the expansion tank level sensor for the T2 cooling system suddenly became noisy. 

Cyclotron fault ref: 16915

Approximately 11:00 am today, the noisy sensor was replaced with a spare, upon which it was learned that the spare sensor is broken (registers 0 level and not detected on PC through USB adapter).

The old (noisy sensor) was then replaced at around 12.00 pm onto the expansion tank and it was found that the noise had subsided.

A possible explanation to the noise would be loose connections/grounding wire.

We will continue to monitor the sensor in the coming days, and order replacement sensors has been placed and we will be able to replace the sensor soon should it become noisy again.

Edit 2024-05-07 - A. Newsome: EPICS monitoring shows the sensor appears to be behaving normally since the aforementioned events. Most likely attributed to improper grounding. The fault will be closed. See attached screenshot.

 The following instructions were provided by Graham Waters:

From diag5pc bootup

Password: cfzx29mkdr

Bring up a terminal (click on "terminal icon" left side of screen)
> su
> xpfk34yzjc
> chmod o+rw /dev/ttyS0
> chmod o+rw /dev/ttys1

Bring up terminal emulator GTKTerm
   From booton left of screen
  Click on Applications->Accessories->GTKTerm 

Three used TNF / 500MeV system cooling system resin cans were removed from the Meson Hall target storage pit (two from Hole #1, one which was stored below block #1).

Two new active cans will be returned to Hole #1 after the TNF resin is changed in a few weeks time.

The Remote Handling Hot Cell Lab Active Sump has been released to city sewer following the procedure in Document-64834.  The water was close to the sump high limit switch (approximately 5800L).  Before release the water was sampled, tested, and approved for release by RPG.  The tritium analysis form has been filled out and returned to RPG,.

It took approximately 5 hours to empty the sump with all valves fully open and the water flowing through the 10" Cuno filter.