Laser tracker scan data from 2017 and 2019 attached as a backup.  Original files from Beamlines Group who also keep copies on \\trwindata\experimental support\alignment\T2-M9_data

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

The T2-MK2 target was transferred from the storage pit to the beam line yesterday (April 16th).  The field was measured to be approximately 165mSv/hr at 0.5m from the target ladder (after approximately 5 months of cool down time).  Vacuum was started this morning and has pumped down normally (1ACG4 at 11mTorr after ~5 hours) indicating that the target is leak tight.  

The T2 profile monitor and protect monitor were 'blip' tested by Bill Rawnsley and Mike Russel from the racks on the Meson Hall mezzanine today.  The response from both monitors appeared normal.

T2 profile monitor and M20 beam blocker actuation were tested from the MCR, both devices actuated smoothly and all limit switches registered properly.

The T2 target is now ready for beam and the T2 monolith may be covered.

 On Thursday May 11th operators attempted to move the T2 profile monitor (T2-MK2) to the IN position but were unsuccessful.  The monitor would leave the OUT limit, but would not reach the IN limit.  This was repeated 2 times. The air amplifier was inspected and nothing out of the ordinary was noted.  The T1 profile monitor was tested and was working properly.  At the time it was suspected that the monitor cassette had become dislodged as occurred earlier this running period (see E-Log #221).

Today monitor actuation was attempted again and it moved to the IN position normally, this was done a total of 3 times..  An electronic check was performed by Mike Russell who reported that it looked the same as when he last checked, meaning the monitor cassette was definitely not unplugged. The monitor was then used with beam and Eric Chapman reported that monitor was functioning normally.  With the beam spot centered vertically on the profile monitor the protect monitor showed more beam on the lower plate, which is expected because the protect monitor center is 2-3mm higher than the profile monitor center (see E-Log #220).

It is unknown why it would not actuate properly last week, but seems to be fine now.  Cyclotron fault #9928 has been returned.

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

A strategy for alignment of the T2 protect monitor, T2 target, and T2 Collimator A was discussed today and decided upon by Yuri Bylinski, Isaac Earle, Yi-Nong Rao, and Roman Ruegg.  The purpose of the adjustment is to try to reduce the amount of beam lost as it passes through the T2 area (presumably being blocked by the collimator because it is not possible to steer the beam through all three devices in their current configuration).  Keeping in mind that changing the elevation of Collimator A would be prohibitively time consuming it was decided that the protect monitor and target both be adjusted upwards 3.0mm compared to where they were during the 2016 running period.  This will result in the following elevations (all w.r.t. the T2 target flange).  The T2 profile monitor elevation will also be adjusted to match the target elevation.  

T2 Protect Monitor:  1833.5mm (1833.2mm from protect monitor flange)

T2 Target: 1836.5mm 

T2 Collimator A:  1834.7mm  (1834.5mm from collimator flange;  This is the current collimator elevation, measurement taken by Tom Lyth in Feb 2009)

These values are similar to what were used in 2009 during which time TNF neutron flux rates were acceptable (>800), and centered beam spots were achieved.  During the 2016 running period the beam spot on target was also roughly centered.  At both times the target was approximately 3mm lower than the protect which seems to indicate the beam is travelling at a downwards angle, possibly not correctable with available tuning.  This is a departure from the strategy in the last few years when the target and protect monitor were adjusted downwards to bring them closer to the beam plane (they had risen due to movement of the T2 monolith).  This is now thought to have been a mistake because it made it difficult to steer beam through the collimator.

Beam will be started with T2 in Position 2 (new target, never seen beam), then after a few days the target assembly will be moved to the hot cell to check the beam spot to confirm alignment and make changes if the strategy needs adjusting.

A PDF copy of Yi-Nong's summary is also attached.

 The T2 water package was reconnected to the target and started after being shut-off since April 12, 2012.  The M9 and M20 groups do not anticipate having to vent the T2 volume, so the package will be left running in preparation for BL1A start-up in early July.  There was no change in T2 vacuum levels during start-up, and therefore no water leaks into the vacuum volume.

The T2 expansion tank water loss observed on May 24 (Meson Hall E-Log #33) seems to have been a one time event, possibly caused by someone starting the T2 water package while the target was disconnected.  The level of the tank has been checked regularly since the water loss, and there have been no further drops in the level.

 The T2 cooling package, the top of the T2 monolith, and the Hansen water connections which connect the package to the target were inspected and no evidence of a water leak was found.  Kim wipes had been stuffed inside the female Hansen fittings after they were disconnected on April 12th, and these were also completely dry.  I checked with control room and there are no log entries regarding anyone starting the cooling package during the relevant dates.  I also contacted Graham Waters (who sometimes works on the water package controls) and he has not turned on the package since it was shut off on April 12th.

The water level in the expansion tank has not changed since last inspection on May 24th.  I will continue to check the water level daily.

T2 target water leak was identified earlier this morning.

Isaac Earle, Ron Kuramoto and Keith Ng co-ordinated Meson Hall blocks removal and target flask transport.

Target assembly was removed from B/L approximately 5pm.

Vacuum "suck tube" (capacity ~ 10L) for removing water in B/L was prepared by Isaac Earle.

Approximately 5 L of water was seen in the bottom of the T2 target ladder vacuum tube enclosure, referenced by the ladder assembly locating spigot (see attached photo).

Most of the water was removed by Grant Minor, Ron Kuramoto, and Keith Ng using the suck tube, job finishing around 8pm November 22nd.

About 1/8" water remains in the bottom of the vacuum tube enclosure, which will have to be left to evaporation overnight and vacuum pumps (see attached photo).

Full suck tube was tagged and left on the north-east side of the Meson Hall blocks near TNF.

Spare target replacement, pump down and block replacement to happen tomorrow, Isaac Earle and Grant Minor to co-ordinate w/ support from Plant Group and Vacuum Group.

 The problem with the T2 vacuum interlock was fixed by restarting the "Crate TG1" control panel microprocessor.  The T1 and T2 vacuum interlocks were retested several times and are functioning properly.  Fault report #5907 was returned.

Graham Waters investigated the faulty signal from T2 expansion tank warning level sensor.  The problem was found to be a loose cable at the panels.  The connection was restored and the level sensors are now displaying correctly.

The Inlet (T1), Outlet (T2), Col.A (T6), and Col. B (T7) thermistors at the T2 cooling package all gave intermittent faulty readings at various times between Oct - Dec 2014.  It is suspected that this was caused by a water leak in the area causing the terminals to get wet.  Inspection of the sensors revealed that they are in poor shape and that the readouts sometimes fluctuate when the probe or connectors are handled.  All four thermistor probes have been replaced.  The plastic interior of the Lemo connectors were found to be crumbling, and were replaced with crimp connections.  All T2 thermistors now appear to be functioning normally.  The T2 cooling package was started (with a bypass instead of the target assembly).  Flow through Collimators A and B has also been started.  The T2 package was inspected while running and no leaks were found.

Cyclotron Fault #7722 has been returned.

 In preparation for extended periods of the T2 volume being vented during the M20 repair, the T2 water package was shut-off, the water lines were disconnected from the target, and the cooling water was removed from the target using compressed air.

The T2 target was moved from Position 4 to Position 0 at 11:15am so the profile monitor could be used during the development shift.

The T2 cooling package tripped regularly throughout the day on high flow while in position 0.

At 10:45pm the T2 target was returned to the previous running position: approximately 2mm above the nominal position 4 elevation (pot ratio fluctuating slightly around 0.799) so that beam could be run overnight.

The T2-MK1 target was moved from the beam line to the target storage pit in position 4.  The target was measured to be 30mSv/hr at 0.5m.  The move went smoothly.

The T2-MK2 target was then moved from the hot cell to the beam line.  The target was 151mSv/hr at 0.5m.  For reasons unknown the target was slightly out of rotation when lowered into the beam line, and could not be fully lowered because the alignment pins were out of position.  It was not possible to twist the target while it was supported by the flask, so 2" aluminum standoffs were placed on the monolith flange, and the target was lowered onto the standoffs.  The target was then lifted without using the flask, the standoffs removed, the target rotated to match the locating pins and then lowered onto the flange.  It is possible to adjust the rotation of the lift T flange on this target if the misalignment is found to be caused by the target and not the flange.

The new target information sheet was delivered to Ops, and XTPAGE 7L was updated.  Graham Waters was notified of the change so that he can update the potentiometer look-up values in the control system.

The target flask catch tray was cleaned using a wet wipe which had 11,000cpm on the 44-2 meter and 300cpm on the RM14 meter.  After cleaning, a dry swipe of the tray inside had 0 counts on the 44-2 and 100cpm RM14.

300cpm contamination was found on Isaac's coveralls and 150 on Ron's.  There was 6000cpm found on Ron's booties.  He had been working on the blocks around the T2 monolith, but had not been down to the monolith itself.  A request will be made for Plant Group to paint the blocks around the T2 monolith.

The T2-MK2 target (currently installed in the beamline) was moved from position 0 to position 4 (10cm Be) using the portable control box.
The T2 cooling package was started after moving the ladder.

There was no change in the BL1A vacuum.  The T2 expansion tank water level dropped slightly, as expected (around 1/16"), and was refilled to ~1/4" above the high level sensor top nut.

30 micro-amps of beam was put down BL1A and the T2 target and cooling system appear to be functioning normally.

It was reported on Dec. 20, 2022 that when Operations attempted to run the target ladder motor, it was unresponsive. A fault report was created: 15477.

During winter shutdown 2023, the T2 area was uncovered and the motor was inspected on 2023-01-11 by Adam Newsome and Maico Dalla Valle. The connectors and cabling appeared OK.

Upon removing the motor from its mounted position, it became clear that the coupling part between the motor and the gearbox had failed. The motor was running properly but not actually connected to the drive shaft and therefore nothing happened when ops tried to move the ladder. We confirmed that the motor runs properly in no-load condition when it is isolated from the rest of the assembly. We were able to move the rest of the drive shaft by hand and it does not appear to have any significant resistance, so it is suspected that this coupling failed due to radiation exposure over a long period of time rather than overload. The coupling part is plastic and designed intentionally to break under high loads (it is thinned out in the middle, where it snapped). Hopefully it did not actually see any high loads (i.e. there is no other root cause of this issue) and it simply degraded so much that it eventually broke.  

To resolve this issue, we will fabricate/purchase and install another one (Albert, Maico) and work with operations to test that all target ladder positions are reachable once it is installed. We will also confirm at that time that the limit switches and potentiometer readback function as expected.

It is recommended to consider the following changes to the drive system going forward to prevent this from happening in the future:

• Replace these plastic coupling parts with metal versions
• Replace the plastic coupling part that is thinned out to intentionally fail under high load with another method of system protection. The same end result could be achieved using a clutch or perhaps a fuse/circuit breaker which would effectively inhibit motion if the motor experienced overload. This would be a significant improvement over breakage because it would not require replacing the part (dose-intensive) and would be more easily recognizable.

For now, this drop-in replacement should suffice because it is not likely that the part will degrade so significantly within the next few years that this would happen again (unless there is a different root cause aside from radiation that we are not aware of!).

[update 2023-01-01] The broken coupler has been replaced by Albert Kong and Adam Newsome. The replacement part was tested - DCR operated the target ladder motor, moving it up and down through all of its positions multiple times. Everything mechanically and electrically appeared OK during motion. DCR confirmed they are able to reach each position and the proper readbacks are present - everything looks OK on their end. The issue is considered resolved and the fault is returned.

 See Cyclotron fault 16553. 

"B1A:T2CS:FGTGT readback is toggling at the warn limit of 3.0 GPM and over the last week has started crossing over the trip threshold of 2.8 GPM, tripping off the water package. Initial Action Taken: 1A is scheduled to take beam on December 13."

Upon reviewing the flow trend over the past semester, the flowrate has been hovering around 3.0 GPM the whole time.

Meanwhile, the water temperatures in/out of the T2 target has remained stable between 24C to 31C with the warn and trip limits > 35C. 

It should therefore be safe to run the target cooling water at a lower flowrate.

As a temporary solution, the low trip limit was adjusted to 2.5 GPM. While the warn was kept at 3.0 GPM.

We will look at how flowrate through the target can be increased in the coming shutdown.

For reference, the T1 target flow trip was set to 2.0 GPM an warn was set to 2.3 GPM (see MH-RH ELOG 331).

UPDATE(June 18, 2024):after shutdown service and replacement of the proteus paddle wheels, flow through the target seems to have recovered and is stable at around 3.7-4 gpm.

The resin flask on the T2 cooling package was exchanged today (new resin can prepped in the morning). 

The newly installed resin flask was placed slightly off-set from the marking on the platform to ensure that the braided hose will clear the blocks when replaced at the end of shutdown.

The spent resin can was dropped off in the hot cell, ready to be prepped for drying. 

See attached image for illustration of the rigging solution used, ~20 ft was covered by two sligs and a shackle to clear the MH mezzanine.

The T2 protect monitor was returned to the beam line using the shielded flask.  The alignment pin did not align with the hole in the vacuum flange, but could be aligned by hand by rotating the monitor assembly while it was hanging from the flange.  The protect monitor vacuum flange was bolted down and all services connected.  The T2 protect and profile monitors were checked by Dave Cameron (both OK).  Vacuum was pumped down and the cooling package started without incident.  After installation of the cap block the M20 beam blocker and the T2 profile monitor were exercised (both OK).  The updated target info, elevations, and beam spot measurements were delivered to the control room operators.

While the monolith was exposed the T2-MK2 target assembly lift hat was rotated slightly to match the orientation of T2 Protect and Col. A.  Hopefully this will improve the alignment for the next time the target is lifted with the flask.