It was reported on Dec. 20, 2022 that when Operations attempted to run the target ladder motor, it was unresponsive. A fault report was created: 15477.

During winter shutdown 2023, the T2 area was uncovered and the motor was inspected on 2023-01-11 by Adam Newsome and Maico Dalla Valle. The connectors and cabling appeared OK.

Upon removing the motor from its mounted position, it became clear that the coupling part between the motor and the gearbox had failed. The motor was running properly but not actually connected to the drive shaft and therefore nothing happened when ops tried to move the ladder. We confirmed that the motor runs properly in no-load condition when it is isolated from the rest of the assembly. We were able to move the rest of the drive shaft by hand and it does not appear to have any significant resistance, so it is suspected that this coupling failed due to radiation exposure over a long period of time rather than overload. The coupling part is plastic and designed intentionally to break under high loads (it is thinned out in the middle, where it snapped). Hopefully it did not actually see any high loads (i.e. there is no other root cause of this issue) and it simply degraded so much that it eventually broke.  

To resolve this issue, we will fabricate/purchase and install another one (Albert, Maico) and work with operations to test that all target ladder positions are reachable once it is installed. We will also confirm at that time that the limit switches and potentiometer readback function as expected.

It is recommended to consider the following changes to the drive system going forward to prevent this from happening in the future:

• Replace these plastic coupling parts with metal versions
• Replace the plastic coupling part that is thinned out to intentionally fail under high load with another method of system protection. The same end result could be achieved using a clutch or perhaps a fuse/circuit breaker which would effectively inhibit motion if the motor experienced overload. This would be a significant improvement over breakage because it would not require replacing the part (dose-intensive) and would be more easily recognizable.

For now, this drop-in replacement should suffice because it is not likely that the part will degrade so significantly within the next few years that this would happen again (unless there is a different root cause aside from radiation that we are not aware of!).

[update 2023-01-01] The broken coupler has been replaced by Albert Kong and Adam Newsome. The replacement part was tested - DCR operated the target ladder motor, moving it up and down through all of its positions multiple times. Everything mechanically and electrically appeared OK during motion. DCR confirmed they are able to reach each position and the proper readbacks are present - everything looks OK on their end. The issue is considered resolved and the fault is returned.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

A fault was reported June 21, 2022 (Fault 15033) regarding T1 inlet pressure warnings... the pressure reading slowly dropped over time as seen in the attached strip tool screenshot (B1A:T1CS:PGIN). The pressure reading issue is attributed to a fault pressure transducer. The transducer was replaced on Oct. 11, 2022 my M. Dalla Valle. It is currently functioning as expected and the system is operational.

The following maintenance tasks were performed during mini-shutdown of 2022 with the goal of resolving the Fault 14966 (target water flow rate low).

• Replace the o-ring, paddle wheel, and shaft on the following flow rate sensors:
  • Q1 (FGTGT)
  • Q2
  • Q4
• Replace the demineralizing flow metering valve with a new needle valve (the old one was flaky and appeared to occasionally restrict flow)
• Re-tune the demin. flow to 1.15 gpm
• Top up the expansion tank due to water loss during the maintenance job (from 27.1 cm level to 34.8 cm)

To test the theory that the demin. flow metering valve was faulty and had an effect on the target water flow rate during operations, the valve was closed to determine if it stopped flow to the target. The result is that it did not, and only flow in the demin. circuit was affected. Therefore, it is likely that the root cause of Fault 14966 was not due to the demin. metering valve, but rather an issue with the pump, heat exchanger, pressure in the lines, or another form of obstruction. It should be noted that when the pump is turned off, all flow rate and pressure sensors drop to zero (except the pressure sensor located in the BL1A Service Tunnel) - so if the fault occurs again and symptoms show this case, it is likely that there is an issue with the pump intermittently turning off or similar.

As an aside: it is recommended in the future when changing circuit boards on the flow rate sensors to use metal screws instead of the plastic screws supplied with the parts.

Dose rates observed in the work area: 300-500 uSv/h at the target cooling package area, 1300 uSv/h at the target monolith area.

The T2-MK2 target was transported from the east hot cell to the storage pit at position 3.

As was noted in the past, position 3 is difficult to align. The flask alignment pins prevent the flask from positioning in the correct location, and it is required to have the flask just above the pins to ensure that the pins did not restrict the flask movement.

It is recommended to install a camera system to assist with the target placement and retrieval, which will help reduce the amount of dose one receives during this operation.

The target elevations measurement and beam spot checks were completed on Sept. 15, 2022. This was done after a July 2022 target exchange for target positions 3 and 5.

Updated target elevation drawing can be found in DocuShare (Collection-33652)

Mechanical Services Group changed the pre-filter in the ventilation system on Aug. 29, 2022 as per work permit C2022-08-29-2.

It is recommended that Mechanical Services change the HEPA filter as soon as possible. These filters are on order with approximately 7 month lead time. 

Edit: as of 2023-05-03, the HEPA filter has been changed.

The water level inside the BL1A active water holding tank was checked today (Aug. 23, 2022). The tank is essentially empty at this time.

On Aug. 18, 2022, Operations informed RH group that an alarm activated for the full status of the MHESA B1 RCR1 Lab holding tank. Additionally another alarm was reported in the RH control room regarding the RH sump full (Southwest Meson Hall Extension - Remote Handling area). However, there is an alarm signal for the RH sump that was seen as OK at the MHESA Lab HMI.

Upon examining the RH active sump it was clear that the water level was in fact quite high, and the red warning light on the local control box was on.

After sampling and testing by RPG the sump contents were released under work permit C2022-08-19-3 as per the procedure in Document-64834. The approximate volume of the full sump is 7270L.

When the water level was below the warning threshold (i.e. red warning light came off, and green light indicating "sump good" came on), a request was made to the control room to reset the alarm. Operations cleared the alarm at the RCR Lab controls and retired the signal defeat that was in place in the DCR (“RCRL warn” signal was defeated).

Edit (2022-08-23): BL1A holding tank water level was checked - it is empty.

It was reported today by Peter Bratt that the nuclear ventilation system for the hot cell is not functioning. At this time, the cause of the issue is unknown. Remote Handling and Electrical Services are investigating. This e-log will be updated when the issue is resolved.

Update: the issue was caused by worn belts which had been smoking. The belts must be replaced.

Some oil was noticed around the hot cell window, on the control panel, and on the ground below the control panel. A leak is suspected.

Suggest tightening all bolts to specified torque along the window, and monitoring overflow oil level (position today was marked). If the problem persists, the gasket must be replaced. Note that the gasket should likely be replaced anyway as it has not been done for some time.

The tubing for the helium wand inside the hot cell is broken and needs to be replaced. More tubing has been ordered and the job will be completed in the coming weeks.

Today myself and Adam Newsome started the leak detection steps for a target exchange and notice that the Helium line was not connected/broken from inside the hotcell.

A new line will need to be installed and the line attached to the helium wand.

This will need to be done when the target has been removed.

We will be installing a temporary line so that we can complete the target exchange first.

The temporary line will be put through the tool port as this seems the most practical and least likely to interfere with operations.

After will will have to schedule some time to install a permanent helium line with some newer flexible tubing. 

Also notice at the operator side that there is a small oil leak that appears to be coming from the hotcell window oil.

We have added an indicator(tape with pen marking) to the oil level to see how much we are loosing.

We will have to check the torques of the window bolts to ensure they are adequate.

If there is still an issue we may have to look at replacing the window seal.

Also Chad Fisher was consulted about the possible solutions 

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

Target T2-MK2 was moved from the storage location (formerly position 3) to the hot cell in preparation for replacement of end of life targets.

The T1-MK2 target was moved from the hot cell to the storage area, in position 5. Note: manual nudging of the flask was required to get it to align with guide pins properly.

The profile monitor actuation tests were completed for T1 and T2. No abnormalities were noticed in vacuum during either test.

Note: an issue was observed with T2-MK1 - see e-log 290. The issue was resolved.

An attempt was made to actuation the T2 profile monitor, however it showed up in EPICS as being not present. Furthermore, the "Remote" indicator light on the physical controller was not lit up despite the switch being in remote mode. The limit switch indicator light was working.

Upon troubleshooting with Erwin Klassen, it was determined that a jumper was missing on the profile monitor control connector (on the target end, rear of bulkhead). The jumper shorts pins A and G on the mil-spec connector on the controller end, which is for 120 VAC supply, to indicate that the device is connected. T1-MK1 was checked - this jumper was present on T1-MK1, across pins E and F of the mil-spec connector, but was not present on T2-MK1. Otherwise the connectors were identical. It is suspected that it was accidentally omitted during maintenance in the past. Regardless, a new jumper was soldered in-place on T2-MK1 across pins E and F to replicate T1-MK1. The system functioned normally after this.

Attached photo shows the completed solder job - the current state of the connector on the rear side of the target bulkhead panel.

The beam blockers for M9 and M20 were actuated, and vacuum was monitored to ensure no disruptions. Both actuation tests were successful. Motion was observed and no abnormalities noted. Cabling and air connections checked: OK.

Note: there was a safety interlock issue with M9 which prevented actuation - a relay or fuse was temporarily removed from the safety system ~10 years ago but it was not replaced. It was replaced for the purpose of this test, and then removed again.

Diagnostics group (Shengli Liu) noted a lack of connectivity between control panel and T2 Protect Monitor via the high voltage cable. Upon inspection it was noted:

• The cable was very brittle due to radiation exposure and it was cracking in multiple places
• In the SHV connector, the pins were slightly bent (bending them back into position seemed to rectify the connectivity issue).
• The cable was bent at the strain relief junction

RH Group has advised that Diagnostics replace the entire cable as soon as possible. For now it appears to be working.