The cyclotron lid was raised starting ~11:45am on 2024-05-02 and finishing at ~12:12pm under work permit C2024-05-02-5.

The raising was stopped a few times up until approximately 5" elevation, for RF group to perform checks, and to inspect the tank seal.

The tank seal was stuck to the lid near jack station 2.

The lid raised smoothly with no stoppages due to out-of-sync stations. No stations needed to be jogged.

Starting master synchro count was 66, but was jogged to 2 prior to starting. The final master synchro count when the lid was fully raised was 47387.

Note: the measuring stick which is affixed to the top of the lid and used to indicate full elevation was bent and should be "re-calibrated".

Summary attached.

A second lid down operation took place on April 30, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47427 (after it was jogged up slightly prior to starting).

Upon startup at 12:51 pm, the system worked fine and lowered the lid. The lowering was stopped at approximately 5" gap as per standard procedure for Operations to perform checks. Furthermore, the lowering was stopped multiple times after that at approximately 0.5" increments for resonator latch checks.

No abnormal observations made. No jogs required. Once again, this was a very smooth lowering process. The lid was fully lowered at 2:00 pm.

Ending master synchro value: 66

See attached summary.

Due to an issue with resonator latching, there was a need to raise the cyclotron lid a second time this shutdown.

The plan at first was to raise the lid by small increments (approximately 1mm at a time up to approximately 25mm) while RF group performed some checks.

After the RF groups completed their checks, it was determined that the lid needed to be raised the full extent.

The starting master synchro count originally was 52, and it was 2480 at the time when the decision was made to fully raise the lid.

The lid raised entirely with zero stoppages or observed issues. No abnormalities observed with any jack station.

The lid was fully raised by 1:49pm with a master synchro count of 47383.

See attached summary.

The lid down operation took place on April 19, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47401. The master synchro was not jogged prior to starting.

Upon startup at 12:45 pm, the system worked fine and lowered the lid. One stop occurred after lowering a few inches but the system was easily restarted with no modification. No abnormal observations made. The system was stopped at 1:05 pm at a master synchro count value of 12385 to check the tank seal, confirm limit switch functionality, etc., then resumed. The lid was lowered at 1:17 pm.

Overall, it seems that the lowering process was more "smooth" than in past years - the jack stations seemed to be well-synchronized and relatively few motor/brake toggles were required during travel (this is just based on memory and visual observations, not actual data). If this is true, it is likely attributed to the fact that during lid up operation this year, each station was synchronized with respect to each other to help reduce stoppages.

See attached summary.

An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.

Today (2024-02-16) when operating the lift trolley to install shields in the cyclotron, the vertical motion was observed to be intermittently working. When the trolley was out of the cyclotron tank and on the carrier, troubleshooting showed that it was likely an issue with the motor relays. The relay box was removed and the relays showed signs of wear due to ageing. The relays were replaced with new ones, and the box was reinstalled. The vertical motion appeared to work correctly after the replacement. The issue is considered resolved.

It is suggested that all motor relays be inspected or replaced as soon as possible, as others are of a similar age and could fail in the near future.

Edit (2024-02-26): last week, further issues were encountered. It was suspected that the upper limit switch was faulty. The limit switch was replaced and the issues did not persist. Note: it is possible that all observed issues could be related to the CAMAC power supply problem discussed in e-log 169

The cyclotron lid was raised starting ~10:30am on 2024-02-06 and finishing at ~3:30pm on 2024-02-08, under work permit C2024-02-06-4.

Upon first attempt to raise the lid, issues were encountered. The system seemed to start as expected but quickly tripped, resulting in no appreciable travel. Upon investigation, it was determined that this was due to a blown fuse which feeds the brakes master relay (see e-log  https://elog.triumf.ca/TIS/RH-Cyclotron/165). The result of this was the the brakes were not properly disengaging, leading to additional friction in the system, and preventing regular motion.

After rectifying this issue, a second attempt was made to raise the lid. This time, issues were encountered with some of the stations being out-of-sync with respect to the others (a result of the troubleshooting which took place for the aforementioned issue). After manually jogging some stations, it was decided to lower the lid all the way, jog motors for individual stations so that they physically match each other in terms of travel distance at a known starting point, manually adjust counters on some stations whose counter values were significantly off so that they matched neighbours, and then jog individual stations such that each synchro measurement was close enough to begin lid up operation.

After this work took place, another attempt to raise the lid was made. This time, the lid traveled fairly smoothly, with a few occasional stops. It seemed that some motors were lagging behind others a bit - namely stations 5, 11, and 12. After raising the lid approximately 4-6", it was decided that the best course of action would be to fully lower it and start again, now that the system had some time to "warm up".

The lid was lowered to the bottom limit again, and then raised. The master synchro count was 0 to start. This time, lid travel was the most smooth, and stops near the start of travel were infrequent. The occasional stop occurred but the system could easily be started again. Jogging individual stations was rarely required. Most stops were due to noise observed on the synchro at station 5 (false trips).
When the lid was approximately halfway up, another unexpected issue occurred: station 10 completely stopped, which caused the entire system to stop. Station 10 could not be jogged in either direction in local mode. After some troubleshooting, it was determined that the motor contactor was causing issues - eventually it was rectified (see e-log https://elog.triumf.ca/TIS/RH-Cyclotron/166).

The lid continued to raise smoothly for the remainder of the travel with very few or no stoppages. It raised all the way, as confirmed by the gap measurement.

Final master synchro count when lid was fully up: 47401

 

See attached inspection checklist which contains information about each jack station during lid up operation, as well as pre-job inspection data.

Recommended actions:
- Replace synchro for station 5 next shutdown
- Inspect all 12 motor contactors prior to next lid down
- Order spare motor contactor
- Install replacement fuse (on order)

It was observed during lid up operation on 2024-02-08, approximately halfway through raising the lid, that the motor contactor for station 10 located within its small motor panel stopped functioning. The symptoms observed were: entire system stopped since station 10 lagged behind (approx. 150 counts on counter), station 10 could not be jogged locally in either direction, the control signal to activate the motor was being sent to the contactor when attempting to jog, but no motion occurred.

After locking out the panel, the contactor was disassembled and reassembled by an electrician. The system was powered back on and the motor functioned normally. The explanation as to why the issue occurred in the first place is unknown but it is likely attributed to years of infrequent usage causing it to "stick" and/or corrosion buildup.

In any case, it is recommended that all contactors be inspected during the year of 2024 and that at least station 10 be replaced, and spares for other stations procured.

It was observed when attempting lid up operation on 2024-02-06 that the brakes for each of the jacks were not operating correctly. It appeared that power was not being applied so the brakes were stuck in the engaged state. This ultimately resulted in issues with the lid up operation.

The issue was traced to relay A1 (the master brake power relay) not toggling as expected, despite the relevant control signals being present to try to enable it. This relay is located in the MCC-E 400V panel. Adjacent to the relay is fuse F4 which supplies the relay. This fuse was blown. The fuse was: English Electric CIS50. 50A rated, 600V, H.R.C. Form II. Replacements were ordered, however the part is obsolete due to its age. In the short term, a temporary workaround was implemented by installing a 30A fuse. This will be replaced when the spare fuses arrive.

The system was powered up again today (2024-02-07) and the brakes energized as expected. It appears that there is no issue. The root cause of why the fuse blew in the first place is unknown at this time. It was originally suspected that it was due to accidental mis-wiring of the brakes when they were reconnected after annual servicing, but if this was true, the fuse would have instantly blown again when relay A1 was energized. The best guess at this point is that the fuse failed due to age (it is likely around 60 years old).

This e-log will be updated if any further developments are made on this issue.

 The annual elevating system maintenance for the 2023-2024 shutdown following Document-220322, 

Work done after electrical lockout of elevating system and motors electrically disconnected: 

Work done on Station #1: 

• Note readings on station counter (71) and gap clearance on yoke (5/8" wrench thickness) 
• Remove all spares(#13)  from station 1 (two jacks, two highfield gear reducers and one electric motor) 
• Reinstall complete serviced original station into station 1
• Set yoke gap and counter to same as previously recorded

Work done on Station #7: 

• Note readings on station counter (71) and gap clearance on yoke (9/16" wrench thickness) 
• Remove originals from station (two jacks, two highfield gear reducers and one electric motor) 
• Install Spare elevating system (station #13) into station #7 
• Set yoke gap and counter to same as previously recorded 
• Inspected and serviced upper bearing: OK 
• the removed originals are taken to the Remote Handling Active Machine shop where each component will be inspected and serviced to be reinstalled next shutdown

Electric motors have been reconnected at stations #1 and #7.

Because of observed noise on the synchro at jack station 11 during recent shutdown lid up/down operations, the synchro at this station was replaced on Jan. 31, 2024. The procedure in Document-234859 was followed.

Prior to starting the replacement, the following counter values and synchro measurements on the station in question and adjacent stations were noted:
The master synchro count value was measured at: 0000
Station 10: Counter: 99392.5, synchro voltage: 5.89 VAC
Station 11: Counter: 0073, synchro voltage: 0.787 VAC (anomaly)
Station 12: Counter: 0068, synchro voltage: 5.98 VAC
 

During the replacement, the master synchro was jogged such that adjacent stations 10 and 12 read a synchro value of approximately 0 volts (as per the nulling procedure). The following observations were made:
The master synchro count value was measured at: 0072
Station 10: synchro voltage: 0.047 VAC
Station 11: synchro voltage prior to swap: 5.70 VAC (anomaly)
Station 12: synchro voltage: 0.282 VAC

After the replacement, before jogging the master synchro back to its starting value of 0000, the following measurements were made:
Station 10: synchro voltage: 0.038 VAC
Station 11: synchro voltage after swap: 0.12 VAC (now correct)
Station 12: synchro voltage: 0.278 VAC

When jogging the master synchro after the swap was completed, it was confirmed that stations 10, 11, and 12 all responded as expected, and their values were synchronized with respect to each other, indicating a successful replacement. The master synchro was jogged back to its original counter value of 0000. The system is now considered operational.

Motor contactors for the cyclotron elevating system jacks were inspected on 2025-01-23 by Joel Semilla under work permit C2024-01-19-1. Joel's assessment is as follows:

"All the twelve magnetic contactors have the same contact tips condition. Last year, I have [sic] to take out the contacts from the housing due to severe "pitting" of the contact's silver surface due to excessive "flash-over" which is common to magnetic contactor on inching/jogging duty. Due to the better condition of the contact tips this year, I can re-surface the contact tips on the magnetic housing itself."

Prior to recent inspections, a recurring inspection at an informed frequency had not been set. It is suggested based on the state of the contactors now, in comparison with the previous inspection, to schedule inspections of these contactors every 4 years. Next inspection due: Jan. 2028.

Edit: on 2024-01-24, Joel Semilla updates: "re-surfacing of stationary and movable contacts for the 12 cyclotron elevating motors have been completed".

 All the bearings on both the upper and lower wheel were replace to see if it will improve the accuracy of the orbital position reading. See drawing SK-2489 for list of bearings used.

The encoder was also replaced to match the style used elsewhere.

 Damaged "H" channel removed from the first festooning car, and spare channel was put in place.

Note: it seems like the aluminum channel is not strong enough for the forces its being put under, and is resulting in deformations (See picture "Festooning car H channel deformation"). We suspect this has happened in the past, notches were cut out where the deformation likely occurred and then the part was flipped and put back in place. There was two notches on the bottom face of the part we took out, and there was already one on the spare part. (see picture "Festooning car H channel bottom face"). A new channel needs to be designed out of steel for the first car to prevent this from reoccurring.

 Service bridge, Outrigger #1, and Outrigger #2 have had the controls system migrated to PLC controls. The 5069-AENTR control unit is located at the centrepost and will be mounted there after wiring is cleaned up. Preliminary testing consisted of operators simulating a shutdown task was performed today and was a success. Minor adjustments to the encoder logic will be made before the commissioning test. The follow tests were done: 

PLC box:

Check Relays toggling over correctly. 

OR1:

Radial In/Out & HMI Feedback Accurate

Vertical Up/Down & HMI Feedback Accurate

Shoulder Home/Out & HMI Feedback Accurate

Elbow Home/Out & HMI Feedback Accurate

Pan CW/CCW & HMI Feedback Accurate

Tilt Up/Down & HMI Feedback Accurate

Cameras Zoom/Focus

4 movements at once

Random camera restart noted (could be tied to grounding issue. will further investigate)

OR2:

Radial In/Out & HMI Feedback Accurate (Encoder Swap direction)

Vertical Up/Down & HMI Feedback Accurate

Shoulder Home/Out & HMI Feedback Accurate

Elbow Home/Out & HMI Feedback Accurate

Pan CW/CCW & HMI Feedback Accurate

Tilt Up/Down & HMI Feedback Accurate

Cameras Zoom/Focus

4 movements at once

Bridge:

Orbital Motor & Encoder HMI Feedback (Orbital Motor controls using Outrigger CP only active when ORs not selected) (set range to 0 to 360 same direction)

Centerpost Camera PTZF

Light

Speed Control

Vid Trolley:

Test operating video trolley with bridge controls and outriggers

Top/Bot LED

Implement radial speed control using pot

The mirror actuation mechanism was observed to be "sticky" during 2023 winter shutdown - it occasionally did not function properly and needed some manual assistance. 

The following items were addressed to resolve the issue: 

• Replacement of the Tyne Myte double acting valve which controls the cylinder (based on troubleshooting tests, it seemed this valve was not functional or intermittently functional)
• Addition of a silencer and in-line flow control on the exhaust of the aforementioned valve - adjusted to allow for smooth operation of the mirror when toggled
• Cleaning of the cylinder's shaft (it had a sticky residue build-up and was notably high friction) - this cylinder should likely be replaced in the near future 
• Replacement of air tubing along the supply line to the valve, and between the valve and cylinder
• Replacement of all fittings (push-connect style) on the valve and cylinders

The mirror actuation was tested after these changes and is behaving normally.

No further repairs required, although it is suggested to change the remainder of the tubing and fittings for the other valves which control different functions on this trolley, and to eventually upgrade the valves themselves to a low profile valve bank. 

The two camera tilt mechanisms on the lift trolley have been upgraded. Upgrades include:

For the individual camera:

• Replaced cylinder
• Replaced fittings for cylinder
• Replaced air tubing from flow control to cylinder
• Visual inspection and fasteners tightened

For the overall platform containing the camera and lighting:

• Replaced cylinder
• Replaced fittings for cylinder
• Replaced valve (5/3 centre closed style)
• Replaced air regulator and set pressure (note: at the time of writing this e-log, it still needs to be fastened in place via its mounting bracket)
• Replaced air tubing
• Visual inspection and fasteners tightened

Outrigger #1 and #2 have had their controls system overhauled as part of the O503 project. The previous controls enclosure and the elements have been replaced with a new aluminum enclosure which contains DIN rail relays, a 5V power supply, and a +-12V Power supply. The new centrepost slave PLC unit handles the control logic of OR1, OR2, the Bridge and is connected to the main PLC in the control room through Ethernet. Both outriggers were tested individually.

Systems Tested:

Vertical Drive - Up, Down, 5V Position (Inches) Potentiometer

Radial Drive - In, Out, Encoder (Inches) 

Shoulder Drive - Home, Out, 5V Position Potentiometer (Deg) 

Elbow Drive - Home, Out, 5V Position Potentiometer (Deg) 

Wrist Drive - Pan Left, Pan Right, Tilt Up, Tilt Down, 5V Position Potentiometer (Deg)

Camera - Zoom In, Zoom Out, Focus Near, Focus Far

Testing Methodology:

We used the control panel which the Outrigger operator will be using during shutdown to confirm that the buttons translate to correct movements of each individual system. Direction control for the motors is done with relay switching of 24V to flip the polarity applied to the motors. +-12V is used for the zoom focus control of the HCZ-6320N camera. Correct movements was observed for both outriggers.

Position feedback was calibrated by adjusting the potentiometer's voltage to 2.5V at the mid point and spanned from 0V at low end to 5V at high end. Correct positions for both outriggers was observed on the HMI screen.

Next Steps

Testing the bridge orbit drive, orbit encoder, centrepost camera, centrepost light, both OR1 and OR2 simultaneously.

The "old" video trolley featuring scissor mechanism and DSLR camera has officially been decommissioned. Going forward, Video Trolley Mk 2 (TRH1089D) will be used.

The original end feet and end caps were removed from the maintenance ladders and replaced by new end caps (TRH1637).