An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.