Today (2024-02-16) when operating the lift trolley to install shields in the cyclotron, the vertical motion was observed to be intermittently working. When the trolley was out of the cyclotron tank and on the carrier, troubleshooting showed that it was likely an issue with the motor relays. The relay box was removed and the relays showed signs of wear due to ageing. The relays were replaced with new ones, and the box was reinstalled. The vertical motion appeared to work correctly after the replacement. The issue is considered resolved.

It is suggested that all motor relays be inspected or replaced as soon as possible, as others are of a similar age and could fail in the near future.

Edit (2024-02-26): last week, further issues were encountered. It was suspected that the upper limit switch was faulty. The limit switch was replaced and the issues did not persist. Note: it is possible that all observed issues could be related to the CAMAC power supply problem discussed in e-log 169

An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.

The lid down operation took place on April 19, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47401. The master synchro was not jogged prior to starting.

Upon startup at 12:45 pm, the system worked fine and lowered the lid. One stop occurred after lowering a few inches but the system was easily restarted with no modification. No abnormal observations made. The system was stopped at 1:05 pm at a master synchro count value of 12385 to check the tank seal, confirm limit switch functionality, etc., then resumed. The lid was lowered at 1:17 pm.

Overall, it seems that the lowering process was more "smooth" than in past years - the jack stations seemed to be well-synchronized and relatively few motor/brake toggles were required during travel (this is just based on memory and visual observations, not actual data). If this is true, it is likely attributed to the fact that during lid up operation this year, each station was synchronized with respect to each other to help reduce stoppages.

See attached summary.

Due to an issue with resonator latching, there was a need to raise the cyclotron lid a second time this shutdown.

The plan at first was to raise the lid by small increments (approximately 1mm at a time up to approximately 25mm) while RF group performed some checks.

After the RF groups completed their checks, it was determined that the lid needed to be raised the full extent.

The starting master synchro count originally was 52, and it was 2480 at the time when the decision was made to fully raise the lid.

The lid raised entirely with zero stoppages or observed issues. No abnormalities observed with any jack station.

The lid was fully raised by 1:49pm with a master synchro count of 47383.

See attached summary.

A second lid down operation took place on April 30, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47427 (after it was jogged up slightly prior to starting).

Upon startup at 12:51 pm, the system worked fine and lowered the lid. The lowering was stopped at approximately 5" gap as per standard procedure for Operations to perform checks. Furthermore, the lowering was stopped multiple times after that at approximately 0.5" increments for resonator latch checks.

No abnormal observations made. No jogs required. Once again, this was a very smooth lowering process. The lid was fully lowered at 2:00 pm.

Ending master synchro value: 66

See attached summary.

The cyclotron lid was raised starting ~11:45am on 2024-05-02 and finishing at ~12:12pm under work permit C2024-05-02-5.

The raising was stopped a few times up until approximately 5" elevation, for RF group to perform checks, and to inspect the tank seal.

The tank seal was stuck to the lid near jack station 2.

The lid raised smoothly with no stoppages due to out-of-sync stations. No stations needed to be jogged.

Starting master synchro count was 66, but was jogged to 2 prior to starting. The final master synchro count when the lid was fully raised was 47387.

Note: the measuring stick which is affixed to the top of the lid and used to indicate full elevation was bent and should be "re-calibrated".

Summary attached.

A third lid down operation took place over May 6-7, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47387 (after it was jogged up slightly prior to starting).

Upon startup on May 6 at 14:42, the system worked fine and lowered the lid. The lowering was paused at 15:28 (master synchro value of 17593).

Jack 6 exhibited issues similar to previous lid up (see e-log 166). It was likely due to an issue with a motor contactor. Electricians were contacted but could not work on the issue until the following day so the lowering was paused.

At 07:30 on May 7, electrician Randy Boehm determined that a cable connector on the motor contactor was loosened. After tightening the cable, jogging motor 6 was tested - OK.

Lid down commenced again at 08:10. Master synchro: 1915.

The lowering was stopped at approximately 1" gap as per standard procedure for Operations to perform checks. The lowering was stopped again multiple times to check resonator latching - OK.

End time: 08:37

Ending master synchro value: 59

Jogged master synchro value to 2, adjusted station 12 to match.

Notes: It was observed that at lid down Jack 12 did not have any gap. Jack 12 was jogged to match synchro values with the master synchro but still exhibited a 0.125” gap instead of the ideal 0.25” gap. It is recommended to keep an eye on Jack #12 during the next operation and check the motor contactors for loose fittings. In the before and after photos of the motor contactor it looks like the fitting on the bottom right corner of the motor contactor could have either come loose from the vibrations or the screw was sheared off. Further investigation is required.

See attached summary.

A safety walkaround for July 2024 was completed for the B1 and B2 levels by A. Newsome, and for the main shop and back shop by R. Sykes.

Results can be found in the master spreadsheet

Major deficiencies, main shop:

• Unpainted lead bricks need to be picked up. This was requested of mechanical services twice already in July 2024. Follow up next inspection if not removed.
• Nearly half bulbs burnt out - enough to warrant replacement. AN requested of electrical services on 2024-07-17. Follow up next inspection if not replaced.

Major deficiencies, back shop:

• Phone not working. AN created help ticket 2024-7-17. Follow up next inspection.

Minor deficiencies, B1 level:

• 4-5 light bulbs will need to be replaced soon, not urgent yet.

Major deficiencies, B2 level:

• On the outboard support frame, the lifting eye fastener was stripped which prevented the nut from tightening all the way, posing a rigging hazard. The damaged lifting eye was replaced with a working one 2024-07-15.
• Light fixture above bootbox - mounting damaged and hanging precariously. Reported to electrical services 2024-07-15 and informed common users of area.
  Update 2024-07-17 - the issue has been resolved.
• Probes group has rad-waste in the middle of the floor. Reminded them to dispose of it soon.

Fault 16710 was initially assigned to Remote Handling: IBOP0206 giving false positive - Al-ALCW water pressure shows "off" when it isn't; The safety system is reporting that Al-ALCW water pressure is "off" (page SVG) when the system is still on and at normal operating pressure.

2024-04-10: initial inspection by A. Newsome yielded the following (email from A. Newsome to K. Genge):

1. First, on an unrelated note, there's a thermometer mounted to the Al-ALCW pipe which is loose and rattling due to vibration of the pipe. There's a little bit of water leak in this area. I don't want to mess around with it but I suggest someone tighten it soon! 
2. I looked around for sensors on the Al-ALCW pipe and I think I found the water pressure sensor which is giving the faulty reading. I have three suggestions for what to look into, in order of importance:

1. The cable for this sensor has a portion of it which is electrical-taped together. I didn't mess with it, but I suspect underneath the electrical tape it could just be disconnected. This would be a super quick check and quick fix. Suggest connectorizing it in the future with a simple Molex connector or similar - let me know if you want us to do that. 
2. The pressure transmitter itself could have failed or could have some sort of internal issue. Suggest disassembly to inspect. First, water will need to be turned off and the isolation valve turned off. If no spare readily available, suggest swapping with another transmitter of the same model to see if that is in fact the issue before ordering one (I think there's one nearby for the Meson Hall water system, beside the electrical panels). 
3. Somewhere along the line, there could be a cable degradation issue, or the cable could be disconnected/pinched/cut. Suggest looking into this last as it is the most tedious and also least likely.
   

2024-07-16: J. Ableseth confirmed the pressure transmitter (3400T Series) appears to be in working condition.

2024-07-24: A. Newsome and J. Zhang investigated the elevating system control panel located in B3 level. The following are findings (email from A. Newsome to K. Genge):

1. Our elevating system panel accepts an overall interlock summary signal from an unknown external source (probably a safety panel located elsewhere). There's no manipulation of the signal at all - it just enables/disables the system from running. 
2. The indicator lights on top of the elevating system panel do not have any electronics nearby - the light wiring comes from an external source via multiconductor cable. They are just indicators and are not connected to any electronics inside our panel at all. In fact this isn't even in our drawing package. 
3. We traced the light signal multiconductor cable out to the tunnel on B3 level - the cable runs along the middle cable tray. Did not follow all the way, but we assume it comes from the safety panel in B2 that Kathleen pointed out. In this panel, there's an exact duplicate of the indicator light display! This one, too, showed the "CYC H2O OFF" light as being on (false logic since the pump is active). It appears that the wiring for these lights is connected to a bank of relays right beside it. My suspicion is that one of these relays has failed, which then causes both of the indicator lights to be stuck in the ON position (this is the "safe" position). Given that there's also a similar cyropanel-related faulty signal, this makes even more sense - the relays are probably very old and/or there was some external issue that could have caused both to fail. Seems more likely than two totally unrelated sensors failing at the same time, that just happen to be routed through the same panel and relay bank. 

• Kevin Ardron investigate the relays in that panel... if it is not obvious the relay has failed, easy thing to do is just swap it with an adjacent one that is known to work and see what happens.
• Probably a good idea to just replace all of them if they are all the same age, because we'll likely see others fail. I think it is a relatively inexpensive and straightforward thing to do that could save some headache and beam trips in the future. 
• If the relay appears to be fine, or if the relays turn out to be totally unrelated to the actaul signal itself, continue tracing the signal back to its ultimate source. This panel on the B2 level is most likely where we'll find the issue.
  
  

[Update] 2024-07-26: Email from K. Genge:

An update from Safety System’s findings re: elevating system interlocks -

1. Very difficult to test the relays given the design/location of them and the cables connecting to them.
2. Kevin A. managed to trace the cryopanel elevating system related fault (#17020) over to the vacuum racks on SAB2 where he coincidently ran into Dimo, Vacuum Group Leader. Turns out that the sensor that gives this signal is connected to old cyclotron cryogenics/vacuum equipment that is no longer in use (and therefore providing the elevating system interlock system with the right info!). During the shutdown, Dimo and his team will move this sensor to the correct location so that it’s providing the correct info. That fault has been redirected.
3. The water ok signal goes to a “water system box”. We don’t know where this box is, but Kevin is willing to provide additional cable info to Joe. I recommend that the next step is for Joe and Kevin Ardron to connect and figure out where this cable goes.

A safety walkaround was completed for the RH Shop, Back Shop, B1, and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main Shop:
  
  • Lifting equipment has overdue inspection
  • Light bulbs burnt out
  • Unpainted lead has not been removed
• B1:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection
• B2:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection

Action has been taken on all deficiencies.

See previous e-log for phase 1: e-log 134

Phase 2 of the bridge lift mechanism refurbishment was completed. The scope of work for this phase includes replacement of the hydraulic pump (including dump valve) and associated controls hardware, plus some portion of the hydraulic tubing which interfaces with the pump itself. Work was completed by Hyseco under work permit C2024-10-02-2.

The pump was integrated such that existing operator control functionality was maintained, except for one difference: with the new system, the pump does not run all the time when the operator turns the "pump on" switch on. Rather, the pump is turned on during the action of raising/lowering. This was deemed acceptable by operators.

During installation, flow rates were adjusted by Hyseco techs to achieve the desired performance in terms of raising and lowering speed. It has been noted that raised appears to be slightly more smooth and even than previously. Lowering behaves similarly to previous. Furthermore, the new pump is much quieter.

Due to the relative simplicity of this system, a full commissioning report was not created, but the following commissioning tests were performed multiple times between 2024-10-14 and 2024-10-15:

• Turn pump enable on
• Raise bridge to maximum upper limit and confirm it stops automatically
• Lower bridge to rest on centre post support stand
• Dump to fully disengage cylinders

These tests were successfully completed and witnessed by A. Newsome, T. Cave, R. Sykes, J. Zhang, N. Jorgenson, M. Dalla Valle, and the Hyseco techs. Therefore, the system is considered commissioned and operational. Some minor wiring cleanup remains to be done and the system will be tested again multiple times prior to shutdown. A maintenance manual will be requested from the vendor.

The old pump will be kept for ~3 years as a spare.

Update 2024-10-21 [AN]:

For reasons unknown, the system is no longer functioning. There are two issues, which may be related or may not be:

1. Previously during the aforementioned testing the hydraulic pump motor was powered via 120 VAC wall outlet. Now it is being powered from our Carrier (ultimately from the blue power supply box in B2 level). There are large line losses associated with this. We measure approximately 35 VAC drop at the input to the pump when attempting to run the system. Because of this, the 24V supply located in the pump's control box used for enabling the motor and for the relays is dropping out, which turns the pump off immediately and causes chattering. When this 24VDC supply is separately powered from wall outlet, it does not drop out, but still the large line loss voltage issue remains for the pump motor. Based on an estimate of around 100 ft long wiring, the line losses measured suggest an inrush current of approximately 50A. This seems relatively high given the pump's FLA of 10A. This should be investigated further. Perhaps it is related to the second issue listed below, or perhaps the motor should be switched to 240 VAC, or perhaps the wiring should be changed from 14 AWG to 10 AWG to reduce losses.

2. When pressing the "down" button, the system moves up. There is no way to move down aside from dumping. Even when disconnecting the control signals entirely from the valves and manually pushing them in, both valves cause upward motion. Suspect an issue with hydraulic line plumbing or the valves themselves.

Hyseco engineer will be contacted and further troubleshooting will take place this week. This log will be updated accordingly.

Update 2024-10-29 [NJ]

The issue of the system moving up when pressing down is resolved. Hyseco engineer returned to repair. It was determined the the relief valves by nature have some leakage and because the flow control valves were so tight more fluid was travelling through the leakage than the control valve. The relief valve was moved to be before the flow control valve, which seemed to resolve the issue.  Additionally one of the pressure gauges was replaced.

A safety walkaround was completed for the RH building B1 and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• B2:
  
  • Low on booties stock (more have been ordered)
  • Rad waste from Probes group remains in the middle of the floor (requested RPG to ask them to remove it)

Action has been taken on all deficiencies.

A safety walkaround was completed for the RH building shop and back shop.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main shop, ground level:
  
  • Lighting still needs to be replaced (this was previously requested of electrical services and they are aware of it)
  • Unpainted lead bricks still need to be picked up (this was previously requested of mechanical services and they are aware of it)
  • Rad waste needs to be picked up (will do soon)