The vault shield door has been closed and seal inflated. The operation was performed with no issue.

The lid up operation took place on Jan. 30, 2023.

The system was not energized well ahead of time to allow to "warm up", as is usually done.

Starting Master Synchro value: 62

The Master Synchro was jogged to a value around 2 prior to starting, to allow it to lag behind the jacks.

When first starting: after a few attempts to raise, it was determined that the master synchro value needed to lag behind the other synchros more than initial suspected. This was changed, and the system operated smoothly for a short time.

During lid travel, station 5 and station 11 both caused trips and needed to be manually jogged to match the rest of the stations. The system was restarted, and more trips occurred. Stations 7 and 10 were adjusted as well. Eventually, after adjusting a few stations, the lid travel smoothed out and no more issues were observed for the remainder of the operation.

The system was stopped at approx. 3 inches of travel (synchro count 5255) to inspect the seal, etc. Everything OK.

No major issues with any jacks. No overheating observed.

Jack 11 showed occasional noise spikes on readback (this happened in 2021 and 2022 as well - suggest replacing this station's synchro next).

When finished, the limit switches toggled as expected, and the gap was measured at 46.5" (correct value).

Ending Master Synchro value: 47388

The attached spreadsheets show all synchro values before and after the operation, as well as jack station infrared heat measurements.

Start time: 9:23 am
End time: 10:53 am (includes gap measurement, etc. - actual lid travel finish was a bit earlier)
Total time to raise was approx. 1.5 hr.

Doses:
Adam Newsome: 0.005 mSv
Maico Dalla Valle: 0.121 mSv
Eric Lahe: 0.056 mSv
Ryan Anderson: 0.017mSv
Kevin Lucow: 0.060 mSv
Ed Parton: 0.007 mSv
Jason Zhang: 0.025 mSv
Nadia Jorgenson: 0.005 mSv
Arthur Leung: 0 mSv
Dan Louie: N/A

[edited 2023-09-8 to fix typo - Adam Newsome]

The lid down operation took place on Mar. 31, 2023.

The system was energized a few minutes ahead of time.

Starting Master Synchro value: 47388. The master synchro was jogged up prior to starting to approximately 47395.

Upon startup at 10:38 am, the system worked fine and lowered the lid. No stops occurred and the lid lowered smoothly. No abnormal observations made. The lid was completely lowered at 11:05 am.

See attached summary.

The vault shield door has been closed and seal inflated. The operation was performed with no issue.

The original end feet and end caps were removed from the maintenance ladders and replaced by new end caps (TRH1637).

An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.

The lid down operation took place on April 19, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47401. The master synchro was not jogged prior to starting.

Upon startup at 12:45 pm, the system worked fine and lowered the lid. One stop occurred after lowering a few inches but the system was easily restarted with no modification. No abnormal observations made. The system was stopped at 1:05 pm at a master synchro count value of 12385 to check the tank seal, confirm limit switch functionality, etc., then resumed. The lid was lowered at 1:17 pm.

Overall, it seems that the lowering process was more "smooth" than in past years - the jack stations seemed to be well-synchronized and relatively few motor/brake toggles were required during travel (this is just based on memory and visual observations, not actual data). If this is true, it is likely attributed to the fact that during lid up operation this year, each station was synchronized with respect to each other to help reduce stoppages.

See attached summary.

A second lid down operation took place on April 30, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47427 (after it was jogged up slightly prior to starting).

Upon startup at 12:51 pm, the system worked fine and lowered the lid. The lowering was stopped at approximately 5" gap as per standard procedure for Operations to perform checks. Furthermore, the lowering was stopped multiple times after that at approximately 0.5" increments for resonator latch checks.

No abnormal observations made. No jogs required. Once again, this was a very smooth lowering process. The lid was fully lowered at 2:00 pm.

Ending master synchro value: 66

See attached summary.

Fault 16710 was initially assigned to Remote Handling: IBOP0206 giving false positive - Al-ALCW water pressure shows "off" when it isn't; The safety system is reporting that Al-ALCW water pressure is "off" (page SVG) when the system is still on and at normal operating pressure.

2024-04-10: initial inspection by A. Newsome yielded the following (email from A. Newsome to K. Genge):

1. First, on an unrelated note, there's a thermometer mounted to the Al-ALCW pipe which is loose and rattling due to vibration of the pipe. There's a little bit of water leak in this area. I don't want to mess around with it but I suggest someone tighten it soon! 
2. I looked around for sensors on the Al-ALCW pipe and I think I found the water pressure sensor which is giving the faulty reading. I have three suggestions for what to look into, in order of importance:

1. The cable for this sensor has a portion of it which is electrical-taped together. I didn't mess with it, but I suspect underneath the electrical tape it could just be disconnected. This would be a super quick check and quick fix. Suggest connectorizing it in the future with a simple Molex connector or similar - let me know if you want us to do that. 
2. The pressure transmitter itself could have failed or could have some sort of internal issue. Suggest disassembly to inspect. First, water will need to be turned off and the isolation valve turned off. If no spare readily available, suggest swapping with another transmitter of the same model to see if that is in fact the issue before ordering one (I think there's one nearby for the Meson Hall water system, beside the electrical panels). 
3. Somewhere along the line, there could be a cable degradation issue, or the cable could be disconnected/pinched/cut. Suggest looking into this last as it is the most tedious and also least likely.
   

2024-07-16: J. Ableseth confirmed the pressure transmitter (3400T Series) appears to be in working condition.

2024-07-24: A. Newsome and J. Zhang investigated the elevating system control panel located in B3 level. The following are findings (email from A. Newsome to K. Genge):

1. Our elevating system panel accepts an overall interlock summary signal from an unknown external source (probably a safety panel located elsewhere). There's no manipulation of the signal at all - it just enables/disables the system from running. 
2. The indicator lights on top of the elevating system panel do not have any electronics nearby - the light wiring comes from an external source via multiconductor cable. They are just indicators and are not connected to any electronics inside our panel at all. In fact this isn't even in our drawing package. 
3. We traced the light signal multiconductor cable out to the tunnel on B3 level - the cable runs along the middle cable tray. Did not follow all the way, but we assume it comes from the safety panel in B2 that Kathleen pointed out. In this panel, there's an exact duplicate of the indicator light display! This one, too, showed the "CYC H2O OFF" light as being on (false logic since the pump is active). It appears that the wiring for these lights is connected to a bank of relays right beside it. My suspicion is that one of these relays has failed, which then causes both of the indicator lights to be stuck in the ON position (this is the "safe" position). Given that there's also a similar cyropanel-related faulty signal, this makes even more sense - the relays are probably very old and/or there was some external issue that could have caused both to fail. Seems more likely than two totally unrelated sensors failing at the same time, that just happen to be routed through the same panel and relay bank. 

• Kevin Ardron investigate the relays in that panel... if it is not obvious the relay has failed, easy thing to do is just swap it with an adjacent one that is known to work and see what happens.
• Probably a good idea to just replace all of them if they are all the same age, because we'll likely see others fail. I think it is a relatively inexpensive and straightforward thing to do that could save some headache and beam trips in the future. 
• If the relay appears to be fine, or if the relays turn out to be totally unrelated to the actaul signal itself, continue tracing the signal back to its ultimate source. This panel on the B2 level is most likely where we'll find the issue.
  
  

[Update] 2024-07-26: Email from K. Genge:

An update from Safety System’s findings re: elevating system interlocks -

1. Very difficult to test the relays given the design/location of them and the cables connecting to them.
2. Kevin A. managed to trace the cryopanel elevating system related fault (#17020) over to the vacuum racks on SAB2 where he coincidently ran into Dimo, Vacuum Group Leader. Turns out that the sensor that gives this signal is connected to old cyclotron cryogenics/vacuum equipment that is no longer in use (and therefore providing the elevating system interlock system with the right info!). During the shutdown, Dimo and his team will move this sensor to the correct location so that it’s providing the correct info. That fault has been redirected.
3. The water ok signal goes to a “water system box”. We don’t know where this box is, but Kevin is willing to provide additional cable info to Joe. I recommend that the next step is for Joe and Kevin Ardron to connect and figure out where this cable goes.

An emergency trolley rescue winch (TRH1527) was developed for the purposes of retrieving trolleys from the cyclotron tank whose radial drive have failed.

This winch was tested today on the carrier in mockup setting with success by Adam Newsome, Dan McDonald, and Travis Cave. The following is a summary of the test:

• Lower resonator trolley was used with additional 1200 lb loading (total weight: approximately 2900 lb)... this represents the worst-case scenario for rescue
• The drive mechanism was bypassed so wheels could freely rotate (this is standard rescue procedure)
• The winch was mounted so that the alignment angle of the cable was close to 0 degrees
• The winch successfully pulled the entire assembly with no issue – it was pulled approximately 5 metres

An ECO will be released for the drawings for mounting components - no redlines were required.

 All 4 Under rail wheels serviced and cleaned. All Timken bearings repacked with grease. No defects in any bearing set. 4 spare Timken 02820 bearings in RH shop.

 Motor/Gearbox unit removed from carrier and fully inspected and serviced. Oil changed on May 29th using Mobil 600 W Super Cylinder Oil. No signs of wear in internal gears, recommend oil service roughly every 5 years after shutdown.

 1,2, 5 Year Service and maintenance package completed on Carrier. All reducers repacked with Molub-Alloy 777-1 ES grease. Main drive train and all bearings relubricated. No issues. Operational.