A safety walkaround was completed for the RH building shop and back shop.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main shop, ground level:
  
  • Lighting still needs to be replaced (this was previously requested of electrical services and they are aware of it)
  • Unpainted lead bricks still need to be picked up (this was previously requested of mechanical services and they are aware of it)
  • Rad waste needs to be picked up (will do soon)

A safety walkaround was completed for the RH building B1 and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• B2:
  
  • Low on booties stock (more have been ordered)
  • Rad waste from Probes group remains in the middle of the floor (requested RPG to ask them to remove it)

Action has been taken on all deficiencies.

See previous e-log for phase 1: e-log 134

Phase 2 of the bridge lift mechanism refurbishment was completed. The scope of work for this phase includes replacement of the hydraulic pump (including dump valve) and associated controls hardware, plus some portion of the hydraulic tubing which interfaces with the pump itself. Work was completed by Hyseco under work permit C2024-10-02-2.

The pump was integrated such that existing operator control functionality was maintained, except for one difference: with the new system, the pump does not run all the time when the operator turns the "pump on" switch on. Rather, the pump is turned on during the action of raising/lowering. This was deemed acceptable by operators.

During installation, flow rates were adjusted by Hyseco techs to achieve the desired performance in terms of raising and lowering speed. It has been noted that raised appears to be slightly more smooth and even than previously. Lowering behaves similarly to previous. Furthermore, the new pump is much quieter.

Due to the relative simplicity of this system, a full commissioning report was not created, but the following commissioning tests were performed multiple times between 2024-10-14 and 2024-10-15:

• Turn pump enable on
• Raise bridge to maximum upper limit and confirm it stops automatically
• Lower bridge to rest on centre post support stand
• Dump to fully disengage cylinders

These tests were successfully completed and witnessed by A. Newsome, T. Cave, R. Sykes, J. Zhang, N. Jorgenson, M. Dalla Valle, and the Hyseco techs. Therefore, the system is considered commissioned and operational. Some minor wiring cleanup remains to be done and the system will be tested again multiple times prior to shutdown. A maintenance manual will be requested from the vendor.

The old pump will be kept for ~3 years as a spare.

Update 2024-10-21 [AN]:

For reasons unknown, the system is no longer functioning. There are two issues, which may be related or may not be:

1. Previously during the aforementioned testing the hydraulic pump motor was powered via 120 VAC wall outlet. Now it is being powered from our Carrier (ultimately from the blue power supply box in B2 level). There are large line losses associated with this. We measure approximately 35 VAC drop at the input to the pump when attempting to run the system. Because of this, the 24V supply located in the pump's control box used for enabling the motor and for the relays is dropping out, which turns the pump off immediately and causes chattering. When this 24VDC supply is separately powered from wall outlet, it does not drop out, but still the large line loss voltage issue remains for the pump motor. Based on an estimate of around 100 ft long wiring, the line losses measured suggest an inrush current of approximately 50A. This seems relatively high given the pump's FLA of 10A. This should be investigated further. Perhaps it is related to the second issue listed below, or perhaps the motor should be switched to 240 VAC, or perhaps the wiring should be changed from 14 AWG to 10 AWG to reduce losses.

2. When pressing the "down" button, the system moves up. There is no way to move down aside from dumping. Even when disconnecting the control signals entirely from the valves and manually pushing them in, both valves cause upward motion. Suspect an issue with hydraulic line plumbing or the valves themselves.

Hyseco engineer will be contacted and further troubleshooting will take place this week. This log will be updated accordingly.

Update 2024-10-29 [NJ]

The issue of the system moving up when pressing down is resolved. Hyseco engineer returned to repair. It was determined the the relief valves by nature have some leakage and because the flow control valves were so tight more fluid was travelling through the leakage than the control valve. The relief valve was moved to be before the flow control valve, which seemed to resolve the issue.  Additionally one of the pressure gauges was replaced.

A safety walkaround was completed for the RH Shop, Back Shop, B1, and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main Shop:
  
  • Lifting equipment has overdue inspection
  • Light bulbs burnt out
  • Unpainted lead has not been removed
• B1:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection
• B2:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection

Action has been taken on all deficiencies.

  Jack 7L pulled apart and inspected for deficiencies, alignment monitored with prussian blue and deemed acceptable wear. All parts in working order, jack was rebuilt and lubricated with Molub-Alloy 777-1 ES. Fully operational for next shutdown period swap.

 Jack 7R pulled apart and inspected for deficiencies, all parts deemed to be in working order, jack was rebuilt and lubricated with Molub-Alloy 777-1 ES. Jack is deemed fully operational for next swap. 

Fault 16710 was initially assigned to Remote Handling: IBOP0206 giving false positive - Al-ALCW water pressure shows "off" when it isn't; The safety system is reporting that Al-ALCW water pressure is "off" (page SVG) when the system is still on and at normal operating pressure.

2024-04-10: initial inspection by A. Newsome yielded the following (email from A. Newsome to K. Genge):

1. First, on an unrelated note, there's a thermometer mounted to the Al-ALCW pipe which is loose and rattling due to vibration of the pipe. There's a little bit of water leak in this area. I don't want to mess around with it but I suggest someone tighten it soon! 
2. I looked around for sensors on the Al-ALCW pipe and I think I found the water pressure sensor which is giving the faulty reading. I have three suggestions for what to look into, in order of importance:

1. The cable for this sensor has a portion of it which is electrical-taped together. I didn't mess with it, but I suspect underneath the electrical tape it could just be disconnected. This would be a super quick check and quick fix. Suggest connectorizing it in the future with a simple Molex connector or similar - let me know if you want us to do that. 
2. The pressure transmitter itself could have failed or could have some sort of internal issue. Suggest disassembly to inspect. First, water will need to be turned off and the isolation valve turned off. If no spare readily available, suggest swapping with another transmitter of the same model to see if that is in fact the issue before ordering one (I think there's one nearby for the Meson Hall water system, beside the electrical panels). 
3. Somewhere along the line, there could be a cable degradation issue, or the cable could be disconnected/pinched/cut. Suggest looking into this last as it is the most tedious and also least likely.
   

2024-07-16: J. Ableseth confirmed the pressure transmitter (3400T Series) appears to be in working condition.

2024-07-24: A. Newsome and J. Zhang investigated the elevating system control panel located in B3 level. The following are findings (email from A. Newsome to K. Genge):

1. Our elevating system panel accepts an overall interlock summary signal from an unknown external source (probably a safety panel located elsewhere). There's no manipulation of the signal at all - it just enables/disables the system from running. 
2. The indicator lights on top of the elevating system panel do not have any electronics nearby - the light wiring comes from an external source via multiconductor cable. They are just indicators and are not connected to any electronics inside our panel at all. In fact this isn't even in our drawing package. 
3. We traced the light signal multiconductor cable out to the tunnel on B3 level - the cable runs along the middle cable tray. Did not follow all the way, but we assume it comes from the safety panel in B2 that Kathleen pointed out. In this panel, there's an exact duplicate of the indicator light display! This one, too, showed the "CYC H2O OFF" light as being on (false logic since the pump is active). It appears that the wiring for these lights is connected to a bank of relays right beside it. My suspicion is that one of these relays has failed, which then causes both of the indicator lights to be stuck in the ON position (this is the "safe" position). Given that there's also a similar cyropanel-related faulty signal, this makes even more sense - the relays are probably very old and/or there was some external issue that could have caused both to fail. Seems more likely than two totally unrelated sensors failing at the same time, that just happen to be routed through the same panel and relay bank. 

• Kevin Ardron investigate the relays in that panel... if it is not obvious the relay has failed, easy thing to do is just swap it with an adjacent one that is known to work and see what happens.
• Probably a good idea to just replace all of them if they are all the same age, because we'll likely see others fail. I think it is a relatively inexpensive and straightforward thing to do that could save some headache and beam trips in the future. 
• If the relay appears to be fine, or if the relays turn out to be totally unrelated to the actaul signal itself, continue tracing the signal back to its ultimate source. This panel on the B2 level is most likely where we'll find the issue.
  
  

[Update] 2024-07-26: Email from K. Genge:

An update from Safety System’s findings re: elevating system interlocks -

1. Very difficult to test the relays given the design/location of them and the cables connecting to them.
2. Kevin A. managed to trace the cryopanel elevating system related fault (#17020) over to the vacuum racks on SAB2 where he coincidently ran into Dimo, Vacuum Group Leader. Turns out that the sensor that gives this signal is connected to old cyclotron cryogenics/vacuum equipment that is no longer in use (and therefore providing the elevating system interlock system with the right info!). During the shutdown, Dimo and his team will move this sensor to the correct location so that it’s providing the correct info. That fault has been redirected.
3. The water ok signal goes to a “water system box”. We don’t know where this box is, but Kevin is willing to provide additional cable info to Joe. I recommend that the next step is for Joe and Kevin Ardron to connect and figure out where this cable goes.

 1,2, 5 Year Service and maintenance package completed on Carrier. All reducers repacked with Molub-Alloy 777-1 ES grease. Main drive train and all bearings relubricated. No issues. Operational.

A safety walkaround for July 2024 was completed for the B1 and B2 levels by A. Newsome, and for the main shop and back shop by R. Sykes.

Results can be found in the master spreadsheet

Major deficiencies, main shop:

• Unpainted lead bricks need to be picked up. This was requested of mechanical services twice already in July 2024. Follow up next inspection if not removed.
• Nearly half bulbs burnt out - enough to warrant replacement. AN requested of electrical services on 2024-07-17. Follow up next inspection if not replaced.

Major deficiencies, back shop:

• Phone not working. AN created help ticket 2024-7-17. Follow up next inspection.

Minor deficiencies, B1 level:

• 4-5 light bulbs will need to be replaced soon, not urgent yet.

Major deficiencies, B2 level:

• On the outboard support frame, the lifting eye fastener was stripped which prevented the nut from tightening all the way, posing a rigging hazard. The damaged lifting eye was replaced with a working one 2024-07-15.
• Light fixture above bootbox - mounting damaged and hanging precariously. Reported to electrical services 2024-07-15 and informed common users of area.
  Update 2024-07-17 - the issue has been resolved.
• Probes group has rad-waste in the middle of the floor. Reminded them to dispose of it soon.

 Highfields 7A,7B post shutdown service complete. Oil change completed, new Mobil 600 W Super Cylinder oil added. Sample collected for RPG of used oil. Both found to be mechanically sound and stored in upper mezzanine. 

 Yearly, 2 Year and 5 Year service completed. All bearings on support frames turn freely and were regreased with Molub Alloy 777-1 ES. Reducer repacked as well. Reinspect pre shutdown and lubricate orbital drive chain before wiping off excess.

 1,2 and 5 Year service completed. Vertical ball screws and chains inspected and regreased with Chemours Krytox EG 2000 suitable for vacuum and clean room situations. Mechanically sound, will do a once over before next shutdown period.

  12, 24 and 60 month service completed. Bearings for drive motor wheels regreased and observed to move freely without load. Reducers repacked with Molub Alloy 777-1 ES. Vertical ball screw scissor mechanism inspected and all operations observed to be in sound mechanical condition.

 24 Month, 60 month service completed. Bearings for drive motor wheels regreased and observed to move freely without load. Reducers repacked with Molub Alloy 777-1 ES. Vertical ball screw mechanism inspected and bearing change postponed due to lack of major usage, recommend change within next 5 years or service. All mechanically sound.

 60 month service completed. Bearings for drive motor wheels regreased, Reducer repacked with Molub Alloy 777-1 ES, all mechanically sound. Recommended to swap out front right drive wheel in 5 years due to minimal deteoration.

 Both outriggers were inspected and were found to be mechanically sound. No damage to components or hardware visible, will revisit with pre-inspection before shutdown period 2025.

 Motor/Gearbox unit removed from carrier and fully inspected and serviced. Oil changed on May 29th using Mobil 600 W Super Cylinder Oil. No signs of wear in internal gears, recommend oil service roughly every 5 years after shutdown.

 Center post bearing on service bridge inspected, turned freely by hand and regreased with Molub-Alloy 777-1 ES

 All 4 Under rail wheels serviced and cleaned. All Timken bearings repacked with grease. No defects in any bearing set. 4 spare Timken 02820 bearings in RH shop.

A third lid down operation took place over May 6-7, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47387 (after it was jogged up slightly prior to starting).

Upon startup on May 6 at 14:42, the system worked fine and lowered the lid. The lowering was paused at 15:28 (master synchro value of 17593).

Jack 6 exhibited issues similar to previous lid up (see e-log 166). It was likely due to an issue with a motor contactor. Electricians were contacted but could not work on the issue until the following day so the lowering was paused.

At 07:30 on May 7, electrician Randy Boehm determined that a cable connector on the motor contactor was loosened. After tightening the cable, jogging motor 6 was tested - OK.

Lid down commenced again at 08:10. Master synchro: 1915.

The lowering was stopped at approximately 1" gap as per standard procedure for Operations to perform checks. The lowering was stopped again multiple times to check resonator latching - OK.

End time: 08:37

Ending master synchro value: 59

Jogged master synchro value to 2, adjusted station 12 to match.

Notes: It was observed that at lid down Jack 12 did not have any gap. Jack 12 was jogged to match synchro values with the master synchro but still exhibited a 0.125” gap instead of the ideal 0.25” gap. It is recommended to keep an eye on Jack #12 during the next operation and check the motor contactors for loose fittings. In the before and after photos of the motor contactor it looks like the fitting on the bottom right corner of the motor contactor could have either come loose from the vibrations or the screw was sheared off. Further investigation is required.

See attached summary.