This is a list of all the trolleys and other equipment witch were checked and serviced prior to work in the Cyclotron.

- Video Trolley cleaned and checked (no adjustments needed)

-Lift Trolley cleaned and checked (no adjustments needed)

- Left Outrigger cleaned and checked

 All of the bearings in the festooning wheels  were changed.

  Drive chain needed adjustment. (modification to wheel mounting bracket so that chain could be adjusted)

-Right Outrigger cleaned and checked

 All of the bearings in the festooning wheels were changed.

 Drive chain needed adjustment. (modification to wheel mounting bracket so that the chain could be adjusted)

-Tool Trolley cleaned and checked

 One of the support wheels was checked and found to be out of alignment. (an adjustment was made to correct the problem) 

-Upper Resonator Trolley cleaned and checked (no adjustments needed)

-Service Bridge cleaned and checked

  Center post mount was rotated by hand. (no adjustment needed)

-Service Bridge Carrier cleaned and checked

 The two hydraulic cylinders witch lift the Service Bridge into the Cyclotron leaked last year.They were rebuilt last year and are not leaking.

 The two drive chains witch run the Carrier North and South were found to be too tight. (an adjustment was made to correct the problem)

All trolleys work electrically and control wise. One encoder was found non functioning on the upper resonator trolley for the vertical drive it was replaced, we've no move replacements of those current type of encoders so a wholesale replacement of all the trolley's encoders will have to be done post shutdown. Also the wiring for the outrigger is in need of replacement but will work for this shutdown it will also have to be replaced post shutdown as well. The upper resonator trolley has also been setup to remove X2C.

Routine maintenance as per Mike Gallop's written procedure took place in December with the only non routine service being the changing of shaft seals on Vacuum trolley drive reduction box.

Trolleys maintained - Lift trolley

                                     - Vacuum trolley / Tool trolley

                                     - Camera trolley

                                     - Upper resonator trolley

Old fittings and broken pneumatic tubing on the FTRF jig have been replaced with new push-connect fittings and polyurethane tubing. The jig was previously not in a functioning state due to damaged tubing. A pneumatic diagram was created in the process of the upgrade. The tubing will be cleaned up within the next few days and the jig tested.

On Feb. 24 while practising a training exercise, the lower resonator trolley unexpectedly experienced a radial drive failure. The trolley did not move in either radial direction (this occurred after approx. 5 minutes in the tank, near the centre post, with FTRF jig installed). The trolley was retrieved from the tank manually for investigation. Upon retrieval and testing on the mockup, the radial drive began to work again.

Investigation consisted of:

• Verifying controls signals reaching motor controller
• Verifying power to motor
• Verifying all connectors and cables intact
• Verifying drive components installed and free of damage (shafts, keys, set screws, etc.)
• Disassembly and inspection of gearbox
• Running the motor for approximately 3 hours with no load to verify no issues

The cause of the issue cannot be found, however it is suspected a thermal shutdown of the motor or motor controller may have occurred, and after a short delay, the circuit was reset back to a functioning state on its own.

In the future prior to using this trolley it is recommended to test run it, with load, in the mockup to verify radial drive functionality.

The two support fingers on the upper resonator trolley were bent due to excessive loading during cylinder toggling.

Fingers were bent back into position such that they are vertical when the cylinder is retracted.

The cylinders used to actuate these fingers were replaced (one was leaking) - replacement allowed for the opportunity for upgrade. The following changes were made:

• Changing reverse spring-return cylinders to double-acting cylinders (formerly American Cylinder 750RVS-2.00, replaced with SMC NCDMB075-0200C)
• Addition of in-line flow controls for both extension and retraction motions
• Tuning of flow control to achieve smooth operation and prevent further damage to the fingers during actuation

 The new finger configuration has been tested and is working.

 60 month service completed. Bearings for drive motor wheels regreased, Reducer repacked with Molub Alloy 777-1 ES, all mechanically sound. Recommended to swap out front right drive wheel in 5 years due to minimal deteoration.

 24 Month, 60 month service completed. Bearings for drive motor wheels regreased and observed to move freely without load. Reducers repacked with Molub Alloy 777-1 ES. Vertical ball screw mechanism inspected and bearing change postponed due to lack of major usage, recommend change within next 5 years or service. All mechanically sound.

  12, 24 and 60 month service completed. Bearings for drive motor wheels regreased and observed to move freely without load. Reducers repacked with Molub Alloy 777-1 ES. Vertical ball screw scissor mechanism inspected and all operations observed to be in sound mechanical condition.

 1,2 and 5 Year service completed. Vertical ball screws and chains inspected and regreased with Chemours Krytox EG 2000 suitable for vacuum and clean room situations. Mechanically sound, will do a once over before next shutdown period.

Today Remote Handling staff (Maico Dalla Valle, Travis Cave) used the Remote Handling 2C probe removal trolley to remove the probe assembly from inside the Cyclotron.  This is a regular yearly MRO task that is done every shutdown.

This task involves lining up the 2C trolley on the service bridge with the 2C probe assembly remotely with cameras, using a set of alignment markings on the trolley and probe assembly.  Once the trolley is lined up with the probe assembly, a set of solenoids are fired to disengage a corresponding set of four QDLs (quick disconnect latches) at different locations on the probe assembly.  A complete procedure is documented by Remote Handling staff.

The four QDLs are actuated by a set of four solenoid actuators on the 2C removal trolley, numbered 3, 4, 5, and 8.

Traditionally in the past, the QDL actuated by solenoid 8 has  jammed and does not automatically disengage when hit with the solenoid actuator.  A member of Probes or Remote Handling has had to suit up with a respirator, run into the Cyclotron tank, and tap this QDL by hand to free it from it's jammed state.  This has been done every year for quite some time, as long as Don Jackson can remember (around 30 years)!

However, today, Remote Handling staff (Travis and Maico) were able to disengage all four QDLs remotely using the solenoid actuators on the trolley!

A few things were done differently this year compared to previous years, which may have contributed to this success:

a.  The centre post bearing on the service bridge was disassembled, cleaned, and lubricated (see previous e-log from this shutdown)

b.  The "AXIAL A" alignment tab on the 2C removal trolley was intentionally misaligned with the felt pen markings on the 2C assembly (see attached picture that points out this "gap").

c.   The solenoids were fired in the following order: 8 (no success on disengagement), 5 (success), 4 (success), 3 (success), then 8 again (success).

After all four QDLs were disengaged, and the probe assembly was lowered away from its fixture points, it was observed that the assembly moved slightly laterally, indicating a slight stress or misalignment between all four QDL pins and their corresponding receptacles in the Cyclotron tank. The "gap" labelled in the attached picture, shrunk a small amount after this lateral movement.

Don Jackson remarked that in his entire tenure here, he had never witnessed successful remote disengagement of all four QDLs!  This is quite a positive result, as it lowers the expected dose and time required to remove the 2C probe assembly.  This approach to the task should be repeated next year and video documented (unfortunately we were not video recording the successful attempt this year).

Cheers,

Grant Minor, Remote Handling

All control to the video trolley was lost during remote photographing of components in the cyclotron. The control system and cameras on the trolley were still operational leading to the suspicion that a power cable was broken either on the trolley or the bridge festooning. Two trips into the tank were made to evaluate the problem and attempt to push the trolley out of the tank. Pushing the trolley only managed to move the trolley a few feet before jamming. A lift mechanism was used to raise the drive wheel without any success. This device was modified and used successfully to remove trolley completely from the tank.

Serviced the Camera trolley this morning as per Mike Gallops service outline.

Maintenance has been performed on the lift trolley, re sealed the main drive gear box it was quite dry inside and in need of new seals and lubrication. Inspected the vertical gear box and resealed it, re tensioned  the vertical drive chain after removing gear box and finding since last year the chain had become somewhat slack causing it to rub on the frame. Purchased a new LED light bar to replace the aging and failing flourescent light that was in place. greased all pillow block bearings and replaced old Zerk ( grease nipple) on some bearings to allow for easier lubrication.

Lower resonator trolley radial motor overcurrent was causing control system to drop out momentarily.  All controlled devices acted as expected and reverted to their fail-safe mode.  When system restarted 1-2 seconds later, all solenoids returned to their previous positions.

The cause was eventually traced to an unusual operation of the trolley.  In an attempt to limit trolley motion the drive switch on the control panel was repeatedly “tapped” in a rapid fashion.  This did not allow the motor to slow down enough to allow the starter winding to kick in which allowed large in-rush current.  The current was enough to load down the entire system including the power supply for the trolley controls, which caused the controls drop-out.

Later the same effect was observed to a lesser extent on the upper resonator trolley which used identical motors and controls.

Proposed solution is to either to limit large inrush current or preferably to replace the existing motor with a DC motor and controller.  A DC motor has the added benefit of now allowing true speed control.

No attempt is being made to modify the system until after the shutdown.  All trolley operators are being instructed to not use this method to attempt to control positioning of the trolley.  It is unlikely that it would help in positioning even if the system could handle the current overload.

We had just finished pulling out the ladders from the cyclotron with the lift trolley. We started to run the lift trolley into the cyclotron when it stopped moving. The radial drive gearbox was the problem. We pulled the gearbox off. Replaced  the key and the lift trolley is all repaired.

Tool Trolley:

The connector faceplate was replaced with a new version to allow for more space for service, and consolidate video lines into a single 9-pin CPC connector. A model and drawing were created and are part of the overall tool trolley assembly (to be added to PDM later): TRH1394

Air tubing that previously ran to the connector faceplate on the tool trolley was replaced as it was very old. New tubing is in 3 different colours for easy identification: blue, black, green.

Fittings were replaced with more modern push-connect style (1/4" tube).

Upper and lower resonator trolleys received mounting plates and brackets to accommodate DC motor controls for updated drive motors.

An issue was observed in which radial motion on the Video Trolley Mk 1 did not respond to commands. After troubleshooting, it was determined that this was due to the "OP LIMIT" interlock - the "OP LIMIT" pushbutton on the console was illuminated in red, and needed to be continuously held down in order to operate the radial in/out motion buttons.

It was determined that the root cause of the issue was a failed radial limit switch which was likely damaged during trolley transport. The limit switch was replaced, and functionality was restored.

All of the old brass air fittings on the Tool Trolley have now been swapped over to push-connect style fittings. As these are plastic fittings, it is recommended to inspect them after ~5 years for degradation.

The "old" video trolley featuring scissor mechanism and DSLR camera has officially been decommissioned. Going forward, Video Trolley Mk 2 (TRH1089D) will be used.

The two camera tilt mechanisms on the lift trolley have been upgraded. Upgrades include:

For the individual camera:

• Replaced cylinder
• Replaced fittings for cylinder
• Replaced air tubing from flow control to cylinder
• Visual inspection and fasteners tightened

For the overall platform containing the camera and lighting:

• Replaced cylinder
• Replaced fittings for cylinder
• Replaced valve (5/3 centre closed style)
• Replaced air regulator and set pressure (note: at the time of writing this e-log, it still needs to be fastened in place via its mounting bracket)
• Replaced air tubing
• Visual inspection and fasteners tightened

The mirror actuation mechanism was observed to be "sticky" during 2023 winter shutdown - it occasionally did not function properly and needed some manual assistance. 

The following items were addressed to resolve the issue: 

• Replacement of the Tyne Myte double acting valve which controls the cylinder (based on troubleshooting tests, it seemed this valve was not functional or intermittently functional)
• Addition of a silencer and in-line flow control on the exhaust of the aforementioned valve - adjusted to allow for smooth operation of the mirror when toggled
• Cleaning of the cylinder's shaft (it had a sticky residue build-up and was notably high friction) - this cylinder should likely be replaced in the near future 
• Replacement of air tubing along the supply line to the valve, and between the valve and cylinder
• Replacement of all fittings (push-connect style) on the valve and cylinders

The mirror actuation was tested after these changes and is behaving normally.

No further repairs required, although it is suggested to change the remainder of the tubing and fittings for the other valves which control different functions on this trolley, and to eventually upgrade the valves themselves to a low profile valve bank. 

Today (2024-02-16) when operating the lift trolley to install shields in the cyclotron, the vertical motion was observed to be intermittently working. When the trolley was out of the cyclotron tank and on the carrier, troubleshooting showed that it was likely an issue with the motor relays. The relay box was removed and the relays showed signs of wear due to ageing. The relays were replaced with new ones, and the box was reinstalled. The vertical motion appeared to work correctly after the replacement. The issue is considered resolved.

It is suggested that all motor relays be inspected or replaced as soon as possible, as others are of a similar age and could fail in the near future.

Edit (2024-02-26): last week, further issues were encountered. It was suspected that the upper limit switch was faulty. The limit switch was replaced and the issues did not persist. Note: it is possible that all observed issues could be related to the CAMAC power supply problem discussed in e-log 169

The cyclotron radiation survey was aborted due to the orbit not working recording the numbers correctly. One shield has been in installed.

Jan 29th: uploaded meeting minutes from January 28th regarding the orbital counter problem. - Grant Minor

200's, 400's, 500's, 600's, and a pallet of 700's have been installed

Started on the removal of shields.

More shields have been removed from the cyclotron only 16 remain. I was also noted that the lift trolley is making an odd noise and will be looked at after the shutdown.

The large shield door on the RH B2 level of the remote handling building has been shut.

The Remote Handling B2 level Shielding door has been closed.

all 200, 400, 500, 600, and some 700 series of shield are in.

The seven copper blockers were removed from the Cyclotron tank by the RH group using the service bridge, lift 
trolley, and attached checklist on 2017-01-30.

The monitor box (NE/SW-5/8/10) and series 200, 400, 500, 600, and 700 shields were installed in the Cyclotron 
tank by the RH group using the service bridge, lift trolley, and attached checklist on 2017-01-31 and 2017-02-
01.

The seven port hole covers were installed in the Cyclotron tank by the RH group using the service bridge, tool 
trolley, and vacuum unit on 2017-01-02.

The two ladders were installed in the Cyclotron tank by the RH group using the service bridge, lift trolley, 
and ladder attachment on 2017-01-02.

=================================================================================================================

The monitor box (NE/SW-5/8/10) and series 200, 400, 500, 600, and 700 shields were removed from the Cyclotron 
tank by the RH group using the service bridge, lift trolley, and attached checklist on 2017-03-07 and 2017-03-
08.

The seven port hole covers were removed and the Cyclotron tank seal was cleaned by the RH group using the 
service bridge, tool trolley, and vacuum unit on 2017-03-09.

The two ladders were removed from the Cyclotron tank by the RH group using the service bridge, lift trolley, 
and ladder attachment on 2017-03-09.

The seven copper blockers were installed in the Cyclotron tank by the RH group using the service bridge, lift 
trolley, and attached checklist on 2017-03-09.

All shadow shields which are remotely placed inside the cyclotron during shutdown were inspected by Adam Newsome and Chris Peterson on Jan. 13, 2021.

The inspection consisted of:

• Verifying feet were damage-free, welds intact (note: only outer feet were checked due to the large amount of effort that would be involved in shuffling shields around to check each foot)
• Inspection of lifting eyes
• Tightening all critical bolts - many of these bolts were slightly loosened from years of vibration (approx. 1/4 turn)
• General check for other miscellaneous damage

It is believed that all shields are fully acceptable for use, except ones which are clearly marked otherwise from previous years.

The inspection was prompted by a shield which had significant damage last year which could have resulted in a high-dose scenario if it were to get stuck inside the cyclotron. That shield has since been repaired. It is suggested that this inspection take place once every year or two years.

 The shield door is now open on the RH B2 level.

Repair of Orbit Limit Switch Plastic Belt

During the fall of 2011, Remote Handling discovered that a plastic toothed belt had stripped in the Cyclotron Bridge orbit limit switch mechanism.  This toothed belt is supposed to wrap around the stationary outer race/chassis of the center-post bearing on the Cyclotron service bridge.  The belt meshes with a smaller spur gear, which connects through a gear reduction system to a potentiometer and a limit readout.  The purpose of this limit switch is to prevent multiple continuous orbits of the bridge in one direction around the Cyclotron, which would twist and stress the trolley electrical wires.  Below is a picture showing the stripped belt condition as of Sept 23, 2011 (photo by Travis Cave).

During November 2011, Maico Dalla Valle and Travis went through the drawings of the service bridge assembly and ordered a replacement belt from the original manufacturer (still in business).  Maico removed the central bearing assembly, disassembled the bearing, replaced the belt, re-lubricated the bearing, and reassembled the unit into the service bridge.  The belt that was installed by Maico was installed with one less link / tooth than the previous belt, to obtain a tighter fit, so that fewer shims had to be used to fit it snugly against the outer race / chassis.  After assembly back into the service bridge, Maico commented that a much better mesh was achieved with the pinion. and the system was much easier to turn by hand.  This was likely due to re-lubrication of the centre post bearing and and improved  mesh / fit of the pinion with the belt.  A photo of the serviced assembly with the new belt is shown below:

Orbit Counter System - Operational Improvement (mid-Jan to Feb 7 2012)

During the 2011 winter shutdown, there had been some observations of inaccuracy in the orbit counter mechanism used for the automated radiation survey of the Cyclotron.  These observations were made by both Remote Handling and RPG staff.  The purpose of this orbit counter is to provide an angular position measurement of the service bridge relative to a known reference point in the Cyclotron vault, within a certain degree of accuracy and precision.  It appeared that the reading coming from the counter wheel that makes contact with the outer sector surfaces and follows the service bridge along it's orbit was skipping at various suspect locations around the perimeter of the Cyclotron. This skip, at it's worst, was deemed to be sizable enough to invalidate the automated survey of the tank.  Several e-mails and reports were generated by multiple parties in Remote Handling and RPG about the problem (attached).  The problem was not repeatable each time.  A photo of the counter wheel riding along the perimeter sector surfaces is shown below (taken with RH outrigger camera):

Various theories on the problem and potential solutions were discussed, including:

- Warped aluminum perimeter sectors protruding a small amount into the path of the orbit wheel, causing the wheel to skip as it passes over these bumps (see photos below)

- Insufficient traction between the counter wheel and the perimeter sector surfaces that the wheel rides on.

However, after the center post bearing had been serviced by Remote Handling, and during the routine service bridge work for the 2012 shutdown (copper blockers, shadow shields, photo survey), the following observations were made by RH staff:

- The orbit counter was operating smoothly and accurately and showing no sign of the suspect skips along it's path, throughout the entire range of orbit speeds and along all outer sectors of the Cyclotron.  In fact, according to Don Jackson, he could not remember the orbit counter ever working so well.

- The images from the outrigger cameras were noticeably more stable, showing less signs of structural vibration from the service bridge during orbit.

In my professional opinion, I believe this marked improvement in the orbit counter stability is due to the cleaning, lubrication and re-fitting of the center post bearing assembly described in the previous section above.  Don Jackson could not remember the center post bearing ever being serviced.  I have certainly not been able to find any written record of such service.  If that bearing was poorly lubricated and not rotating smoothly, then vibrations, seizing, and sticking would have result during rotation.  Any skip, jump, or vibration from such a sticky bearing would be amplified across the 30+ foot lever of the service bridge to the point at which the orbit counter wheel contacts the bridge sector.  These vibrations could likely have contributed to false orbit counter readings.  See photo below:

In any case, the orbit counter system has been observed to be operating normally and adequately by RH staff for several weeks of the 2012 shutdown.  I recommend that an automated tank survey is attempted as normal procedure, with both RH and RPG staff present to observe and document the behaviour of the orbit counter.

Thanks,

Grant Minor, Remote Handling

It was observed during the last time the Cyclotron service bridge was used the orbit drive chain had a lot of slack. The drive motor had to be shimmed to tighten the slack in the chain. 

The hydraulic cylinder on the West side of the service bridge (used to raise the bridge for insertion into the Cyclotron) failed this morning, leaking oil in the vault tunnel.  Most likely this was due to a failed seal or gasket.

Maico Dalla Valle has removed both the East and West cylinders for refurbishment and is pursuing repair kits from Hyseco.

The purple coloured cylinder on the East side has been slowly leaking oil for some time.  A gasket in the West cylinder (yellow) seems to have failed and a broken piece of the gasket was removed during the disassembly, shown in the photo "Hydraulic Cylinder Gasket 4 Feb 2013 008.jpg" attached.

Cylinder model number is ENERPAC RC-254.

http://www.hyseco.com/

On Monday February 4TH  the Service Bridge was being lifted into  the Cyclotron with the Carrier. It was noticed that one of the two Hydraulic Jacks was leaking. All work stopped and the Service Bridge and Carrier were moved to the Remote Handling B2 level for repair. The Service Bridge and Carrier were separated and the Hydraulic Jacks were inspected. It was noticed that both Hydraulic Jacks were leaking. Two overhaul kits were ordered. Two other Hydraulic Jacks were found and replaced. Work continued in the Cyclotron with the Bridge and Carrier. The two  Leaking Hydraulic Jacks were overhauled. Once most of the work in the Cyclotron was done and there was time that the Service Bridge and Carrier were out of the Cyclotron and not in use the two overhauled Hydraulic Jacks  would be replaced. They were tested without the Service Bridge and then with the Service Bridge. There were no leaks and all operation was back to normal.   

The festooning and bearings for the festooning cars for both outriggers has been replaced and it is in working order.

Noticed oil leaking out of reduction box, scan and swipe, disconnect e motor and all related cables from bridge, remove motor and reduction box assembly with over head crane, disassemble reduction box, grease broken down and separated likely cause of leak. Inspect bearings and internals for wear -all good

re assemble with new grease and gaskets (we added two grease nipples to the bearing housing to aid lubrication), test e motor on its own- good. Mount motor and box as unit adjust chain for tension and final test - good

outrigger 1 had a power failure, bridge was removed power restored and outrigger parked, outrigger will be looked at once the shield are done hopefully.

Outrigger 2 vertical drive repair is complete - ball screws were replaced to resolve a jamming issue (this work was done in the summer).

The outrigger has been recalibrated and all limit switches and feedback signals tested. All ranges of motion were tested multiple times as well.

It has been reinstalled on the bridge and is ready for service for 2020 shutdown.

The following items from Phase 1 of 2 of Bridge Lift Mechanism Refurbishment (O504) are complete:

• Replacement of pneumatic tubing
• Replacement of hydraulic cylinders for lifting, and procurement of spare

The newly refurbished system has been tested by Dan McDonald (June 16, 2021) and verified as functioning.

Phase 2 of the refurbishment will include replacement of the hydraulic pump and creation of pneumatic/hydraulic drawings. This is expected to be complete by end of 2022.

[Edit 2024-10-16]: Phase 2 has been completed. See e-log 191

The Hinged section of the festoon rail at Cyclotron has a piston to keep rail in the up position for shutdown work (Bridge and Trolley in tank work).

A plastic liner for the piston has fallen apart and needs to be replace.

A spare one has been temporarily put into  place until another one has been machined and reinstalled.

During the 2023 shutdown outrigger 1 failed. While performing the HE Probe removal we found that the outrigger could only travel vertically about one inch in each direction.

Upon further inspection we found that the linear bearings had been pulled off the ball screws. And 2/3 of the ball screws had been bent. Additionally one of the linear bearings broke and was missing some of the balls, the parts were later found inside the tank.

The event that caused the failure is unknown, but one guess is that while the outrigger was in the home position the shoulder joint was opened before the elbow joint.

The outrigger was repaired on March 21st, using parts from the old outrigger. After the shutdown the parts should be replaced again with new parts. Additionally it was observed that the linear bearings are quite old so they should be replaced on outrigger 2 also.  See photo for details on which parts where repairs or replaced.

Update 2023-09-01: Both outrigger 1 and 2 had the linear bearings replaced with brand new bearings. Note that the same brand and part number was used (Thomson super-12-OPN ball bushing), however the quality of the new ones seems to have decreased and likely will not last as long as the previous ones. It is recommended that we review if they need to be replaced again in 5-10 years. 

Outrigger #1 and #2 have had their controls system overhauled as part of the O503 project. The previous controls enclosure and the elements have been replaced with a new aluminum enclosure which contains DIN rail relays, a 5V power supply, and a +-12V Power supply. The new centrepost slave PLC unit handles the control logic of OR1, OR2, the Bridge and is connected to the main PLC in the control room through Ethernet. Both outriggers were tested individually.

Systems Tested:

Vertical Drive - Up, Down, 5V Position (Inches) Potentiometer

Radial Drive - In, Out, Encoder (Inches) 

Shoulder Drive - Home, Out, 5V Position Potentiometer (Deg) 

Elbow Drive - Home, Out, 5V Position Potentiometer (Deg) 

Wrist Drive - Pan Left, Pan Right, Tilt Up, Tilt Down, 5V Position Potentiometer (Deg)

Camera - Zoom In, Zoom Out, Focus Near, Focus Far

Testing Methodology:

We used the control panel which the Outrigger operator will be using during shutdown to confirm that the buttons translate to correct movements of each individual system. Direction control for the motors is done with relay switching of 24V to flip the polarity applied to the motors. +-12V is used for the zoom focus control of the HCZ-6320N camera. Correct movements was observed for both outriggers.

Position feedback was calibrated by adjusting the potentiometer's voltage to 2.5V at the mid point and spanned from 0V at low end to 5V at high end. Correct positions for both outriggers was observed on the HMI screen.

Next Steps

Testing the bridge orbit drive, orbit encoder, centrepost camera, centrepost light, both OR1 and OR2 simultaneously.

 Service bridge, Outrigger #1, and Outrigger #2 have had the controls system migrated to PLC controls. The 5069-AENTR control unit is located at the centrepost and will be mounted there after wiring is cleaned up. Preliminary testing consisted of operators simulating a shutdown task was performed today and was a success. Minor adjustments to the encoder logic will be made before the commissioning test. The follow tests were done: 

PLC box:

Check Relays toggling over correctly. 

OR1:

Radial In/Out & HMI Feedback Accurate

Vertical Up/Down & HMI Feedback Accurate

Shoulder Home/Out & HMI Feedback Accurate

Elbow Home/Out & HMI Feedback Accurate

Pan CW/CCW & HMI Feedback Accurate

Tilt Up/Down & HMI Feedback Accurate

Cameras Zoom/Focus

4 movements at once

Random camera restart noted (could be tied to grounding issue. will further investigate)

OR2:

Radial In/Out & HMI Feedback Accurate (Encoder Swap direction)

Vertical Up/Down & HMI Feedback Accurate

Shoulder Home/Out & HMI Feedback Accurate

Elbow Home/Out & HMI Feedback Accurate

Pan CW/CCW & HMI Feedback Accurate

Tilt Up/Down & HMI Feedback Accurate

Cameras Zoom/Focus

4 movements at once

Bridge:

Orbital Motor & Encoder HMI Feedback (Orbital Motor controls using Outrigger CP only active when ORs not selected) (set range to 0 to 360 same direction)

Centerpost Camera PTZF

Light

Speed Control

Vid Trolley:

Test operating video trolley with bridge controls and outriggers

Top/Bot LED

Implement radial speed control using pot

 Damaged "H" channel removed from the first festooning car, and spare channel was put in place.

Note: it seems like the aluminum channel is not strong enough for the forces its being put under, and is resulting in deformations (See picture "Festooning car H channel deformation"). We suspect this has happened in the past, notches were cut out where the deformation likely occurred and then the part was flipped and put back in place. There was two notches on the bottom face of the part we took out, and there was already one on the spare part. (see picture "Festooning car H channel bottom face"). A new channel needs to be designed out of steel for the first car to prevent this from reoccurring.

 All the bearings on both the upper and lower wheel were replace to see if it will improve the accuracy of the orbital position reading. See drawing SK-2489 for list of bearings used.

The encoder was also replaced to match the style used elsewhere.

 Center post bearing on service bridge inspected, turned freely by hand and regreased with Molub-Alloy 777-1 ES

 Both outriggers were inspected and were found to be mechanically sound. No damage to components or hardware visible, will revisit with pre-inspection before shutdown period 2025.

 Yearly, 2 Year and 5 Year service completed. All bearings on support frames turn freely and were regreased with Molub Alloy 777-1 ES. Reducer repacked as well. Reinspect pre shutdown and lubricate orbital drive chain before wiping off excess.

See previous e-log for phase 1: e-log 134

Phase 2 of the bridge lift mechanism refurbishment was completed. The scope of work for this phase includes replacement of the hydraulic pump (including dump valve) and associated controls hardware, plus some portion of the hydraulic tubing which interfaces with the pump itself. Work was completed by Hyseco under work permit C2024-10-02-2.

The pump was integrated such that existing operator control functionality was maintained, except for one difference: with the new system, the pump does not run all the time when the operator turns the "pump on" switch on. Rather, the pump is turned on during the action of raising/lowering. This was deemed acceptable by operators.

During installation, flow rates were adjusted by Hyseco techs to achieve the desired performance in terms of raising and lowering speed. It has been noted that raised appears to be slightly more smooth and even than previously. Lowering behaves similarly to previous. Furthermore, the new pump is much quieter.

Due to the relative simplicity of this system, a full commissioning report was not created, but the following commissioning tests were performed multiple times between 2024-10-14 and 2024-10-15:

• Turn pump enable on
• Raise bridge to maximum upper limit and confirm it stops automatically
• Lower bridge to rest on centre post support stand
• Dump to fully disengage cylinders

These tests were successfully completed and witnessed by A. Newsome, T. Cave, R. Sykes, J. Zhang, N. Jorgenson, M. Dalla Valle, and the Hyseco techs. Therefore, the system is considered commissioned and operational. Some minor wiring cleanup remains to be done and the system will be tested again multiple times prior to shutdown. A maintenance manual will be requested from the vendor.

The old pump will be kept for ~3 years as a spare.

Update 2024-10-21 [AN]:

For reasons unknown, the system is no longer functioning. There are two issues, which may be related or may not be:

1. Previously during the aforementioned testing the hydraulic pump motor was powered via 120 VAC wall outlet. Now it is being powered from our Carrier (ultimately from the blue power supply box in B2 level). There are large line losses associated with this. We measure approximately 35 VAC drop at the input to the pump when attempting to run the system. Because of this, the 24V supply located in the pump's control box used for enabling the motor and for the relays is dropping out, which turns the pump off immediately and causes chattering. When this 24VDC supply is separately powered from wall outlet, it does not drop out, but still the large line loss voltage issue remains for the pump motor. Based on an estimate of around 100 ft long wiring, the line losses measured suggest an inrush current of approximately 50A. This seems relatively high given the pump's FLA of 10A. This should be investigated further. Perhaps it is related to the second issue listed below, or perhaps the motor should be switched to 240 VAC, or perhaps the wiring should be changed from 14 AWG to 10 AWG to reduce losses.

2. When pressing the "down" button, the system moves up. There is no way to move down aside from dumping. Even when disconnecting the control signals entirely from the valves and manually pushing them in, both valves cause upward motion. Suspect an issue with hydraulic line plumbing or the valves themselves.

Hyseco engineer will be contacted and further troubleshooting will take place this week. This log will be updated accordingly.

Update 2024-10-29 [NJ]

The issue of the system moving up when pressing down is resolved. Hyseco engineer returned to repair. It was determined the the relief valves by nature have some leakage and because the flow control valves were so tight more fluid was travelling through the leakage than the control valve. The relief valve was moved to be before the flow control valve, which seemed to resolve the issue.  Additionally one of the pressure gauges was replaced.

A simulated removal of the LE2 Probe from the Cyclotron tank was performed by the RH and Probes group using the 
service bridge, upper resonator trolley, LE probe attachment, and the (soon to be) attached LE Probe removal 
procedure on 2017-02-23.

The LE1/LE2 Probe Frame was mounted on the Upper Resonator Trolley in the mockup with the intention of confirming functionality and making some updates to the probe installation/removal procedure document.

The following updates were made to the LE1/LE2 Probe Frame and the Upper Resonator Trolley:

• Additional holes drilled and grommets installed in the trolley's valve bank mounting plate to support all of the air cylinders on the Probe Frame
  
• All old brass fittings replaced with quick-connect style fittings
• All air tubing replaced with new polyurethane tubing
• Flow control valves installed
  

The air cylinder functionality was tested in the mockup and working as expected. In the future, a full mockup procedure will be performed in order to confirm camera views as well as finish updating the procedure document.

A new remote crane pendant was installed to replace the existing wired pendant on the B1 level of Remote Handling building (electrical lab). Two remote pendants exist (spare). The previously used pendant will be kept, at least temporarily, as a backup if needed.

The pendant was installed by M. Faragher, and tested by M. Faragher, A. Newsome, and T. Cave. It is fully functional.

TRIUMF Work Request 5684 describes the upgrade in more detail.

A new remote crane pendant was installed to replace the existing wired pendant on the B2 level of Remote Handling building (basement area - vault access). Two remote pendants exist (spare). The previously used pendant will be kept, at least temporarily, as a backup if needed.

The pendant was installed by M. Faragher, and tested by M. Faragher and T. Cave. It is fully functional.

TRIUMF Work Request 5684 describes the upgrade in more detail.

The following outlets were installed by Electrical Services Group in order to facilitate upgrading power supplies for Cyclotron Remote Handling, as per Work Request 5289:

• 2x 15 Amp outlets
• 2x 20 Amp outlets

The four breakers for these outlets are located in the panel adjacent to the main entry door to the electrical lab. The outlets are located in behind the control console.

The vault crane annual inspection was completed by Tom Kauss.

The wired pendant was replaced with a remote pendant by Tom Kauss.

A safety walkaround for July 2024 was completed for the B1 and B2 levels by A. Newsome, and for the main shop and back shop by R. Sykes.

Results can be found in the master spreadsheet

Major deficiencies, main shop:

• Unpainted lead bricks need to be picked up. This was requested of mechanical services twice already in July 2024. Follow up next inspection if not removed.
• Nearly half bulbs burnt out - enough to warrant replacement. AN requested of electrical services on 2024-07-17. Follow up next inspection if not replaced.

Major deficiencies, back shop:

• Phone not working. AN created help ticket 2024-7-17. Follow up next inspection.

Minor deficiencies, B1 level:

• 4-5 light bulbs will need to be replaced soon, not urgent yet.

Major deficiencies, B2 level:

• On the outboard support frame, the lifting eye fastener was stripped which prevented the nut from tightening all the way, posing a rigging hazard. The damaged lifting eye was replaced with a working one 2024-07-15.
• Light fixture above bootbox - mounting damaged and hanging precariously. Reported to electrical services 2024-07-15 and informed common users of area.
  Update 2024-07-17 - the issue has been resolved.
• Probes group has rad-waste in the middle of the floor. Reminded them to dispose of it soon.

A safety walkaround was completed for the RH Shop, Back Shop, B1, and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main Shop:
  
  • Lifting equipment has overdue inspection
  • Light bulbs burnt out
  • Unpainted lead has not been removed
• B1:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection
• B2:
  
  • Crane has overdue inspection
  • Lifting equipment has overdue inspection

Action has been taken on all deficiencies.

A safety walkaround was completed for the RH building B1 and B2 levels.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• B2:
  
  • Low on booties stock (more have been ordered)
  • Rad waste from Probes group remains in the middle of the floor (requested RPG to ask them to remove it)

Action has been taken on all deficiencies.

A safety walkaround was completed for the RH building shop and back shop.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Main shop, ground level:
  
  • Lighting still needs to be replaced (this was previously requested of electrical services and they are aware of it)
  • Unpainted lead bricks still need to be picked up (this was previously requested of mechanical services and they are aware of it)
  • Rad waste needs to be picked up (will do soon)

Shut down work for 2012.Replaced serviced Jacking Station #6. Tapered alignment pins were damaged. They were skimmed and checked and installed. Station #4 was removed and the spare Station was put in its place.Station # 4 Upper Bearing was inspected and serviced.Station #4 had a problem with the Jack on the right side.The coupling hub than joins the Jack to the High Field was installed backwards.(Not sure how long the hub has been like this maybe 15-20 years?). The coupling and hub had to be cut/split apart to get them off of the Jacks shaft. A new coupling and hub was machined and installed. Anew shaft seal was installed next to the new coupling hub. All the old grease was removed from the Jack. The Jack was inspected and then new grease was replaced. The left Jack was also serviced. It was taken apart and the old grease was removed. The Jack was inspected. The shaft seal on the other side of the coupling hub was replaced. New grease was replaced. The two High Fields had the oil drained. They were inspected and new oil was replaced. #4 Stations motor has four mounting bolt holes and two tapered pin holes for alignment. The one tapered pin hole in the front right side of the motor had a small hairline crack. The paint was removed in the area where the crack was so it could be welded.(The motor base is made of Cast Iron). The crack was  welded with 309L SS welding rod. After welding the tapered pin hole was checked with the tapered pin.(It was OK). The #4 motor was bolted to the service area work bench. An Electrician hooked the motor up for testing. The motor was greased  and then  it was  started.  It ran nice and smooth.           

Lid-up was first attempted yesterday (Jan 28th) by Dan Louie, Arthur Leung, and Maico Dalla Valle.  They experienced problems with Station 9 continually tripping the system and were not successful.

This morning again there were problems raising the lid above ~1 inch due to tripping at station 9.  It was observed that the mechanical counter at Station 9 was increasing about 5 to 6 times faster than the counters at the other stations.  There may be a mechanical problem with the gear reduction for this counter.  However, the mechanical counter is just a readout and does not control any aspect of the lid up.   Comparisons between stations 9 and 10 at the 1 inch trip point are shown below.  Station 10 is representative of the other stations within about 100 counts. 

The lid was lowered fully and the gap between the yoke and the upper structure primary beams was measured by Maico at one spot per station with 1/4" stock and feeler gauges (see attached reference photo from 2012).

Not much could be concluded from the measurements.  The control system is supposed to keep the jacks parallel within about 50 thou, however Stations 1 and 11 appeared to be out by 180 thou.  It is difficult to say without further study whether this larger discrepancy is due to overshoot at each station after the lower limit switches are reached.

Grant Minor, Dan, Maico, and Arthur met in the Main Control Room at around 1:30pm to assess.  Dan commented that Station 9 was continually lagging in the system and causing the ~1 inch trip.   The decision was made to drive the whole system to the ~1inch trip point and then jog Station 9 manually to bring it back into the "go" band of tolerance, and continue on this way until the "sticky" portion of travel was overcome.  In Grant's opinion, the transmission system at each station (motor, reducers, jacks, and upper bearing) sits dormant in one position for a full year each year and settles in a single position.  Some asymmetry in the static and dynamic friction from station to station is expected after such a long idle period.  Additionally, the lubrication at station 9 has not been serviced since 1996 (18 years).  Grant also felt that jogging the system in small increments to bring it back into the tolerance band should not damage the jacks, motors, or reducers provided they were carefully monitored for abnormal noises or overheating.

The sequence of events following the 1:30pm meeting is outlined below:

- The system tripped as expected around 1 inch.

- Station 9 was jogged upwards 20 thou, measured with the dial indicator.  Station 9 counter increased from 9255 to 9429.

- The entire system was run upwards by about another inch with several more trips, some from Station 9 and some from other stations.  Dan reset the system after each trip.

- At around 2 inches, Station 9 was jogged again.  Maico reset the dial indicator and attempted to jog manually but the dial did not move.  This might have been due to the lag at Station 9 creating some amount of backlash before the load was taken again.  He reset the dial again and this time measured a jog of 25 thou.  Dan asked us to back off by 20 thou, then he reset the system and we continued to raise.

- The system was raised to about 3" in small increments with several more trips, each time reset by Dan.  Station 9 was jogged 3 more times during this period by 10 thou each time, at approx. counter readings 16 362, 16 802, and 17 134.

- The system then ran properly for quite some time and tripped at around 6 inches.  We stopped to inspect the tank seal, which was OK.

- The system was reset by Dan and run again to about 10 inches, at which point station 1 tripped.  Station 9 counter read 60 775, Station 11 counter read 10 829 (again a factor of 6 discrepancy).

- Dan requested a jog of 10 thou at Station 1, and then we continued to run.  The system ran with only a couple of trips up to about 1 foot from the upper limit.

- At around 1 foot from the upper limit, Station 7 started to cause tripping.  It was jogged 10 thou and we continued all the way to the upper limit.

(Total travel is approximately 48 inches or 4 feet).

During the course of the work, it was observed that the motor at station 9 was getting hot early (around 2 to 3 inches of travel), but the motors at other stations (e.g. Station 11, Station 1) reached about the same temperature to touch after the system was running properly.

Report by Grant Minor, P.Eng.

Dan Louie's e-mail comments from 30 Jan 2013 are attached.

- Grant

Due to concerns raised regarding the state of the lubrication of the power screws at Station 9, the lubrication was visually inspected by me and Ron Kuramoto on February 4th.

Photos are attached.

The power screws for these stations were serviced and re-lubricated on the following dates:

Station 9 - 1996

Station 10 - 2001

Station 11 - 1997

Station 6 - 2011

There did not appear to be any major visual differences in the lubrication between any of these stations.  The grease at Station 9 appeared to coat the power screw uniformly through its range of travel, and there were no major signs of drying, hardening, or separation of the constituents relative to the other stations inspected.

There is a slight apparent difference in the opacity of the grease between Station 9 and Station 6 in the photos attached, but this may be due to the lighting and camera flash at the angle taken.

There did not appear to be any flecks of metal, contaminants, or other signs of abnormal wear in the grease.

The mechanical properties of the grease at Station 9 vs. virgin grease would have to be assessed by a tribologist in a laboratory setting from samples.

Samples of the grease were not collected at this time, but could possibly be collected in the future when Station 9 is removed for service.

Attached is a compiled record showing the years of service of the jacks, upper bearing, and gear reducers, prepared by Grant Minor on December 17 2012.

Attached is a scan of a handwritten record from 1994 showing yoke-to-upper-structure gap measurements from 1994, both L and R sides, with tank under vacuum and vented.

The measurements appear to be made with a 0.25" shim and 0.01" pieces of shim stock as they are only in increments of 10 thou.

The 1994 gap appears to vary from 0.25" to 0.33" (80 thou max variation) under vacuum, and from 0.29" to 0.33" vented (40 thou max variation).

Our 2013 measurements, made with a 0.25" piece of stock and a feeler gauge, varied from 0.265" at Station 1 to 0.445" at Station 11, (180 thou max variation, see Jan 28 e-log).

I have not yet found any other records of this measurement but I will keep searching.  The gaps may have varied by as much as 180 thou for many years.

I recommend checking the clearance at all stations and jogging individual stations such that the gaps are within a 40 thou band prior to the next lift.

The dimension between the yokes and the Cyclotron upper structure primary beams was set to .25" nominal at all jack stations today.

At each station:

1. The jacks were jogged manually upwards until the yokes hard-stopped on a piece of .25" nominal shim stock (see attached reference photo, measured accurate within .005").
2. The dial indicator was set and zeroed, and the station was backed off by .005" to 0.010" in order to remove the .25" reference stock (except at station 4 where the dial indicator mount was damaged).
3. After the adjustment, the mechanical counter was decoupled from the output shaft of the jack (without affecting the synchro) and set to "00100".

This work was performed by Maico Dalla Valle and Daniel Wright in co-ordination with Dan Louie.

The defective mechanical counters at stations 5 and 9 were replaced, with a brand new counter at one station and a used repaired counter at the other station.

The yokes at all jacking stations should now be .250" +/- .015" from the upper structure.

There was a brief access period to the Cyclotron vault during April 9 to 12, 2013 due to other Cyclotron issues.

This time was utilized to perform the following Elevating System tasks:

- Drilling of 2 (two) holes for concrete anchors (DBI Sala Product 2100085) for self-retracting lifelines (see TRIUMF purchase order TR197399)

- Installation of 2 (two) ceiling-hung self-retracting lifeline units, one above the climb-up access point to the bearing, and the other above the recently installed outboard safety platform (see TRH1217)

- Complete service of the upper bearing assembly at Station 9 (dis-assembly of bearing races, cleaning off old grease, inspection of races and guide tube, application of new grease, re-assembly of races)

- Tear down of safety handrails for some minor modifications (see upcoming Rev B to TRH1217 and sub-components)

 All work was completed on Wednesday April 10th by Dan Wright and Maico Dalla Valle.

Maico Dalla Valle and Grant Minor, supported by Dan Louie and Arthur Leung, entered the Cyclotron Vault on January 15th, 2014 to check the indexing of the jacking stations.  520MeV work permit 2014-01-15-4 was filled out for this work.

Yoke gaps at all stations were found to be within .005" to .010" of the .25" shim stock, and all mechanical counters were found to read zero.  This is consistent with the recalibration done on March 19th, 2013.

The system is properly indexed mechanically, and ready for lid-up.

Grant

Yesterday Grant Minor (me), Maico Dalla Valle, Dan Louie, and Arthur Leung worked with Operations to perform the Cyclotron Lid-Up task for the 2014 winter maintenance.

Work Permit 2014-01-22-2 was issued for this job.

In general, the lid up operation was rough at the beginning.  The system began to trip-out within about 500 counts on the mechanical counter, just when the jacks started taking the load, then continued to trip within the first ~ 3 inches of travel.  Multiple station jogs, complete lowerings, and complete raisings were required to get beyond the rough spot. 

I tried as best as possible to record the mechanical counters for each trip event, as well as confirm the master synchro count and culprit station with Arthur.  The data I collected is summarized below:

Start of Operation: All stations were indexed with yoke gaps within .005" and .010" of the .250" nominal shim stock (see previous e-logs).  Mechanical counters were all set to 100 counts (each count is approximately .001").

1. First lid up attempt tripped at less than 1" of travel.  Station 9 was found to be lagging adjacent stations.  It as jogged in local mode as shown:

2. Lid was raised again and stalled.  Station 11 was found to be lagging and was jogged:

3. Lid was raised again and stalled.  The decision was made to lower completely and raise again.  All station counters were recorded before lowering:

4. Lid stalled on the way down.  Station 9 was found to be lagging and was jogged:

5. Lid was lowered completely and then raised again.  It tripped at Station 9.  It was examined and found not to be lagging a great deal.

6. System was reset and lid was raised further.  It continued to trip.  Station 9 was examined and not found to be out.

7. Lid was lowered completely and then raised again.  It tripped again, and Station 9 was found to be out.  It was jogged with some intentional overshoot.

8. Lid was raised again and tripped.  The decision was made to lower lid completely.  Some counter values were recorded when the lid had reached the lower limit, for comparison to the starting reference.

9. Lid was raised again and traveled more smoothly until tripping at around 850 counts.  6 counter values were recorded.  Station 9 was found to be lagging and was jogged:

10. Lid was raised again and tripped.  Station 9 was found to be lagging and was jogged:

11. Lid was raised again and tripped.  Station 9 was found to lag very slightly (but within the normal variation band), and was jogged:

12. Lid was raised further and tripped again.  Four of the counters recorded and the group broke for lunch (approximately 1 hr).  The lid was left at this position over lunch.

13. After lunch, the lid was raised up to about 1090 counts with 4 trips.  Counters were read as below:

14.  Lid was raised but tripped almost immediately.  Decision was made to lower again.  Counters at bottom limit were read again:

15. Lid was raised with 3 trips.  Station 9 was found to be lagging and was jogged:

16. Lid was lowered completely, then raised.  It went up fairly smoothly and tripped out at about 1350 counts.  All counters were recorded:

17.  A few more attempts were made to raise the lid with a few more trips.  The system traveled in longer bursts and more smoothly between trips.  Counters were recorded and stations 9 and 11 were jogged:

18. System was reset and lid was raised again with a few more trips close together.  The decision was made to lower the lid completely and raise again.  Master synchro was recorded at 2285 before lowering.

19. Lid was lowered completely and raised fairly smoothly up to a master count of 2363 before tripping out on stations 1 & 9.

20. A few attempts were made to lower the lid slightly (not all the way) and then raise again.  This did not seem to reduce the frequency of trips.  The system tripped out around 2820 and all counters were recorded before taking a short break to allow the contactors to cool down:

21. After the break, there was a raise and lower cycle within about 3600 counts that I was not present for.  When I had returned the system was still rising in small bursts and tripping out on station 9 mostly.  Over a span of 5 trips, Station 9 was jogged 5 times by between 20 and 50 counts, with 20-count overshoot.

22. The master synchro passed 3600 counts and then the system tripped.  Stations 9 and 10 were recorded and Station 9 was jogged:

23.  The master passed 3800 counts, and then they system went up smoothly for the first time by about 1500 counts before stopping. Some counters were recorded:

24.  Lid was raised again and stopped around 6350.  The group paused to check the tank seal and all counters were recorded:

25. The lid was raised further with 2 trips at station 9.  Stations 9 and 10 were checked and station 9 was jogged:

26. Lid was raised again with 2 trips initially and then smooth travel up to a Master count of 9600.  After 3 more trips and resets, I began recording just the station trips and master counts at each trip.  No jogs were required during this period.

27. Jacks 1 & 5 tripped at 10,200 counts on the master.

28. Jacks 1 & 9 tripped at 11,500 counts on the master.

29. Jack 9 tripped at 13,400 counts on the master.  Several trips occurred immediately after this.  Counter readings were taken and station 1 was jogged:

30. Lid was raised again with trips at station 1, 5, 7 & 9.  Master synchro count was 15,600.

31. Lid was raised again with trip at stations 1 & 7.  Master synchro count was 18,800.

32. Lid was raised again with trips at stations 5, 7, and 9.  Master synchro count was 21,650.

33. Lid was raised again with trips at stations 1, 3, 5, 7, 9, 11.  Master synchro count was 23,360.

34. Lid was raised again with trips at stations 1, 5, 7, 9. Master synchro count was 23,700.

35. Lid was raised again with trip at station 9.  Master synchro count was 25,420.

36. Lid was raised again with trip at station 7. Master synchro count was 27,700.

37. Lid was raised again with trip at station 5.  Master count was 27,700.

38. Lid was raised again with trip at station 7. Master count was 30,300.

39. Lid was raised again with trip at stations 1 & 3. Master count was 33,650.

40. Lid was raised with trip at station 7.  Master count was 37,400.

41.  A moaning / groaning noise was heard from the structure around 38,000 counts.  This noise was also observed during the 2013 lid-up.

42. Lid was raised with trip at station 7. Master count was 39,722.

43. Lid was raised with trip at station 1.  Master count was 41,600.

44. Lid was raised with trip at station 9.  Master count was 42,857.

45.  All counters were read and selected stations were jogged:

46. Lid was raised but tripped at station1.  Station one count was found to be behind and was jogged from 42,804 to 42,880.

47. Lid was raised until limit switches were contacted.  All final counter readings were recorded:

***

Photos of the limit switches and the reference "stick" relative to the roof beam are attached.  The stick appears to be between 1/16" and 1/8" clear of the roof beams.

The Cyclotron Lid Down was successfully completed today on Work Permit C2014-03-03-6.

The system only tripped twice on the way down.  Both trips were at Station 9, at the following Master Syncro readings: 43194, 39198.

Oil was observed on the Jack #1 platform, but it was determined that it was coming from the crane when it's parked in the beam-on position, and not actually from the elevating system.

The final counter readings after the lower limit was reached were recorded:

The Master Syncro was recorded at 28.  All station yoke gaps were very loose on the .25" shim, which makes sense, as they are about 70 thou lower than the starting position before lid-up this year (100 counts at each station).

The system was NOT re-indexed.  The decision was made to wait until lid-up next year to re-index.  The lit should be raised until the Master is at about 100 counts, then the yoke gaps checked and adjusted as required.

Regards,

Grant

The cyclotron lid was raised on June 27th, 2014.  Start time was 1:05pm end time was 2:17pm (good!) Dan McDonald collected and recorded numbers from the mechanical counters for the lid up sequence, I am entering the data into the e-Log on his behalf.

Below are the mechanical counter values from the start and finish of the lid-up job:

There were 5 local jogs made to the jacks during the lid up process: 

Cyclotron lid down was successfully completed on July 2 2014. Lid went partially down before jack nine had to be jogged to catch up to system. After thirty min of continual system trips Maico Dalla Valle and I Jogged all stations to 42745. Still encountering continual system trips Gord Roy and Dan Louie removed and reset all relays in system, lid proceeded to lower with out problem. Final jack position recording done at 19:45.

5 x TRH1217 Upper Bearing Service Platform assemblies were installed between January 6th and January 13th at Stations 8, 10, 3, 4, and 5 (see attached photos).

The assembly went very smoothly.  Dan McDonald and Maico Dalla Valle worked together to install the platforms and wore harnesses and lanyards while working on top of the vault crane at each station (see attached photo).

Some notes were observed during assembly:

- There were not enough McMaster Carr 3949T1 3/4" Zinc Plated High Strength Snaps.  Assembly drawing TRH1217 Rev C only calls out 2 of these snaps, but this is incorrect, as 6 snaps are required per assembly.

- The ultra-coated 3/8-16 UNC bolts and nuts ( ref McMaster p/n 91257A634) were found to be very difficult to fasten.  In the future, zinc plated or mild-steel hardware should be used.

Both of the above items must be revised on TRH1217 towards Rev D.

Horizontal DBI Sala lifelines are now installed as well between Stations 8, 9, and 10, and between Stations 4 and 5, to permit access to the stations that do not have a direct ascent up the Cyclotron magnet structure.

All platforms and handrails are now installed on stations 8, 9, and 10 and stations 3, 4, and 5.  This permits full jack and gear reducer service at stations 9 and 4 (as adjacent stations are required for chainfall installation), as well as upper bearing service at all stations with platforms.

Still remaining is the work to drill the holes required for the ceiling mounted Self-Retracting Lifeline units that permit the first ascent and setup of the guardrails at each station.

The Cyclotron Elevating System Shut Down  work done this year was done on Station #4 and Station #9.

 Work done on Station #4

                       - note the readings on the Station counter and the gap clearance on the Yoke.

                       - Remove all the spares from Station #4 Elevating system (Spares are made up of two Jacks, two Highfield gear reducers,and one electric motor)

                       - Reinstall the complete serviced original Station into Station #4.

                       - Set the noted gap on the Yoke and the same readings on the Station counter.

Work done on Station #9

                        - Note the readings on the Station counter and the gap clearance on the Yoke.

                        - Remove the complete Station #9 Elevating system.

                        - Install the complete Spare Elevation system in Station #9.

                        - Set the noted gap on the Yoke and the same reading on the Station counter.

                        - The complete original Station #9 is taken to the Remote Handing Active Machine shop where each component will be inspected and serviced.(This Station will be ready to be reinstalled during next years Shut Down)

 Electric Motors have been reconnected on Station #9 and #4. We have not been able to test then until all the Interlocks have been satisfied. This may happen on Monday January 26th.

The Upper Bearing are needed to be serviced on Station #5 and #10. We wound like to do this work this Shut Down.This work can be done after the Cyclotron lid has come back down after service and maintenance has been done inside the Cyclotron.

All of the Fall arrest Platform Railing has been Collapsed for Cyclotron Lid up.

Lid-up started approximately 10am Monday 26 Jan 2015.

The first attempt to raise the lid was halted at a mechanical counter reading of approximately 1050 because a large discrepancy was observed in the mechanical counter reading at Station 11.  One jog at station 9 was employed leading up to the halt:

Station 9 Jog:

8              732

9              652/735               

10           739

Halt:

1              1048

2              1052

3              1025

4              1053

5              1026

6              1055

7              1038

8              1042

9              987

10           1051

11           2968

12           1052

All jacks were lowered back to the starting position, and the mechanical counter was replaced at station 11.  The new counter was set to the balance between adjacent stations 10 and 12.

All stations were jogged manually to index the yoke gap at ¼” with shims and the counters were recorded.  It was discovered that station 11 was out by about 30 counts.   Dan Louie wanted to check the discrepancy with the control system so we lowered manually back to start position “A”. 

After Dan’s OK we jogged manually up again to ¼” yoke gap and noticed the same ~30 count discrepancy with station 11.  It was determined that both the counter and the syncro at station 11 were out by about 30 counts with respect to the master.   Maico and Dan de-coupled the syncro and counter box from the output shaft of the jack, and adjusted both the syncro and the counter together until the count on the counter and the control system was adjusted about 30 counts back into the range of the other jacks.

New start “A”:                   ¼” shim - #1       ¼” shim #2

1              44                           115                         116

2              58                           145                         143        

3              44                           137                         133

4              55                           146                         144

5              46                           133                         133

6              47                           127                         124

7              47                           129                         125

8              49                           105                         110

9              62                           129                         126

10           54                           124                         124

11           55*                         162                         157 to 130**

12           55                           116                         114

* 55 set value after counter replacement

** both syncro and mech counter decoupled from jack worm gear shaft and adjusted to 130

The lid-up procedure then started again.  Two jogs of station 9 were required as recorded below, then the system travel from that point forward was smooth.

Station                  Value    Jog #1                   Value    Jog #2

8                              1439                                       3499

9                              1426       1446                       3476       3506

10                           1448                                       3506

The lid up sequence was stopped at 3pm half way through at around 19,000 counts because it was noticed that the new fall protection platform at station 8 would potentially interfere with an overhang in the ceiling for the vault lights. This interference occurs only at that particular station, and the station directly opposite 8.

The interference was measured by Dan McDonald and confirmed (see attached photos).

The platform was removed and lid-up started again at 4pm and was completed by about 4:23 pm.

Final Counts:

1              47393

2              47404

3              47391

4              47404

5              47403

6              47387

7              47401

8              47379

9              47397

10           47403

11           47397

12           47401

Lid down started around 2:30 pm today (March 6th, 2015)

Master Start: 47422

Mechanical Counter Readings:

The operation was mostly smooth and no station jogs were required.

At around 4800 counts, a stainless braid hose between stations 2 and 3 knocked into an electrical box (perhaps a new box?) and knocked the cover off, which made a loud clang (photo attached).  The lid down was stopped until the source of the clang was identified.

Fall equipment from the lockers in the RH shop have been checked for contamination and the lockers themselves have been swiped. All items checked in the target hall cave.

No contamination found.

Upon discussion with Dan it was mentioned the harness that I used was not used in the vault, and its origin is of question at this point.

The harness, carabineer, and shock packed used during the ISAC hot cell swipe have been checked seperately by Mike and he has found that the carabineer was active (feild of ~3 uSv/h on contact) not contaminated. All other components are clean and non active.

The assumption being that this carabineer most have come from the cyclotron vault.

Lid was raised today (2nd time during 2015 shutdown) for additional inflector work.

Dan and Maico report that the lid went up smoothly and no jogs were required.  The operation took about 30 minutes.

- Grant

Unusual wear markings were observed on the Station 9 jack (not sure if L or R jack?) during a visual inspection.

The grease looked to be in good condition and no sign of chips or contamination was found in the grease, so it is possible that the wear markings are older.

More investigation is required before using this worm gear assembly again.

Grant Minor, P.Eng.

The following people were consulted regarding the unusal wear noticed on jack station #9 right side

Neil Wong:  Neil inspected the jack parts.  Based on the wear and condition of the grease he thought that the wear wasn't recent and that the parts are ok to use.  Neil supported assembling the parts w/o grease and measuring clearances and backlash  worm gears & jack screw to nut 

Guy Stanford:  Guy came in and inspected the parts.  Based on the condition of the grease and the parts he also thought that the parts are ok to use and that the backlash/clearance tests should be done.  Guy could not recall when and which stations were damaged.

Ron Mullen:  Ron was contacted by phone.  He suggested putting in one of the new phosphor bronze jack nuts.  He also supported doing the tests

Bill Chapman:  Bill was contacted by phone.  Bill remembered discoloration of some of the stations.  He also agreed with doing the tests.  He thought the worn one should be ok to use.  He suggested cleaning the old one, reassembling and installing, then pulling out again after 1 year of use to inspect.

There are 3 spare phosphor bronze jack nuts on the shelf in the RH machine shop (2 left, 1 right).  No documentation has been found regarding these spares.  The right side spare was put onto the jack screw (no problems), but when it was put into the jack housing we realized that the nut requires machining to accommodate the bearing races.  The jack case and the jack lid also need to be modified to accommodate the rings at each end of the jacking nut  (this was confirmed when Station 9 Jack Left Side was disassembled - see next E-Log entry).  Machining these parts would likely need to be done off site due to the size.

It was noted that on the inner thread of the jacking nut the thin edge was slightly sheared off.  A piece of the thread was removed to prevent this piece from shearing into the jacking gearbox - see attached photos.

The original right side jack parts were cleaned and reassembled without grease. The following measurements were taken:
Nut in jack case - vertical play: 0.010"  (vertical when installed)

Nut in jack case - tilt: 0.010"
Worm gear backlash: 0.014"  (travel of key on the worm shaft before nut moves)

Nut on jack screw - backlash: 0.023"  (vertical travel when lifting nut)
Nut on jack screw - horizontal play on top of nut: 0.027"
Nut on jack screw - horizontal play on bottom of nut 0.022"

One of the new spare jack nuts was installed on the jack screw.  The following measurements were taken:
Nut on jack screw - backlash: 0.005"
Nut on jack screw - horizontal play on top of nut: 0.004"
Nut on jack screw - horizontal play on bottom of nut: 0.005"

Outer diameters of the old jack nut were measured:
Top bearing race: 7.473"
Bottom bearing race: 7.472"
Top jack case lid: 7.461"
Lower jack nut housing bore: 7.463"

Outer diameters of a new unused jack nut were measured:
Top bearing race: 7.479"
Bottom bearing race: 7.480"

 

It was decided that we should disassemble the left side and inspect before proceeding.  Work will continue on the right side when Maico returns from holidays on Oct 22.  The current plan is to clean off the wear, reassemble, install, run it for one year, and then remove and inspect in Shutdown 2017.

Photos of the Station 9 Jack Right Side parts are attached.

The left side jack was disassembled and cleaned.  It was discovered that the left side had been rebuilt with a new jack nut (similar to the three spare jack nuts in the RH machine shop).  This nut had been machined as is required to fit the bearing races on and fit the nut into the jack nut case.  The jacking case and lid also were machined to accommodate the different size rings for both ends of the nut  This rebuild may have been done after the jack failure that occurred in the late 70s / early 80s due to incorrect bearings and grease in the jacks.

There was less grease in the left side, and one of the worm bearings was on the dry side.  All other components still seemed lubricated, but not as much as the right side.  It seemed that some grease was stuck on the outsides of the jacking case and not mixing with the other grease.  The jacking case was approximately 70% full of grease.

There was a normal amount of wear visible on the surfaces of all jack parts according to Maico.  No unusual wear or colouration such as on the right side observed.  A leak in the shaft seal opposite the drive coupling was observed - this was replaced upon reassembly.

The jack was reassembled without grease .  The following measurements were taken:
Nut in jack case - vertical play:  0.008"  (vertical when installed)
Nut in jack case - tilt:  0.008"
Worm gear backlash:  0.017"

Nut on jack screw - backlash:  0.040"
Nut on jack screw - horizontal play on top of nut:  0.031"
Nut on jack screw - horizontal play on bottom of nut:  0.031"

The left side jack was fully reassembled with grease and is ready for installation.
 

 Station #9 Cleaning of  discoloration-

The right side Jack screw nut and worm screw (mating parts in the right side Jack) had some discoloration from may years ago it is thought.

These two parts were cleaned with Scotch Brite.

The Jack was reassembled with grease.Station #9 has been fully serviced and is ready for 2016 Shut Down to be put back in place.

The installation of the upper bearing fall protection equipment was completed last week by Dan and Maico.

- Full safety platform assemblies (ref. TRH1217) were installed on stations 1, 6, 7, 11, 12

- Short safety platform assemblies (ref. TRH1248) were installed on stations 2 and 8 to avoid interference with the lights

- Inboard anchors and SRL (self retracting lifeline) units were installed on 1, 2, 6, 7, 8, 11, 12, to permit the setup of the inboard railings at each station

- Outboard anchors were drilled and installed at all stations, using either D-ring swivel loops or Saf-Lok wedge anchors (both DBI Sala products), to allow safe setup of the outboard railings at each station

Also note:

- One railing at Station 1 had to be shortened to avoid interference with the guardrail of the aluminum stairs at that station, see attached photo.

- The beam clamp for the horizontal lifeline at Station 1 has an extension that protrudes into the working area.  This extension has been marked with pink flagging tape.

- It was noted that one more horizontal lifeline and two more beam clamps are needed to permit safe access to a station.  Grant will ask the equipment provider if they can make a shorter beam clamp to use at station 1, then transfer that longer standard clamp to the station that requires it (along with another new longer clamp).

- It was noted that we were short 2 carabiners.  Grant will order extra.

The Cyclotron Elevating System Shut Down  work done this year was done on Station #9 and Station #10.

 Work done on Station #9

                       - note the readings on the Station counter and the gap clearance on the Yoke.

                       - Remove all the spares from Station #9 Elevating system (Spares are made up of two Jacks, two Highfield gear reducers,and one electric motor)

                       - Reinstall the complete serviced original Station into Station #9.

                       - Set the noted gap on the Yoke and the same readings on the Station counter.

                      - Station #9  Jacks were found to have some discoloration on the worm and worm gear. Next shut down they will be taken out and inspected and reinstalled.

Work done on Station #10

                        - Note the readings on the Station counter and the gap clearance on the Yoke.

                        - Remove the complete Station #10 Elevating system.

                        - Install the complete Spare Elevation system(Station#14) into Station #10.

                        - Set the noted gap on the Yoke and the same reading on the Station counter.

                        - The complete original Station #10 is taken to the Remote Handing Active Machine shop where each component will be inspected and serviced.(This Station will be ready to be reinstalled during next years Shut Down)

 Electric Motors have been reconnected on Station #9 and #10. We have not been able to test them until all the Interlocks have been satisfied. This may happen on Monday February 1st. 

Annual jack maintenance : Jack 10b upon inspection it was found that the output shaft seal on 10b was in need of replacement. In past years RH personal had to remove a LoveJoy coupling which required cutting of the coupler due to galling and possible interference tolerances being out. We sourced sample split seals from Motion Canada that can be installed with out coupling removal we will install and monitor to determine the effectiveness and the possibility of converting all jack stations to this seal design . Attached are pictures of old seal style, new style seal, worm, wheel housing, current coupling on shaft, old coupling that has been cut off and various components. The jack components appear to be in good working order, no extra ordinary maintenance needs to be performed besides seal replacement. 

The Cyclotron Elevating System planned maintenance work this year was performed on Jack Station #3, #8, #9, and #10.

Start date: January 10, 2017
End date: January 23, 2017

Lockout procedure
 - Performed electrical lockout of Cyclotron Elevating System

Jack Station #10
 - Disconnected power from electric motor
 - Noted initial station counter reading and yoke gap
 - Removed limit switch
 - Removed complete spare set #14 from station (2x Duff-Norton screw jacks, 2x Highfield gear reducers, and 1x electric 
motor)
 - Installed complete original set #10 onto station (2x screw jacks, 2x Highfield gear reducers, and 1x electric motor)
 - Installed limit switch
 - Set yoke gap and station counter reading according to initial values
 - Reconnected power to electric motor


Jack Station #9
 - Noted initial station counter reading and yoke gap
 - Removed 2x screw jacks from original set #9 for early inspection
 - Lifted 2x screw jacks to RH active shop
 - Broke-out worm-screw from screw jack #9R
 - Inspected worm-screw and worm-gear surfaces: observed discoloured grease, similar to that found while servicing this 
screw 
jack in 
September 
2015
 - Decided not to reinstall screw jacks from original set #9 so that they can be serviced again after shutdown
 - Installed 2x screw jacks from spare set #14
 - Set yoke gap and station counter reading according to initial values

Jack Station #8
 - Disconnected power from electric motor
 - Noted initial station counter reading and yoke gap
 - Removed complete original set #8 from station (2x screw jacks, 2x Highfield gear reducers, and 1x electric motor)
 - Installed complete spare set #13 onto station (2x Duff-Norton screw jacks, 2x Highfield gear reducers, and 1x electric 
motor)
 - Set yoke gap and station counter reading according to initial values
 - Reconnected power to electric motor
 - Lifted complete original set #8 to RH active shop for inspection and servicing
 - Inspected and serviced upper bearing: ok

Jack Station #3
 - Unable to remove screw jacks from original set #3 for inspection and servicing since there no remaining spare screw 
jacks 
available
 - Decided to postpone removal of complete original set #3 until January 2018
 - Inspected and serviced upper bearing: ok

Pre lid-up preparations
 - Tested Jack Station #8 and #10 to verify correct motor phasing
 - Indexed all jack stations to 0.25"

On January 8, 2018, while troubleshooting the Cyclotron Elevating System during the lid-up operation, the
insulation on the three motor drive wires were repaired (w/ electrical tape) and the two motor brake wires were
replaced (w/ crimped butt splices) between the horizontal conduit below the Cyclotron and the terminal block on
the Station #9 motor. The wiring was severely damaged over an 18" long interval that crosses the mid-plane of
the Cyclotron; outside of the 18" damage zone, the wiring insulation was ductile and showed no signs of cracking
or embrittlement.

On February 22, 2018, after the in-tank work was completed and the lid was lowered, the wiring for the synchro
and bump switch on Station #9 was pulled for inspection, which revealed that the insulation on the five synchro
wires had almost completely disintegrated, and the insulation on the six bump switch wires appeared to had
melted together to form a single wire over the typical 18" interval that roughly corresponds to the gap between
the Cyclotron resonators.

On February 23, 2018, the three motor drive wires on Station #9 were pulled and replaced, using crimped butt
splices to the existing wiring in the horizontal conduit below the Cyclotron, as the electrical tape repair job
during the lid-up operation was only a temporary fix. Also, the motor drive, motor brake, synchro, and bump switch
wires on Station #3 were inspected and replaced due to insulation damage.

On February 27, 2018, the synchro and bump switch wires on Station #5 were inspected and replaced due to insulation
damage. The motor brake and drive wires were inspected and re-connected as the insulation showed no signs of damage.

Prior to commencing work on jacking stations we installed a wall mounted come along plate as per Jason kapalkas instructions   Completed spare jack swap on station 11 and 12, station 11 was swapped with spare set # 13 and station 12 now has # 14 jack set in its place. Once jack stations where swapped we proceeded to clean, inspect and re grease 11-12 upper bearings.

Station 11-12 where swapped for station 13-14 during 2018-2019 shutdown. Once lock out and motor disconnect was complete we commenced removal of spares and installation of original equipment. Station 13 was swapped in for station 3 and the upper bearing was serviced.  In addition to regular maintenance we built and installed a cover for the void in between station 10-11, this void in shielding measured approx 2'x4'. RH borrowed a demo Milwaukee 1' cordless impact from Motion industries. Our goal in trying this tool was to compare break out force between it and a pneumatic 1' impact in the hopes of removing all pneumatic tooling from our elevating system work. We tried it against a number of bolts and found it comparable with a 75% weight savings. In the past number of years we have made investments in tooling, equipment and safety gear in the drive to remove hazards and mitigate risks.

Under work permit C2022-10-04-1, the motor contactors for jack motors 1 through 12 were replaced. The contactors showed signs of corrosion.

 Annual Jack Inspection: The annual elevation system inspection has been completed (Jan 31,2023).

At station 1, Highfield gear Jack 13R needs a coupler seal replacement. It is ok for this shutdown, but should be replaced before next shutdown.

During the lift, station 7, Right Highfield was running a little rough - should also be checked before next shutdown. 

All motor and gearbox temperatures recorded during lift were consistent to temperatures recorded in the past. 

The full service to two jacks are not being completed this year. 

Motor contactors for the cyclotron elevating system jacks were inspected on 2025-01-23 by Joel Semilla under work permit C2024-01-19-1. Joel's assessment is as follows:

"All the twelve magnetic contactors have the same contact tips condition. Last year, I have [sic] to take out the contacts from the housing due to severe "pitting" of the contact's silver surface due to excessive "flash-over" which is common to magnetic contactor on inching/jogging duty. Due to the better condition of the contact tips this year, I can re-surface the contact tips on the magnetic housing itself."

Prior to recent inspections, a recurring inspection at an informed frequency had not been set. It is suggested based on the state of the contactors now, in comparison with the previous inspection, to schedule inspections of these contactors every 4 years. Next inspection due: Jan. 2028.

Edit: on 2024-01-24, Joel Semilla updates: "re-surfacing of stationary and movable contacts for the 12 cyclotron elevating motors have been completed".

Because of observed noise on the synchro at jack station 11 during recent shutdown lid up/down operations, the synchro at this station was replaced on Jan. 31, 2024. The procedure in Document-234859 was followed.

Prior to starting the replacement, the following counter values and synchro measurements on the station in question and adjacent stations were noted:
The master synchro count value was measured at: 0000
Station 10: Counter: 99392.5, synchro voltage: 5.89 VAC
Station 11: Counter: 0073, synchro voltage: 0.787 VAC (anomaly)
Station 12: Counter: 0068, synchro voltage: 5.98 VAC
 

During the replacement, the master synchro was jogged such that adjacent stations 10 and 12 read a synchro value of approximately 0 volts (as per the nulling procedure). The following observations were made:
The master synchro count value was measured at: 0072
Station 10: synchro voltage: 0.047 VAC
Station 11: synchro voltage prior to swap: 5.70 VAC (anomaly)
Station 12: synchro voltage: 0.282 VAC

After the replacement, before jogging the master synchro back to its starting value of 0000, the following measurements were made:
Station 10: synchro voltage: 0.038 VAC
Station 11: synchro voltage after swap: 0.12 VAC (now correct)
Station 12: synchro voltage: 0.278 VAC

When jogging the master synchro after the swap was completed, it was confirmed that stations 10, 11, and 12 all responded as expected, and their values were synchronized with respect to each other, indicating a successful replacement. The master synchro was jogged back to its original counter value of 0000. The system is now considered operational.

 The annual elevating system maintenance for the 2023-2024 shutdown following Document-220322, 

Work done after electrical lockout of elevating system and motors electrically disconnected: 

Work done on Station #1: 

• Note readings on station counter (71) and gap clearance on yoke (5/8" wrench thickness) 
• Remove all spares(#13)  from station 1 (two jacks, two highfield gear reducers and one electric motor) 
• Reinstall complete serviced original station into station 1
• Set yoke gap and counter to same as previously recorded

Work done on Station #7: 

• Note readings on station counter (71) and gap clearance on yoke (9/16" wrench thickness) 
• Remove originals from station (two jacks, two highfield gear reducers and one electric motor) 
• Install Spare elevating system (station #13) into station #7 
• Set yoke gap and counter to same as previously recorded 
• Inspected and serviced upper bearing: OK 
• the removed originals are taken to the Remote Handling Active Machine shop where each component will be inspected and serviced to be reinstalled next shutdown

Electric motors have been reconnected at stations #1 and #7.

It was observed when attempting lid up operation on 2024-02-06 that the brakes for each of the jacks were not operating correctly. It appeared that power was not being applied so the brakes were stuck in the engaged state. This ultimately resulted in issues with the lid up operation.

The issue was traced to relay A1 (the master brake power relay) not toggling as expected, despite the relevant control signals being present to try to enable it. This relay is located in the MCC-E 400V panel. Adjacent to the relay is fuse F4 which supplies the relay. This fuse was blown. The fuse was: English Electric CIS50. 50A rated, 600V, H.R.C. Form II. Replacements were ordered, however the part is obsolete due to its age. In the short term, a temporary workaround was implemented by installing a 30A fuse. This will be replaced when the spare fuses arrive.

The system was powered up again today (2024-02-07) and the brakes energized as expected. It appears that there is no issue. The root cause of why the fuse blew in the first place is unknown at this time. It was originally suspected that it was due to accidental mis-wiring of the brakes when they were reconnected after annual servicing, but if this was true, the fuse would have instantly blown again when relay A1 was energized. The best guess at this point is that the fuse failed due to age (it is likely around 60 years old).

This e-log will be updated if any further developments are made on this issue.

It was observed during lid up operation on 2024-02-08, approximately halfway through raising the lid, that the motor contactor for station 10 located within its small motor panel stopped functioning. The symptoms observed were: entire system stopped since station 10 lagged behind (approx. 150 counts on counter), station 10 could not be jogged locally in either direction, the control signal to activate the motor was being sent to the contactor when attempting to jog, but no motion occurred.

After locking out the panel, the contactor was disassembled and reassembled by an electrician. The system was powered back on and the motor functioned normally. The explanation as to why the issue occurred in the first place is unknown but it is likely attributed to years of infrequent usage causing it to "stick" and/or corrosion buildup.

In any case, it is recommended that all contactors be inspected during the year of 2024 and that at least station 10 be replaced, and spares for other stations procured.

The cyclotron lid was raised starting ~10:30am on 2024-02-06 and finishing at ~3:30pm on 2024-02-08, under work permit C2024-02-06-4.

Upon first attempt to raise the lid, issues were encountered. The system seemed to start as expected but quickly tripped, resulting in no appreciable travel. Upon investigation, it was determined that this was due to a blown fuse which feeds the brakes master relay (see e-log  https://elog.triumf.ca/TIS/RH-Cyclotron/165). The result of this was the the brakes were not properly disengaging, leading to additional friction in the system, and preventing regular motion.

After rectifying this issue, a second attempt was made to raise the lid. This time, issues were encountered with some of the stations being out-of-sync with respect to the others (a result of the troubleshooting which took place for the aforementioned issue). After manually jogging some stations, it was decided to lower the lid all the way, jog motors for individual stations so that they physically match each other in terms of travel distance at a known starting point, manually adjust counters on some stations whose counter values were significantly off so that they matched neighbours, and then jog individual stations such that each synchro measurement was close enough to begin lid up operation.

After this work took place, another attempt to raise the lid was made. This time, the lid traveled fairly smoothly, with a few occasional stops. It seemed that some motors were lagging behind others a bit - namely stations 5, 11, and 12. After raising the lid approximately 4-6", it was decided that the best course of action would be to fully lower it and start again, now that the system had some time to "warm up".

The lid was lowered to the bottom limit again, and then raised. The master synchro count was 0 to start. This time, lid travel was the most smooth, and stops near the start of travel were infrequent. The occasional stop occurred but the system could easily be started again. Jogging individual stations was rarely required. Most stops were due to noise observed on the synchro at station 5 (false trips).
When the lid was approximately halfway up, another unexpected issue occurred: station 10 completely stopped, which caused the entire system to stop. Station 10 could not be jogged in either direction in local mode. After some troubleshooting, it was determined that the motor contactor was causing issues - eventually it was rectified (see e-log https://elog.triumf.ca/TIS/RH-Cyclotron/166).

The lid continued to raise smoothly for the remainder of the travel with very few or no stoppages. It raised all the way, as confirmed by the gap measurement.

Final master synchro count when lid was fully up: 47401

 

See attached inspection checklist which contains information about each jack station during lid up operation, as well as pre-job inspection data.

Recommended actions:
- Replace synchro for station 5 next shutdown
- Inspect all 12 motor contactors prior to next lid down
- Order spare motor contactor
- Install replacement fuse (on order)

Due to an issue with resonator latching, there was a need to raise the cyclotron lid a second time this shutdown.

The plan at first was to raise the lid by small increments (approximately 1mm at a time up to approximately 25mm) while RF group performed some checks.

After the RF groups completed their checks, it was determined that the lid needed to be raised the full extent.

The starting master synchro count originally was 52, and it was 2480 at the time when the decision was made to fully raise the lid.

The lid raised entirely with zero stoppages or observed issues. No abnormalities observed with any jack station.

The lid was fully raised by 1:49pm with a master synchro count of 47383.

See attached summary.

The cyclotron lid was raised starting ~11:45am on 2024-05-02 and finishing at ~12:12pm under work permit C2024-05-02-5.

The raising was stopped a few times up until approximately 5" elevation, for RF group to perform checks, and to inspect the tank seal.

The tank seal was stuck to the lid near jack station 2.

The lid raised smoothly with no stoppages due to out-of-sync stations. No stations needed to be jogged.

Starting master synchro count was 66, but was jogged to 2 prior to starting. The final master synchro count when the lid was fully raised was 47387.

Note: the measuring stick which is affixed to the top of the lid and used to indicate full elevation was bent and should be "re-calibrated".

Summary attached.

A third lid down operation took place over May 6-7, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47387 (after it was jogged up slightly prior to starting).

Upon startup on May 6 at 14:42, the system worked fine and lowered the lid. The lowering was paused at 15:28 (master synchro value of 17593).

Jack 6 exhibited issues similar to previous lid up (see e-log 166). It was likely due to an issue with a motor contactor. Electricians were contacted but could not work on the issue until the following day so the lowering was paused.

At 07:30 on May 7, electrician Randy Boehm determined that a cable connector on the motor contactor was loosened. After tightening the cable, jogging motor 6 was tested - OK.

Lid down commenced again at 08:10. Master synchro: 1915.

The lowering was stopped at approximately 1" gap as per standard procedure for Operations to perform checks. The lowering was stopped again multiple times to check resonator latching - OK.

End time: 08:37

Ending master synchro value: 59

Jogged master synchro value to 2, adjusted station 12 to match.

Notes: It was observed that at lid down Jack 12 did not have any gap. Jack 12 was jogged to match synchro values with the master synchro but still exhibited a 0.125” gap instead of the ideal 0.25” gap. It is recommended to keep an eye on Jack #12 during the next operation and check the motor contactors for loose fittings. In the before and after photos of the motor contactor it looks like the fitting on the bottom right corner of the motor contactor could have either come loose from the vibrations or the screw was sheared off. Further investigation is required.

See attached summary.

 Highfields 7A,7B post shutdown service complete. Oil change completed, new Mobil 600 W Super Cylinder oil added. Sample collected for RPG of used oil. Both found to be mechanically sound and stored in upper mezzanine. 

 Jack 7R pulled apart and inspected for deficiencies, all parts deemed to be in working order, jack was rebuilt and lubricated with Molub-Alloy 777-1 ES. Jack is deemed fully operational for next swap. 

  Jack 7L pulled apart and inspected for deficiencies, alignment monitored with prussian blue and deemed acceptable wear. All parts in working order, jack was rebuilt and lubricated with Molub-Alloy 777-1 ES. Fully operational for next shutdown period swap.

2C has been installed located on all QDLs and all the numbers, continuity check done and passed. Bridge has been removed from the cyclotron tank.

Setup of the HE 1 probe is difficult has we have no mockup of it to properly setup cameras and equipment for it. This is problem for both extraction and installation of the probe. Also with the setup some of the broken micro switch from the HE 1 was found on the HE 1 probe jig after the jig was deconned and moved up the B2 level. Also for setup required camera adjustment to be made with the heavy contaminated probe being very close to the person setting up the cameras.

Now installation of the probe worked but it was very fiddly and touchy feely on the install, which something one doesn't want for remote installation. It required lots of movements and adjustment and a little luck and the probe went in, even with a air cylinder that did decide to work in the end.

On the extraction of HE 1 was mess and was damaged upon extraction, this was due to improper camera setup (no mock up of probe), and to the fact that removal is very fiddly and not east to the current design of the jig, and the way the probe is locked in place on the lid.

Note on contamination on extraction the probe jig was contaminated with about 20 000 cpm in spots it was removable and the jig cleaned. Contamination was also found on the jig for the installation of the jig it was about 800 cpm jig was cleaned again. Plastic was also laid on the platform of the upper ressonator trolley as to prevent the spread of contamination to the trolley, this appeared to work.  AS A NOTE NONE OF REMOTE HANDLING'S TROLLEYS AND CYCLOTRON EQUIPMENT WAS DESIGNED TO HANDLE CONTAMINATION. If the trolley ever get or contamination is ever found on them they will be impossible to clean.

This morning during routine service bridge work in the Cyclotron tank, it was observed by Travis Cave that the cable tray for the Remote Handling service bridge was catching on something on the top inside surface of the Cyclotron tank.  The cable tray was observed to visibly snag on this protrusion and then bounce around.

Isaac Earle and Grant Minor conducted an inspection from the edge of the Cyclotron tank with cameras and binoculars.  The contact point was identified and photos and video were recorded (see attached).

After a meeting with Yuri Bylinski it was identified that the item interfering with the cable tray was a beam scraper foil (part of the Resinator D Gap), close to the center post area.

A number of causes for the snag were suspected:

1. The upper tank lid might not have been at its upper limit (i.e. not raised completely by the elevation jacking system)
2. The center post bearing for the service bridge might not be seated properly
3. The cable tray mounting bracketry or air cylinder may have been displaced

After inspections of the service bridge before and after removal from the tank, it was determined that neither 2 nor 3 were the cause of the problem.

It was observed by the Remote Handling group that the upper limit switches located at stations 1 and 10 had not advanced high enough to mechanically "click", even though their wheels were contacting the reference surface.  There was about 0.5 to 0.75 inch of travel remaining in these switches (see attached photos) before clicking.

It was also observed that the wooden stick normally set flush with the Cyclotron roof beams at max height was about 0.5" below the roof beams (measured with a Starrett scale by Grant Minor).

After further assessment, it was also determined by Dan Louie and Yuri Bylinski that the limit switch lamp on the elevating control system was actually not lit, even though it appeared so.  The dull glow in the lamp was due to the neon style of the bulb, which can glow a small amount even at very low voltages and appear to be lit.  A test of raising the switches while watching the lamp revealed that the lamp switched on brighter after the limit clicked.

The upper tank lid was raised around 6pm by Dale N. with Yuri Bylinski and Grant Minor present.  The stick advanced upwards about 0.5" almost completely to the ceiling (see attached photos).

Until a systematic method of measurement of the vertical displacement of the system is developed (this may be the purpose of the dial indicators at each station, but requires further investigation), this problem can be avoided again by always checking both the limit switches and the distance from the calibrated "wooden stick" to the cyclotron roof beams.

This morning Don Jackson commented to Grant Minor that he believed the tank lid was still not raised high enough to safely operate the Remote Handling bridge and trolleys, despite it having been raised 0.5" yesterday to the point of engaging the limit switches (see e-log from yesterday).  Don said he believed this to be the case as a result of a visual inspection with binoculars from the outside of the tank, as well as from the views achievable with the remote cameras.

Don also believed he had caused further damage to the equipment inside the tank by orbiting with the bridge.

After a teleconference with Klaus Reiniger after lunch (Yuri Bylinski, Dan Louie, Don Jackson, Grant Minor present), the following was confirmed:

• There is a complete set of limit switches at station 1, and duplicate set at station 10 for redundancy.
• Both stations 1 and 10 have an operator upper limit switch, as well as an over-travel limit set slightly higher than the upper limit.
• The lid has traditionally been raised to the point of the upper limit switch tripping, then the lift procedure stops there.
• The wooden stick was traditionally set to just touch the roof beams with the lid at the upper limit.  This means if the stick is contacting the roof beams within 1/32nd inch or so and the upper limit trips, the lid has been raised fully.
• Klaus could only remember one incident of the lid being raised so high before tripping the upper limit that the wooden stick actually bowed under pressure from the roof beams.  In other words, this was not a normal occurrence.
• Klaus also reminded us that the limit switches have some mechanical hysteresis and need to be actuated through their full range of motion to reset them.

Don Jackson indicated that another 1/4" of clearance would be adequate to safely operate the RH equipment in his opinion.

At around 5pm Grant Minor and Ron Kuramoto reset the upper limit switches at both stations 1 and 10 by adjusting the block that both upper limit and over-travel are mounted to, see attached photos.

• A line was scribed in the primer paint to mark the original position of the limit switch mounting block.
• The bolt was loosened, and the block was raised fully to reset the switches, and then set 1/4" higher than the scribed line, measured with a Starrett scale accurate within about 1/64"
• The bolt was re-tightened.

Dan Louie and a shift operator reset the wooden stick 1/4" lower than the roof beams, this was double checked by Grant.

Dan Louie started up the system and confirmed the limit switches were properly reset.  He then raised the lid until the limit switches tripped again, confirmed by checking the illumination of the upper limit lamp (see attached photo).

After the raising of the lid, the wooden stick was confirmed to be contacting the roof beams by Grant Minor.

Dan Louie also confirmed that the master synchro counter had advanced almost exactly 250 counts (proportional to about 250 thousands of an inch).

Mechanical counter readings at each station (where accessible) were taken before and after this 1/4" rise by Dan Wright and Grant Minor:

Grant Minor had dosimeter 208 today, which was signed out at 0.61 mSv and signed in at 0.69 mSv.  Thus elevating system assessments and adjustments resulted in approximately 0.08 mSv dose.

Today Grant Minor, Dan Wright, Arthur Leung, and Dan Louie checked the height difference between the tank lid upper travel limit switch and the tank lid over travel limit switch at both stations 1 and 10.

In the attached photos, the upper travel limit is the lever on the right.  The over travel limit is the lever on the left.

The over travel limit is a safety switch designed to halt the system and lock it out if there is some problem with the upper travel limit.

The following tasks were performed:

1. The new reference positions for the brackets after the March 11th adjustment of 0.25" (see previous e-log) were scribed onto the mounting posts for Stations 1 and 10
2. The brackets at both Station 1 and Station 10 were unbolted and raised up completely to disengage the switches.  This had to be done due to mechanical hysteresis in the switches.  Dan Louie confirmed when the upper limit / over travel limit signal lamps had turned off at the control rack.
3. The Station 1 bracket was bolted up higher temporarily so that both switches were disengaged at that Station.
4. The Station 10 bracket was positioned back at the new scribed reference line from step 1.
5. A 1/8" nominal machinists parallel was slid under the lever arm of the over-travel limit switch at Station 10, and Dan Louie confirmed that the over travel lamp had turned on.
6. The Station 10 bracket was unbolted and raised to disengage both switches, and bolted in this higher position.
7. The Station 1 bracket was then positioned back at the reference line from step 1.
8. Step 5 was repeated for Station 1, confirming that the over travel lamp engaged after the 1/8" parallel was inserted under the switch.
9. Both brackets were positioned back at the reference positions from step 1 and bolted in place.

This procedure confirmed that the over travel limit switches at both stations 1 and 10 engage less than 1/8" higher than the upper limit switches.

In tank survey completed.  A few items of interest were found.  Ladders need to moved manually outboard to #5 resonators and the items noted in attached document need to be addressed.

Prior to the lid-down operation today, the Cyclotron was lowered by 0.75" and then raised again to confirm that the upper limit switches were working.  Dan Louie confirmed this from the control racks.

Maico and Grant walked around the perimeter of the cyclotron to inspect each station and ensure that there were no interferences, e.g. dial indicator rods, tread plates, etc.

The lid was lowered successfully by Gord Roy and Dan Louie.

Mechanical counter numbers recorded after lid down are shown below:

Measurements of the cyclotron magnet face gap were taken on March 15, 2013 at two locations around the tank with the cyclotron lid in the fully raised position (the lid had been raised until the upper limit switches caused the control system to stop travelling).  The purpose of the measurements is to establish an independent means of verifying correct lid height before beginning in-tank remote handling activities at the start of a shutdown period.

The height of the lid at the time of measurement was confirmed to be adequate for the bridge to perform a full orbit without any interference between the bridge or cable tray and devices on the lid.

Measurement locations were labelled using permanent marker.  A plumb-bob was used to ensure the measurement location on the upper and lower faces were aligned vertically.  The measurements were performed using a tape measure hung from the upper magnet face and read at the lower magnet face as shown.

Location #1: RH bridge entry/exit point (~50° position), Measured gap: 46-1/2" +/- 1/16"

Location #2: East side of jack station #1 (~215° position), Measured gap: 46-9/16" +/- 1/16"

Photographs of the upper limit switches on jack stations #1 and #10 were taken as shown below:

At this lid height, the reference stick on the top of the cyclotron lid was found to be touching the concrete ceiling, but was not bowed.  See photo below.

 The distance between the cyclotron tank lid and the RH bridge was then measured using a meter stick attached to the lift trolley.  The trolley was moved into the tank, and the stick raised until it contacted the tank lid.  This position was recorded, the stick was lowered, the trolley was removed from the tank, then the stick raised again to the recorded position.  A piece of aluminum angle was used to translate the measurement from the center of the trolley to the edge of the bridge as shown below:

Total Distance from RH Bridge Rail to Cyclotron Tank Lid: 15-7/16" + 6-5/16" = 21-3/4" +/- 1/8"

The copper blockers have been removed from the cyclotron and placed into the coffins.

 The Shielding and the Port Hole Covers have all been installed. No issues with the installation have caused any problems. This orbit, radial, and vertical read back continues to be an issue.

All shields, pothole covers, and ladders have been installed into the cyclotron.

2C has been successfully removed from the cyclotron and is now currently sitting in the B2 level of the RH building. A tank access was needed to feel the stuck QDL.

2C has been installed into the cyclotron.

 One of the beam spill monitor boxes was removed and placed into a lead coffin. The fields around the coffin prior to the monitor being placed in were 145 uSv/hr just above the box and 33.5 uSv/hr inside of it. A swipe was taken inside the box at this time and a count of zero was found. The tray for the monitor box was then placed into the coffin and the monitor removed from the cyclotrons. The field above the coffin with the monitor box was 1.0 Sv/hr. Safety hinted that it may be possible to put this in the decay storage tunnel. The copper block was then returned to the coffin. The fields above the the coffin were now 9 Sv/hr. The lid was returned to the coffin and returned to it storage place. The new monitor box was then installed into the Cyclotron. The box only hung up on the pins once and there was some slight difficulties unlatching. From previous experience once fully unlatched a slight tap in the latch direction seems to allow it to release easier.  Overall the installation was considered very smooth. 

During the electrical test the was no contact.  To help investigate further the Montitor box was removed from the Cyclotron. It was found to have zero counts. The issue is believed to have to do with the fit of the box but confirmation of the writing harness continuity needs to be completed. This will continue on Monday. 

Bevan Moss wrote:

One of the beam spill monitor boxes was removed and placed into a lead coffin. The fields around the coffin prior to the monitor being placed in were 145 uSv/hr just above the box and 33.5 uSv/hr inside of it. A swipe was taken inside the box at this time and a count of zero was found. The tray for the monitor box was then placed into the coffin and the monitor removed from the cyclotrons. The field above the coffin with the monitor box was 1.0 Sv/hr. Safety hinted that it may be possible to put this in the decay storage tunnel. The copper block was then returned to the coffin. The fields above the the coffin were now 9 Sv/hr. The lid was returned to the coffin and returned to it storage place. The new monitor box was then installed into the Cyclotron. The box only hung up on the pins once and there was some slight difficulties unlatching. From previous experience once fully unlatched a slight tap in the latch direction seems to allow it to release easier.  Overall the installation was considered very smooth.

all shields, porthole covers and ladder out and copper blockers in. Some conmination was found on the pads of the ladders (about a 1000 count or so) and was removed.

Day one of RH in tank work for the cyclotron (shields and blockers stuff like that) many pictures were taken.

All copper blocker removed from the cyclotron and first radiation survey done. Shields start tomorrow.

Over 50% of the shields are in some 700's left and the monitor box covers.

All shields, ladders and pothole covers are in. One rad survey left to do.

Final Radiation Survey done. All in tank work completed bridge removed from the tank.

2C has been removed from the cyclotron.

2C extraction probe has been re-installed in the cyclotron. After some problems on the first install attempt, do to a busted micro-switch that was causing the 24 Volt DC system to short out.

All shields and ladders have been removed from the tank, all the port holes have been uncovered, the tank seal has been vacuumed, and the copper blockers installed into the cyclotron.

The Upper bearings on Station #5 and Station #10 have been serviced. The Upper Bearing were disassembled. The old gear was removed and wiped clean. All parts were inspected.All parts were greased and then reassembled.All Fall Protection Platform railing was collapsed on both stations #5and #10.

Lid up recap for  March 26th 2015.

Start time 14:50        Master- 33                               Finish 15:25        Master -47413             Zero jogs to system.

• 1- 36                                                                47386                                                          
• 2-38                                                                 47378
• 3-33                                                                 47374                                                       
• 4-36                                                                 47382
• 5-36                                                                 47394
• 6-36                                                                 47385
• 7-37                                                                 47398
• 8-33                                                                 47373
• 9-38                                                                 47395
• 10-38                                                               47395
• 11-38                                                               47395
• 12-36                                                               47393

Copper Blockers have been removed from the cyclotron and one ladder has been installed.

Lid down Start at 11:17, lid went down without any issues or stop's. Lid finish at 11:45

Start count              Finish count

1- 47386                   47

2- 47378                   49

3- 47374                   45

4- 47382                   46

5- 47394                   47

6- 47385                   48

7- 47398                   49

8- 47375                   45

9- 47395                   50

10- 47395                 49

11- 47395                  51

12- 47393                 49

The shield door on the B2 level has been opened.

Lid up procedure Started at  12:00pm. Initial numbers. # 1-47,2-49,3-45,4-47,5-47,6-47,7-49,8-45,9-50,10-50,11-51,12-49. After a number of unsuccessful attempts at jogging the system and trouble shooting it was decided to suspend operations record numbers  and resume on Feb 2nd. Final numbers for Feb 1st #1-1289,2-1297,3-1288,4-1295,5-1283,6-1298,7-1287,8-1289,9-1290,10-1296,11-1287,12-1296.

Feb 2nd commence lid up operations at 11:30 am .When Maico Dalla Valle and my self arrived Dan Louie and Arther Leung had attempted to trouble shoot elevating issues and lowered the lid in hopes that raising would be smoother . The counter numbers on our arrival where as follows station   # 1-2856,2-2858,3-2848,4-2857,5-2855,6-2861,7-2856,8-2857,9-2852,10-2862,11-2860,12-2859. After Discussion between Grant Minor, Dan Louie and all involved it was decided to lower the lid re gap the yokes and reset counters to base numbers ensuring they correlated with master synchro. lowered lid numbers #1-55,2-59,3-55,4-57,5-57,6-54,7-50,8-54,9-59,10-59,11-58,12-59.

At this point we re gapped to .250" with counters at #1-119,2-177,3-123,4-144,5-137,6-120,7-123,8-111,9-127,10-148,11-128,12-117. After comparison with Dan L and the Master Synchro, stations #2,3,4,5,10 where jogged to better match the average on the counters and synchro these stations where put at 120 on the station counters. Once this was completed we started final lid up operations stopping at 5" to check tank seal, once it was confirmed the seal was not sticking we continued uninterrupted  to final lid up position. #1-16947,2-47366,3-47351,4-47365,5-47362,6-47365,7-17495,8-47358,9-47360,10-47366,11-47362,12-47363. Counters #1 #7 appear to be failing, they will be changed out and indexed to match as needed.

All shadow shields have been installed. Ladders, the pothole covers and the second rad survey will be done tomorrow.

All cyclotron in tank work is completed.

The HE1 Probe was removed from the cyclotron this morning following the procedure in Document-126165 (currently in draft status, to be released after the probe has been reinstalled).  The procedure took approximately 2 hours from when the trolley was put into the tank until when it was removed from the tank with the probe.  The procedure took from 8am - noon including putting the bridge into the cyclotron, and removing the probe from the trolley into the Probes Group work area on the B2 level.

No major surprises were encountered during the procedure.  Camera view screen captures were taken at each step to be added to the procedure.

Possible improvements to the procedure and equipment include:
1. A flow restrictor should be added to the inner-member stop tab air line to slow movement
2. The inboard guide views on both sides (camera views #7 and #8) would be more useful if they were more zoomed in
3. It would be useful to be able to turn off the trolley fluorescent light at the inboard end once the bridge is in the tank (it caused glare in some views)
4. The opposite side latch camera view (camera view #5) seems to not be secure enough and was bumped slightly out of position during trolley move, try to secure better
5. In Section 1 of the report it would be better for a final check of each view with the probe mockup to be done after the trolley has been lowered to the bridge
6. The probe didnt lower freely immediately after unhooking the outboard latch.  After slight lowering of the frame and a few up and down movements it became free.  It is suspected this is because of too much upward force on the probe from the frame due to the inboard tip being tilted too high.  Steps have been added to the procedure to prevent this in the future.  Another possible improvement to the equipment would be load cells on the trolley or between the trolley and frame

 The HE1 Probe was reinstalled into the cyclotron this morning following Section 3.2 in Document-126165 (to be reviewed and released after camera view screen captures added).  The procedure took approximately 1 hour from when the trolley was moved into the cyclotron tank until when it was removed.  Camera view screenshots were captured throughout the procedure and the probe head was inspected with the left outrigger camera after installation to check for abnormalities - none found.

In Step 3.2.7 of the procedure view #6 was checked to ensure the inner-member was fully retracted before actuating the stop tab - consensus was reached among Remote Handling and Probes Group personnel that the inner-member was fully retracted, but in fact what appeared to be the pulleys at the end of the inner-member in the camera view was actually part of the inner member frame.  The stop tab hit the inner-member when it was actuated pushing it backwards.  No damage was caused, but a note will be added to the procedure to avoid this in the future.  Other than this the installation went smoothly and there were no deviations from the procedure.

Ex 2C has been removed from the cyclotron. A trip into the tank was needed to free the probe, hung up on the usual spot.

EX 2C has been installed back into the cyclotron.

The new EX4N housing assembly drawing TEC0071 (unfinished) was installed and connected to the EX4 tank side port flange over a 3 day period from Feb29 – Mar 2, 2016. This was a hands on installation spread amongst 5 people with the total accumulated dose was 2.97 mSv.  See W/P C2016-02-29-5
 

Attached are;

- Rad field survey map.
- Updated installation procedure (rev4) with additional notes/comments during the installation  Revisions in Red.  Competed steps in Green. 
- Photo of the EX4N housing assembly.

Recent photos, are on Dimo's S drive :S\dyosifov\Extraction 4\Photos 2016

Lid down Started at 12:15, station and count numbers. Lid went down with out incident as station 1 and 7 show the counters will need replacing. We are going to change counters before the end of this shutdown to ensure proper orientation in case of emergency lid up scenario. 

Start          12:15                   Finish    13:15                 Master #47383   Master final 38

1-16047                    76303

2-47366                           44

3-47351                          41  

4-47365                          43

5-47362                          42

6-47365                          43

7-17495                  49568

8-47358                         40

9-47360                          44

10-47366                       44

11-47362                        45

12-47393                      43

The shield door on the B2 level of the RH building has been closed.

200, 400, 500, 600, and a pallet of 700 series of shield have been installed in the cyclotron.

 The HE2 probe was removed from the cyclotron today following the procedure in Document #126165.  An orbital bridge position of 247.35° provided better alignment than 247.2 from the procedure.  The probe signal head was intentionally left in the "up" position by Diagnostics Group which did not cause any issues with removal.  Trolley elevation when the frame contacted the base of the probe was 53.8".  No major issues were encountered. The probe will be returned to the tank of Diagnostics Group have converted the radial drive to a bellows feedthru.

 The HE2 probe was reinstalled in the cyclotron this morning following the procedure in Document #126165.  An orbital bridge position of 247.35° was used (same as for the removal on Feb 9th).  The installation went smoothly.

Performed continuity test at Cyclotron center post with RH and Operations group members using the service bridge, 
tool trolley, and probe attachment on 2017-03-03.

Lower tank:
Quad #1: All six plates PASSED.
Quad #2: All eight plates PASSED.
Quad #3: All six plates PASSED.
Quad #4: All eight plates PASSED.

Upper tank:
Quad #1: Had some repeat-ability challenges with all six screw-heads/wires. Retest required.
Quad #2: All eight plates PASSED.
Quad #3: All six screw-heads PASSED.
Quad #4: All eight plates FAILED. Retest required.

Performed extra continuity test at Cyclotron center post with RH and Operations on 2017-03-06: achieved a 
successful test on all upper quad #1 and #4 screws/plates. One unconfirmed explanation for the successful retest 
is that the multimeter cable was firmly grounded against the computer monitor rack in the RH service bridge 
control room (see attached picture), which may not have been the case during the initial test.

The start of shutdown photo and video survey was performed in the Cyclotron tank by the RH group using the 
service bridge and video trolley on 2017-01-27 (see attached pictures).
 
The end of shutdown photo and video survey was performed in the Cyclotron tank by the RH, Probes, and 
Operations group using the service bridge and video trolley on 2017-03-08 and observed the following:
- Observed a small piece of melted plastic and wire near the 180° lower tank D-gap; this item has been present 
since the 2016 shutdown based on photographic evidence (see attached picture).
- Observed a large strip of paper on Q2L2, close to where the ladders were placed beside the CP (see attached 
picture). 
- Observed a bent periscope alignment target near Q4U7 (see attached picture).

A proposed guideline for conducting future video surveys is attached (via djacks@triumf.ca and genge@triumf.ca).

The unshielded radiation survey was performed in the Cyclotron tank by the RH group using the service bridge, 
tool trolley, and radiation survey attachment on 2017-01-30 (see attached plot).

The shielded radiation survey was performed in the Cyclotron tank by the RH group using the service bridge, 
tool trolley, and radiation survey attachment on 2017-02-02 (see attached plot).

A obsolete version of the radiation survey procedure (from Don Jackson) that needs revising and the MATLAB code 
for generating a radiation survey map (from Maxim Kinakin) is attached.

A simulated removal of a lower resonator from the Cyclotron tank was attempted by the RH group using the 
service bridge, lower resonator trolley, lower resonator attachment, and the (soon to be) attached lower 
resonator removal procedure on 2017-02-27. Prior to aligning the attachment over a lower resonator, a small 
piece of paper debris was observed falling off the trolley. A successful recovery operation was performed using 
the vacuum trolley; however, the lower resonator exercise had to be abandoned until a future shutdown due to 
scheduling constraints.

Lid up (Attempt #1)

Pre-indexed 

Start @ 2017-01-25 13:05
End   @ 2017-01-25

Master   56

Jack #01 00117 to ABORTED OPERATION
Jack #02 00123 to ABORTED OPERATION
Jack #03 00119 to ABORTED OPERATION
Jack #04 00114 to ABORTED OPERATION
Jack #05 00118 to ABORTED OPERATION
Jack #06 00121 to ABORTED OPERATION
Jack #07 00118 to ABORTED OPERATION
Jack #08 00118 to ABORTED OPERATION
Jack #09 00115 to ABORTED OPERATION
Jack #10 00120 to ABORTED OPERATION
Jack #11 00117 to ABORTED OPERATION
Jack #12 00110 to ABORTED OPERATION


Lid up (Attempt #3)

Start @ 2017-01-26 14:51
End   @ 2017-01-26 15:15

Master   -18 to 47365

Jack #01 00049 to 47335
Jack #02 00049 to 47332
Jack #03 00045 to 47337/41
Jack #04 00047 to 47334
Jack #05 00049 to 47341
Jack #06 00049 to 47343
Jack #07 00050 to 47348
Jack #08 00046 to 47324
Jack #09 00049 to 47348
Jack #10 00050 to 47346
Jack #11 00052 to 47334/6
Jack #12 00051 to 47341

Metre stick: 2 mm from vault ceiling
The Highfield input shaft on 10L was louder than the other stations while raising the lid.


Lid down

Start @ 2017-03-09 14:44
End   @ 2017-03-09 15:25
Master   47365 to 00040
Jack #01 47335 to 00045
Jack #02 47332 to 00043
Jack #03 47337 to 00042
Jack #04 47334 to 00044
Jack #05 47341 to 00044
Jack #06 47343 to 00044
Jack #07 47348 to 00045
Jack #08 47324 to 00043
Jack #09 47348 to 00044
Jack #10 47346 to 00046
Jack #11 47334 to 00048
Jack #12 47341 to 00045

Remote Handing is ready for January, 8 2018 Cyclotron lid up. Elevating system service has been completed.

A Cyclotron Lid raise was attempted on Monday, January 8 at 12:00, however this attempt was aborted after lifting the lid several millimeters due the motor
brake on Station #9 that would not release properly. After troubleshooting the elevating system for several hours, motor brake wires were disconnected from
the motor terminal block at the station and pulled through the vertical conduit down the B2 level below the Cyclotron for further inspection. The inspection
showed severe damage to the wire insulation and conductor over an 18" interval, which corresponded to the position directly across the mid-plane of the
cyclotron. New wires were spliced to the existing wires at the bottom of the vertical conduit section, which were run through the conduit and reconnected to
the motor terminal block.

The motor drive wires on Station #9 were also removed for inspection, which revealed major damage to the wire insulation over the same 18" interval;
however, the conductor remained intact and was covered in electrical tape and re-run through the conduit vertical and reconnected to the terminal block. The
elevating system was then tested and the motor brake on Station #9 released as expected.

The Cyclotron Lid was successfully raised on Monday, January 8 at 20:00. The initial station readings are shown below:

00075 - Master
00045 - Station #1
00044 - Station #2
00042 - Station #3
00045 - Station #4
00044 - Station #5 
00044 - Station #6
00045 - Station #7
00044 - Station #8
00044 - Station #9
00047 - Station #10
00048 - Station #11
00045 - Station #12

The final lid-up station readings are shown below:

47422 - Master
- Station #1
- Station #2
- Station #3
- Station #4
- Station #5 
- Station #6
- Station #7
- Station #8
- Station #9
- Station #10
- Station #11
- Station #12


The Cyclotron Lid was lowered on Tuesday, February 20, between 11:30 and 12:00. The final station readings are shown below:

00081 - Master
00086 - Station #1
00089 - Station #2
00054 - Station #3
00016 - Station #4
00087 - Station #5
00087 - Station #6
00085 - Station #7
00083 - Station #8
00086 - Station #9
00087 - Station #10
00089 - Station #11
00088 - Station #12

While lowering the lid, the following observations were made:

- The left High Field gear reducer on Station #1 made noise intermittently during the middle of the lid lowering operation
- The right High Field gear reducer on Station #2 made noise intermittently during the end of the lid lowering operation
- The right High Field gear reducer on Station #11 made noise intermittently during the beginning of the lid lowering operation

2018-01-09
07:00-10:00 - Pre-job meeting, informed main control room of operation startup, lowered inflector (Operations 
Group)
10:00-12:00 - Connected cable chain rail, installed service bridge into tank, ran video trolley onto service 
bridge, performed initial circumnavigation of service bridge in tank, made contact with newly installed cable 
bracket (Probes Group) near jack station #9, assessed damage to cable and bracket (service bridge OK) 
12:00-13:00 - Performed photo survey of HE1 probe / 2A extraction foil / 1A extraction foil w/ Probes Group 
representative
13:00-14:30 - Performed photo survey of center post / lower and upper D-gaps / lower outer perimeter
14:30-15:00 - Powered down service bridge control room, informed main control room of job progress

2018-01-10
07:00-07:30 - Pre-job meeting, powered up SBCR, informed MCR, troubleshot video trolley
07:30-09:00 - Performed video survey of two cryovac modules / upper and lower RF booster / 2A exit port / 2x 
lower and upper scraper foils
09:00-10:00 - Removed video trolley and installed lift trolley onto carrier
10:00-12:00 - Removed 7x copper blockers from tank and placed into 4x coffins
12:00-12:45 - Removed lift hook extension and installed radiation meter onto lift trolley
12:45-13:45 - Calibrated radiation meter
13:45-15:00 - Post-job meeting and preparation for in-tank radiation survey, informed MCR

2018-01-11
07:00-07:30 - Pre-job meeting, powered up SBCR, informed MCR
07:30-08:00 - Calibrated radiation meter w/ Safety Group representative
08:00-10:30 - Performed in-tank radiation survey w/o lead shields (27 circumnavigation passes inside tank 
from 350" to 80" radial position)
10:30-11:00 - Removed radiation meter from lift trolley
11:00-14:30 - Installed 200 & 400 series lead shields
14:30-15:00 - Powered down SBCR, informed MCR

2018-01-12
07:00-07:30 - Pre-job meeting, powered up SBCR, informed MCR
07:30-09:30 - Installed 600 & 500 series lead shields
09:30-14:30 - Installed 700 series lead shields
14:30-15:00 - Powered down SBCR, informed MCR

2018-01-15
07:00-15:30 - Pre-job meeting, powered up SBCR, informed MCR, installed 700 series shields, installed ladders,
installed port hole covers, powered down SBCR, informed MCR

2018-01-16
07:00-15:30 - Pre-job meeting, powered up SBCR, informed MCR, calibrated radiation meter w/ Safety Group
representative, powered down SBCR, informed MCR

2018-01-17
07:00-15:30 - Prepared for upper resonator 2C Extraction Probe

 All cyclotron in tank work has been completed.

 The HE1 probe was removed from the cyclotron today using the procedure in Document 126165.  The reason for removal is so that Probes Groups can attempt to repair a problem with stuck probe head drive.  Due the failure, the probe head was almost at the inboard travel limit (towards cyclotron center post) at the time of removal. Additional cut-outs were added to the HE probe frame to accommodate the probe head and ribbon cable wheel which were not at the normal position.

Location of the frame below the probe was done according to the procedure.  While moving the frame up some resistance was encountered about 1" from contact between the frame upper surface and the base of the probe.  After some investigation with outrigger cameras and direct viewing from the tank edge it was determined that the probe head was interfering slightly with the removal frame.  The probe head was driven completely to the inboard limit (a small amount of travel was still possible).  After moving the probe head the frame could be raised up to the probe without issue, and then it was removed from the tank as per the procedure.

The probe is scheduled to be reinstalled next week after the repair is complete.

 HE1 probe was reinstalled in the cyclotron this morning as per the procedure in Document 126165.  Hardware for the probe head motion had been repaired/replaced by Probes Group, so the probe head could be put in the fully retracted position.  The job went smoothly and nothing unusual was encountered.

 The Blue Shield Door on the B2 level of the RH building has been closed.

Lid up operations took place on January 21st starting at 10:25 lid went up without issue. Couple things to note are station 11 had a number of spikes in the first ten inches of travel and number five had a couple in the first five inches of travel. We will monitor these stations and perform maintenance as needed. Station three which has spares #13 saw above normal heat generation in the right side High field gear box, we will investigate next week.

Lid down operations for 2019/2020 shutdown went ahead as planned on March 10th 2020 with Dan Louie and Arthur Leung absent due to a conference. Please see attached file for station counter numbers. Station two right side Highfield gear box was making intermittent growling from outboard bearing, this station is next on scheduled maintenance and will be inspected post removal next shutdown. Station 3 has spare jack set 13 in place, using Operations thermal camera we where able  to observe higher than normal running temperatures from right side gear box. Upon removal next year we will investigate bearing end play tolerances as this is the suspected cause of heat.

The pendant used for controlling the orbital motion of the bridge mockup was upgraded. Features of the new pendant include:

• One cable to controller (formerly two cables required)
• Ergonomic and robust form factor of pendant
• Easily modifiable button/contact setup, with spare button space
• Extended cable to reach anywhere in the room

Pictures attached show before and after.

The lid up operation took place on Jan. 22, 2021.

The system was not energized well ahead of time to allow to "warm up", as is usually done.

Starting Master Synchro value: 44

After gap set on all jacks, master synchro: 131

Notes on Jacks: Watch Jack 11, cause a few trips during lid down but was able to run again without jogging synchros

Jack 5 and 11 had spikes early but no trips

Jack 2 and 3 lagged early and caused trips (but no spikes)

After jogged 2 and 3 back, still had issues with 7,9,11

7, 9, 11 were jogged to match and then ran for half an inch, then 1, 5, 11 showed some spread

Lowered all the way down and spread was much improved (starting Master Synchro reading: 3)

Started raising again and went much smoother

Jacks 7 and 11 showed some occasional noise on readback

stopped to inspect vacuum seal (12:30 - Master Synchro 7016)

No trips on second attempt going up until 40"

Jack 1 kept falling behind, causing Jack 2 to trip

Finished at 2:03PM, Master Synchro at 47406

Meter stick did not reach ceiling (off by 1/4")

The meter stick versus limit switch discrepancy could be due to the stick shifting slightly by accident (due to vibrations or accidentally being bumped by someone). The limit switches were trusted as it would be highly unlikely for them to both disagree with the meter stick simultaneously if there was an adjustment setting issue with them.

The newly developed Ladder Handling Tool (TRH1401), which is used to move the maintenance ladders from the #5 resonator area closer to the centre post, was tested inside the cyclotron tank to perform the actual operation. This is a task that was historically performed by operators, resulting in dose received (typically 0.1 - 0.3 mSv for two people, twice per annual shutdown). This tool completely eliminates the need for people to receive this dose - only a small fraction of this dose is involved for one person in setting up the tool for use.

Overall, the test was deemed successful; both ladders were successfully moved which validated the functionality of the tool. There was difficulty in unlatching the lifting eye interlock for the first ladder (this was expected) but it was eventually achieved without human intervention. After the first ladder was moved, the entire tool was brought out of the tank, and some minor adjustments were made (ex. clutch adjusted). An attempt was then made to move the second ladder, and it was a significant improvement over the first attempt.

Feedback was obtained from this test and work will be performed to improve some aspects of the design (mechanical, viewing, procedural). The tool will be tested at least once more before the end of 2021 shutdown, and we will aim to have the important feedback items implemented by then.

The vault shield door has been closed and seal inflated. The operation was performed with no issue.

The lid up operation took place on Feb. 4, 2022.

The system was not energized well ahead of time to allow to "warm up", as is usually done. Upon starting, brakes were not energized - this was because of the master synchro position - once it was adjusted, interlocks for motor brakes toggled as expected.

Starting Master Synchro value: 61

Minor issue when first starting: after a few attempts to raise, jack 12 needed to be manually jogged to adjust its synchronicity with the others. After it was adjusted, the entire operation went smoothly and the system did not automatically stop again.

The system was stopped at approx. 3 inches of travel to inspect the seal, etc. Everything OK.

No major issues with any jacks. No overheating observed.

Jack 11 showed occasional noise spikes on readback (this happened in 2021 as well - suggest replacing this station's synchro next).

Meter stick hit the ceiling and bent slightly. The meter stick versus limit switch discrepancy could be due to the stick shifting slightly by accident (due to vibrations or accidentally being bumped by someone). The limit switches were trusted as it would be highly unlikely for them to both disagree with the meter stick simultaneously if there was an adjustment setting issue with them.The limit switches toggled as expected, and the gap was measured at 46.5" (correct value).

Ending Master Synchro value: 47387

Total time to raise was approx. 40 minutes.

Doses:
Arthur: #150, 0
Dan L: #310, 0.02
Dan M: #819, 0.04
Nadia: #868, 0.02
Adam: #862, 0.014
Kyle: #111, 0.09
Ryan: #115, 0.05
Chris: #840, 0.043

When attempting to perform the radiation survey, the program displayed an error at startup stating that the software was not able to keep up with the incoming hardware samples, and suggested to increase sampling rate or buffer size. The default sampling rate of 500 ms was not used - rather, 100 ms was used. No issues were observed with this rate. As far as I am aware, this issue had not occurred in the past, and the cause is unknown.

The lid down operation took place on Apr. 27, 2022.

The system was not energized well ahead of time to allow to "warm up", as is usually done.

Starting Master Synchro value: 47387

Upon startup, the system worked fine and lowered the lid.

After some time, jack station 12 became out of sync with the master synchro. It needed to be manually jogged to adjust its synchronicity. After it was adjusted, the entire operation went smoothly and the system did not automatically stop again.

No major issues with any jacks. No overheating observed.

The rope and pulley system that is used to raise the festooning rail across the vault shield door has been re-routed. It now makes use of all three pulleys and is easier to raise and lower without interference. 

 We removed the left bushing that is around the air cylinder on the festooning rail, right outside the cyclotron. It was cracked. Will need to replace next year when the vault door opens again.

 The vault shield door has been closed and seal inflated. The operation was performed with no issue.

The lid up operation took place on Jan. 30, 2023.

The system was not energized well ahead of time to allow to "warm up", as is usually done.

Starting Master Synchro value: 62

The Master Synchro was jogged to a value around 2 prior to starting, to allow it to lag behind the jacks.

When first starting: after a few attempts to raise, it was determined that the master synchro value needed to lag behind the other synchros more than initial suspected. This was changed, and the system operated smoothly for a short time.

During lid travel, station 5 and station 11 both caused trips and needed to be manually jogged to match the rest of the stations. The system was restarted, and more trips occurred. Stations 7 and 10 were adjusted as well. Eventually, after adjusting a few stations, the lid travel smoothed out and no more issues were observed for the remainder of the operation.

The system was stopped at approx. 3 inches of travel (synchro count 5255) to inspect the seal, etc. Everything OK.

No major issues with any jacks. No overheating observed.

Jack 11 showed occasional noise spikes on readback (this happened in 2021 and 2022 as well - suggest replacing this station's synchro next).

When finished, the limit switches toggled as expected, and the gap was measured at 46.5" (correct value).

Ending Master Synchro value: 47388

The attached spreadsheets show all synchro values before and after the operation, as well as jack station infrared heat measurements.

Start time: 9:23 am
End time: 10:53 am (includes gap measurement, etc. - actual lid travel finish was a bit earlier)
Total time to raise was approx. 1.5 hr.

Doses:
Adam Newsome: 0.005 mSv
Maico Dalla Valle: 0.121 mSv
Eric Lahe: 0.056 mSv
Ryan Anderson: 0.017mSv
Kevin Lucow: 0.060 mSv
Ed Parton: 0.007 mSv
Jason Zhang: 0.025 mSv
Nadia Jorgenson: 0.005 mSv
Arthur Leung: 0 mSv
Dan Louie: N/A

[edited 2023-09-8 to fix typo - Adam Newsome]

The lid down operation took place on Mar. 31, 2023.

The system was energized a few minutes ahead of time.

Starting Master Synchro value: 47388. The master synchro was jogged up prior to starting to approximately 47395.

Upon startup at 10:38 am, the system worked fine and lowered the lid. No stops occurred and the lid lowered smoothly. No abnormal observations made. The lid was completely lowered at 11:05 am.

See attached summary.

The vault shield door has been closed and seal inflated. The operation was performed with no issue.

The original end feet and end caps were removed from the maintenance ladders and replaced by new end caps (TRH1637).

An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.

The lid down operation took place on April 19, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47401. The master synchro was not jogged prior to starting.

Upon startup at 12:45 pm, the system worked fine and lowered the lid. One stop occurred after lowering a few inches but the system was easily restarted with no modification. No abnormal observations made. The system was stopped at 1:05 pm at a master synchro count value of 12385 to check the tank seal, confirm limit switch functionality, etc., then resumed. The lid was lowered at 1:17 pm.

Overall, it seems that the lowering process was more "smooth" than in past years - the jack stations seemed to be well-synchronized and relatively few motor/brake toggles were required during travel (this is just based on memory and visual observations, not actual data). If this is true, it is likely attributed to the fact that during lid up operation this year, each station was synchronized with respect to each other to help reduce stoppages.

See attached summary.

A second lid down operation took place on April 30, 2024.

The system was energized a few minutes ahead of time.

Starting master synchro value: 47427 (after it was jogged up slightly prior to starting).

Upon startup at 12:51 pm, the system worked fine and lowered the lid. The lowering was stopped at approximately 5" gap as per standard procedure for Operations to perform checks. Furthermore, the lowering was stopped multiple times after that at approximately 0.5" increments for resonator latch checks.

No abnormal observations made. No jogs required. Once again, this was a very smooth lowering process. The lid was fully lowered at 2:00 pm.

Ending master synchro value: 66

See attached summary.

Fault 16710 was initially assigned to Remote Handling: IBOP0206 giving false positive - Al-ALCW water pressure shows "off" when it isn't; The safety system is reporting that Al-ALCW water pressure is "off" (page SVG) when the system is still on and at normal operating pressure.

2024-04-10: initial inspection by A. Newsome yielded the following (email from A. Newsome to K. Genge):

1. First, on an unrelated note, there's a thermometer mounted to the Al-ALCW pipe which is loose and rattling due to vibration of the pipe. There's a little bit of water leak in this area. I don't want to mess around with it but I suggest someone tighten it soon! 
2. I looked around for sensors on the Al-ALCW pipe and I think I found the water pressure sensor which is giving the faulty reading. I have three suggestions for what to look into, in order of importance:

1. The cable for this sensor has a portion of it which is electrical-taped together. I didn't mess with it, but I suspect underneath the electrical tape it could just be disconnected. This would be a super quick check and quick fix. Suggest connectorizing it in the future with a simple Molex connector or similar - let me know if you want us to do that. 
2. The pressure transmitter itself could have failed or could have some sort of internal issue. Suggest disassembly to inspect. First, water will need to be turned off and the isolation valve turned off. If no spare readily available, suggest swapping with another transmitter of the same model to see if that is in fact the issue before ordering one (I think there's one nearby for the Meson Hall water system, beside the electrical panels). 
3. Somewhere along the line, there could be a cable degradation issue, or the cable could be disconnected/pinched/cut. Suggest looking into this last as it is the most tedious and also least likely.
   

2024-07-16: J. Ableseth confirmed the pressure transmitter (3400T Series) appears to be in working condition.

2024-07-24: A. Newsome and J. Zhang investigated the elevating system control panel located in B3 level. The following are findings (email from A. Newsome to K. Genge):

1. Our elevating system panel accepts an overall interlock summary signal from an unknown external source (probably a safety panel located elsewhere). There's no manipulation of the signal at all - it just enables/disables the system from running. 
2. The indicator lights on top of the elevating system panel do not have any electronics nearby - the light wiring comes from an external source via multiconductor cable. They are just indicators and are not connected to any electronics inside our panel at all. In fact this isn't even in our drawing package. 
3. We traced the light signal multiconductor cable out to the tunnel on B3 level - the cable runs along the middle cable tray. Did not follow all the way, but we assume it comes from the safety panel in B2 that Kathleen pointed out. In this panel, there's an exact duplicate of the indicator light display! This one, too, showed the "CYC H2O OFF" light as being on (false logic since the pump is active). It appears that the wiring for these lights is connected to a bank of relays right beside it. My suspicion is that one of these relays has failed, which then causes both of the indicator lights to be stuck in the ON position (this is the "safe" position). Given that there's also a similar cyropanel-related faulty signal, this makes even more sense - the relays are probably very old and/or there was some external issue that could have caused both to fail. Seems more likely than two totally unrelated sensors failing at the same time, that just happen to be routed through the same panel and relay bank. 

• Kevin Ardron investigate the relays in that panel... if it is not obvious the relay has failed, easy thing to do is just swap it with an adjacent one that is known to work and see what happens.
• Probably a good idea to just replace all of them if they are all the same age, because we'll likely see others fail. I think it is a relatively inexpensive and straightforward thing to do that could save some headache and beam trips in the future. 
• If the relay appears to be fine, or if the relays turn out to be totally unrelated to the actaul signal itself, continue tracing the signal back to its ultimate source. This panel on the B2 level is most likely where we'll find the issue.
  
  

[Update] 2024-07-26: Email from K. Genge:

An update from Safety System’s findings re: elevating system interlocks -

1. Very difficult to test the relays given the design/location of them and the cables connecting to them.
2. Kevin A. managed to trace the cryopanel elevating system related fault (#17020) over to the vacuum racks on SAB2 where he coincidently ran into Dimo, Vacuum Group Leader. Turns out that the sensor that gives this signal is connected to old cyclotron cryogenics/vacuum equipment that is no longer in use (and therefore providing the elevating system interlock system with the right info!). During the shutdown, Dimo and his team will move this sensor to the correct location so that it’s providing the correct info. That fault has been redirected.
3. The water ok signal goes to a “water system box”. We don’t know where this box is, but Kevin is willing to provide additional cable info to Joe. I recommend that the next step is for Joe and Kevin Ardron to connect and figure out where this cable goes.

An emergency trolley rescue winch (TRH1527) was developed for the purposes of retrieving trolleys from the cyclotron tank whose radial drive have failed.

This winch was tested today on the carrier in mockup setting with success by Adam Newsome, Dan McDonald, and Travis Cave. The following is a summary of the test:

• Lower resonator trolley was used with additional 1200 lb loading (total weight: approximately 2900 lb)... this represents the worst-case scenario for rescue
• The drive mechanism was bypassed so wheels could freely rotate (this is standard rescue procedure)
• The winch was mounted so that the alignment angle of the cable was close to 0 degrees
• The winch successfully pulled the entire assembly with no issue – it was pulled approximately 5 metres

An ECO will be released for the drawings for mounting components - no redlines were required.

 All 4 Under rail wheels serviced and cleaned. All Timken bearings repacked with grease. No defects in any bearing set. 4 spare Timken 02820 bearings in RH shop.

 Motor/Gearbox unit removed from carrier and fully inspected and serviced. Oil changed on May 29th using Mobil 600 W Super Cylinder Oil. No signs of wear in internal gears, recommend oil service roughly every 5 years after shutdown.

 1,2, 5 Year Service and maintenance package completed on Carrier. All reducers repacked with Molub-Alloy 777-1 ES grease. Main drive train and all bearings relubricated. No issues. Operational.