Powered down ARIEL Target Hall Crane today by switching off main power switch on B1 level in preparation for site-wide BC Hydro power outage on Dec. 27-29.

The non-existent safety limit switch for the target hall door which prevents remote motion has been bypassed in the crane PLC control panel to allow for remote operation of the crane. This will be un-bypassed when a switch is installed. Specifically, a jumper was placed from terminal 18001 to 20121 and 20141 which will enable the hall lock safety relays 2012SR and 2014SR. The crane is currently operational in both local and remote mode.

In preparing to troubleshoot the issue described in e-log #54, the crane was power cycled, and the pendant battery removed, to get to a "fresh start" state. Prior to this, the crane functioned normally aside from the incorrect E/W direction control on the North Aux Hoist.

Upon powering the crane back on, the pendant did not work to control the crane in any mode. The HMI had a fault present which read "305. Radio Control Receiver in Fault (3700RC)". This was accompanied by a red light in the safety control relay status summary screen, for "Radio Control Converter Communication Fault With Receiver (3714CONV)".

If the crane was placed in remote mode, the safety control relay status summary all turned green, and the reset button could be pressed, enabling remote mode. This means the issue was specific to the pendant/receiver communication specifically.

The receiver (in the cabinet in the control room) was inspected and the power status LED was on, the RF LED was off (which means there is no issue, OK), and the "OK to receive signal" LED was on - this indicates that the receiver was functioning as expected.

To further troubleshoot the issue - the crane was powered off, the receiver's antenna was disconnected and reconnected, and the multipin connector for the receiver was disconnected and reconnected. The system was powered back on, and the reset button was pressed. The fault message and the safety relay issue were cleared, and the pendant functioned again as normally expected (aside from the persistent issue from e-log #54).

It is unclear exactly what the issue was and how it was resolved, but could have been related to either a loose connector, or power cycling and resetting the system.

[edit 2023-01-03] The crane was powered back on this morning - there do not appear to be any faults on the HMI and it seems that the crane was unaffected by the power outage.

The ARIEL target hall overhead crane was powered off for the holidays. In the new year, throw the main disconnect in the crane control room to power back on.

An issue was observed in which the pendant did not work to control the crane. This is the second time in recent memory this has happened. See e-log 55 for details. The same sequence of steps outlined in e-log 55 was followed, and the pendant control was re-established. It is still unclear what the root cause of this issue is, but it is suspected that the crane pendant loses communication with the receiver for some reason, and even though communication can be re-established, the reset button on the crane control console does not clear the fault message, and the system must be power-cycled in order to fully reset. This is just speculation based on what is observed. Further investigation required.

Today while operating the crane with load, the crane stopped moving. Upon investigating, all safety signals appeared to be OK. Fault messages regarding drive faults for various VFDs were present, which seemed to stem from an ethernet adapter card fault (see attached photo). It is unknown how this occurred, but suspected that it was due to a brief blip in network communications or the external drive power supply control signal. It is worth noting that the diesel generator tests took place today, and there could have been some affect from a power surge because of this.

The crane was power-cycled using the main disconnect, and safety system reset. The faults disappeared. The crane was operational again.

If this issue occurs again, it should be investigated more thoroughly. It would be worth checking if there were any generator tests or other things which could cause some sort of power surge.

A safety walkaround for July 2024 was completed for the B2 level by A. Newsome. No deficiencies to report.

Results can be found in the master spreadsheet

Upon powering on the crane today and attempting to use the main hoist, the following faults were present:

1. Main Hoist East Drum North Motor Drive Fault (700VFD)
2. Main Hoist West Drum South Motor Drive Fault (900VFD)
3. Main Hoist East Drum North Motor Drive Not Ready (700VFD)

Note that the auxiliary hoists functioned as normal.

Upon power cycling the crane to reset, the faults disappeared, and the main hoist functioned normally again. The root cause of the issue is unknown. It is suspected to be related to either prolonged inactivity, improper safety reset sequence, or the battery being removed from the pendant. This issue will be monitored for re-occurrence.

A safety walkaround was completed for the ARIEL Hot Cell area.

The resulting spreadsheet can be found on DocuShare as Document-242733.

No major deficiencies identified.

The APTW front end was moved into the hot cell, on the lift table, following procedures outlined in Document-244862.

The mock-up frame was transferred from the east turntable to the west turntable above the front end. The east turntable is covered with the aluminum plate. During covering we accidentally broke the o-ring for the turntable plate flange. The new o-ring seal should be made ~1" longer in circumference to avoid difficulties with seating in the o-ring groove.

When we attempted to move the front-end with the manipulator, we noticed that the front end tended to carry on sliding down the slope of the hot cell (see TRI-DN-23-20 / Document-238536). The manipulators were subsequently unable to arrest the front end, likely because they cannot apply a load towards the CG of the front end. Therefore, for heavy-duty jigs in the hot cell it is advisable to avoid using wheels/rollers that allow the heavy assemblies to move freely on the lift table.

--------------------------------------------------------------------------------

UPDATE Nov 21, 2024

• FE attachment related tests completed:
  • replacement of shim plates
  • mock-up alignment of FE with shield plug (adjustment made in person, hot cell operator instruct and give direction with remote camera views
• Difficulties encountered:
  • keeping track of which hole/screws are seen by the cameras during alignment was difficult for the operator
  • shim plate awkward to handle if not familiar
  • incompatible coupling screw and nut threads (M24 course vs fine threads) prevent completion of coupling and uncoupling tests
  • 'loose' service tray requires separate alignment check when raising the FE with the lift table
• Lessons learned:
  • shim plate replacement easier if FE is semi-attached (screw lifts FE, leaving small gap for shim plate to be inserted)
  • need a cover plate to prevent dropping shim plate and associated screws
  • rotation clearance of FE checked, will clear
  • need to also check service tray alignment when raising
  • magnetic cameras would be beneficial for this operation
• Follow up items:
  • crane-based positioning of FE for replacement to be tested after concrete pours/hand-testing/when APTW is exchanged for AETE.
  • future design of protective covering when performing shim plate removal
  • optionally implement magnetic camera array to check hole alignment by looking directly at the holes

At end of shift, mock up and FE was separated, markings made on floor to specify where FE needs to be to couple.

--------------------------------------------------------------------------------

UPDATE Nov 25, 2024

Replacement nuts arrived (M24x3) for the coupling screws. Tests were completed to try and couple/uncouple the front end to the cart/mock up.

• For taking the FE on/off the cart the cart should be able to rotate with the FE so that we can secure the coupling screws.
• For dropping the FE off from the shield plug, we should make use of the alignment features (cup and cones), once screws are undone and the FE rests on the cones on the cart, it should be secured.
• If markings on floor is made to assist with alignment, we need to do two sets; one for unloaded cart, one for loaded cart, this is because the lift table will tilt differently when there is a load.
• The threads on the coupling screws are fine threaded in the real case, there is a slight concern with cross-threading when turning them onto the nuts remotely (from above).
• Existing camera views sufficient to check for clearance between coupling plates and FE structures.
• To allow the FE to guide freely into the alignment features (cup/cone), it may be a good idea to still use the air skates, but supplement it with structural 'stopping' features to keep it from rolling off the table.

--------------------------------------------------------------------------------

UPDATE Nov 26, 2024

We returned the front-end to the cart to allow the shield-plug mock up to be rotated by the target hall crane (turntable still not functioning as of this date).

During this operation we noticed that the front end did not fully return to the alignment cones on the cart (see images), somehow the contact pad screws made contact first, and so the two-faced alignment cone did not come into contact.

For the real support frame we should use a minimal number of contact points. Also,more clearance at the coupling screws between the cart to the front end should be added.

--------------------------------------------------------------------------------

UPDATE Nov 27, 2024

We rotated the mock-up frame manually to have the HTV side face the hot cell window. During which the o-ring fell and we decided to cut it for convenience, this should be replaced with an equivalent o-ring but made ~1" longer in circumference.

--------------------------------------------------------------------------------

UPDATE Nov 28, 2024

Today we attempted to re-couple the front end to the mock up frame in the orientation where the HTV faces the hot cell window. The misalignment between the FE on the cart to the mock-up frame is worse in this orientation than in the previous (90 CCW facing down). There was approximately a 1" gap difference between the opposite ended screws. Factors contributing to the misalignment include uneven seating of the FE onto the bosch frame and uneven 'brake' engagement on the bosch frame wheels. During the attachment process the the screws had to be tightened very unevenly, with the HTV side being tightened far more than the extraction electrode side.

Additionally, the service tray shifted significantly from handling in the previous days and it was difficult to align the locating pins/cylinders. This further complicated the coupling process, in the future the service tray attachment should be made a separate operation to the process of coupling the  FE to the shield plug.

Under these conditions, during tightening, the inner screw at the HTV-side somehow jammed against the nut and the nut conversely jammed into the bracket. Subsequently, that screw could not be tightened further and required great difficulty to be removed, even after removing the bracket. In the aftermath, the nut bracket became forced apart (see picture) and the threads on the screw deformed beyond use. This is a critical failure mode for this coupling system. Note that the screw was hand started onto the nuts on the brackets prior to fully tightening the screws.

We believe the factors contributing to this failure include:

1. bracket gap being too large and allows the nut to wedge itself during tightening,
2. screw threads not being hardened/of proper grade,
3. tightening the screws too unevenly and sharply (too large turns per-screw when bringing the FE up, 1/2-1 turn) / allowing some screws being slack,
4. high misalignment (pitch and roll) between the FE plate and the mock-up frame,
5. the brackets not being able to accommodate the observed misalignment,
6. hole for screw is ~short and creates possibility for jamming when there is misalignment (pitch and roll),
7. aligning the service tray locating features while aligning the screw holes for the FE coupling was difficult,
8. possibly the extra lead in (chamfer) on the coupling screws is excessive and a non-standard design, which may cause problems starting the nut, likely a standard chamfer on the end of a standard screw would be sufficient for this operation

In the real operation, the long length of the screw will help prevent this failure from happening but some things to keep in mind/improve on would be:

1. bracket re-design to use standard wrench/nut gaps to avoid nut jamming
2. use hardened threads
3. try to see if no-lead in/chamfer/using standard screw profiles is sufficient
4. re-design the coupling system to not require a nut at the bottom and avoid cross-threading/thread jamming in the hot cell (move threads and nuts all above the FE assembly)
5. extra: the screw should not be fine threaded in the real case to avoid cross threading
6. minimize misalignment when trying to couple in the future (rotate turntable to minimum misalignment orientation)
7. tighten the screws more incrementally to have them go together evenly, don't allow screws to be fully slack if possible to prevent jamming
8. fully remove the service tray and attach only after the FE is attached

Of the above recommendations we strongly recommend re-designing the nut bracket - even when coupling was smooth (without misalignment) the nuts made a visible dent onto the bracket at the highest point (see image). A smaller gap (standard wrench gap) would prevent such a mark from forming.

We replaced the deformed screw with a standard (off the shelf) one to keep the FE secure to the mock-up.

We also noted after completing coupling that the extraction electrode, farthest out screw had damaged threads (see image).

It is recommended to perform repeated coupling/decoupling operations in the hot cell after some process/design changes are made/implemented to work out any possible failure modes with this operation.

November 27, 2024

Testing completed:

• Removal and re-insertion of 2x multi-pin coupling cable for HTV (see pictures, left most and right most, middle had no wire at the time).
• Tested using 'wire' method to pull cable once removed back through the original path.
• Test without rotating table, on cart, some obstruction that does not reflect reality in operation, rotated cart by hand, will require rotation during operation.

Observations/notes:

• Right connector has wires come across and may interfere with the middle connector, will require removal of multiple connectors to exchange one depending on configuration.
• Locating pins on high voltage multi-pin connector needs to be short for voltage bias reasons.
• Right-most black multi-pin cylindrical connector has higher risk of machined block end catching when fishing from the HTV side.

Recommendations:

• May be a good idea to add a holder connector to go into the multi-pins and give the manipulators a better grip for handling. Directly grabbing on the cylinder of the multi-pin connector is ok though.
• Some strain relief at the machined connector blocs (service tray side) will help prevent damage during wire replacement/routing.
• Possibly re-route some of the electrical connectors to the back side of the HTV column for easier access and shorter length.
• Will need chart mapping where the cables go and indexing different connections for future hot cell work (will be updated on SW model).
• Locking pins on multi-pin connectors can be removed and replaced manually with the manipulator.
• Adding a 'slippy' sheath onto the replacement cable for routing and to prevent catching would be a good idea in the future.

 -----------------------------------------------------------------------

Dec 02, 2024

Testing completed:

• Removal and re-insertion of 1x ceramic-ended wires specifically the one obstructed by the gas line on the right side of the HTV connection array as viewed from the hot cell window.
• Removal and re-insertion of main multi-pin connector cable bundle from the HVFT connector array on the service tray.
• Removal and re-insertion of the gas line obstructing the right-side ceramic electrical connectors << very difficult/impossible to do (recommend modifications + tooling).

Observations/notes:

• Electrical wires/connectors generally straightforward to exchange.
• Plastic/ceramic components on connectors are easy to damage during exchange operation.
• Most connectors require the metal/cylindrical bits to be 'grabbed' by the manipulators if no custom tool is used
• Rectangular multi-pin connector bracket if single is difficult to return to slot since not supported on both sides
• Need documentation/reference material to make sure connectors get returned to where they need to be
• Screws on connector are deep into the FE assembly, creates viewing difficulty
• Cameras hard to use for these screws because of beam dump being highly active when FE is in operation
• Ceramic (mass marker) connectors fine to grab on the conductor part
• When trying to re-align the ceramic bodies to line up the screw holes, found that it was really easy to damage/break the ceramics, avoid using hard tooling/manipulators on the ceramic parts
• Plastic components on multi-pin rectangular brackets got damaged after handling with manipulators
• Full sized pin detent tool not compatible with smaller pins, should make smaller sized pin detent tool, smallest clearance found on HVFT side of multi-pin connector bundle
• Routing HVFT side of multi-pin bundle through wires poses risk of damaging exposed pins

Recommendations:

• Should buy longer bits/ends for torque control tool to get better view angles on screws.
• Maybe captive screw for the gas is a good idea to prevent the screw from dropping.
• Torque tool should be set properly (torque and rotation rate) before performing screw exchanges involving the HTV connectors on the front end
• Return of the gas line (on bracket) to its location on the FE is difficult, suggested modifications include:
  • Changing orientation of cutout on bracket to let stainless bent/welded tube to be returned separate from the bracket.
  • Changing screws for the stainless part on the gas line to be standard size (M6) for hot cell exchange operations.
  • Changing the unused screw hole on the FE to be pinned and allow for better alignment of the bracket onto the FE
  • Removing burr on sheet metal part on hose/wire opening at the bottom to prevent the braided hose from catching.
• Should make a rotator tool (3D printed) to help adjust the ceramic end of the mass marker connector and have the screw hole and thread aligned
• Maybe plastic covers for water/gas connectors to prevent scratching of outer surface
• Make smaller pin detent tool or settle for old tool already made
• Possibly design cover to protect exposed pins on multi-pin bundle during replacement

 -----------------------------------------------------------------------

Dec 12, 2024:

Testing completed:

• Tested removal and return of the gas line obstructing some ceramic electrical cables on the right side of the HTV connector array on the front end, now with 6mm pin (McMaster 97493A162) threaded into the previously unused hole to help guide the previously  unused hole to help guide the bracket.

Observations/notes:

• Chad Fisher commented that even without the pin it should be possible to complete the removal and return operation if a 3D printed (soft )tool is made to grab onto the tubes and guide it with the bracket into the alignment hole/slot.
• Old recommendations still stand though + maybe make the pin longer/custom with more lead in.

Recommendations:

• Remove the retaining tab at the bottom to allow the hose to be positioned more freely.
• De-burr the sheet metal part to prevent catching on the braided hose.

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The North Aux hoist Trolley was stuck in a position up against the East wall.

Present during troubleshooting: Adam N, Maico D, Tom K, Allon M, Travis C

After troubleshooting, it was determined that:

• There were no fault messages on the HMI display relating to this issue
• The HMI indicated "Bridge Not in North Limit and/or Auxiliary Trolley North Not in East or West Limit" - the Bridge was not in the north limit as described, and this message implies the PLC thought the North Aux hoist was somewhere along the middle of its travel
• The West run signal was reaching the remote receiver from the crane pendant
• The West and East run signals were correctly reaching their respective input pins on the VFD (DI1, DI2)... upon swapping wires and commanding it to run East, the West direction still did not function
• The VFD was displaying the correct output frequency information on its display when commanded to run East, but not west
• The limit switch was not related to this issue

Upon swapping out the VFD with the working one from the South Aux hoist, the North Aux hoist Trolley was able to move both East and West, as normal. It was therefore determined that the VFD had failed and must be replaced.

Current state of system as of July 3, 2020:

• The North Aux hoist is fully functioning (contains VFD from South one)
• The South Aux hoist is disabled (no VFD, breakers shut off)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Adam Newsome requested quote for replacement drive (Delta Electronics TDN007E1100WM0) - will advise when replacement arrives
• Install replacement drive into South Aux hoist upon arrival (note: check EMC level modification)
• Verify entire system functioning as normal
• Reinstall North Aux hoist limit switch flag

See E-Log 44 for details on North Aux Hoist failure.

A replacement VFD was ordered by Adam Newsome and installed by Tom Kauss.

Adam and Tom replaced the VFD in the South Aux hoist (which had been relocated to the North Aux hoist). The original braking resistor was used. No functionality changes to the system.

Both Aux hoists were tested: E/W travel, up/down travel. Each hoist is functioning correctly. E/W speeds were compared between the two hoists and found to be the same (~14 Hz for slow speed, ~95 Hz for fast speed).

The replacement job is complete and the issue outlined in E-Log 44 is resolved, although the original cause for failure has not been determined.

Current state of system as of Sept 9, 2020:

• The North Aux hoist is fully functioning (contains original VFD from South one)
• The South Aux hoist is fully functioning (contains newly replaced VFD with original braking resistor)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Reinstall North Aux hoist limit switch flag (TBD when Target Hall installation work is done)

The load cell which is located on the crane itself to display the load to operators was reported not working during the week of November 15. It is planned to investigate on Nov. 25 and this e-log will be updated with the status.

Edit (2022-02-02): the load cell is functioning again. No changes were made.

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The up/down motion was also functioning. The North Aux Hoist Trolley failed during regular usage, seemingly randomly, when it was about 70% towards the West end of travel. Note that the West motion VFD inside the North Aux Hoist is the one previously taken from the South Aux Hoist (the same issue with the North Aux Hoist was observed last year, and the VFD was swapped, and a new one was installed in the South Aux Hoist).

A replacement VFD (Delta TDN007E1100WM0) will be ordered and installed, and this e-log will be updated.

Edit 2022-03-15: the replacement VFD has been installed and the system is operational. See e-log 52.

The North Aux Hoist VFD which had failed (e-log 50) was replaced today by Tom Kauss and Adam Newsome on Work Permit A2022-03-15-1.

The hoist's motions were all tested - it is functioning normally again.

It was observed today that the crane's west light was not functioning.