On 2024-04-22 around 3:30 pm it was discovered that the crane's main hoist was in a faulted state. The operator (Adam Newsome) experienced the fault after performing a safety reset and attempting to lower the hoist. There were no visible signs of any issues, mechanically. This fault had not been previously reported by any other crane operators. The crane was previously unused, sitting idle. Note: the auxiliary hoists appeared to function normally.

Fault messages on the HMI indicated:

"135. Main Hoist West Drum North Motor Drive Fault (1000VFD)"
"104. Main Hoist East Drum South Motor Drive Fault (800VFD)"
"103. Main Hoist East Drum North Motor Drive Fault (700VFD)"
"138. Main Hoist West Drum South Motor Drive Not Ready (900VFD)"
"136. Main Hoist West Drum South Motor Drive Fault (900VFD)"

It is clear from this that there is either some sort of common issue seen across all these main hoist VFDs, or one issue with one of them which caused a cascading series of faults.

On 2024-04-23, upon investigation of these faults by going online with the PLC to determine fault logic, it was clear that all faults were indeed present, but there was no obvious indication as to what it was. Inspection of the VFDs in the control panel showed that all of the aforementioned drives displayed a fault code. This was fault code 4 which indicates DC bus undervoltage.

After researching this fault online, it appears this is typically caused by an issue with the input mains supply (480VAC @ 3 phase in this case), or by the drive's input DC filter/buffer circuitry. When measuring the DC bus voltage, it is expected for it to be approximately 1.414 times the AC supply voltage. In this case, 1.414*480 = 678 VDC. The first troubleshooting step was to measure to confirm the DC bus voltage on a known working and not working drive.

The bus voltage was probed on drive 500VFD which, based on lack of fault message, was expected to be functioning properly. With the safety off due to E-stop condition, the voltage was nearly zero. When a safety reset was pressed to enable the drive, the voltage changed to 690 VDC. This is close enough to the expected 678 VDC. The voltage did not change, even when the fault message for the other drives appeared again.

Next, the bus voltage on two known faulted drives, 900VFD and 1000VFD, was probed. In both cases, after the safety reset, the bus voltage started at 690 VDC but then slowly dropped down towards nearly zero (somewhere around 18 V) over a period of 3-5 seconds. It was during this transition when the voltage dropped off that the undervoltage condition became true, which triggered the fault.

As of 2024-04-24, the root cause of this issue is unknown. It will be investigated further and this e-log will be updated when a solution is found.

Update 2024-04-24: the root cause has been identified. The undervoltage fault was just a symptom as a result of the input contactors for these drives switching off, causing the DC bus voltage to drop slowly due to capacitance in the input filtering circuit. The reason for the input contactors switching off is attributed to safety signals dropping out due a fault observed specifically on 900VFD. When looking at the drive itself, a fault with code 4030 was displayed: Enc1 open wire.
This fault implies the encoder may be disconnected. The encoder wiring was checked: OK. The encoder cable was swapped with a unknown working one from an adjacent drive - the issue remained with 900VFD, which suggested the problem is with the drive itself. The encoder board (20-750-DENC-1) was swapped with an adjacent drive and the problem followed the board. The encoder board was then switched with a brand new spare (note: jumpers needed to be set!).
Upon power-cycling the system, the fault did not persist. Therefore, it is suspected that the encoder board had failed (in fact this happened previously with the same drive - see e-log 42). Upon inspection, one of the capacitors on the board appeared to be cracked - this will be investigated further to see if replacing it fixes the problem.

A spare encoder board will be ordered. This issue should be monitored in the future - it seems as if the drive itself is perhaps causing the encoder boards to fail.

Note: upon powering the system on again, another issue was noticed: 500VFD shows "drive not ready" fault. This is not displayed on the HMI though - it was just not possible to reset the safety system and this was only discovered from going online with the PLC. This will be investigated further.

Update 2024-04-26: the 500VFD "drive not ready fault" was investigated. It was determined that the SP+ (safety power +) signal wire was loose, which meant the safety signals to the drive were not getting through, causing it to remain in a "not ready" state. The wiring issue was corrected. Performing a safety reset resulted in successful drive enables across all drives. However, upon attempting to move the crane, even though all safety signals were green and it appeared to be able to move, it did not.
It turned out that there was a crane pendant fault: "Fault 305. Radio Control Receiver in Fault (3700RC)". This is exactly what happened previously in a similar situation (see e-log 55). The steps mentioned in that e-log were followed (power cycle, reconnect antenna/connector). Upon powering back up, everything worked as normal. The crane was tested in local mode - all three hoists up/down (main hoist run to upper limit), all trolley travels in each direction, and bridge travel in each direction.

As of now, everything is operating normally and all issues are considered to be resolved. A spare encoder board has been ordered.

A safety walkaround for July 2024 was completed for the B2 level by A. Newsome. No deficiencies to report.

Results can be found in the master spreadsheet

Upon powering on the crane today and attempting to use the main hoist, the following faults were present:

1. Main Hoist East Drum North Motor Drive Fault (700VFD)
2. Main Hoist West Drum South Motor Drive Fault (900VFD)
3. Main Hoist East Drum North Motor Drive Not Ready (700VFD)

Note that the auxiliary hoists functioned as normal.

Upon power cycling the crane to reset, the faults disappeared, and the main hoist functioned normally again. The root cause of the issue is unknown. It is suspected to be related to either prolonged inactivity, improper safety reset sequence, or the battery being removed from the pendant. This issue will be monitored for re-occurrence.

The spacer flanges were installed on both turntables today.

See photos and information in the following DocuShare Collection: Collection-39816

Important note: the fitment was quite tight due to interference with universal joints and grease nipples. The flanges were still able to be installed with a bit of difficulty. They are not posing any immediate issues, but it is predicted that the adapter flanges will certainly interfere with drive system components. It is necessary to change the design of the adapter flanges to allow for the appropriate clearance.

A safety walkaround was completed for the ARIEL Hot Cell area.

The resulting spreadsheet can be found on DocuShare as Document-242733.

No major deficiencies identified.

It was reported that the hot cell turntables would not rotate and the access hatch could not be operated. The turntables' elevation motion was working normally, and other cell functions were working.

Upon investigation, after going online with the PLC and checking the interlocks for the turntable, it was determined that motion was being prevented because the turntables were in the "critical position" in which they could interfere with the access hatch raising/lowering. However, the access hatch was fully closed, so this logic should not have been actively preventing motion. After inspecting the target access hatch limit switches, it was determined that the upper limit switch's Normally Open contacts were behaving normally, but the Normally Closed ones were not. When the switch was toggled, the NO contacts did not switch over. Thus, the sensor was in an unknown state causing conflicting logic in the PLC (the program thought the access hatch was in an intermediate state between opened and closed, thus preventing motion... and the access hatch could not move because the turntables were in the "critical position"). After testing the switch multiple times, jiggling the wiring/contacts, and rewiring one of the screw terminals, the NO contacts started working normally again. It is suspected that a wire was loose. 

The issue has been resolved and tested - the access hatch logic is functioning regularly and both turntables can fully rotate and elevate.

After a recent power cycle, the y-axis referencing was lost for the 1000 kg crane. The position readout was also incorrect, somewhere on the order of > 30,000 mm. On 2024-11-29, A. Newsome re-referenced the y-axis. The position readout correctly reset to zero, and both +y and -y motion is functioning correctly, but the y-axis position readout on the HMI constantly displays 0 and does not change. To be investigated.

Update 2024-12-02:

After going online with the PLC, it was determined that the reason for the display of 0 as the position is that the two values used for calibration of the +y and -y limits were actually the same, meaning the scaling factor (the difference between these two values) was 0, which resulted in the displayed value being 0. The root cause of this is that the encoder was not functioning correctly so its value was not changing when the crane moved and it was stuck at one value. Upon investigation of the encoder input card, the red "ERR" light was on. This indicates the encoder signals are not properly reaching the input card. Once this was discovered, it was remembered that this happened in July 2024 as well (no e-log was written). In July, the root cause was identified as being a loose encoder signal wire in a junction box. Junction box CJB1-BC, located on the bridge crane near its disconnects, was opened and investigated. Upon checking each wire, it was determined that the red wire seemed not to be making full contact. The wire was removed and re-inserted, and the "ERR" indicator on the encoder card turned off. The crane's +y and -y limits were re-referenced. The crane is functioning normally after re-referencing. (Note: if something like this happens again, during the re-referencing process, the displayed value on the HMI will be incorrect.. this is because the PLC's scaling factor is not fully adjusted until both +y and -y limits are reached. The actual encoder measurement taken at those limits is used for the scaling factor. This is not an ideal way to program the system, but this is how it works with all ARIEL hot cell subsystems. So incorrectly displayed values can be ignored, in general, until full referencing is completed).

Two externally mounted jib cranes were installed on the ARIEL hot cell - one on the West side above the tool port, and one on the North side above the tool port. These cranes are supplied by AC DC Cranes (see attached quote for info). They are 250 kg WLL, 2 m span, powered hoist, manual pivot. They were mounted using M24x3 x 50mm bolts.

Engineering analysis and BC P Eng sign off for the design of these cranes has been handled by ROBATEL Industries as part of the ARIEL hot cell contract. TRIUMF scope of work included procurement and installation of the cranes.

At the time of writing this e-log, the cranes have not been powered and therefore have not been tested. They are not considered commissioned and operational at this point and must not be used. Facility coordinators have been informed.

Future work (to be done by A. Newsome): coordinate electrical services to supply power to the cranes, perform initial functionality and load testing, add cranes to calibration/inspection index, and include the logbook and list of qualified operators down in the hot cell area.

A safety walkaround was completed for the ARIEL Hot Cell area.

The resulting spreadsheet can be found on DocuShare as Document-242733.

No major deficiencies identified.

December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam 

Tests:

Gas lines removal: 

• Starting with the most exterior connection, VCR connection loosened off with open ended conventional wrench
• Once loose, the nut can be un threaded with manipulator finger (rolling nut technique) 
• Once gas line unhooked, gasket removed by bringing female end outside the service tray footprint and jiggling until the gasket was removed
  • Also possible to use a pick if needed
• The same procedure was conducted for the interior VCR connection

Gas lines installation: 

• Starting with the inner most connection
• Gasket placed on 3D printed tool and clipped into position on VCR male end
• At first an M10 bolt was inserted into the elbow below VCR connection, but without rotational authority, the makeshift handle is not worth using
• Griping the elbow with one manipulator and rolling the nut onto the threads with the other proved successful
• Nut was tightened using conventional open ended 19mm wrench

Observations/Notes:

• Service Tray was not in a fully connected position, so even less space will be afforded.
  • This may affect the ability to get 2 manipulators on the same connection
• Piston modules were missing some limit wires on the side. These constrain movement horizontally and will either need to be removed as part of the procedure or, make the procedure more difficult
• Mass markers and various other connectors not installed, and these could slightly restrict movement as well
• Service tray pin was restricting the movement of the left manipulator during install
• Lighting was inadequate under the piston 

Recommendations/follow up items/questions:

• All metal Parker VGR style gaskets to replace plastic retainer versions
  • This is so that the degraded plastic doesn't break off and end up in the gas lines (upside down connections)
• A modified wrench with flats for handles and with more length would make the above procedures easier 
• Along with raising and lower the service tray, the service tray pin being oriented towards the beam entry direction, would make life easier 
• Still need to test gasket install on "other" Piston gas line connection
  • Chad will redesign some new gasket tools to be tested
    • side load and axial load gaskets, low and high clearance, aluminum construction 
• Lighting positioned to flood the service tray area will be needed
• labeling the gas lines with a more permanent solution will be needed
• Once the high voltage feed-through parts have come in, we can re-test and see if the piston module can be lowered to aid in target removal situations
• Aaron will test Parker style seals to see if they can be removed as easily once brought up to specified torque 

------------------------------------------------------------------------------------------------------------------------------------------------------------------

 December 16, 2024

• Upon inspecting the piston modules on AETE in TISA, we became aware of how the wires are arranged for the limit switches, it may be a good idea to look into how these wires are handled when using the piston module jig.

 December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam, Austin Hagen

Tests:

Loose piece connection and removal 

• By hand, the two connections were brought together and removed carefully

Observation/notes:

• Copper extensions are fragile and can bend when removing extraction electrode. 
  • This makes this part fairly limited in its capacity to be re-used
  • Copper extensions will need to be tuned before inserting into hotcell for replacement 

Recommendations/follow up items/questions:

• Bigger lead-ins would be appreciated
• What is the exchange frequency?
• the flexible nature of the copper extensions makes re-use limited
• How many connections can we expect from the aluminum fingers?
• Would a closer locating feature specific to the pins help?

December 12, 2024 - Michael Genix, Aaron Tam 

• Guide pins for the extraction electrode contact before the banana plugs
  • As these pins are low tolerance, this will act as the guiding for the plugs
  • Copper extensions have a much smaller diameter than the plugs, so positioning can be less precise in this area 
• Exchange frequency is still TBD
• Pin and plug life is still TBD
• Potentially the copper extensions can be tuned in the hotcell

 Previously prepped - A. Kong, S. Liu, A. Tam

• Gate valve assembly put together
•  All bolts torqued to >=55Nm
• Gate valve Cart moved into hotcell

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

April 23, 2025 - A. Kong, C. Fisher, S. Liu, M. Genix, A. Tam 

• Moved standoffs to the taller position on the cart
• Weight added to cart to bring CG towards centre of cart.
  • Cart rolling away and hitting lip of lift table could tip (especially with gate valve in higher position)
• Gate valve moved onto cart via 1000kg crane
  • Sling used between openings in gate valve assembly frame members
• Complete gate valve assembly with hard stop and bellows movement retainer
• Dust cover removed, sealing (o-ring) flange removed,
• vacuum and sealing surfaces cleaned,
• hard stops (gate valve – shield plug) installed, bellows limit brackets installed
• Bagged all items removed to keep them clean
  

QDS seal installed BY HAND onto gate valve tapered flange

• Seal tabs align with a cutout feature on the flange (3 tabs, 120O spacing @ 10:00, 2:00, 6:00)
• These cutouts allow the tabs to bend towards the flange without obstructing sealing surfaces

RH Assumptions (UNTESTED)

• in the situation of a gate valve assembly replacement:
  • Seal should be pre-installed onto gate valve flange using soft jawed clamps (bought, designed, modified)
  • Assembly should come bagged and bag removed at the point of installation to reduce contamination on  vacuum and sealing surfaces
• if seal replacement on its own (eg. during extraction electrode replacement)
  • Seal should arrive on a clean stand through tool port and oriented in installation position
  • two manips to place seal onto flange; grabbing top two tabs (UNTESTED but confident because of taper on seal retainer)
  • clamps are TBD (bought, designed, modified)               

Installing gate valve assembly proposed sequence of operations:

• Using Hotcell lift table, raise gate valve assembly so that Shield-plug-to-gate-valve screw can engage, leaving ~1/8 inch between Shield-plug and gate-valve-assembly
• Slide gate-valve with cart into rough x-y position (use camera views for alignment)
• Start thread on shield-plug-to-gate-valve screw, until weight is released from cart.
• Release gate valve from cart by unscrewing 2 bolts holding it
• Torque shield-plug-to-gate-valve screw fully to align hard-stop threaded holes
• Lightly install hard-stop screws
• Crack loosen the shield plug-gate valve screws
• Final torque on hardstop screws
• Final torque on shield plug-gate valve screws

Additional Notes:

• Tested manipulator access/reach to the hard-stop screws)
  • (better access may be achieved by lowering torque tool handle)
• Nut housing for shield plug to gate valve assembly screws/nuts not used as final bolts will react differently than test shield-plug

To be tested:

• Gate valve seal chain clamp using torque multiplier and reaction arm.   

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

April 30, 2025 - C. Fisher, A. Kong, K. Ng, A. Tam 

Preparations:

• Chad programmed 4 torque ratchet wrench settings in CW (12 Nm & 24 Nm) and CCW (12 Nm & 24 Nm)
• Chad had procured and dry fit Norbar HT3-1000 torque multiplier for this operation

Chain Clamp Sealing/Tightening procedure:

• Shielding on the outside of the RIB side structure needs to be taken off to allow QDS reaction head tool to have access to structural components
• Chain clamp brought together lightly by turning turnbuckle screws lightly by hand (this would be done with a smaller air or electric ratcheting wrench in the future)
  • Using several camera angles to verify that the bellow flange is engaging properly within the chain clamp 
  • A soft tipped tool held by one manipulator can be used to push the flanges together, while the other manipulator starts cinching down on one of the chain clamp turnbuckle screws
• Once positioning had been confirmed, turnbuckle screws were tightened on the first 12 Nm program with the the Norbar HT3-1000 torque multiplier to achieve ~60Nm
  • With the torque multiplier, the right manipulator held onto the torque ratchet wrench and the left manipulator brought the reaction head into contact with structural components on the FE
    
  • Bottom screw reaction against FE kinematic mount, top screw reaction against RIB side of gate valve bracket.
• Once both screws have been tightened to ~60Nm, they were both tightened to their final 120 Nm (24Nm on the electric torque ratchet wrench)
  

Notes:

• torque multiplier is heavy, may be a good idea to use a tool balancer with the torque tool
• Stepping up the torque in a more increments may yield a more even clamp for the gasket to seal 
• As a check, a 20-200Nm click style torque wrench was used to verify that approximately 120Nm was reached 
  • 110 Nm - 120 Nm was required to continue tightening the turnbuckle screws