On 2024-04-22 around 3:30 pm it was discovered that the crane's main hoist was in a faulted state. The operator (Adam Newsome) experienced the fault after performing a safety reset and attempting to lower the hoist. There were no visible signs of any issues, mechanically. This fault had not been previously reported by any other crane operators. The crane was previously unused, sitting idle. Note: the auxiliary hoists appeared to function normally.

Fault messages on the HMI indicated:

"135. Main Hoist West Drum North Motor Drive Fault (1000VFD)"
"104. Main Hoist East Drum South Motor Drive Fault (800VFD)"
"103. Main Hoist East Drum North Motor Drive Fault (700VFD)"
"138. Main Hoist West Drum South Motor Drive Not Ready (900VFD)"
"136. Main Hoist West Drum South Motor Drive Fault (900VFD)"

It is clear from this that there is either some sort of common issue seen across all these main hoist VFDs, or one issue with one of them which caused a cascading series of faults.

On 2024-04-23, upon investigation of these faults by going online with the PLC to determine fault logic, it was clear that all faults were indeed present, but there was no obvious indication as to what it was. Inspection of the VFDs in the control panel showed that all of the aforementioned drives displayed a fault code. This was fault code 4 which indicates DC bus undervoltage.

After researching this fault online, it appears this is typically caused by an issue with the input mains supply (480VAC @ 3 phase in this case), or by the drive's input DC filter/buffer circuitry. When measuring the DC bus voltage, it is expected for it to be approximately 1.414 times the AC supply voltage. In this case, 1.414*480 = 678 VDC. The first troubleshooting step was to measure to confirm the DC bus voltage on a known working and not working drive.

The bus voltage was probed on drive 500VFD which, based on lack of fault message, was expected to be functioning properly. With the safety off due to E-stop condition, the voltage was nearly zero. When a safety reset was pressed to enable the drive, the voltage changed to 690 VDC. This is close enough to the expected 678 VDC. The voltage did not change, even when the fault message for the other drives appeared again.

Next, the bus voltage on two known faulted drives, 900VFD and 1000VFD, was probed. In both cases, after the safety reset, the bus voltage started at 690 VDC but then slowly dropped down towards nearly zero (somewhere around 18 V) over a period of 3-5 seconds. It was during this transition when the voltage dropped off that the undervoltage condition became true, which triggered the fault.

As of 2024-04-24, the root cause of this issue is unknown. It will be investigated further and this e-log will be updated when a solution is found.

Update 2024-04-24: the root cause has been identified. The undervoltage fault was just a symptom as a result of the input contactors for these drives switching off, causing the DC bus voltage to drop slowly due to capacitance in the input filtering circuit. The reason for the input contactors switching off is attributed to safety signals dropping out due a fault observed specifically on 900VFD. When looking at the drive itself, a fault with code 4030 was displayed: Enc1 open wire.
This fault implies the encoder may be disconnected. The encoder wiring was checked: OK. The encoder cable was swapped with a unknown working one from an adjacent drive - the issue remained with 900VFD, which suggested the problem is with the drive itself. The encoder board (20-750-DENC-1) was swapped with an adjacent drive and the problem followed the board. The encoder board was then switched with a brand new spare (note: jumpers needed to be set!).
Upon power-cycling the system, the fault did not persist. Therefore, it is suspected that the encoder board had failed (in fact this happened previously with the same drive - see e-log 42). Upon inspection, one of the capacitors on the board appeared to be cracked - this will be investigated further to see if replacing it fixes the problem.

A spare encoder board will be ordered. This issue should be monitored in the future - it seems as if the drive itself is perhaps causing the encoder boards to fail.

Note: upon powering the system on again, another issue was noticed: 500VFD shows "drive not ready" fault. This is not displayed on the HMI though - it was just not possible to reset the safety system and this was only discovered from going online with the PLC. This will be investigated further.

Update 2024-04-26: the 500VFD "drive not ready fault" was investigated. It was determined that the SP+ (safety power +) signal wire was loose, which meant the safety signals to the drive were not getting through, causing it to remain in a "not ready" state. The wiring issue was corrected. Performing a safety reset resulted in successful drive enables across all drives. However, upon attempting to move the crane, even though all safety signals were green and it appeared to be able to move, it did not.
It turned out that there was a crane pendant fault: "Fault 305. Radio Control Receiver in Fault (3700RC)". This is exactly what happened previously in a similar situation (see e-log 55). The steps mentioned in that e-log were followed (power cycle, reconnect antenna/connector). Upon powering back up, everything worked as normal. The crane was tested in local mode - all three hoists up/down (main hoist run to upper limit), all trolley travels in each direction, and bridge travel in each direction.

As of now, everything is operating normally and all issues are considered to be resolved. A spare encoder board has been ordered.