Powered down ARIEL Target Hall Crane today by switching off main power switch on B1 level in preparation for site-wide BC Hydro power outage on Dec. 27-29.

The non-existent safety limit switch for the target hall door which prevents remote motion has been bypassed in the crane PLC control panel to allow for remote operation of the crane. This will be un-bypassed when a switch is installed. Specifically, a jumper was placed from terminal 18001 to 20121 and 20141 which will enable the hall lock safety relays 2012SR and 2014SR. The crane is currently operational in both local and remote mode.

In preparing to troubleshoot the issue described in e-log #54, the crane was power cycled, and the pendant battery removed, to get to a "fresh start" state. Prior to this, the crane functioned normally aside from the incorrect E/W direction control on the North Aux Hoist.

Upon powering the crane back on, the pendant did not work to control the crane in any mode. The HMI had a fault present which read "305. Radio Control Receiver in Fault (3700RC)". This was accompanied by a red light in the safety control relay status summary screen, for "Radio Control Converter Communication Fault With Receiver (3714CONV)".

If the crane was placed in remote mode, the safety control relay status summary all turned green, and the reset button could be pressed, enabling remote mode. This means the issue was specific to the pendant/receiver communication specifically.

The receiver (in the cabinet in the control room) was inspected and the power status LED was on, the RF LED was off (which means there is no issue, OK), and the "OK to receive signal" LED was on - this indicates that the receiver was functioning as expected.

To further troubleshoot the issue - the crane was powered off, the receiver's antenna was disconnected and reconnected, and the multipin connector for the receiver was disconnected and reconnected. The system was powered back on, and the reset button was pressed. The fault message and the safety relay issue were cleared, and the pendant functioned again as normally expected (aside from the persistent issue from e-log #54).

It is unclear exactly what the issue was and how it was resolved, but could have been related to either a loose connector, or power cycling and resetting the system.

[edit 2023-01-03] The crane was powered back on this morning - there do not appear to be any faults on the HMI and it seems that the crane was unaffected by the power outage.

The ARIEL target hall overhead crane was powered off for the holidays. In the new year, throw the main disconnect in the crane control room to power back on.

An issue was observed in which the pendant did not work to control the crane. This is the second time in recent memory this has happened. See e-log 55 for details. The same sequence of steps outlined in e-log 55 was followed, and the pendant control was re-established. It is still unclear what the root cause of this issue is, but it is suspected that the crane pendant loses communication with the receiver for some reason, and even though communication can be re-established, the reset button on the crane control console does not clear the fault message, and the system must be power-cycled in order to fully reset. This is just speculation based on what is observed. Further investigation required.

Today while operating the crane with load, the crane stopped moving. Upon investigating, all safety signals appeared to be OK. Fault messages regarding drive faults for various VFDs were present, which seemed to stem from an ethernet adapter card fault (see attached photo). It is unknown how this occurred, but suspected that it was due to a brief blip in network communications or the external drive power supply control signal. It is worth noting that the diesel generator tests took place today, and there could have been some affect from a power surge because of this.

The crane was power-cycled using the main disconnect, and safety system reset. The faults disappeared. The crane was operational again.

If this issue occurs again, it should be investigated more thoroughly. It would be worth checking if there were any generator tests or other things which could cause some sort of power surge.

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The North Aux hoist Trolley was stuck in a position up against the East wall.

Present during troubleshooting: Adam N, Maico D, Tom K, Allon M, Travis C

After troubleshooting, it was determined that:

• There were no fault messages on the HMI display relating to this issue
• The HMI indicated "Bridge Not in North Limit and/or Auxiliary Trolley North Not in East or West Limit" - the Bridge was not in the north limit as described, and this message implies the PLC thought the North Aux hoist was somewhere along the middle of its travel
• The West run signal was reaching the remote receiver from the crane pendant
• The West and East run signals were correctly reaching their respective input pins on the VFD (DI1, DI2)... upon swapping wires and commanding it to run East, the West direction still did not function
• The VFD was displaying the correct output frequency information on its display when commanded to run East, but not west
• The limit switch was not related to this issue

Upon swapping out the VFD with the working one from the South Aux hoist, the North Aux hoist Trolley was able to move both East and West, as normal. It was therefore determined that the VFD had failed and must be replaced.

Current state of system as of July 3, 2020:

• The North Aux hoist is fully functioning (contains VFD from South one)
• The South Aux hoist is disabled (no VFD, breakers shut off)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Adam Newsome requested quote for replacement drive (Delta Electronics TDN007E1100WM0) - will advise when replacement arrives
• Install replacement drive into South Aux hoist upon arrival (note: check EMC level modification)
• Verify entire system functioning as normal
• Reinstall North Aux hoist limit switch flag

See E-Log 44 for details on North Aux Hoist failure.

A replacement VFD was ordered by Adam Newsome and installed by Tom Kauss.

Adam and Tom replaced the VFD in the South Aux hoist (which had been relocated to the North Aux hoist). The original braking resistor was used. No functionality changes to the system.

Both Aux hoists were tested: E/W travel, up/down travel. Each hoist is functioning correctly. E/W speeds were compared between the two hoists and found to be the same (~14 Hz for slow speed, ~95 Hz for fast speed).

The replacement job is complete and the issue outlined in E-Log 44 is resolved, although the original cause for failure has not been determined.

Current state of system as of Sept 9, 2020:

• The North Aux hoist is fully functioning (contains original VFD from South one)
• The South Aux hoist is fully functioning (contains newly replaced VFD with original braking resistor)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Reinstall North Aux hoist limit switch flag (TBD when Target Hall installation work is done)

The load cell which is located on the crane itself to display the load to operators was reported not working during the week of November 15. It is planned to investigate on Nov. 25 and this e-log will be updated with the status.

Edit (2022-02-02): the load cell is functioning again. No changes were made.

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The up/down motion was also functioning. The North Aux Hoist Trolley failed during regular usage, seemingly randomly, when it was about 70% towards the West end of travel. Note that the West motion VFD inside the North Aux Hoist is the one previously taken from the South Aux Hoist (the same issue with the North Aux Hoist was observed last year, and the VFD was swapped, and a new one was installed in the South Aux Hoist).

A replacement VFD (Delta TDN007E1100WM0) will be ordered and installed, and this e-log will be updated.

Edit 2022-03-15: the replacement VFD has been installed and the system is operational. See e-log 52.

The North Aux Hoist VFD which had failed (e-log 50) was replaced today by Tom Kauss and Adam Newsome on Work Permit A2022-03-15-1.

The hoist's motions were all tested - it is functioning normally again.

It was observed today that the crane's west light was not functioning.

There was an issue with a 20 A circuit breaker used to control motor assist functions for the telemanipulators (specifically, the rightmost CB in the panel for the right (East) manipulator set). This circuit breaker was moved to the panel for the left (West) manipulator set in the rightmost position, and then subsequently replaced with a new one, which is functioning correctly. It is suspected that the former CB which was not working was calibrated towards the low end from the factory, based on discussions with the manufacturer and extensive testing of multiple scenarios.

On August 1, 2023 HC1 was pumped down to -1200 Pa (gauge) and then valved off to test the leak rate after turntables had been sealed with backer rod in the gaps.

The sealing with backer rod allowed the cell to reach the test pressure of 1000 Pa. The cell pressure equalized within 30-40 seconds which is an improvement but still not sufficient to meet the goals we have set.

Next step is to pump down to -500 Pa and use smoke generator and perform thorough investigation to identify any leaks.

On 2024-04-22 around 3:30 pm it was discovered that the crane's main hoist was in a faulted state. The operator (Adam Newsome) experienced the fault after performing a safety reset and attempting to lower the hoist. There were no visible signs of any issues, mechanically. This fault had not been previously reported by any other crane operators. The crane was previously unused, sitting idle. Note: the auxiliary hoists appeared to function normally.

Fault messages on the HMI indicated:

"135. Main Hoist West Drum North Motor Drive Fault (1000VFD)"
"104. Main Hoist East Drum South Motor Drive Fault (800VFD)"
"103. Main Hoist East Drum North Motor Drive Fault (700VFD)"
"138. Main Hoist West Drum South Motor Drive Not Ready (900VFD)"
"136. Main Hoist West Drum South Motor Drive Fault (900VFD)"

It is clear from this that there is either some sort of common issue seen across all these main hoist VFDs, or one issue with one of them which caused a cascading series of faults.

On 2024-04-23, upon investigation of these faults by going online with the PLC to determine fault logic, it was clear that all faults were indeed present, but there was no obvious indication as to what it was. Inspection of the VFDs in the control panel showed that all of the aforementioned drives displayed a fault code. This was fault code 4 which indicates DC bus undervoltage.

After researching this fault online, it appears this is typically caused by an issue with the input mains supply (480VAC @ 3 phase in this case), or by the drive's input DC filter/buffer circuitry. When measuring the DC bus voltage, it is expected for it to be approximately 1.414 times the AC supply voltage. In this case, 1.414*480 = 678 VDC. The first troubleshooting step was to measure to confirm the DC bus voltage on a known working and not working drive.

The bus voltage was probed on drive 500VFD which, based on lack of fault message, was expected to be functioning properly. With the safety off due to E-stop condition, the voltage was nearly zero. When a safety reset was pressed to enable the drive, the voltage changed to 690 VDC. This is close enough to the expected 678 VDC. The voltage did not change, even when the fault message for the other drives appeared again.

Next, the bus voltage on two known faulted drives, 900VFD and 1000VFD, was probed. In both cases, after the safety reset, the bus voltage started at 690 VDC but then slowly dropped down towards nearly zero (somewhere around 18 V) over a period of 3-5 seconds. It was during this transition when the voltage dropped off that the undervoltage condition became true, which triggered the fault.

As of 2024-04-24, the root cause of this issue is unknown. It will be investigated further and this e-log will be updated when a solution is found.

Update 2024-04-24: the root cause has been identified. The undervoltage fault was just a symptom as a result of the input contactors for these drives switching off, causing the DC bus voltage to drop slowly due to capacitance in the input filtering circuit. The reason for the input contactors switching off is attributed to safety signals dropping out due a fault observed specifically on 900VFD. When looking at the drive itself, a fault with code 4030 was displayed: Enc1 open wire.
This fault implies the encoder may be disconnected. The encoder wiring was checked: OK. The encoder cable was swapped with a unknown working one from an adjacent drive - the issue remained with 900VFD, which suggested the problem is with the drive itself. The encoder board (20-750-DENC-1) was swapped with an adjacent drive and the problem followed the board. The encoder board was then switched with a brand new spare (note: jumpers needed to be set!).
Upon power-cycling the system, the fault did not persist. Therefore, it is suspected that the encoder board had failed (in fact this happened previously with the same drive - see e-log 42). Upon inspection, one of the capacitors on the board appeared to be cracked - this will be investigated further to see if replacing it fixes the problem.

A spare encoder board will be ordered. This issue should be monitored in the future - it seems as if the drive itself is perhaps causing the encoder boards to fail.

Note: upon powering the system on again, another issue was noticed: 500VFD shows "drive not ready" fault. This is not displayed on the HMI though - it was just not possible to reset the safety system and this was only discovered from going online with the PLC. This will be investigated further.

Update 2024-04-26: the 500VFD "drive not ready fault" was investigated. It was determined that the SP+ (safety power +) signal wire was loose, which meant the safety signals to the drive were not getting through, causing it to remain in a "not ready" state. The wiring issue was corrected. Performing a safety reset resulted in successful drive enables across all drives. However, upon attempting to move the crane, even though all safety signals were green and it appeared to be able to move, it did not.
It turned out that there was a crane pendant fault: "Fault 305. Radio Control Receiver in Fault (3700RC)". This is exactly what happened previously in a similar situation (see e-log 55). The steps mentioned in that e-log were followed (power cycle, reconnect antenna/connector). Upon powering back up, everything worked as normal. The crane was tested in local mode - all three hoists up/down (main hoist run to upper limit), all trolley travels in each direction, and bridge travel in each direction.

As of now, everything is operating normally and all issues are considered to be resolved. A spare encoder board has been ordered.

Tuesday, November 04, 2014, 16:58

This is a test.

Two small (approx. 5 cm across) puddles of green fluid were discovered on the floor, directly below the crane trolley.  The hook block was lowered and inspected.  No fluid found on hook block.
The plant group (Neil Wong) was notified and asked to schedule an inspection when the maintenance crew is on site to inspect the ISAC target hall crane.

The crane bridge and trolley were moved to another position on Friday evening and parked over the weekend.  The floor directly below the crane trolley was inspected on Monday morning.  Three very small (1-3cm diameter) puddles of green fluid were found.