ELOG RH-ISAC

E-mail correspondence on the Target Hall crane fault diagnosis between October 11th and 15th:

Hello all,

Mike Smith from Norelco arrived this morning and spent some time with Travis from RH and Mike Rowe from Controls group examining the crane controls and electrical schematics. A common understanding of the wiring and control signal scheme for the thermal sensor circuits was reached. A scanned copy of Norelco / Seattle Crane drawing E5 - 1/1 with markups by Dave Morris (Oct 11th) and Mike Smith (Oct 15th) is attached.

The group then worked on top of the crane bridge to systematically isolate the thermistor circuits in each drive motor. The thermistor readings were all normal / as-expected (about 180 Ohms for 3 in series), i.e. there was no indication of failure in any of the motor thermistors. Fall arrest equipment was used to access the motors where required.

Next, the shunt circuit in each temperature sensor module was measured (terminals T1 and T2 on each module, on drawing E5 - 1/1). The shunt in the operational module was measured at 500 kOhms. The shunt in the suspected faulty module was measured at 79 kOhms.

It was concluded that there is most likely a fault in the module with the low shunt resistance. The suspect module was left wired in bypass (shorted from terminals 14 to 11). Mike Smith advised at that point that it would be safe to operate the crane in this mode temporarily until a replacement module can be installed. I discussed this with him and Don Dale, and we agreed the risk is low, as we have not ever experienced a motor overheat scenario in the operating lifetime of the crane.

Mike Smith called me around 1pm to advise that he has located a replacement unit from their supplier in the US and I asked him to order 2 units on expedited overnight. Mike expects the components to arrive Friday this week or Monday next week latest, factoring in delays at customs and local delivery times. The units will be sent to Norelco first, then to TRIUMF locally.

Remote Handling is now continuing with Target Hall operations, using the crane as normal with the thermal module bypassed until the replacement module arrives.

Thanks to Dave Morris / Mike Lowe for their assistance in diagnosing this fault.

Here is a short summary of the Fault:

Cause of fault
- Most likely a failed thermal sensor module for one of the two pairs of bridge drive motors

Diagnosis
- Crane company (Norelco) consulted, with on-site inspections by tech Mike Smith on October 9th and October 15th
- Controls group / Mike Smith measured resistances in motor thermistors and sensors to systematically isolate fault to thermal sensor

Remedy actions taken
- Thermal sensor bypassed to temporarily restore crane function
- Risk of operating without sensor determined to be low through discussions with Norelco / Don Dale / Grant Minor
- Replacement thermal sensor + 1 spare (~ $300 ea + shipping) ordered from parts supplier through Norelco - expected delivery Monday October 21st

How could this fault have been prevented?
- Cause of failure within thermal sensor module unknown at this time
- Annual crane inspection performed by Norelco in March 2013 did not indicate any problems with this sensor (crane drive system functioned as normal)
- Suggested for future: routine inspection of sensor module shunt resistance to determine if there are any long-term signs of slow degradation

What can be done to prevent the fault in the future?
- The ISAC crane controls were designed in the late 1990's by Norelco sub-contractor Seattle Crane (now defunct)
- The crane is supposed to have two pairs of independent motors with separate thermal sensors, each capable of driving the crane if the other fails due to thermal trip
- My understanding is that the controls logic design should have been designed so that failure of one pair of motors does not prevent crane operation... it does not seem to be the case with the current design
- The design of the control logic related to these sensors could be re-evaluated and modified so that a trip from one thermal module does not prevent operation of the bridge if the other module is OK
- This is a project that will require some time and some controls and engineering resources
- In the interim, if the same failure occurs again, it may be possible to jumper the thermal sensor from the crane control room and continue operating normally - I will confirm with controls group / Norelco
- However, this is not advisable as it would be difficult in some cases to determine if the trip was due to a faulty thermal module, or an actual motor overheat scenario
- I will discuss further actions with Don Dale

Thanks and best regards,

Grant

On 11/10/2013 4:29 PM, Grant Minor wrote:

Hello all,

I spoke with the Norelco tech Mike Smith this afternoon regarding the failed thermal sensor component / circuit. He will call Kone on holiday Monday (their office is not closed for Canadian Thanksgiving) and determine if they have a replacement module.

I explained (to the best of my understanding) the diagnosis performed by Dave Morris / controls group and the test that was done to isolate the thermal sensor unit. Unfortunately, Mike Smith's drawings from Seattle crane contain notes and revisions that don't match the drawing that Dave Morris provided me (Seattle Crane dwg E5 - 1/1 revision 1 02-10-99), and he was unable to confirm in his opinion whether the bypass performed by Dave Morris has correctly isolated the problem components. In addition, he believes that the bypass he performed on Wednesday isolated portions of circuitry that are inconsistent with this thermal sensor circuit.

Mike Smith will return Tuesday morning to review the bypass and schematics. He recommended not operating the crane until he has a chance to confirm this thermal sensor issue.

From the drawings I have in hand and from discussions with Dave Morris, the failed component might be the thermistor sets inside the motor housing which are in series with the suspect module, or the module itself. My understanding is that the crane is designed with two pairs of motors for the bridge drive, and each pair is protected by one of these thermal modules (thus there are two modules total). Both modules would be the same age and both might be susceptible to the same failure mode.

As it is unclear to me the exact nature of the drive system failure, and I have a recommendation from Norelco to hold on operation of the crane, I must unfortunately state that the crane should not be used and we will have to wait until Tuesday to gather more information.

I am not so concerned about temporary loss of the thermal protection circuitry, as we have not had an overheating scenario in those motors in the lifetime of the crane (to my knowledge). I am more concerned about getting 100% confirmation that we have addressed the problem. If we attempt to use the crane and experience another failure during a lift that prevents us from placing the load down safely with the crane stuck in position over a target station, then we would be in much deeper trouble.

My apologies for this situation.

Best regards and have a good weekend,

Grant

On 11/10/2013 1:28 PM, Grant Minor wrote:
Thanks Dave,

I've contacted Norelco, Mike Smith and I are planning a path forward for further diagnosis and repair.

The crane MUST NOT be used to carry a load until it is identified whether the module or the sensors have failed, the reason for the failure, and the functionality of the thermal sensors has been restored.

Without the thermal sensor circuit we lose protection from motor overheat, an event which could have much more serious failure implications to the crane.

Cheers,

Grant

On 11/10/2013 1:04 PM, David Morris wrote:
The fault has been identified in the Target Hall Crane as a failed motor temperature module, or motor temperature sensor, on the East-West motion. The status contact in the module was bypassed allowing crane motion. There was no fault with the controls.

Dave