This morning Bill Richert pressurized the beam dump water cooling system with 30psi air and held for 2 hours.  He reported no change in pressure which indicates the flexible hose connections installed last week are leak tight.

On December 2 the upper half of the RH water blocks were installed (4 total) using a 0.030" thick 6061-T6 Al shim torqued to 60ft*lbs as was tested in Document-114260.

Bypass loops for testing the cooling system piping were installed as shown in the attached photo.

 The upper lead shield was installed this morning.  Also the water hoses for lower lead shield, beam dump, and upper lead shield were all installed.  See attached photos and hose installation information PDF.

Tuesday, November 04, 2014, 16:58

This is a test.

The following work was completed today on the EHD water leak sensors:

• Mike Vogel installed a 1/4" copper tube which leads from the east side of the EHDT shielding down to the catch tray below the water blocks.  This can be used for future testing of the in-tray sensors without removing the shield blocks.  It may also be possible to test the other sensors if enough water is poured into the tray for it to overflow and eventually get them wet.  The routing of the copper tube is not expected to interfere with the vertical shield plug
• Ray Mendoza re-located the SMT circuit boards to a rack on the north wall of the E-Hall, east of the concrete shield wall.  The push-connect terminal on one board was damaged, so there is currently no electronics installed for EHD:LEAK06  (After various debugging over the last couple weeks it was eventually determined that the previous problems with these sensors was caused by a shared power source for the relays and the sensors themselves.  Ray installed a dedicated 9V power supply for the sensors, with 24V supply to the relays still coming from the PLC)
• We swapped the wiring of two of the sensors so that LEAK01 and LEKA02 are the strips located in the tray as specified in interlock spec Document-112445 (see below for details of final installation)
  • Inside tray east side sensor is cable #4, wired to EHD:LEAK01
  • Inside tray west side sensor is cable #2, wired to EHD:LEAK02
  • Outside tray east side sensor is cable #3, wired to EHD:LEAK03
  • Outside tray west side sensor is cable #1, wired to EHD:LEAK04
• After the wiring was completed we shorted across each sensor in the junction box near the beam dump.  EHD:LEAK01 through LEAK05 responded as expected

• Water was poured into the 1/4" copper tube to test the sensor strips in the tray.  After approximately 50mL sensor #2 triggered, and #1 triggered after approximately 100mL total.  Both sensors were still in alarm 5 minutes later.

• Water was applied to sensor #3 (outside tray, east side) using a snake camera with plastic tube attached.  Approximately 50mL was applied;  The sensor triggered once after a small amount of water reached the strip, came out of alarm, then went back into alarm when more water was applied.  It stayed in alarm for approximately 1 minute then cleared

• EHD:LEAK04 (outside tray west side) was triggered by pouring water onto the ground in the general area of the sensor (direct application with the snake camera was not possible). 125mL was initially poured which did not trigger the sensor.  An additional 125mL was applied which triggered it.  More water was added to the LEAK03 sensor to confirm that the system behaves properly with all 4 sensors in alarm

•  Air was forced down towards the sensors using a shop-vac hooked up in reverse.  LEAK03 and LEAK04 cleared within a couple minutes.  LEAK01 and LEAK02 cleared in approximately 2 hours.

On 2024-04-22 around 3:30 pm it was discovered that the crane's main hoist was in a faulted state. The operator (Adam Newsome) experienced the fault after performing a safety reset and attempting to lower the hoist. There were no visible signs of any issues, mechanically. This fault had not been previously reported by any other crane operators. The crane was previously unused, sitting idle. Note: the auxiliary hoists appeared to function normally.

Fault messages on the HMI indicated:

"135. Main Hoist West Drum North Motor Drive Fault (1000VFD)"
"104. Main Hoist East Drum South Motor Drive Fault (800VFD)"
"103. Main Hoist East Drum North Motor Drive Fault (700VFD)"
"138. Main Hoist West Drum South Motor Drive Not Ready (900VFD)"
"136. Main Hoist West Drum South Motor Drive Fault (900VFD)"

It is clear from this that there is either some sort of common issue seen across all these main hoist VFDs, or one issue with one of them which caused a cascading series of faults.

On 2024-04-23, upon investigation of these faults by going online with the PLC to determine fault logic, it was clear that all faults were indeed present, but there was no obvious indication as to what it was. Inspection of the VFDs in the control panel showed that all of the aforementioned drives displayed a fault code. This was fault code 4 which indicates DC bus undervoltage.

After researching this fault online, it appears this is typically caused by an issue with the input mains supply (480VAC @ 3 phase in this case), or by the drive's input DC filter/buffer circuitry. When measuring the DC bus voltage, it is expected for it to be approximately 1.414 times the AC supply voltage. In this case, 1.414*480 = 678 VDC. The first troubleshooting step was to measure to confirm the DC bus voltage on a known working and not working drive.

The bus voltage was probed on drive 500VFD which, based on lack of fault message, was expected to be functioning properly. With the safety off due to E-stop condition, the voltage was nearly zero. When a safety reset was pressed to enable the drive, the voltage changed to 690 VDC. This is close enough to the expected 678 VDC. The voltage did not change, even when the fault message for the other drives appeared again.

Next, the bus voltage on two known faulted drives, 900VFD and 1000VFD, was probed. In both cases, after the safety reset, the bus voltage started at 690 VDC but then slowly dropped down towards nearly zero (somewhere around 18 V) over a period of 3-5 seconds. It was during this transition when the voltage dropped off that the undervoltage condition became true, which triggered the fault.

As of 2024-04-24, the root cause of this issue is unknown. It will be investigated further and this e-log will be updated when a solution is found.

Update 2024-04-24: the root cause has been identified. The undervoltage fault was just a symptom as a result of the input contactors for these drives switching off, causing the DC bus voltage to drop slowly due to capacitance in the input filtering circuit. The reason for the input contactors switching off is attributed to safety signals dropping out due a fault observed specifically on 900VFD. When looking at the drive itself, a fault with code 4030 was displayed: Enc1 open wire.
This fault implies the encoder may be disconnected. The encoder wiring was checked: OK. The encoder cable was swapped with a unknown working one from an adjacent drive - the issue remained with 900VFD, which suggested the problem is with the drive itself. The encoder board (20-750-DENC-1) was swapped with an adjacent drive and the problem followed the board. The encoder board was then switched with a brand new spare (note: jumpers needed to be set!).
Upon power-cycling the system, the fault did not persist. Therefore, it is suspected that the encoder board had failed (in fact this happened previously with the same drive - see e-log 42). Upon inspection, one of the capacitors on the board appeared to be cracked - this will be investigated further to see if replacing it fixes the problem.

A spare encoder board will be ordered. This issue should be monitored in the future - it seems as if the drive itself is perhaps causing the encoder boards to fail.

Note: upon powering the system on again, another issue was noticed: 500VFD shows "drive not ready" fault. This is not displayed on the HMI though - it was just not possible to reset the safety system and this was only discovered from going online with the PLC. This will be investigated further.

Update 2024-04-26: the 500VFD "drive not ready fault" was investigated. It was determined that the SP+ (safety power +) signal wire was loose, which meant the safety signals to the drive were not getting through, causing it to remain in a "not ready" state. The wiring issue was corrected. Performing a safety reset resulted in successful drive enables across all drives. However, upon attempting to move the crane, even though all safety signals were green and it appeared to be able to move, it did not.
It turned out that there was a crane pendant fault: "Fault 305. Radio Control Receiver in Fault (3700RC)". This is exactly what happened previously in a similar situation (see e-log 55). The steps mentioned in that e-log were followed (power cycle, reconnect antenna/connector). Upon powering back up, everything worked as normal. The crane was tested in local mode - all three hoists up/down (main hoist run to upper limit), all trolley travels in each direction, and bridge travel in each direction.

As of now, everything is operating normally and all issues are considered to be resolved. A spare encoder board has been ordered.

The main trolley's E/W motion was observed to be slowed on 2024-02-13.
Upon investigation, the following faults were present:
- 200. Rotating Hook Velocity Difference between SSI Encoder and Drive Speed Feedback
- 82. Main Hoist Slack Rope

It is suspected that these faults are irrelevant in this situation. Upon inspection of the main trolley status screen, it was evident that both the SLOW EAST and SLOW WEST indicators were on (which matches observations). Clicking one of these status icons showed "Main Trolley Encoder Not Homed 2253ENC".

Logging in as an administrator allowed for homing the encoder.

The trolley was moved all the way East, to the limit, and the encoder was homed at position 0.389m (encoder count = 6886940). Note: the alignment arrows were not used as this would have required gaining access to the crane which was difficult at the time. It is suggested to properly home it in the future.

The E/W motions were fully tested to confirm proper slow down and stopping at the expected positions. The crane appears to be operational as of 2024-02-14.

 

It was observed recently that the main hoist's wires seemed skewed, and there was occasionally some noise heard when raising and lowering the hook block. It was suspected that this was due to wires having jumped in their tracks at some point. This is typically caused by side loading. It is possible that wire wrap skew occurred during the incident outlined in e-log 46. 

Gordon Crane was contacted for a service call. Crane inspector Ali from Gordon Crane was supervised/assisted by Maico Dalla Valle. They lowered the hoist down into the target pit and supported it on saw horses, then let the wires run all the way out. Ali confirmed that there was a wire wrapping issue, and re-seated the wires properly. The crane is now functioning normally. It is recommended to re-apply lubrication as soon as possible.  

Note: Ali from Gordon Crane has advised that it is acceptable for some degree of wire skew to occur - they can handle the abrasion if there's a small angle between them. However, wires should not be rubbing across each other at a large angle (ex. more than approx. 10 degrees). 

When the EHDT shielding service stand shield plug was installed in January it was found that the Beam Dump Outlet Hose (TEL5079) was interfering with the plug.  The following remedial action was taken:

- Upper lead water blocks disconnected and upper lead shield removed w/ hose assemblies and upper half of water blocks attached
- BD hose assembly TEL5079 removed by disconnecting Swagelok fittings
- A 1-1/8" spacer was added between the ceramic break and the 45 degree elbow
- The finished assembly was helium leak checked:  no leaks
- Final dimensions of the hose assembly were recorded for drawing updates (Isaac's notebook)

- All water blocks were disconnected (previous install used gaskets only w/o retainers so that a leak check could be performed)
- New gaskets manufacutred by Dan McDonald on Feb 2 (TEL4263)
- Gaskets cleaned w/ acetone and put into retainers (nice fit)
- Sealing surfaces on lower and upper half of BD water blocks cleaned with acetone, gaskets, installed, and torqued to 60ft*lbs w/ torque wrench and crows foot wrench
- TEL5079 reinstalled.  Swagelok nuts tightened to same rotational position as initial install
- Upper lead shield reinstalled (Feb 3)
- Upper lead water block sealing surfaces cleaned, blocks installed and torrqued to 60ft*lbs as w/ BD blocks

Before installation of upper lead water blocks I attempted to move, install, and uninstall the upper half of the water blocks using a 4' pole tool.  The blocks were fairly hard to manipulate due to the stiffness of the 1" hose.  The original design was to use a 1/2" hose, but we switched to 1" hose during a very rushed part of the project because of shorter lead times for 1" hoses.

As there is currently no beam planned for the next 4-6 weeks I will investigate replacing the upper lead shield and beam dump flex hoses with smaller diameter more flexible hoses.
 

The position limit modification described in e-log 49 has been reverted. Currently the South soft limit is set back to 0.36m.

The limit was tested and confirmed (craned stopped at 0.33m position which is acceptable due to coasting). The entire bridge range of motion was checked to ensure there were no adverse effects - all is OK. The South bumpers need to be reinstalled - this will take place in the coming weeks.

Changes noted in crane log book.

Relevant work permit: A2022-05-30-3

Edit (2022-06-08): the bumpers have been re-installed.

 In August 2014 gasket corrosion tests were prepared by placing a 1" Swagelok VCR gasket in a 500mL beaker with de-ionized water and a 1"x3/4"x1/8" piece of 6061 aluminum.  Tests were prepared for a silver plated 316L SS gasket (Swagelok PN: SS-16-VCR-2), a silver plated nickel gasket (PN: NI-16-VCR-2), and a TRIUMF made 0.060" thick 6061 Al gasket.  The top of the beakers were sealed, and they were left to sit until March 2016.

The Nickel and SS gaskets show no noticeable corrosion.  The aluminum gasket became dull in colour, with some corrosion visible on the surface.  See attached photos of each gasket.  The gaskets have been bagged and stored with beam dump spare parts in the NW corner of the RH Meson Hall hot cell lab.

The non-existent safety limit switch for the target hall door which prevents remote motion has been bypassed in the crane PLC control panel to allow for remote operation of the crane. This will be un-bypassed when a switch is installed. Specifically, a jumper was placed from terminal 18001 to 20121 and 20141 which will enable the hall lock safety relays 2012SR and 2014SR. The crane is currently operational in both local and remote mode.

The location of the already installed blocks including the TSH0392 protruding lift bails were remeasured, and it was found that modification of the Standard Lock-Block Flat Top block was not required.  A Standard Lock-Block Flat Top block was installed in the place of TSH0487 in drawing TSH0372, and the remainder of the blocks on top were installed.  It was a tight fit for the lower two Half Height Lock-Blocks, but future removal of the BD shielding and SS plug should still be possible.  Drawing TSH0372 will be updated to reflect as-built status.

The remote handling 4" marman flange clamp was reinstalled today with a used seal.  It is "Clamp B" as tested in Document-114623.  The purpose of installing the clamp now is to prepare for the upcoming remote handling test of the vacuum joint and one of the water blocks.

During installation it was found that it will be necessary to manipulate both the BD flange and the EHDT beam pipe flange to have the seal mated on both of them.  Clamp installation will therefore be a 2 or 3 person job when done remotely.  The BD can be moved upstream by pushing on the back of the clamp mounting collar.  The EHDT beam pipe can be moved downstream by threading a pole tool into the collar just upstream of the vacuum flange.

If the clamp becomes stiff during early travel it means that the flanges aren't seated in the jaws properly and the position of the clamp or flanges must be adjusted.  Do not back off the clamp drive nut or lift the clamp in this case.  This procedure was followed during today's installation.

 A 4" Remote Handling Marman Flange Clamp was delivered to Doug Preddy today to be used to clamp the EHDT beam pipe section to the beam dump during beam pipe installation.

The clamp is one of the two manufactured in 2014, Identified with marker "B"

 Powered down ARIEL Target Hall Crane today by switching off main power switch on B1 level in preparation for site-wide BC Hydro power outage on Dec. 27-29.

Naimat Khan noticed after using the crane that some grease dropped onto a spreader bar, and believes it is a leak from the main hoist (see attached photo). The hoist should be inspected.

Edit: Dan McDonald notes that this has been observed since the crane was installed and is likely due to excess grease dropping out. Perhaps not a major concern.

Two small (approx. 5 cm across) puddles of green fluid were discovered on the floor, directly below the crane trolley.  The hook block was lowered and inspected.  No fluid found on hook block.
The plant group (Neil Wong) was notified and asked to schedule an inspection when the maintenance crew is on site to inspect the ISAC target hall crane.

The crane bridge and trolley were moved to another position on Friday evening and parked over the weekend.  The floor directly below the crane trolley was inspected on Monday morning.  Three very small (1-3cm diameter) puddles of green fluid were found.

An issue was observed in which the pendant did not work to control the crane. This is the second time in recent memory this has happened. See e-log 55 for details. The same sequence of steps outlined in e-log 55 was followed, and the pendant control was re-established. It is still unclear what the root cause of this issue is, but it is suspected that the crane pendant loses communication with the receiver for some reason, and even though communication can be re-established, the reset button on the crane control console does not clear the fault message, and the system must be power-cycled in order to fully reset. This is just speculation based on what is observed. Further investigation required.