Powered down ARIEL Target Hall Crane today by switching off main power switch on B1 level in preparation for site-wide BC Hydro power outage on Dec. 27-29.

Naimat Khan noticed after using the crane that some grease dropped onto a spreader bar, and believes it is a leak from the main hoist (see attached photo). The hoist should be inspected.

Edit: Dan McDonald notes that this has been observed since the crane was installed and is likely due to excess grease dropping out. Perhaps not a major concern.

Two small (approx. 5 cm across) puddles of green fluid were discovered on the floor, directly below the crane trolley.  The hook block was lowered and inspected.  No fluid found on hook block.
The plant group (Neil Wong) was notified and asked to schedule an inspection when the maintenance crew is on site to inspect the ISAC target hall crane.

The crane bridge and trolley were moved to another position on Friday evening and parked over the weekend.  The floor directly below the crane trolley was inspected on Monday morning.  Three very small (1-3cm diameter) puddles of green fluid were found.

December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam 

Tests:

Gas lines removal: 

• Starting with the most exterior connection, VCR connection loosened off with open ended conventional wrench
• Once loose, the nut can be un threaded with manipulator finger (rolling nut technique) 
• Once gas line unhooked, gasket removed by bringing female end outside the service tray footprint and jiggling until the gasket was removed
  • Also possible to use a pick if needed
• The same procedure was conducted for the interior VCR connection

Gas lines installation: 

• Starting with the inner most connection
• Gasket placed on 3D printed tool and clipped into position on VCR male end
• At first an M10 bolt was inserted into the elbow below VCR connection, but without rotational authority, the makeshift handle is not worth using
• Griping the elbow with one manipulator and rolling the nut onto the threads with the other proved successful
• Nut was tightened using conventional open ended 19mm wrench

Observations/Notes:

• Service Tray was not in a fully connected position, so even less space will be afforded.
  • This may affect the ability to get 2 manipulators on the same connection
• Piston modules were missing some limit wires on the side. These constrain movement horizontally and will either need to be removed as part of the procedure or, make the procedure more difficult
• Mass markers and various other connectors not installed, and these could slightly restrict movement as well
• Service tray pin was restricting the movement of the left manipulator during install
• Lighting was inadequate under the piston 

Recommendations/follow up items/questions:

• All metal Parker VGR style gaskets to replace plastic retainer versions
  • This is so that the degraded plastic doesn't break off and end up in the gas lines (upside down connections)
• A modified wrench with flats for handles and with more length would make the above procedures easier 
• Along with raising and lower the service tray, the service tray pin being oriented towards the beam entry direction, would make life easier 
• Still need to test gasket install on "other" Piston gas line connection
  • Chad will redesign some new gasket tools to be tested
    • side load and axial load gaskets, low and high clearance, aluminum construction 
• Lighting positioned to flood the service tray area will be needed
• labeling the gas lines with a more permanent solution will be needed
• Once the high voltage feed-through parts have come in, we can re-test and see if the piston module can be lowered to aid in target removal situations
• Aaron will test Parker style seals to see if they can be removed as easily once brought up to specified torque 

------------------------------------------------------------------------------------------------------------------------------------------------------------------

 December 16, 2024

• Upon inspecting the piston modules on AETE in TISA, we became aware of how the wires are arranged for the limit switches, it may be a good idea to look into how these wires are handled when using the piston module jig.

An issue was observed in which the pendant did not work to control the crane. This is the second time in recent memory this has happened. See e-log 55 for details. The same sequence of steps outlined in e-log 55 was followed, and the pendant control was re-established. It is still unclear what the root cause of this issue is, but it is suspected that the crane pendant loses communication with the receiver for some reason, and even though communication can be re-established, the reset button on the crane control console does not clear the fault message, and the system must be power-cycled in order to fully reset. This is just speculation based on what is observed. Further investigation required.

In preparing to troubleshoot the issue described in e-log #54, the crane was power cycled, and the pendant battery removed, to get to a "fresh start" state. Prior to this, the crane functioned normally aside from the incorrect E/W direction control on the North Aux Hoist.

Upon powering the crane back on, the pendant did not work to control the crane in any mode. The HMI had a fault present which read "305. Radio Control Receiver in Fault (3700RC)". This was accompanied by a red light in the safety control relay status summary screen, for "Radio Control Converter Communication Fault With Receiver (3714CONV)".

If the crane was placed in remote mode, the safety control relay status summary all turned green, and the reset button could be pressed, enabling remote mode. This means the issue was specific to the pendant/receiver communication specifically.

The receiver (in the cabinet in the control room) was inspected and the power status LED was on, the RF LED was off (which means there is no issue, OK), and the "OK to receive signal" LED was on - this indicates that the receiver was functioning as expected.

To further troubleshoot the issue - the crane was powered off, the receiver's antenna was disconnected and reconnected, and the multipin connector for the receiver was disconnected and reconnected. The system was powered back on, and the reset button was pressed. The fault message and the safety relay issue were cleared, and the pendant functioned again as normally expected (aside from the persistent issue from e-log #54).

It is unclear exactly what the issue was and how it was resolved, but could have been related to either a loose connector, or power cycling and resetting the system.

[edit 2023-01-03] The crane was powered back on this morning - there do not appear to be any faults on the HMI and it seems that the crane was unaffected by the power outage.

The ARIEL target hall overhead crane was powered off for the holidays. In the new year, throw the main disconnect in the crane control room to power back on.

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The North Aux hoist Trolley was stuck in a position up against the East wall.

Present during troubleshooting: Adam N, Maico D, Tom K, Allon M, Travis C

After troubleshooting, it was determined that:

• There were no fault messages on the HMI display relating to this issue
• The HMI indicated "Bridge Not in North Limit and/or Auxiliary Trolley North Not in East or West Limit" - the Bridge was not in the north limit as described, and this message implies the PLC thought the North Aux hoist was somewhere along the middle of its travel
• The West run signal was reaching the remote receiver from the crane pendant
• The West and East run signals were correctly reaching their respective input pins on the VFD (DI1, DI2)... upon swapping wires and commanding it to run East, the West direction still did not function
• The VFD was displaying the correct output frequency information on its display when commanded to run East, but not west
• The limit switch was not related to this issue

Upon swapping out the VFD with the working one from the South Aux hoist, the North Aux hoist Trolley was able to move both East and West, as normal. It was therefore determined that the VFD had failed and must be replaced.

Current state of system as of July 3, 2020:

• The North Aux hoist is fully functioning (contains VFD from South one)
• The South Aux hoist is disabled (no VFD, breakers shut off)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Adam Newsome requested quote for replacement drive (Delta Electronics TDN007E1100WM0) - will advise when replacement arrives
• Install replacement drive into South Aux hoist upon arrival (note: check EMC level modification)
• Verify entire system functioning as normal
• Reinstall North Aux hoist limit switch flag

The North Aux Hoist VFD which had failed (e-log 50) was replaced today by Tom Kauss and Adam Newsome on Work Permit A2022-03-15-1.

The hoist's motions were all tested - it is functioning normally again.

See E-Log 44 for details on North Aux Hoist failure.

A replacement VFD was ordered by Adam Newsome and installed by Tom Kauss.

Adam and Tom replaced the VFD in the South Aux hoist (which had been relocated to the North Aux hoist). The original braking resistor was used. No functionality changes to the system.

Both Aux hoists were tested: E/W travel, up/down travel. Each hoist is functioning correctly. E/W speeds were compared between the two hoists and found to be the same (~14 Hz for slow speed, ~95 Hz for fast speed).

The replacement job is complete and the issue outlined in E-Log 44 is resolved, although the original cause for failure has not been determined.

Current state of system as of Sept 9, 2020:

• The North Aux hoist is fully functioning (contains original VFD from South one)
• The South Aux hoist is fully functioning (contains newly replaced VFD with original braking resistor)
• The North Aux hoist limit switch flag has been temporarily bypassed to allow for crane work in the target hall (not a safety issue - extended bumpers are installed and the regular bridge limit switches are functioning normally... this is only to allow the bridge to run its full length of travel when the North Aux hoist is at its fully East position)

Next steps:

• Reinstall North Aux hoist limit switch flag (TBD when Target Hall installation work is done)

Update 2022-10-06: During routine inspection of the crane, Tom Kauss investigated the direction swap issue. It was determined that there was likely an issue with the initial installation of the new VFD back in March 2022, and that it was unnoticed (directional control wires crossed). The issue has been resolved by Tom Kauss and is no longer a concern. 

It was observed during operation today (Aug. 23, 2022) that the motion axes were reversed for East/West travel on the North Aux Hoist. i.e. when pressing the joystick on the local pendant in the East direction, the hoist traveled West, and vice-versa. This behaviour was not noticed (or at least not reported) before. It could potentially be related to the recent replacement which was done in March 2022 (https://elog.triumf.ca/TIS/RH-ARIEL/52), although the motion was checked after replacement. It is possible that an oversight was made in the testing at that time though.

The controls should be checked and the issue should be rectified as soon as possible. The Aux Hoist controls are not run through the PLC, but rather handled directly within the hoist's control box. It is recommended to go online with the PLC and check if there is any hint as to why this is occurring, before making any modifications. If no info can be gained, it is recommended to try swapping the VFDs for East and West motion, and check if that rectifies the issue. It is possible that the VFDs are programmed specifically for their direction output, and they simply need to be swapped. 

An issue was observed in which the North Aux hoist failed to travel West when commanded to, but the East motion was functioning. The up/down motion was also functioning. The North Aux Hoist Trolley failed during regular usage, seemingly randomly, when it was about 70% towards the West end of travel. Note that the West motion VFD inside the North Aux Hoist is the one previously taken from the South Aux Hoist (the same issue with the North Aux Hoist was observed last year, and the VFD was swapped, and a new one was installed in the South Aux Hoist).

A replacement VFD (Delta TDN007E1100WM0) will be ordered and installed, and this e-log will be updated.

Edit 2022-03-15: the replacement VFD has been installed and the system is operational. See e-log 52.

Edit: as of May 30, 2022, the modification has been reversed, and the crane is functioning normally as per the manufacturer's limits. 

Modifications have been made to the crane to allow for the South Aux Hoist to reach closer to the South wall during the ARIEL Hot Cell installation period. These modifications are temporary, and were considered to be a safer and simpler alternative to the modification proposed by the hot cell contractor workers, which involved installing a sort of come-along on the wall and pulling the hoist closer to obtain the desired reach. The following modifications were made:

1. Removal of the bumpers on the south side (T. Kauss). There are no bumpers on the South side of the crane’s bridge currently.

2. Modification of the PLC program to reduce the software limit by approximately 4 inches (A. Newsome). The previous soft limit was set at 0.36 m and it was reduced to 0.26 m (~3.9 inch reduction).

 These changes have been tested – the crane stops in the expected position. However, it is very close to the wall now. If you are using the crane anywhere near the South wall: exercise extreme caution. The crane automatically slows down when approaching the South wall, and there is plenty of time for the operator to react and further slow the motion by only holding the joystick down at about 20% speed. I strongly recommend operating at about 20% speed when close to the wall (on top of the automatic speed reduction I mentioned).  

• These changes are based on Work Permit A2022-01-06-1
• Changes have been noted in the crane’s logbook

See attached screenshot of PLC program indicating that the new limit has been reached.

Contact me (Adam Newsome) for more detailed information.

A 1/8" SS tube was routed from the Swagelok fitting on the front of the lower lead shield to the service stand water catch tray as shown in the attached photos.

The tube was routed to fit in the cut-out and chamfered corner gaps of block TSH0387 in the de-rated EHDT shielding assembly TSH0293.

 The lower half of the beam dump EHDT shielding was installed yesterday by Plant Group.  After installation of the service stand plug, the position of the south and east faces of the plug were checked with the laser tracker w.r.t. EHAT MB4.  The south face was within approximately 2mm of nominal, and the east face was approximately 6mm east of nominal position.  It was decided that this positioning was acceptable.  The modified 6 foot block (TSH0387) was then installed with approximately 4mm clearance from the plug according to TSH0293.

Beamlines Group may now proceed with installation of the EHDT section of beam pipe.

Photos and results of the laser tracker measurements are attached.

The load cell which is located on the crane itself to display the load to operators was reported not working during the week of November 15. It is planned to investigate on Nov. 25 and this e-log will be updated with the status.

Edit (2022-02-02): the load cell is functioning again. No changes were made.

On August 1, 2023 HC1 was pumped down to -1200 Pa (gauge) and then valved off to test the leak rate after turntables had been sealed with backer rod in the gaps.

The sealing with backer rod allowed the cell to reach the test pressure of 1000 Pa. The cell pressure equalized within 30-40 seconds which is an improvement but still not sufficient to meet the goals we have set.

Next step is to pump down to -500 Pa and use smoke generator and perform thorough investigation to identify any leaks.

On Tuesday, February 14th I finished modifying Jason's boxes for the remote crane tests by adding shelves to mount cameras... Apart from given the edges a quick sanding (so no one gets slivers)...which I will try and do tomorrow.

On November 28th the lower lead shield was installed with 3/8" shimming at the front end, and 11/32" shimming at the rear end.  The south and east faces were aligned as flush as possible with the lower center block.  The beam dump insert was then placed on the rails in the lower lead shield.

Laser tracker measurements were performed by Michael Vogel, and he reported that the beam dump was 1.7mm high at the rear, and 0.8mm high at the front.

The lower lead shield was lifted, and 1/32" shimming was removed from the front, and 3/32" from the rear.  The block was then placed precisely using the laser tracker with targets on the beam dump insert.

After placing the lower lead shield, Michael aligned the beam dump by shimming and adjusting the rail track, and was able to get all fiducial points within 0.2mm of the nominal position.