We successfully performed a removal and replacement of a power busbar stabuli connector on the front end plate where the HTV connects to:

• The middle-lower connector was tested as the others are more straightforward to work with.
• It may be a good idea to use a 3D printed jig to wedge into the connectors without damaging the flexures/pins.
• A longer hex bit on the hot cell torque tool would greatly help this operation.
• Even without special tooling the bottom connector returned and mated back onto the locating features relatively easily with just the manipulators.
• Longer hex bits purchased, 3D printed too left for future.
• Screws and washers are effectively captive in the connectors which helps

Some issues were encountered with the turntable controls on the west hot cell, as a result the festooning post for the turntable was damaged. While the post was out of commission, the turntable was operated without actively monitoring the top of the hot cell and some cables ended up catching on the limit sensors, causing some slight fraying. The cables were duct-taped to prevent shorting for now but should be inspected and replaced accordingly in the future. We should also check that the sensor bracket did not move from where itshould be.

The four water connections on the front end plate of the APTW that will connect to the HTV were tested for replacement with the manipulators:

• The pin on these connectors were fairly easy to remove and replace with the old pin detent tool.
• The pin detent tool could also be used to manipulate the water lines while protecting the sealing face.
• The horizontal VCR joint on the DRIVE BEAM CONE (ATA3674) could be undone with the manipulator and standard wrenches, it is advisable to use 'semi-closed' wrenches on the nut to prevent it from dropping when torquing or undoing the joint.
• Of the four water lines, only the top line could be taken out and returned by hand, testing with manipulator still to be done.
  • It would be nice for this top most line to have a clamp or something to keep the VCR nut on the bottom U-bend from falling.
• The second line from the top interferes with the bus bar when trying to be removed by hand.
• The second line from the bottom has a VCR connection obstructed by the same bus bar when trying to remove by hand.
• The bottom line catches on the other lines when trying to remove by hand, the other lines likely need to be removed first to allow this line to be exchanged.
• is the intent for the bus bar need to be removed to exchange these lines?
• The VCR nuts on the water lines located by the bottom of the bus bars will require the turntable to be raised most of the way up to access with the manipulators.

 Testing will resume in the following week for these water lines.

The four water connections on the front end plate of the APTW that will connect to the HTV were tested for replacement with the manipulators:

• The pin on these connectors were fairly easy to remove and replace with the old pin detent tool.
• The horizontal VCR joint on the DRIVE BEAM CONE (ATA3674) could be undone with the manipulator and standard wrenches, it is advisable to use 'semi-closed' wrenches on the nut to prevent it from dropping when torquing or undoing the joint.
• Of the four water lines, only the top line could be taken out and returned by hand, testing with manipulator still to be done.
  • It would be nice for this top most line to have a clamp or something to keep the VCR nut on the bottom U-bend from falling.
• The second line from the top interferes with the bus bar when trying to be removed by hand.
• The second line from the bottom has a VCR connection obstructed by the same bus bar when trying to remove by hand.
• The bottom line catches on the other lines when trying to remove by hand, the other lines likely need to be removed first to allow this line to be exchanged.
• is the intent for the bus bar need to be removed to exchange these lines?

 Testing will resume in the following week for these water lines.

 December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam, Austin Hagen

Tests:

Loose piece connection and removal 

• By hand, the two connections were brought together and removed carefully

Observation/notes:

• Copper extensions are fragile and can bend when removing extraction electrode. 
  • This makes this part fairly limited in its capacity to be re-used
  • Copper extensions will need to be tuned before inserting into hotcell for replacement 

Recommendations/follow up items/questions:

• Bigger lead-ins would be appreciated
• What is the exchange frequency?
• the flexible nature of the copper extensions makes re-use limited
• How many connections can we expect from the aluminum fingers?
• Would a closer locating feature specific to the pins help?

December 12, 2024 - Michael Genix, Aaron Tam 

• Guide pins for the extraction electrode contact before the banana plugs
  • As these pins are low tolerance, this will act as the guiding for the plugs
  • Copper extensions have a much smaller diameter than the plugs, so positioning can be less precise in this area 
• Exchange frequency is still TBD
• Pin and plug life is still TBD
• Potentially the copper extensions can be tuned in the hotcell

December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam 

Tests:

Gas lines removal: 

• Starting with the most exterior connection, VCR connection loosened off with open ended conventional wrench
• Once loose, the nut can be un threaded with manipulator finger (rolling nut technique) 
• Once gas line unhooked, gasket removed by bringing female end outside the service tray footprint and jiggling until the gasket was removed
  • Also possible to use a pick if needed
• The same procedure was conducted for the interior VCR connection

Gas lines installation: 

• Starting with the inner most connection
• Gasket placed on 3D printed tool and clipped into position on VCR male end
• At first an M10 bolt was inserted into the elbow below VCR connection, but without rotational authority, the makeshift handle is not worth using
• Griping the elbow with one manipulator and rolling the nut onto the threads with the other proved successful
• Nut was tightened using conventional open ended 19mm wrench

Observations/Notes:

• Service Tray was not in a fully connected position, so even less space will be afforded.
  • This may affect the ability to get 2 manipulators on the same connection
• Piston modules were missing some limit wires on the side. These constrain movement horizontally and will either need to be removed as part of the procedure or, make the procedure more difficult
• Mass markers and various other connectors not installed, and these could slightly restrict movement as well
• Service tray pin was restricting the movement of the left manipulator during install
• Lighting was inadequate under the piston 

Recommendations/follow up items/questions:

• All metal Parker VGR style gaskets to replace plastic retainer versions
  • This is so that the degraded plastic doesn't break off and end up in the gas lines (upside down connections)
• A modified wrench with flats for handles and with more length would make the above procedures easier 
• Along with raising and lower the service tray, the service tray pin being oriented towards the beam entry direction, would make life easier 
• Still need to test gasket install on "other" Piston gas line connection
  • Chad will redesign some new gasket tools to be tested
    • side load and axial load gaskets, low and high clearance, aluminum construction 
• Lighting positioned to flood the service tray area will be needed
• labeling the gas lines with a more permanent solution will be needed
• Once the high voltage feed-through parts have come in, we can re-test and see if the piston module can be lowered to aid in target removal situations
• Aaron will test Parker style seals to see if they can be removed as easily once brought up to specified torque 

------------------------------------------------------------------------------------------------------------------------------------------------------------------

 December 16, 2024

• Upon inspecting the piston modules on AETE in TISA, we became aware of how the wires are arranged for the limit switches, it may be a good idea to look into how these wires are handled when using the piston module jig.

Dec 10, 2024

Testing completed:
 - Used hydraulic scissor jack cart to test raising/lowering service tray onto front-end, with only manipulators, placement of jack table done by hand because it isn't freely moving on the plane of the lift table.
 - Removal and insertion of the center post pin with indirect view to FE.
 - Application and removal of VCR gaskets on HVFT water connection array with 3D printed tool, only for 1/2" size (service tray side of VCR joints).
 - Checking whether the VCR joints can be done up with the service tray 'lifted' (not fully connected but the screws on the HVFT bracket are slid onto their slots)


Observation/notes:
 - Lift table slope creates some difficulty bringing the service tray up.
 - Effective misalignment of service tray changes with the degree in which it is 'brought up' because the ribbon connectors start to compress and exert load, tipping the service tray. There isn't a 'CG' that we can place the table under which will balance the service tray throughout the attachment operation.
 - Service tray is large and will not be fully supported by the base jig without custom jig on top.
 - Using a clamped piece of 2x4 wood on the jack cart worked to increase the supported region during testing, something similar with the base jig may work.
 - The service tray needs to be brought up such that the HVFT is ~1" from being fully engaged before the screws on the bracket can be upturned and start to engage. Likely the bus bar connectors will have started engaging before this is achieved (bus bar connectors roughly 2" in depth), this can create complications because the loading on the base jig will be really uneven at that point.
 - The pin for the center post can be easier to handle if we added a handle or an extender. A makeshift handle was made using tape and scrap U-channel nearby.
 - We managed to remove and attach the pin with the HTV table facing East using a remote camera view, but in practice it would be ideal if we could rotate the front end with service tray supported by some jig between orientations where the HVFT and center post are more easily accessed. This will require the ribbons and the alignment cylinders on the HVFT to transmit some force onto the lift table to overcome any friction the base jig may have on the lift table during rotation.
 - With the brackets on the HVFT on, we were able to start and even tighten the VCR joint closest to the corner bracket (using low clearance wrench, no photo).
 - Difficult to get at all brackets on HVFT and do up, will need cameras irrespective of the orientation, did not do up the bracket yet so we don't know how careful we need to be when inching each up and establishing the HVFT connection.
 - Difficult to get at and rotate screws on HVFT bracket, with torque tool due to proximity with HVFT structure, a longer bit on the torque tool will be beneficial. Note, using an Allen key with the manipulator gripper is difficult because of the same issue.
 - VCR wrenches slippery without catching features, will add pin for better grip
 - If 'fence' on HVFT for water lines is taken off it is quite easy to loose track of where the lines are supposed to return to, should have markings/labels for where the water lines need to return.
 - HTV side ribbons bend inward instead of outward.

Recommendations/follow up items/questions + answers as appropriate:
 - Consider adding grabbing features and an extender attachment for the center post pin: (Dec 16, 2024) Michael agreed to cut the tab and cross-pin the center post pin to match the HTV water connector pins.
 - Can we change the screws used on the HVFT bracket to be longer so it can engaging before the bus bars? (Dec 16, 2024) Michael will look into.
 - Can we add a lip to the HVFT top brackets to keep the screws in place and not fall out during tightening? (Dec 16, 2024) Michael will look into.
 - Consider more items that may help simplify the brackets on the HVFT
 - Is it possible to re-orient the center post holes to be toward the HVFT instead of aligned with the HTV platform axis, this may avoid the need for re-orienting the FE when raising/lowering the service tray. (Dec 16, 2024) Michael will look into.
 - Need to capture effect of bus bars connection in future testing. (Dec 16, 2024) Albert will design and send something to the shop to work with Phoenix's mock up.
 - Need to repeat service tray testing with base jig.
 - Check if ribbons are bending in the right direction. (Dec 16, 2024) Yes.
 - Perform full service tray coupling with actual machined brackets to inform coupling operation (3D printed brackets currently not strong enough and deflect unrealistically).(Dec 16, 2024) received spares from Michael.
 - Need modification to wrench to prevent slippage. (Dec 16, 2024) Albert and Aaron working on this (will simply add cross-pin)
 - VCR gasket tool for smaller size VCR's to be made.(Dec 16, 2024) Chad to do.
 - Real deal VCR gasket tools to be made out of soft metal to prevent scratching sealing surfaces in operation
 - If not possible to do with base jig, possibly a suspended plate with turnbuckles instead will be sufficient to support the service tray during attachment/removal operations. (Dec 16, 2024) Discussed with Michael the possibility of changing the HVFT 'cables' to threaded rod to also facilitate ejection of the service tray on the HVFT side (necessary to disconnect bus bar connections).

 

After a recent power cycle, the y-axis referencing was lost for the 1000 kg crane. The position readout was also incorrect, somewhere on the order of > 30,000 mm. On 2024-11-29, A. Newsome re-referenced the y-axis. The position readout correctly reset to zero, and both +y and -y motion is functioning correctly, but the y-axis position readout on the HMI constantly displays 0 and does not change. To be investigated.

Update 2024-12-02:

After going online with the PLC, it was determined that the reason for the display of 0 as the position is that the two values used for calibration of the +y and -y limits were actually the same, meaning the scaling factor (the difference between these two values) was 0, which resulted in the displayed value being 0. The root cause of this is that the encoder was not functioning correctly so its value was not changing when the crane moved and it was stuck at one value. Upon investigation of the encoder input card, the red "ERR" light was on. This indicates the encoder signals are not properly reaching the input card. Once this was discovered, it was remembered that this happened in July 2024 as well (no e-log was written). In July, the root cause was identified as being a loose encoder signal wire in a junction box. Junction box CJB1-BC, located on the bridge crane near its disconnects, was opened and investigated. Upon checking each wire, it was determined that the red wire seemed not to be making full contact. The wire was removed and re-inserted, and the "ERR" indicator on the encoder card turned off. The crane's +y and -y limits were re-referenced. The crane is functioning normally after re-referencing. (Note: if something like this happens again, during the re-referencing process, the displayed value on the HMI will be incorrect.. this is because the PLC's scaling factor is not fully adjusted until both +y and -y limits are reached. The actual encoder measurement taken at those limits is used for the scaling factor. This is not an ideal way to program the system, but this is how it works with all ARIEL hot cell subsystems. So incorrectly displayed values can be ignored, in general, until full referencing is completed).

It was reported that the hot cell turntables would not rotate and the access hatch could not be operated. The turntables' elevation motion was working normally, and other cell functions were working.

Upon investigation, after going online with the PLC and checking the interlocks for the turntable, it was determined that motion was being prevented because the turntables were in the "critical position" in which they could interfere with the access hatch raising/lowering. However, the access hatch was fully closed, so this logic should not have been actively preventing motion. After inspecting the target access hatch limit switches, it was determined that the upper limit switch's Normally Open contacts were behaving normally, but the Normally Closed ones were not. When the switch was toggled, the NO contacts did not switch over. Thus, the sensor was in an unknown state causing conflicting logic in the PLC (the program thought the access hatch was in an intermediate state between opened and closed, thus preventing motion... and the access hatch could not move because the turntables were in the "critical position"). After testing the switch multiple times, jiggling the wiring/contacts, and rewiring one of the screw terminals, the NO contacts started working normally again. It is suspected that a wire was loose. 

The issue has been resolved and tested - the access hatch logic is functioning regularly and both turntables can fully rotate and elevate.

November 27, 2024

Testing completed:

• Removal and re-insertion of 2x multi-pin coupling cable for HTV (see pictures, left most and right most, middle had no wire at the time).
• Tested using 'wire' method to pull cable once removed back through the original path.
• Test without rotating table, on cart, some obstruction that does not reflect reality in operation, rotated cart by hand, will require rotation during operation.

Observations/notes:

• Right connector has wires come across and may interfere with the middle connector, will require removal of multiple connectors to exchange one depending on configuration.
• Locating pins on high voltage multi-pin connector needs to be short for voltage bias reasons.
• Right-most black multi-pin cylindrical connector has higher risk of machined block end catching when fishing from the HTV side.

Recommendations:

• May be a good idea to add a holder connector to go into the multi-pins and give the manipulators a better grip for handling. Directly grabbing on the cylinder of the multi-pin connector is ok though.
• Some strain relief at the machined connector blocs (service tray side) will help prevent damage during wire replacement/routing.
• Possibly re-route some of the electrical connectors to the back side of the HTV column for easier access and shorter length.
• Will need chart mapping where the cables go and indexing different connections for future hot cell work (will be updated on SW model).
• Locking pins on multi-pin connectors can be removed and replaced manually with the manipulator.
• Adding a 'slippy' sheath onto the replacement cable for routing and to prevent catching would be a good idea in the future.

 -----------------------------------------------------------------------

Dec 02, 2024

Testing completed:

• Removal and re-insertion of 1x ceramic-ended wires specifically the one obstructed by the gas line on the right side of the HTV connection array as viewed from the hot cell window.
• Removal and re-insertion of main multi-pin connector cable bundle from the HVFT connector array on the service tray.
• Removal and re-insertion of the gas line obstructing the right-side ceramic electrical connectors << very difficult/impossible to do (recommend modifications + tooling).

Observations/notes:

• Electrical wires/connectors generally straightforward to exchange.
• Plastic/ceramic components on connectors are easy to damage during exchange operation.
• Most connectors require the metal/cylindrical bits to be 'grabbed' by the manipulators if no custom tool is used
• Rectangular multi-pin connector bracket if single is difficult to return to slot since not supported on both sides
• Need documentation/reference material to make sure connectors get returned to where they need to be
• Screws on connector are deep into the FE assembly, creates viewing difficulty
• Cameras hard to use for these screws because of beam dump being highly active when FE is in operation
• Ceramic (mass marker) connectors fine to grab on the conductor part
• When trying to re-align the ceramic bodies to line up the screw holes, found that it was really easy to damage/break the ceramics, avoid using hard tooling/manipulators on the ceramic parts
• Plastic components on multi-pin rectangular brackets got damaged after handling with manipulators
• Full sized pin detent tool not compatible with smaller pins, should make smaller sized pin detent tool, smallest clearance found on HVFT side of multi-pin connector bundle
• Routing HVFT side of multi-pin bundle through wires poses risk of damaging exposed pins

Recommendations:

• Should buy longer bits/ends for torque control tool to get better view angles on screws.
• Maybe captive screw for the gas is a good idea to prevent the screw from dropping.
• Torque tool should be set properly (torque and rotation rate) before performing screw exchanges involving the HTV connectors on the front end
• Return of the gas line (on bracket) to its location on the FE is difficult, suggested modifications include:
  • Changing orientation of cutout on bracket to let stainless bent/welded tube to be returned separate from the bracket.
  • Changing screws for the stainless part on the gas line to be standard size (M6) for hot cell exchange operations.
  • Changing the unused screw hole on the FE to be pinned and allow for better alignment of the bracket onto the FE
  • Removing burr on sheet metal part on hose/wire opening at the bottom to prevent the braided hose from catching.
• Should make a rotator tool (3D printed) to help adjust the ceramic end of the mass marker connector and have the screw hole and thread aligned
• Maybe plastic covers for water/gas connectors to prevent scratching of outer surface
• Make smaller pin detent tool or settle for old tool already made
• Possibly design cover to protect exposed pins on multi-pin bundle during replacement

 -----------------------------------------------------------------------

Dec 12, 2024:

Testing completed:

• Tested removal and return of the gas line obstructing some ceramic electrical cables on the right side of the HTV connector array on the front end, now with 6mm pin (McMaster 97493A162) threaded into the previously unused hole to help guide the previously  unused hole to help guide the bracket.

Observations/notes:

• Chad Fisher commented that even without the pin it should be possible to complete the removal and return operation if a 3D printed (soft )tool is made to grab onto the tubes and guide it with the bracket into the alignment hole/slot.
• Old recommendations still stand though + maybe make the pin longer/custom with more lead in.

Recommendations:

• Remove the retaining tab at the bottom to allow the hose to be positioned more freely.
• De-burr the sheet metal part to prevent catching on the braided hose.

The APTW front end was moved into the hot cell, on the lift table, following procedures outlined in Document-244862.

The mock-up frame was transferred from the east turntable to the west turntable above the front end. The east turntable is covered with the aluminum plate. During covering we accidentally broke the o-ring for the turntable plate flange. The new o-ring seal should be made ~1" longer in circumference to avoid difficulties with seating in the o-ring groove.

When we attempted to move the front-end with the manipulator, we noticed that the front end tended to carry on sliding down the slope of the hot cell (see TRI-DN-23-20 / Document-238536). The manipulators were subsequently unable to arrest the front end, likely because they cannot apply a load towards the CG of the front end. Therefore, for heavy-duty jigs in the hot cell it is advisable to avoid using wheels/rollers that allow the heavy assemblies to move freely on the lift table.

--------------------------------------------------------------------------------

UPDATE Nov 21, 2024

• FE attachment related tests completed:
  • replacement of shim plates
  • mock-up alignment of FE with shield plug (adjustment made in person, hot cell operator instruct and give direction with remote camera views
• Difficulties encountered:
  • keeping track of which hole/screws are seen by the cameras during alignment was difficult for the operator
  • shim plate awkward to handle if not familiar
  • incompatible coupling screw and nut threads (M24 course vs fine threads) prevent completion of coupling and uncoupling tests
  • 'loose' service tray requires separate alignment check when raising the FE with the lift table
• Lessons learned:
  • shim plate replacement easier if FE is semi-attached (screw lifts FE, leaving small gap for shim plate to be inserted)
  • need a cover plate to prevent dropping shim plate and associated screws
  • rotation clearance of FE checked, will clear
  • need to also check service tray alignment when raising
  • magnetic cameras would be beneficial for this operation
• Follow up items:
  • crane-based positioning of FE for replacement to be tested after concrete pours/hand-testing/when APTW is exchanged for AETE.
  • future design of protective covering when performing shim plate removal
  • optionally implement magnetic camera array to check hole alignment by looking directly at the holes

At end of shift, mock up and FE was separated, markings made on floor to specify where FE needs to be to couple.

--------------------------------------------------------------------------------

UPDATE Nov 25, 2024

Replacement nuts arrived (M24x3) for the coupling screws. Tests were completed to try and couple/uncouple the front end to the cart/mock up.

• For taking the FE on/off the cart the cart should be able to rotate with the FE so that we can secure the coupling screws.
• For dropping the FE off from the shield plug, we should make use of the alignment features (cup and cones), once screws are undone and the FE rests on the cones on the cart, it should be secured.
• If markings on floor is made to assist with alignment, we need to do two sets; one for unloaded cart, one for loaded cart, this is because the lift table will tilt differently when there is a load.
• The threads on the coupling screws are fine threaded in the real case, there is a slight concern with cross-threading when turning them onto the nuts remotely (from above).
• Existing camera views sufficient to check for clearance between coupling plates and FE structures.
• To allow the FE to guide freely into the alignment features (cup/cone), it may be a good idea to still use the air skates, but supplement it with structural 'stopping' features to keep it from rolling off the table.

--------------------------------------------------------------------------------

UPDATE Nov 26, 2024

We returned the front-end to the cart to allow the shield-plug mock up to be rotated by the target hall crane (turntable still not functioning as of this date).

During this operation we noticed that the front end did not fully return to the alignment cones on the cart (see images), somehow the contact pad screws made contact first, and so the two-faced alignment cone did not come into contact.

For the real support frame we should use a minimal number of contact points. Also,more clearance at the coupling screws between the cart to the front end should be added.

--------------------------------------------------------------------------------

UPDATE Nov 27, 2024

We rotated the mock-up frame manually to have the HTV side face the hot cell window. During which the o-ring fell and we decided to cut it for convenience, this should be replaced with an equivalent o-ring but made ~1" longer in circumference.

--------------------------------------------------------------------------------

UPDATE Nov 28, 2024

Today we attempted to re-couple the front end to the mock up frame in the orientation where the HTV faces the hot cell window. The misalignment between the FE on the cart to the mock-up frame is worse in this orientation than in the previous (90 CCW facing down). There was approximately a 1" gap difference between the opposite ended screws. Factors contributing to the misalignment include uneven seating of the FE onto the bosch frame and uneven 'brake' engagement on the bosch frame wheels. During the attachment process the the screws had to be tightened very unevenly, with the HTV side being tightened far more than the extraction electrode side.

Additionally, the service tray shifted significantly from handling in the previous days and it was difficult to align the locating pins/cylinders. This further complicated the coupling process, in the future the service tray attachment should be made a separate operation to the process of coupling the  FE to the shield plug.

Under these conditions, during tightening, the inner screw at the HTV-side somehow jammed against the nut and the nut conversely jammed into the bracket. Subsequently, that screw could not be tightened further and required great difficulty to be removed, even after removing the bracket. In the aftermath, the nut bracket became forced apart (see picture) and the threads on the screw deformed beyond use. This is a critical failure mode for this coupling system. Note that the screw was hand started onto the nuts on the brackets prior to fully tightening the screws.

We believe the factors contributing to this failure include:

1. bracket gap being too large and allows the nut to wedge itself during tightening,
2. screw threads not being hardened/of proper grade,
3. tightening the screws too unevenly and sharply (too large turns per-screw when bringing the FE up, 1/2-1 turn) / allowing some screws being slack,
4. high misalignment (pitch and roll) between the FE plate and the mock-up frame,
5. the brackets not being able to accommodate the observed misalignment,
6. hole for screw is ~short and creates possibility for jamming when there is misalignment (pitch and roll),
7. aligning the service tray locating features while aligning the screw holes for the FE coupling was difficult,
8. possibly the extra lead in (chamfer) on the coupling screws is excessive and a non-standard design, which may cause problems starting the nut, likely a standard chamfer on the end of a standard screw would be sufficient for this operation

In the real operation, the long length of the screw will help prevent this failure from happening but some things to keep in mind/improve on would be:

1. bracket re-design to use standard wrench/nut gaps to avoid nut jamming
2. use hardened threads
3. try to see if no-lead in/chamfer/using standard screw profiles is sufficient
4. re-design the coupling system to not require a nut at the bottom and avoid cross-threading/thread jamming in the hot cell (move threads and nuts all above the FE assembly)
5. extra: the screw should not be fine threaded in the real case to avoid cross threading
6. minimize misalignment when trying to couple in the future (rotate turntable to minimum misalignment orientation)
7. tighten the screws more incrementally to have them go together evenly, don't allow screws to be fully slack if possible to prevent jamming
8. fully remove the service tray and attach only after the FE is attached

Of the above recommendations we strongly recommend re-designing the nut bracket - even when coupling was smooth (without misalignment) the nuts made a visible dent onto the bracket at the highest point (see image). A smaller gap (standard wrench gap) would prevent such a mark from forming.

We replaced the deformed screw with a standard (off the shelf) one to keep the FE secure to the mock-up.

We also noted after completing coupling that the extraction electrode, farthest out screw had damaged threads (see image).

It is recommended to perform repeated coupling/decoupling operations in the hot cell after some process/design changes are made/implemented to work out any possible failure modes with this operation.

Two tarps were installed above the hot cell roof to protect against concrete dust while testing in the hot cell.

Zip ties were used to hook the tarp onto tubing brackets on the back wall.

A safety walkaround was completed for the ARIEL Hot Cell area.

The resulting spreadsheet can be found on DocuShare as Document-242733.

No major deficiencies identified.

The spacer flanges were installed on both turntables today.

See photos and information in the following DocuShare Collection: Collection-39816

Important note: the fitment was quite tight due to interference with universal joints and grease nipples. The flanges were still able to be installed with a bit of difficulty. They are not posing any immediate issues, but it is predicted that the adapter flanges will certainly interfere with drive system components. It is necessary to change the design of the adapter flanges to allow for the appropriate clearance.

Upon powering on the crane today and attempting to use the main hoist, the following faults were present:

1. Main Hoist East Drum North Motor Drive Fault (700VFD)
2. Main Hoist West Drum South Motor Drive Fault (900VFD)
3. Main Hoist East Drum North Motor Drive Not Ready (700VFD)

Note that the auxiliary hoists functioned as normal.

Upon power cycling the crane to reset, the faults disappeared, and the main hoist functioned normally again. The root cause of the issue is unknown. It is suspected to be related to either prolonged inactivity, improper safety reset sequence, or the battery being removed from the pendant. This issue will be monitored for re-occurrence.

A safety walkaround for July 2024 was completed for the B2 level by A. Newsome. No deficiencies to report.

Results can be found in the master spreadsheet

On 2024-04-22 around 3:30 pm it was discovered that the crane's main hoist was in a faulted state. The operator (Adam Newsome) experienced the fault after performing a safety reset and attempting to lower the hoist. There were no visible signs of any issues, mechanically. This fault had not been previously reported by any other crane operators. The crane was previously unused, sitting idle. Note: the auxiliary hoists appeared to function normally.

Fault messages on the HMI indicated:

"135. Main Hoist West Drum North Motor Drive Fault (1000VFD)"
"104. Main Hoist East Drum South Motor Drive Fault (800VFD)"
"103. Main Hoist East Drum North Motor Drive Fault (700VFD)"
"138. Main Hoist West Drum South Motor Drive Not Ready (900VFD)"
"136. Main Hoist West Drum South Motor Drive Fault (900VFD)"

It is clear from this that there is either some sort of common issue seen across all these main hoist VFDs, or one issue with one of them which caused a cascading series of faults.

On 2024-04-23, upon investigation of these faults by going online with the PLC to determine fault logic, it was clear that all faults were indeed present, but there was no obvious indication as to what it was. Inspection of the VFDs in the control panel showed that all of the aforementioned drives displayed a fault code. This was fault code 4 which indicates DC bus undervoltage.

After researching this fault online, it appears this is typically caused by an issue with the input mains supply (480VAC @ 3 phase in this case), or by the drive's input DC filter/buffer circuitry. When measuring the DC bus voltage, it is expected for it to be approximately 1.414 times the AC supply voltage. In this case, 1.414*480 = 678 VDC. The first troubleshooting step was to measure to confirm the DC bus voltage on a known working and not working drive.

The bus voltage was probed on drive 500VFD which, based on lack of fault message, was expected to be functioning properly. With the safety off due to E-stop condition, the voltage was nearly zero. When a safety reset was pressed to enable the drive, the voltage changed to 690 VDC. This is close enough to the expected 678 VDC. The voltage did not change, even when the fault message for the other drives appeared again.

Next, the bus voltage on two known faulted drives, 900VFD and 1000VFD, was probed. In both cases, after the safety reset, the bus voltage started at 690 VDC but then slowly dropped down towards nearly zero (somewhere around 18 V) over a period of 3-5 seconds. It was during this transition when the voltage dropped off that the undervoltage condition became true, which triggered the fault.

As of 2024-04-24, the root cause of this issue is unknown. It will be investigated further and this e-log will be updated when a solution is found.

Update 2024-04-24: the root cause has been identified. The undervoltage fault was just a symptom as a result of the input contactors for these drives switching off, causing the DC bus voltage to drop slowly due to capacitance in the input filtering circuit. The reason for the input contactors switching off is attributed to safety signals dropping out due a fault observed specifically on 900VFD. When looking at the drive itself, a fault with code 4030 was displayed: Enc1 open wire.
This fault implies the encoder may be disconnected. The encoder wiring was checked: OK. The encoder cable was swapped with a unknown working one from an adjacent drive - the issue remained with 900VFD, which suggested the problem is with the drive itself. The encoder board (20-750-DENC-1) was swapped with an adjacent drive and the problem followed the board. The encoder board was then switched with a brand new spare (note: jumpers needed to be set!).
Upon power-cycling the system, the fault did not persist. Therefore, it is suspected that the encoder board had failed (in fact this happened previously with the same drive - see e-log 42). Upon inspection, one of the capacitors on the board appeared to be cracked - this will be investigated further to see if replacing it fixes the problem.

A spare encoder board will be ordered. This issue should be monitored in the future - it seems as if the drive itself is perhaps causing the encoder boards to fail.

Note: upon powering the system on again, another issue was noticed: 500VFD shows "drive not ready" fault. This is not displayed on the HMI though - it was just not possible to reset the safety system and this was only discovered from going online with the PLC. This will be investigated further.

Update 2024-04-26: the 500VFD "drive not ready fault" was investigated. It was determined that the SP+ (safety power +) signal wire was loose, which meant the safety signals to the drive were not getting through, causing it to remain in a "not ready" state. The wiring issue was corrected. Performing a safety reset resulted in successful drive enables across all drives. However, upon attempting to move the crane, even though all safety signals were green and it appeared to be able to move, it did not.
It turned out that there was a crane pendant fault: "Fault 305. Radio Control Receiver in Fault (3700RC)". This is exactly what happened previously in a similar situation (see e-log 55). The steps mentioned in that e-log were followed (power cycle, reconnect antenna/connector). Upon powering back up, everything worked as normal. The crane was tested in local mode - all three hoists up/down (main hoist run to upper limit), all trolley travels in each direction, and bridge travel in each direction.

As of now, everything is operating normally and all issues are considered to be resolved. A spare encoder board has been ordered.

The main trolley's E/W motion was observed to be slowed on 2024-02-13.
Upon investigation, the following faults were present:
- 200. Rotating Hook Velocity Difference between SSI Encoder and Drive Speed Feedback
- 82. Main Hoist Slack Rope

It is suspected that these faults are irrelevant in this situation. Upon inspection of the main trolley status screen, it was evident that both the SLOW EAST and SLOW WEST indicators were on (which matches observations). Clicking one of these status icons showed "Main Trolley Encoder Not Homed 2253ENC".

Logging in as an administrator allowed for homing the encoder.

The trolley was moved all the way East, to the limit, and the encoder was homed at position 0.389m (encoder count = 6886940). Note: the alignment arrows were not used as this would have required gaining access to the crane which was difficult at the time. It is suggested to properly home it in the future.

The E/W motions were fully tested to confirm proper slow down and stopping at the expected positions. The crane appears to be operational as of 2024-02-14.

 

On August 1, 2023 HC1 was pumped down to -1200 Pa (gauge) and then valved off to test the leak rate after turntables had been sealed with backer rod in the gaps.

The sealing with backer rod allowed the cell to reach the test pressure of 1000 Pa. The cell pressure equalized within 30-40 seconds which is an improvement but still not sufficient to meet the goals we have set.

Next step is to pump down to -500 Pa and use smoke generator and perform thorough investigation to identify any leaks.

It was observed recently that the main hoist's wires seemed skewed, and there was occasionally some noise heard when raising and lowering the hook block. It was suspected that this was due to wires having jumped in their tracks at some point. This is typically caused by side loading. It is possible that wire wrap skew occurred during the incident outlined in e-log 46. 

Gordon Crane was contacted for a service call. Crane inspector Ali from Gordon Crane was supervised/assisted by Maico Dalla Valle. They lowered the hoist down into the target pit and supported it on saw horses, then let the wires run all the way out. Ali confirmed that there was a wire wrapping issue, and re-seated the wires properly. The crane is now functioning normally. It is recommended to re-apply lubrication as soon as possible.  

Note: Ali from Gordon Crane has advised that it is acceptable for some degree of wire skew to occur - they can handle the abrasion if there's a small angle between them. However, wires should not be rubbing across each other at a large angle (ex. more than approx. 10 degrees).