It was reported that the hot cell turntables would not rotate and the access hatch could not be operated. The turntables' elevation motion was working normally, and other cell functions were working.

Upon investigation, after going online with the PLC and checking the interlocks for the turntable, it was determined that motion was being prevented because the turntables were in the "critical position" in which they could interfere with the access hatch raising/lowering. However, the access hatch was fully closed, so this logic should not have been actively preventing motion. After inspecting the target access hatch limit switches, it was determined that the upper limit switch's Normally Open contacts were behaving normally, but the Normally Closed ones were not. When the switch was toggled, the NO contacts did not switch over. Thus, the sensor was in an unknown state causing conflicting logic in the PLC (the program thought the access hatch was in an intermediate state between opened and closed, thus preventing motion... and the access hatch could not move because the turntables were in the "critical position"). After testing the switch multiple times, jiggling the wiring/contacts, and rewiring one of the screw terminals, the NO contacts started working normally again. It is suspected that a wire was loose. 

The issue has been resolved and tested - the access hatch logic is functioning regularly and both turntables can fully rotate and elevate.

After a recent power cycle, the y-axis referencing was lost for the 1000 kg crane. The position readout was also incorrect, somewhere on the order of > 30,000 mm. On 2024-11-29, A. Newsome re-referenced the y-axis. The position readout correctly reset to zero, and both +y and -y motion is functioning correctly, but the y-axis position readout on the HMI constantly displays 0 and does not change. To be investigated.

Update 2024-12-02:

After going online with the PLC, it was determined that the reason for the display of 0 as the position is that the two values used for calibration of the +y and -y limits were actually the same, meaning the scaling factor (the difference between these two values) was 0, which resulted in the displayed value being 0. The root cause of this is that the encoder was not functioning correctly so its value was not changing when the crane moved and it was stuck at one value. Upon investigation of the encoder input card, the red "ERR" light was on. This indicates the encoder signals are not properly reaching the input card. Once this was discovered, it was remembered that this happened in July 2024 as well (no e-log was written). In July, the root cause was identified as being a loose encoder signal wire in a junction box. Junction box CJB1-BC, located on the bridge crane near its disconnects, was opened and investigated. Upon checking each wire, it was determined that the red wire seemed not to be making full contact. The wire was removed and re-inserted, and the "ERR" indicator on the encoder card turned off. The crane's +y and -y limits were re-referenced. The crane is functioning normally after re-referencing. (Note: if something like this happens again, during the re-referencing process, the displayed value on the HMI will be incorrect.. this is because the PLC's scaling factor is not fully adjusted until both +y and -y limits are reached. The actual encoder measurement taken at those limits is used for the scaling factor. This is not an ideal way to program the system, but this is how it works with all ARIEL hot cell subsystems. So incorrectly displayed values can be ignored, in general, until full referencing is completed).

Dec 10, 2024

Testing completed:
 - Used hydraulic scissor jack cart to test raising/lowering service tray onto front-end, with only manipulators, placement of jack table done by hand because it isn't freely moving on the plane of the lift table.
 - Removal and insertion of the center post pin with indirect view to FE.
 - Application and removal of VCR gaskets on HVFT water connection array with 3D printed tool, only for 1/2" size (service tray side of VCR joints).
 - Checking whether the VCR joints can be done up with the service tray 'lifted' (not fully connected but the screws on the HVFT bracket are slid onto their slots)


Observation/notes:
 - Lift table slope creates some difficulty bringing the service tray up.
 - Effective misalignment of service tray changes with the degree in which it is 'brought up' because the ribbon connectors start to compress and exert load, tipping the service tray. There isn't a 'CG' that we can place the table under which will balance the service tray throughout the attachment operation.
 - Service tray is large and will not be fully supported by the base jig without custom jig on top.
 - Using a clamped piece of 2x4 wood on the jack cart worked to increase the supported region during testing, something similar with the base jig may work.
 - The service tray needs to be brought up such that the HVFT is ~1" from being fully engaged before the screws on the bracket can be upturned and start to engage. Likely the bus bar connectors will have started engaging before this is achieved (bus bar connectors roughly 2" in depth), this can create complications because the loading on the base jig will be really uneven at that point.
 - The pin for the center post can be easier to handle if we added a handle or an extender. A makeshift handle was made using tape and scrap U-channel nearby.
 - We managed to remove and attach the pin with the HTV table facing East using a remote camera view, but in practice it would be ideal if we could rotate the front end with service tray supported by some jig between orientations where the HVFT and center post are more easily accessed. This will require the ribbons and the alignment cylinders on the HVFT to transmit some force onto the lift table to overcome any friction the base jig may have on the lift table during rotation.
 - With the brackets on the HVFT on, we were able to start and even tighten the VCR joint closest to the corner bracket (using low clearance wrench, no photo).
 - Difficult to get at all brackets on HVFT and do up, will need cameras irrespective of the orientation, did not do up the bracket yet so we don't know how careful we need to be when inching each up and establishing the HVFT connection.
 - Difficult to get at and rotate screws on HVFT bracket, with torque tool due to proximity with HVFT structure, a longer bit on the torque tool will be beneficial. Note, using an Allen key with the manipulator gripper is difficult because of the same issue.
 - VCR wrenches slippery without catching features, will add pin for better grip
 - If 'fence' on HVFT for water lines is taken off it is quite easy to loose track of where the lines are supposed to return to, should have markings/labels for where the water lines need to return.
 - HTV side ribbons bend inward instead of outward.

Recommendations/follow up items/questions + answers as appropriate:
 - Consider adding grabbing features and an extender attachment for the center post pin: (Dec 16, 2024) Michael agreed to cut the tab and cross-pin the center post pin to match the HTV water connector pins.
 - Can we change the screws used on the HVFT bracket to be longer so it can engaging before the bus bars? (Dec 16, 2024) Michael will look into.
 - Can we add a lip to the HVFT top brackets to keep the screws in place and not fall out during tightening? (Dec 16, 2024) Michael will look into.
 - Consider more items that may help simplify the brackets on the HVFT
 - Is it possible to re-orient the center post holes to be toward the HVFT instead of aligned with the HTV platform axis, this may avoid the need for re-orienting the FE when raising/lowering the service tray. (Dec 16, 2024) Michael will look into.
 - Need to capture effect of bus bars connection in future testing. (Dec 16, 2024) Albert will design and send something to the shop to work with Phoenix's mock up.
 - Need to repeat service tray testing with base jig.
 - Check if ribbons are bending in the right direction. (Dec 16, 2024) Yes.
 - Perform full service tray coupling with actual machined brackets to inform coupling operation (3D printed brackets currently not strong enough and deflect unrealistically).(Dec 16, 2024) received spares from Michael.
 - Need modification to wrench to prevent slippage. (Dec 16, 2024) Albert and Aaron working on this (will simply add cross-pin)
 - VCR gasket tool for smaller size VCR's to be made.(Dec 16, 2024) Chad to do.
 - Real deal VCR gasket tools to be made out of soft metal to prevent scratching sealing surfaces in operation
 - If not possible to do with base jig, possibly a suspended plate with turnbuckles instead will be sufficient to support the service tray during attachment/removal operations. (Dec 16, 2024) Discussed with Michael the possibility of changing the HVFT 'cables' to threaded rod to also facilitate ejection of the service tray on the HVFT side (necessary to disconnect bus bar connections).

 

December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam 

Tests:

Gas lines removal: 

• Starting with the most exterior connection, VCR connection loosened off with open ended conventional wrench
• Once loose, the nut can be un threaded with manipulator finger (rolling nut technique) 
• Once gas line unhooked, gasket removed by bringing female end outside the service tray footprint and jiggling until the gasket was removed
  • Also possible to use a pick if needed
• The same procedure was conducted for the interior VCR connection

Gas lines installation: 

• Starting with the inner most connection
• Gasket placed on 3D printed tool and clipped into position on VCR male end
• At first an M10 bolt was inserted into the elbow below VCR connection, but without rotational authority, the makeshift handle is not worth using
• Griping the elbow with one manipulator and rolling the nut onto the threads with the other proved successful
• Nut was tightened using conventional open ended 19mm wrench

Observations/Notes:

• Service Tray was not in a fully connected position, so even less space will be afforded.
  • This may affect the ability to get 2 manipulators on the same connection
• Piston modules were missing some limit wires on the side. These constrain movement horizontally and will either need to be removed as part of the procedure or, make the procedure more difficult
• Mass markers and various other connectors not installed, and these could slightly restrict movement as well
• Service tray pin was restricting the movement of the left manipulator during install
• Lighting was inadequate under the piston 

Recommendations/follow up items/questions:

• All metal Parker VGR style gaskets to replace plastic retainer versions
  • This is so that the degraded plastic doesn't break off and end up in the gas lines (upside down connections)
• A modified wrench with flats for handles and with more length would make the above procedures easier 
• Along with raising and lower the service tray, the service tray pin being oriented towards the beam entry direction, would make life easier 
• Still need to test gasket install on "other" Piston gas line connection
  • Chad will redesign some new gasket tools to be tested
    • side load and axial load gaskets, low and high clearance, aluminum construction 
• Lighting positioned to flood the service tray area will be needed
• labeling the gas lines with a more permanent solution will be needed
• Once the high voltage feed-through parts have come in, we can re-test and see if the piston module can be lowered to aid in target removal situations
• Aaron will test Parker style seals to see if they can be removed as easily once brought up to specified torque 

------------------------------------------------------------------------------------------------------------------------------------------------------------------

 December 16, 2024

• Upon inspecting the piston modules on AETE in TISA, we became aware of how the wires are arranged for the limit switches, it may be a good idea to look into how these wires are handled when using the piston module jig.

 December 12, 2024 - Chad Fisher, Albert Kong, Aaron Tam, Austin Hagen

Tests:

Loose piece connection and removal 

• By hand, the two connections were brought together and removed carefully

Observation/notes:

• Copper extensions are fragile and can bend when removing extraction electrode. 
  • This makes this part fairly limited in its capacity to be re-used
  • Copper extensions will need to be tuned before inserting into hotcell for replacement 

Recommendations/follow up items/questions:

• Bigger lead-ins would be appreciated
• What is the exchange frequency?
• the flexible nature of the copper extensions makes re-use limited
• How many connections can we expect from the aluminum fingers?
• Would a closer locating feature specific to the pins help?

December 12, 2024 - Michael Genix, Aaron Tam 

• Guide pins for the extraction electrode contact before the banana plugs
  • As these pins are low tolerance, this will act as the guiding for the plugs
  • Copper extensions have a much smaller diameter than the plugs, so positioning can be less precise in this area 
• Exchange frequency is still TBD
• Pin and plug life is still TBD
• Potentially the copper extensions can be tuned in the hotcell

The four water connections on the front end plate of the APTW that will connect to the HTV were tested for replacement with the manipulators:

• The pin on these connectors were fairly easy to remove and replace with the old pin detent tool.
• The horizontal VCR joint on the DRIVE BEAM CONE (ATA3674) could be undone with the manipulator and standard wrenches, it is advisable to use 'semi-closed' wrenches on the nut to prevent it from dropping when torquing or undoing the joint.
• Of the four water lines, only the top line could be taken out and returned by hand, testing with manipulator still to be done.
  • It would be nice for this top most line to have a clamp or something to keep the VCR nut on the bottom U-bend from falling.
• The second line from the top interferes with the bus bar when trying to be removed by hand.
• The second line from the bottom has a VCR connection obstructed by the same bus bar when trying to remove by hand.
• The bottom line catches on the other lines when trying to remove by hand, the other lines likely need to be removed first to allow this line to be exchanged.
• is the intent for the bus bar need to be removed to exchange these lines?

 Testing will resume in the following week for these water lines.

The four water connections on the front end plate of the APTW that will connect to the HTV were tested for replacement with the manipulators:

• The pin on these connectors were fairly easy to remove and replace with the old pin detent tool.
• The pin detent tool could also be used to manipulate the water lines while protecting the sealing face.
• The horizontal VCR joint on the DRIVE BEAM CONE (ATA3674) could be undone with the manipulator and standard wrenches, it is advisable to use 'semi-closed' wrenches on the nut to prevent it from dropping when torquing or undoing the joint.
• Of the four water lines, only the top line could be taken out and returned by hand, testing with manipulator still to be done.
  • It would be nice for this top most line to have a clamp or something to keep the VCR nut on the bottom U-bend from falling.
• The second line from the top interferes with the bus bar when trying to be removed by hand.
• The second line from the bottom has a VCR connection obstructed by the same bus bar when trying to remove by hand.
• The bottom line catches on the other lines when trying to remove by hand, the other lines likely need to be removed first to allow this line to be exchanged.
• is the intent for the bus bar need to be removed to exchange these lines?
• The VCR nuts on the water lines located by the bottom of the bus bars will require the turntable to be raised most of the way up to access with the manipulators.

 Testing will resume in the following week for these water lines.

Some issues were encountered with the turntable controls on the west hot cell, as a result the festooning post for the turntable was damaged. While the post was out of commission, the turntable was operated without actively monitoring the top of the hot cell and some cables ended up catching on the limit sensors, causing some slight fraying. The cables were duct-taped to prevent shorting for now but should be inspected and replaced accordingly in the future. We should also check that the sensor bracket did not move from where itshould be.

We successfully performed a removal and replacement of a power busbar stabuli connector on the front end plate where the HTV connects to:

• The middle-lower connector was tested as the others are more straightforward to work with.
• It may be a good idea to use a 3D printed jig to wedge into the connectors without damaging the flexures/pins.
• A longer hex bit on the hot cell torque tool would greatly help this operation.
• Even without special tooling the bottom connector returned and mated back onto the locating features relatively easily with just the manipulators.
• Longer hex bits purchased, 3D printed too left for future.
• Screws and washers are effectively captive in the connectors which helps

Today RH (CF and AK) agreed upon using the following style of gaskets for the ARIEL front ends:

• All metal rings to be used to avoid plastic retaining rings deteriorating due to radiation and leaving bits stuck in the nuts
• Parker style (4 VGR-SS and 8 VGR-SS) gaskets for all VCR joints, except
• For the piston module and hanging water line VCR joints (found in the 'back', these should use SwageLok side-load gaskets (SS-4-VCR-2-ZC-VS and SS-8-VCR-2-ZC-VS)
  • The primary reasons for which are clearance issues to fit normal loading VCR tools in these joints.

 Hardness testing was performed on 1 sample each of parts TBP1822 and TBP1823 (part of the 35-deg magnet stand assembly TBP1804),  They were found to be 60-60.5 Rc and 60-61 Rc respectively which is within spec.  The alloy used to fabricate these parts was AISI #8620. The testing was performed by Precision Heat Treat in Surrey who also performed the heat treatment at an earlier date.

Adjustment blocks were installed on the second dipole bender today (labeled "No. 03" "C15-III-57/15").  The mounting holes on the magnet were not aligned properly, so modification was required for two of the blocks.  The counterbored holes on TBP1809 were slotted 0.10" vertically, and on TBP1808, 0.06" horizontal slotting was required.  A stress analysis was performed to confirm that the blocks are still strong enough to withstand the seismic loads (PDF attached).

Assembly of two complete bender stands was completed on Friday June 2nd as well as installation of new hardware on the dipole magnets for support, adjustment, and seismic restraint.  All parts were installed according to drawing TBP1804.

One of the frames was assembled with the caster wheel configuration.  This frame was pushed from the proton hall B2 level to its final installation location in the ARIEL tunnel.  The frame was a tight fit around the first corner around BPM 26 but it could still pass by without having to remove any items including the electrical boxes on the right side of the tunnel (when facing towards ARIEL).  Part of the grouting for the BPM 26 support stand and the next three stands after that was ground away to allow more clearance for the frame.  The nut/bolt on the frame casters on one side were swapped which also provides an additional ~1/8" clearance.  After these modifications the stand was pushed out past this area and then back to its installation spot confirming that there is now ample clearance.

The other frame was assembled in the configuration with stand legs.  This frame has been moved to the proton hall extension loading bay to be used for magnet field testing.

Both dipole magnets have been assembled with the new adjuster blocks and support legs.  For the magnet labeled “No.01” the original support legs were removed, cleaned in the ultrasonic cleaner, and then reassembled with way oil  The support leg support pads (TBP1822) were replaced with new parts, but all other parts were reused.  For the other magnet, labeled “No.03”, new support legs were fabricated because it was supported differently in its previous location and did not have these parts.  The fasteners attaching the adjuster blocks and support legs to the magnets were torqued to the recommended torque for each fastener based on the fastener grade and size.

Various minor changes to the design were made since the release of drawings, primarily to improve ease of manufacture.  As-built drawings will be released after the magnets have been installed on the stands in case additional changes arise.

 BL4N 35-deg bender magnet "No. 1" has been installed onto the magnet stand with "stand leg" configuration in the Proton Hall Extension loading bay area.  The magnet fit as expected, and with the support legs resting on the ball transfers the magnet position could be easily adjusted by turning one of the adjuster leg fasteners (with the others loose).  After experimenting with position adjustment, the fasteners were torqued to 70ft*lbs.

The engineering analysis (Document #138415) as well as drawings TBP1804, 1805, 1806, 1811, and 1820 are in the process of being updated to reflect all as-built changes.

Tuesday, November 04, 2014, 16:58

This is a test.

I received a call today from Jim Adamson at Kaltech regarding modifications to shielding block TSH0225.

They had finished removing 19mm from the bottom of the block, as well as removing 10mm of material from the side plates (TSH0246).  He was wondering whether it was also necessary to remove material from the back plate (TSH0253) and weld the back plate to the 6" plate (TSH0252).  I instructed him that it is not necessary to machine and weld the back plate.  This decision is based on weld strength calculations I performed last week which showed that the existing welds are much stronger than what is required to take the load applied when lifting the block (FOS >> 10)

Testing was performed on the ceramic water breaks to be used on the beam dump water hose assemblies.  The ceramic breaks are a custom MDC product with the following description:  3kV ceramic break, 1" OD tube 0.083" wall thickness, 2" length, 304SS tube stub each end, 250psi water rating (see TRIUMF PO#  TR199315).

A load was applied perpendicular to the part, 12" away from the center of the ceramic, and increased incrementally until failure.  Two parts were tested, which failed at 125lbf and 170lbf. Equivalent to 125ft*lb and 170ft*lb when converted to torque.  Test data, photos of the failed specimens, and a more detailed description of the test setup are attached.

The purpose of the tests was to gain an understanding of how fragile the ceramic parts are.  The results show that the ceramic breaks should be able to withstand the expected loads caused by the flexible hoses being pulled as the RH water blocks are installed or removed remotely using pole tools, however this operation should still be done cautiously.  The ceramic breaks are at risk of failure if a torque of over 100ft*lb is applied, such as if someone were to stand on the ceramic breaks or the attached hose.

Painting and relabeling of the modified lower center block (TSH0461) was completed on Monday Nov 24.  The height of the modified block was measured at the four corners using a tape measure and found to be approximately 533mm at all corners.

The block was lowered to the E-Hall and aligned on Tuesday Nov 25.   No cantilever lift was required because the catwalk above the hatch had been removed.  During the block installation one of the crane cables was contacting the concrete on the edge of hatch opening.  Future installations should use a long sling or cable to prevent this.  Photos of the installation are attached.  Beamlines Group assisted with alignment using their laser tracker.  Vertical alignment was achieved using 3x3" metal shims of various thicknesses placed at 4 corners of the block.  North-south, east-west alignment was achieved by installing based on marks drawn on the E-Hall floor by Beamlines group.  After initial placement, measurements were taken with the laser tracker, and the position was adjusted.  The nominal positions of the beam dump according to installation drawing TSH0372 are:

East Face: x=-5926.3mm from EHBT CoP
South Face:  y=2223.6mm from EHBT CoP
Top Face: z=-208.0mm from EHBT CoP

The laser tracker measurements of block position after final placement are as follows:

East Face: x = -5926.2 ~ -5926.9mm
South Face: y = 2220.9 ~ 2224.7mm
Top Face: z = -209.6 ~ 211.6mm

The East face is within 0.6mm of nominal position which is excellent (note that the east-west direction cannot be corrected using the alignment rail track).  The south face position is within 2.7mm.  This is fine because adjustment can be made to the north-south position of the rail track.  The top face of the block is a maximum of 3.6mm below the nominal position which is also fine, as the lower lead shield block can be shimmed up to it's nominal position.  Data from Beamlines Group is attached (note that 19.05mm must be added to each measurement to account for the distance from target base to center).  The location of the measurement points on the top face are as follows.

                 North Side
--------------------------------------------
| 6                  4                  2 |
|                                           |
| 5                  3                  1 |
--------------------------------------------
 

We will aim to have the bottom of the lower lead shield at z = -204.8mm (this leaves nominal 2mm space to be shimmed under the rail track to bring the BD flange center to beam height).  Therefore the following thickness shims should be placed on the top of the lower center block.

Position 1: 5.8mm (suggest 1/8 + 1/16 + 1/32" shim = 5.6mm)
Position 2: 5.9mm (suggest 1/8 + 1/16 + 1/32" shim = 5.6mm)
Position 3: 6.8mm
Position 4: 6.0mm
Position 5: 5.1mm (suggest 1/8 + 1/16" shim = 4.8mm)
Position 6: 4.8mm (suggest 1/8 + 1/16" shim = 4.8mm)

Position 3 and 4 do not need shims, as the outer four corners should be adequate.

On November 28th the lower lead shield was installed with 3/8" shimming at the front end, and 11/32" shimming at the rear end.  The south and east faces were aligned as flush as possible with the lower center block.  The beam dump insert was then placed on the rails in the lower lead shield.

Laser tracker measurements were performed by Michael Vogel, and he reported that the beam dump was 1.7mm high at the rear, and 0.8mm high at the front.

The lower lead shield was lifted, and 1/32" shimming was removed from the front, and 3/32" from the rear.  The block was then placed precisely using the laser tracker with targets on the beam dump insert.

After placing the lower lead shield, Michael aligned the beam dump by shimming and adjusting the rail track, and was able to get all fiducial points within 0.2mm of the nominal position.

On December 2 the upper half of the RH water blocks were installed (4 total) using a 0.030" thick 6061-T6 Al shim torqued to 60ft*lbs as was tested in Document-114260.

Bypass loops for testing the cooling system piping were installed as shown in the attached photo.