Yesterday, Grant entered the Ante Room with David Wang to visually inspect the two sets of failed c-seals from the Heat Shield circuit, to remove the second set from the Heat Shield water blocks, and to leak check the heat shield circuit with o-rings and a water-block-to-leak-detector fitting.

Some photos of the inspected seals are in the attached design review presentation.  Some unusual marks were observed on both sets of failed seals.

The heat shield lines and water block assembly leak checked successfully to the bottom of the leak rate range ("UNDER" ~1x10-9 atm cc / sec on the Varian 979) with a great flood of helium on all joints with no response anywhere.

Bevan Moss wrote:

Today Maico completed the new leak testing tool and blank off. He tested them with C-seals without springs and all was leak tight. He then inspected the crushed seals and noticed that there was an area that was crushed less on both seals (more prominent in one,  20130906_tm3sourtrayefurb_P1020975) and that this reduced crushed zone only appears on one side of the seal. This is similar to the failure seen on both sets of the heat shield lines. On the seal with the greatest change in crush zone the average crush zone was ~0.025 and the reduced crush zone was ~0.012". He inspected the leak tester and the blank off and determined they were within tolerance and that they had a total seal goove height of 0.074" which is nominal. New seals were selected and one had a visible dimple prior to crushing (20130906_tm3sourtrayefurb_P1020978)  and the other had some defects on the inside (20130906_tm3sourtrayefurb_P1020962). These defects were marked and crushed using the same leak testing tool and blank off. The seal with the dimple showed a reduced crush zone in the same area (20130906_tm3sourtrayefurb_P1030021) and the seal with the defects on the inside showed less or undetectable change in crush zone. Maico then inspected (20130906_tm3sourtrayefurb_P1020991) and crushed the "thicker seals" (more indium coating). When inserting the seals he noticed that they fit tight on the counter bore of the blank off. He then attempted to crush the seal to the point where the copper faces would touch (as design intent) but the seal locked. The gap between the 2 copper faces was measured to be ~0.001". This setup was leak tested and found to be leak tight. When inspecting the crushed thicker seals it was found that the material had actually been pushed sideways causing a lip to form around the crush zone (20130906_tm3sourtrayefurb_P1030003). Maico then fitted the wires for the testing of the module and the heat shield line.

 Hello all,

As an addendum to Bevan's e-Log:

Maico also prepared eight (8) new retainer spring windings out of the .025" diameter stainless welding wire for the next seals that we will attempt in the Hot Cell and Ante Room

Maico, Bevan, Keith and I had a discussion about the bolt torque related to the c-seal and water block compression:
- It was found that not much torque is required to compress the "standard" design "thin" c-seals (Ultra-Seal P/N 50606 .001 - .0015 thou indium plating) - basically hand tight only with an allen wrench
- By hand-tightening the bolts with an allen key until the faces of the blocks came together, and then measuring the torque with a torque wrench, Maico discovered that about 8 foot-lbs = 96 inch-lbs (or 5/8ths of a turn past finger tight) was required
- The c-seals are fully compressed when the faces of the blocks are contacting... any additional torque applied is only to pre-load the bolts to ensure that they do not come loose due to temperature cycling and mechanical vibration
- Chad's Hot Cell torque tool is nominally set to about 168 - 180 inch-lbs (or about 14 to 15 foot lbs) based on an e-mail update from him 3-June-2013 - this torque is normal chart torque for a 1/4"-28 UNF SAE Grade 8 bolt pre-load of 3,250 lbs
- see http://www.imperialsupplies.com/pdf/A_FastenerTorqueCharts.pdf
- This amount of bolt pre-load torque may not be required if the bolts are SAE Grade 5.  I recommend at this point that the minimum possible pre-load torque be applied to achieve the chart recommended pre-load for the grade of the bolt, which should be investigated
- This is to ensure that we do not over-stress the thread inserts on the water blocks... if these inserts are damaged, we will basically render the service chase unusable and ruin the module

Cheers,

Grant

I have compiled the e-mail comments from people involved in High-Power Conditioning of TM3 w/ the new Rev 3 source tray and NiO#2 since Bevan Moss' last log on October 2nd below:

On October 2nd at 6:35pm Friedhelm Ames wrote:
TM3 heaters have been tested already yesterday in the conditioning station. Current and voltage readings were in the normal operating range.
The heaters were kept on over the night and no changes except a slow improvement in the vacuum has been seen. Today the high voltage was started. A high bias current of up to 150 micro Amps at 10 kV and an oscillation of the current with a frequency of about 3/minutes has been seen. Going higher in voltage resulted in an erratic behavior. After a visual inspection in the target hall some water leakage on top of the module was detected. it could be traced to 2 of the water connectors. They have been replaced. After this the high voltage current was down to normal values and the oscillation disappeared. With the heaters off the voltage could be raised to 22 kV, after which sparking occurred.
Earlier with the heaters on the functioning of the source has been tested. Extracted beam could be seen on the Faraday cup and the dependance on both the anode voltage and the coils could be verified.
One of the steerer plates (bottom) appears to have a short. As there is some redundancy in the steering elements in the target station optics, I don't believe this will effect the beam delivery very much.
A full test of all the source systems is planned for tomorrow including a careful attempt to rise the high voltage to a higher value. For the acceleration of 10C we will need 20.4 kV.

On October 3rd at 9:32am Bevan Moss wrote:

FYI Module is holding 24kV with zero sparks as of yet  I will continue with updates as I go

On October 3rd at 10:18am Bevan Moss wrote:

Sparking started at ~28kV I have had to dial it back to 22kV to hold stable voltage. I will continue with updates.

On October 3rd at 11:17am Bevan Moss wrote:

Sparking has forced the voltage down to 20kV. Friedhelm has asked me to turn everything on and start ramping the heaters.

On October 3rd at 4:15pm Bevan Moss wrote:

The heaters have been brought online and the Bias raised to 25kV. It has been sitting there for the last 10 minutes with 1 spark during this time. All other PS are on and beam is being extracted.

On October 3rd at 4:37pm Bevan Moss wrote:

Sparking has just occurred and the Vacuum is now getting worse significantly faster. Voltage was lowered to 21KV where it has been stable for the last 5 minutes. I am going to turn off the HV leave the module overnight with the heaters on. Hopefully we will have good Vacuum again in the morning.

On October 4th at 2:20pm Bevan Moss wrote:

I have finished completing another complete leak test on the module and it is leak tight. Once the heaters were turned off the vacuum did improve it is currently sitting at 9.3 x 10^{^-7} Torr in the service cap, 5.71 x 10^{^-6} Torr in the diagnostics box and 4.0 x 10^{^-6} in in the secondary.  I have raise the bias voltage and it can hold 20kV for 5 minutes but did have 1 spark. At 22.5kV it appears to be doing some conditioning as the current is unstable. I would not recommend moving TM3 to ITE on Monday until we can establish that the module can hold 25kV for at least 1 hour preferably with the heaters on. I have left the turbo pumps running in case there is more testing you would like to do. If you choose to move to ITE on Monday please have someone stop the turbo pumps otherwise I must wait an hour before venting on Monday. The TCS laptop on my desk should not have the screen saver come on so TCS can be operated through it.

Here is the change in the TCS vacuum after heaters were turned off earlier today:

2:20 pm readings

TCS:IG1 (service cap) 9.3 x 10^{^-7} Torr

TCS:IG2 (diagnostics box) 5.71 x 10^{^-6} Torr

TCS:IG1S (secondary vacuum) 4.0 x 10^{^-6} Torr

7:17 pm readings

TCS:IG1 (service cap) 8.69 x 10^{^-7} Torr

TCS:IG2 (diagnostics box) 5.0 x 10^{^-6} Torr

TCS:IG1S (secondary vacuum) 3.38 x 10^{^-6} Torr

Grant Minor wrote this e-mail today:

***

Hello all,

During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke.

We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars.

Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?)

Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator.

Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment.

Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section.

Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them.

I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed.

Thank you all for your assistance,

Grant

***

A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

Grant Minor wrote:

Grant Minor wrote this e-mail today: *** Hello all, During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke. We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars. Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?) Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator. Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment. Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section. Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them. I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed. Thank you all for your assistance, Grant *** A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

E-mail from today:

Hello Dave, Jane, Franco, Michael, Don (controls + Franco), Remy,

We have lost the East-West bridge drive control functionality of the ISAC Target Hall crane, and unfortunately this very much prevents us from operating the ISAC facility.

The problem was originally thought to be a stuck bridge wheel but this was not actually the case.

Mike Smith from Norelco was on-site for most of today working with Michael Rowe in an attempt to diagnose the problem.

Mike believes the problem to be one of the following three items:
- input PLC board on the crane bridge
- output PLC board in the interface panel in B1 level Room 5 (ISAC remote crane control room)
- communication between the two boards (festooning hardware)

We are in a pretty dire situation without the ISAC target hall crane, and we lack the expertise within the Remote Handling group in PLC hardware to diagnose this on our own.  I need to ask for somebody's help from the controls group in diagnosing and repairing this problem.

I have notes from my telephone conversation with Mike Smith which I can discuss with a controls representative in the morning.  There is a pretty muddled history of the design and implementation of these controls but essentially Norelco outsourced the PLCs and festooning to a third party contractor, Seattle Crane in the US, which no longer exists anymore.  Neither Kone, Seattle Crane, or Norelco made appropriate as-built drawings of the wiring of the PLCs.  Mike Smith at Norelco had a stack of hand-written notes about the as-built configuration he made at the time of installation in the late 1990's but these notes have not been located yet.

The PLCs appear to be Allan Bradley but the modules are from approximately 1997 and we're not sure if spare units are available.

Mike Smith says the easiest thing to do at this point is install replacement PLC modules on the crane bridge and in the control room that have confirmed functionality.  If control is not restored then the problem is likely in the festooning.

Control of the crane bridge was temporarily attained by bypassing the controls (speed control / end limits, etc.) and sending drive signal directly to the drive motor controller in order to return the bridge to its home position at the access ladder.  This mode of operation is not safe as the crane has no speed reduction / limits to prevent driving the bridge into the walls at max speed.

Jane / Remy / Franco: please let me know who I should correspond with on the controls side and I will talk with them tomorrow morning.

Thanks,

Grant

E-mail correspondence on the Target Hall crane fault diagnosis between October 11th and 15th:

On 15/10/2013 3:34 PM, Grant Minor wrote:

Hello all,

I spoke with the Norelco tech Mike Smith this afternoon regarding the failed thermal sensor component / circuit.  He will call Kone on holiday Monday (their office is not closed for Canadian Thanksgiving) and determine if they have a replacement module.

I explained (to the best of my understanding) the diagnosis performed by Dave Morris / controls group and the test that was done to isolate the thermal sensor unit.  Unfortunately, Mike Smith's drawings from Seattle crane contain notes and revisions that don't match the drawing that Dave Morris provided me (Seattle Crane dwg E5 - 1/1 revision 1 02-10-99), and he was unable to confirm in his opinion whether the bypass performed by Dave Morris has correctly isolated the problem components.  In addition, he believes that the bypass he performed on Wednesday isolated portions of circuitry that are inconsistent with this thermal sensor circuit.

Mike Smith will return Tuesday morning to review the bypass and schematics.  He recommended not operating the crane until he has a chance to confirm this thermal sensor issue.

From the drawings I have in hand and from discussions with Dave Morris, the failed component might be the thermistor sets inside the motor housing which are in series with the suspect module, or the module itself.  My understanding is that the crane is designed with two pairs of motors for the bridge drive, and each pair is protected by one of these thermal modules (thus there are two modules total).  Both modules would be the same age and both might be susceptible to the same failure mode.

As it is unclear to me the exact nature of the drive system failure, and I have a recommendation from Norelco to hold on operation of the crane, I must unfortunately state that the crane should not be used and we will have to wait until Tuesday to gather more information.

I am not so concerned about temporary loss of the thermal protection circuitry, as we have not had an overheating scenario in those motors in the lifetime of the crane (to my knowledge).  I am more concerned about getting 100% confirmation that we have addressed the problem.  If we attempt to use the crane and experience another failure during a lift that prevents us from placing the load down safely with the crane stuck in position over a target station, then we would be in much deeper trouble.

My apologies for this situation.

Best regards and have a good weekend,

Grant


On 11/10/2013 1:28 PM, Grant Minor wrote:
Thanks Dave,

I've contacted Norelco, Mike Smith and I are planning a path forward for further diagnosis and repair.

The crane MUST NOT be used to carry a load until it is identified whether the module or the sensors have failed, the reason for the failure, and the functionality of the thermal sensors has been restored.

Without the thermal sensor circuit we lose protection from motor overheat, an event which could have much more serious failure implications to the crane.

Cheers,

Grant

On 11/10/2013 1:04 PM, David Morris wrote:
The fault has been identified in the Target Hall Crane as a failed motor temperature module, or motor temperature sensor, on the East-West motion. The status contact in the module was bypassed allowing crane motion. There was no fault with the controls.

Dave

The blocks above the pre-separator section of the ISAC beam lines were removed today in the Target Hall by Bevan Moss and Chad Fisher with Lynne LeMessurier present from RPG.

The two blocks in the layer below the top layer above the pre-separator were damaged on removal (see attached photos).  This was due to an attempt to lift them in the incorrect sequence caused by misinterpretation of the labeling / numbering system drawn on the top of the blocks.  The blocks are labelled for the replacement sequence, but not for the removal sequence.  An attempt was made to lift a block that had an overlap with an adjacent block that was not visible, causing damage to two blocks.  A note was added in black marker to the critical block to indicate that it should be removed last (shown in attached photo).  The block replacement sequence will be photographed and a procedure document will be written by Bevan, with a bound copy of the procedure provided in the Target Hall, in order to prevent such an incident from occurring again.  Broken pieces of the blocks were collected and placed in a 5-gallon pail.

The removal of all of the blocks caused the depression in the target station to fluctuate around the trip point (17 inches of water column), which caused the ventilation alarm to trip on and off constantly.  Operations contacted me to ask if this could be remedied.  Bevan closed the damper to ITW, which brought the depression to a low enough value so that the alarm could be defeated properly.  This was around 2pm.

Preliminary co-ordination of the ITE Harp 9 removal job was discussed by Scott Kellogg and Ron Kuramoto around lunch time.  Keith Ng (RH) and Scott searched for the blank-off that was used during the last Harp 9 repair (removed from B/L in March 2013) but they could not locate it.  I was later able to find the blank-off bagged in a grey rubbermaid bin at the bottom of the stairs in the target storage vault pit (under the SHC / NHC ventilation and filters).  The blank-off was found bagged and tagged as surveyed at 100 counts @ 0.5m on March 27, 2013.  I gathered Scott, Ron Kuramoto, and Danka Krsmanovic to co-ordinate the Harp 9 removal and blank-off installation, starting at around 2:30 pm.

For the Target Hall Harp work, Ron Kuramoto and Scott prepared the required tools and plastic coverings / bags.  I acted as crane operator.  Danka was the surveyor present.  Anders Mjos joined the group for photos and moral support.

Scott disconnected services from the Harp device, and prepared it for lift.  I called operations to confirm that the section of B/L had been vented.  Ron and Scott used the RH tools to disconnect the 2 vacuum flange bullet bolts.  Ron directed the crane with signals and I lowered the chain rigging to the Harp, it was connected and removed from the BL with the main hoist.  (I made the mistake of directing the rigging of the harp to the main 20T hoist, when in fact the 3T Aux hoist should have been used, as the main hoist can't reach to the vacuum / storage pit on the west side of the hall.)

Danka performed several swipes and measurements of the Harp after removal.  Alpha contamination was discovered on the bottom of the Harp flange.

Ron and Scott installed the blank-off flange, then I called operations to indicate that pumping should start.

Ron and Scott retrieved the vacuum test flask jig / support structure from the West storage pit and brought it down to the pre-separator pit.  I lowered the Harp into this flask jig, and Scott used the side-wall location bolts from the top of the harp assembly to fasten the flanges of the Harp and the jig.

I had trouble at first activating the Aux hoist but soon realized that it's necessary to press the Aux pendant "ON" button down for several seconds before the power actually turns on to this hoist.

We then removed the main hoist hook and attached the Aux hoist to the Harp / flask.  The assembly was transported to the West storage pit.

Some of the handrails had to be removed to facilitate initial movements of the Harp assembly.  These handrails should be replaced tomorrow morning.

Scott also submitted a Work Order for a back-up blank-off today, job number 40303.

The pre-separator blocks have NOT been replaced yet, we will do this tomorrow morning.

It was a busy day, thanks to all for their efforts and for staying a bit late to get the blank-off installed.

Best regards,

Grant

The pre-separator-area blocks were replaced today by Bevan and Travis, with Grant present for photos and observation.

Photos are attached of each block replacement.

Some difficulty was encountered in replacing the "Layer #3" blocks (photo 9 attached).  An interference occurred between the outside of the west "Layer #3" block and an aluminum channel directly beside the rubber pad that supports the block.  The block was initially resting on this channel but slipped off, breaking off some concrete at the edge.   See photo 12 attached and notice crumbled concrete inside this channel around the area where the interference occurred. Photo 14 shows the damaged block edge.

It should be noted that when these two blocks are resting flat, there should be a gap between them (i.e. they should NOT be touching as shown in photo 9).

Anders Mjos wrote:

The conductance check on the electrical check was found to be outside of accepted values for the TBHT at 7.0 mOhm. A successful attempt to reduce the conductance was made by re-tightening the bolts on the TGHT legs in the SHC. Grant operated the manipulator and torqued the bolts to about 1/8 of a turn with the air ratchet. The conductance across the TGHT (A-B) could be improved from 7.0 mOhm to 6.2 mOhm by re-tightening the bolts. The value is still somewhat higher than usual, but was deemed to be acceptable. Measurements attached.

1. TGHT leg bolts were loosened, and resistance was measured very high, as expected.

2. TGHT leg bolts were then tightened, loosened, and tightened again, and the resistance improvement was then measured (from 7.0 mOhm to 6.2 mOhm).

3. Another attempt was then made to loosen and tighten to see if there would be further improvement.  The right bolt was loosened, but the left bolt was found to be quite stiff and was left as-is (not fully loosened) for fear of breaking the supporting insulator.  Both bolts were then re-tightened, and resistance was checked to be about the same as achieved during the previous loosening / tightening attempt (step 2 above).

To tighten the bolts in all cases: the air ratchet was used first to turn the bolts "air tight" (air pedal only, no manual force) and then with the air pedal held down, manual torque was applied to turn no more than 1/8th of a turn.

Friedhelm Ames commented to me that there appears to be a short between High Voltage and Ground at TM3 at ITE.

Friedhelm, Anders and I initiated a survey by Andrew from RPG, and entered the Target Hall at approximately 5pm to visually assess the HV cover at the MAA.  There was no clear indication of a problem.  We opted to wait for David to return as we weren't sure exactly what we were looking for.

Friedhelm and Anders confirmed in the Electrical Room that the short is in the Target Hall.

There were some problems this morning with provision of FEBIAD gas to the the target at ITE.  It was suspected that the gas line was improperly connected to the module, or that the valves at the module were not opened.

Remote Handling was asked to open the HV cover on TM3 at ITE and check the connection of the FEBIAD gas line.  Grant Minor, Maico Dalla Valle, Dan McDonald, and Anders Mjos entered the Target Hall before lunch on Work Permit I2014-07-07-1.

Maico entered ITE with suit and respirator, disengaged / disconnected the interlocks and removed the HV cover.  He inspected the FEBIAD gas line and found that it was properly connected, and that the valves were both open.  The gas line is shown in the attached image, with both valves marked in red.

Grant and Anders went into the electrical room to examine the supply gas bottle and supply circuit, and found that the supply gas pressure to the regulating valve was low.  It was concluded that the problem was not at the Target Module / Target Station.

Grant returned to the Target Station and instructed Maico to replace the HV cover.  After this was done, Grant confirmed with Jon Aoki over the phone that the interlock switch signals were both reading "OK".

Friedhelm Ames made an entry in today's ISAC Ops e-log regarding the supply pressure to the regulating valve.

ISAC E-log

Target Ionizer tube cooling lines "C" and "D" in Target Module 2 were both leak checked at the Hot Cell.  The lines were both blanked-off.

Bill Paley was present with David Wang in the Target Hall, operating the leak detector.  Grant Minor and Anders Mjos were present with Chad Fisher at the Hot Cell.

The "D" circuit was already blanked off with a copper blank containing silver brazed stainless steel inserts and indium plated inconel c-seals.

The "C" circuit was already blanked off with an aluminum blank and rubber o-rings.

The cooling lines are indicated in the attached screencapture from ITA6021 Rev .14, and are also labelled in the attached photo.

The "D" circuit was checked first, starting at around 12pm.

1.  The helium flow was tested by blowing dust on the work table.

2. The base leak rate on the Varian leak detector was confirmed in the "D" circuit to be stable at 10^-9 atm cc / sec and the pressure was 10^-4 Torr.

3. Each soldered joint on the "D" circuit was sprayed with a 2-second blast of helium - no response at the Varian.

4. The joint between the module-half of the water block and the copper blank-off was sprayed with a 2-second burst - no response.

5. A general area flood of helium sprayed for 10 seconds was applied all around both the "D" and "C" blocks - no response.

It was concluded that the "D" circuit had passed this leak check.  The "C" circuit was checked next.  The detector was changed to the "C" circuit and the circuit was pumped down.  At around 2pm the leak check resumed.

6. The base leak rate on the Varian leak detector was confirmed in the "C" circuit to be stable at 10^-9 atm cc / sec and the pressure was 10^-4 Torr.

7. The solder joint area on the "C" circuit was sprayed with an 8-second burst of helium - no response.

8. The joint between the two halves of the water blocks was sprayed with a 6-second helium burst - no response.

It was concluded that the "C" circuit had passed this leak check.

The electrical continuity of the "C" and "D" circuits were confirmed in the Target Hall at the service cap, and in the module at the Hot Cell, to double check that the correct lines has been pumped and leak checked - this was OK.

The ionizer tube heater lines appear to be leak tight at the Hot Cell.

Hello all,

Access was required this morning to the TCS TK2 Expansion Tank to open a valve and re-fill the tank.

As the access path to this tank lacked any form of Fall Protection, Fall Arrest was employed.

Worksafe BC Part 11 Section 11.2 requires fall protection to be employed at elevations greater than 3 meters.  The expansion tank is located at an elevation far exceeding this regulation.
http://www2.worksafebc.com/publications/ohsregulation/part11.asp

The worker, Allon Messenberg, was fitted with a harness, personal self-retracting lifeline / lanyard with shock-pack, and carried two steel rope ring-ended slings with him to tie off to the structural beams overhead.  All products used were DBI-Sala safety products, which are kept in the Remote Handling machine shop Fall Protection Equipment lockers.  Several RH group members have keys to this locker, including me.

Three configurations were required to gain access to the tank (photo attached):
Configuration 1: Traversing the section between the 2nd floor offices and the ventilation duct.
Configuration 2: Traversing the section between the ventilation duct and the "cross point" of the structural beams.
Configuration 3: Tied-off to the "cross point" of the structural beams and working within the reach limits of the personal SRL and shock-pack.

Transitions between the three configurations required brief moments of un-clipping from the harnesses.  This will be solved in the future via the use of additional carabiners and lanyards.

Operations has advised me that they have observed either TRIUMF workers or contractors working in this area without wearing Fall Arrest equipment.  My request is for staff and operations to stay vigilant in observing access to this area. 

Any work performed in this area without the usage of fall arrest must be reported in an NCR.

I welcome any comments from those who have ideas for improving the safe method of access to this tank.

The cooling package is described in the TCS operating manual, Document-103881.  Page 8 states clearly that Fall Arrest is required to access this valve.

Regards,

Grant

The MK4 optics tray prepared for TM2 (as per ITA6082) had an undersized bore for the banana plug, making insertion of the plug extremely difficult / impossible.

Reference stores parts were collected to check the fit:  male plug 3-4/01152 and female jack 1-6/1201.  The size of the female jack was checked with a #21 drill at .159".

The bore on the optics tray was drilled out dry using the #21 drill to the same depth as the original blind hole, by Maico and Dan McDonald.  Chips were carefully collected in a rag, and the hole was blown out with compressed air.

The fit was checked to be snug but possible, using the male plug and the Remote Handling ultem base (ref ITA6023).

Redlines and photos are attached.

The water blocks soldered to the TM2 cooling lines (see Ref. Assy. ITA6144) were inspected.

See attached inspection reports and drawings.

It was found that only 18% of the seal groove diameters and 55% of the depths were in spec.

The source tray must be tested on the test stand with c-seals to ensure that these blocks will seal properly on the module.

Hello,

For reference, I have attached a list of all solder joints in the TM2 Source Tray, and the alloys used for the most recent build (as per ITA6144).   I have also provided data sheets for the alloys, and PDF copies of all the drawings calling out the joints.

Certanium 34 C (222 deg C melt point) was used for all low-temperature solder applications, except for the steel inserts in the water blocks.  Previous revisions of the Source Tray drawings called out a higher-temperature McMaster lead-based solder (304 deg C melt point).  The change to Certanium 34 C on the prints was made in Bevan's era, during the transitional update of the source tray drawings between the TM1 and TM3 source tray projects, just before I took over updating the drawing packages.  I believe Bevan made this change on general advice from Guy Stanford, who stated that Certanium 34 C is a much easier solder to work with and makes a more mechanically reliable joint.  However, the change really should have had more review and validation testing before being implemented.

This said, we now have a source tray that uses Certanium 34 C, and rework of every joint would set us back too far in the schedule to meet the deadline, so we will have to run with it as it is.

Of particular concern are the joints on the target oven and coil heating blocks, which are now using a lower temperature solder (222 deg C melt vs 304 deg C melt).  We have experienced a coil block failure on TM4 due to a failed solder joint, although the failure mode is not known for certain.  We also had two recent failures on the test stand of a coil joint soldered with Certanium 34 C, but this was due to an error in the connection of cooling water to the joints (i.e. the joints had no coolant flow and the solder melted... this is expected!).

Anders and I have performed independent thermal analysis on the joints due to the radiative and electric heating from the target, and have concluded that even with a lower temperature solder, there is a lot of margin, so the risk seems to be low.  This work will be published in a separate report to be circulated later.

Thank you and regards,

Grant

Don Jackson brought to my attention some potential HV minimum spark gap issues with the source tray cooling lines.  It appears:

- The gap between the Ground Electrode cooling lines (ground) and Heat Shield cooling lines (HV) does not meet the spec of 0.668" MIN called out on the print

- The gap between the Window cooling lines (ground) and the Mounting Support Plate cooling lines (HV) does not meet the spec of 0.668" MIN called out on the print.

A copy of the tube bending drawing is attached with the spec highlighted in green.

The 0.668" MIN gap spec came from a check I asked Mark Llagan to perform to find the minimum gap when he was preparing the model for the tube bending drawing.  This gap was approved Jan 28th, 2015 by Friedhelm Ames on ECO-3587.

The source tray with the tubes bent as it is currently has been tested to 55 kV at the test stand by Anders, see:
https://elog.triumf.ca/TIS/Test-Stand/447

I am away June 12th to June 24th on vacation inclusive.  Isaac Earle is in charge of Remote Handling in my absence.

I am delegating Don Jackson responsible for resolving this TM2 HV issue, collecting and documenting the approval to proceed from the T/IS group leader with the source tray as-is, or to try to re-work the cooling lines and delay the project.

The source tray is NOT to be installed in the Hot Cell until this issue has been resolved.

Regards,

Grant

-- 
Grant Minor, M.A.Sc., P.Eng.
TRIUMF Remote Handling Group Leader
Nuclear Engineer
4004 Wesbrook Mall, Vancouver
BC, Canada, V6T2A3
gminor@triumf.ca
(604) 222-7359(604) 222-7359
http://www.triumf.ca/profiles/4557

A few additional higher-resolution photos of the TM2 source tray with new spacer ITA6270, and bent copper ground line clip.

Also attached is a photo of a 3/8" copper line placed in between the loose ECR line and the Mounting Support Plate line, for dimensional reference.