Around 7pm tonight I opened the main valve approximately 1.5 turns on the Argon bottle used for venting ITE, located in the ISAC electrical room. The cylinder pressure gauge increased to about 1500 psi. I adjusted the secondary regulator (pressure relief valve) until it read 2psi. I opened the needle valve in the vent line completely. The pressure on the pressure relief valve dropped slightly and I adjusted it again until it read 2psi. The Argon supply should be ready for venting ITE w/ TM4. The station should be vented around midnight to prepare for TM4 disconnect tomorrow morning. If there are problems with the venting process, an operator should check the pressure on the secondary regulator / pressure relief valve to ensure it is reading 2psi. - Grant Minor

TM4 was successfully moved from the South Hot Cell to the South-West Silo remotely by Grant Minor, with help from David Wang and Don Jackson.  The move was without problems.  See ISAC Work Permit 2012-12-04-1.

Bevan attempted to start pumping down on TM4 today remotely but there was an issue.  He will report on Monday.

I went into the Target Hall this afternoon while on the phone with Bevan and performed the following:

- Checked the knurled vent valves on the side of each turbo, both were found to be in the tightened position (all the way clockwise)

- Closed the VAT valve between the scroll pump and the nuclear ventilation

Attached are some photos of TM4 in the Conditioning Station after the no-module HV test completion.  Only vacuum services are connected.

See attached TAPC 477 documentation.

The water block sealing surfaces on the module-side of TM3 were polished this week by Chad Fisher, using his air-ratchet controlled rotating head tools (single and double-surface polishing tools).

Photos of the polishing tools in-use are attached (from August 16 - 21, 2013)

Scotch-brite pads were used, followed by 2000 grit sandpaper, then white felt with isopropanol, then lint-free dry polishing pads, each on detachable heads that are used with the polishing tool.

The module-side water block surfaces were inspected by Grant Minor on 21 Aug 2013 using a level telescope and camera looking through the hot cell window.  Results can be viewed here, with photos indicating which block is being viewed in sequence.

https://documents.triumf.ca/docushare/dsweb/View/Collection-11251

After a great struggle with the water block jigs, the TM3 source tray was finally advanced fully into the service tray and all water blocks were connected and torqued to spec in the hot cell by Chad Fisher.

Notes and description of problems encountered during the installation:

- The water lines for the optics tray interfered with the module-side water block jig, and had to be bent out of the way to clear while advancing the tray (see attached photos)

- The water block jig on the source tray side was too low relative to the module side, and had to be eventually unbolted completely from it's mounting bracket in order to raise it up into alignment

- A piece of aluminum plate was placed under the jig, and the pneumatic table was raised in order to move the jig upwards

- After many repeated attempts to engage the blocks by pressing the jigs together, it was discovered that some of the blocks had rotational misalignment with their respective counterparts on the module side, preventing the pins from engaging properly

- By looking through the bolt holes on several of the blocks, it was possible to determine which direction they had to be rotated

- Chads polishing tool was used to apply some torque to rotate the source-tray-side water blocks slightly so proper alignment of the pins could be achieved

- See the attached photo which indicates which blocks had to be rotated (blocks 5, 6, 9, 10, and 11)

- The blocks had the bolts installed and were also torqued in the numbered order shown in the attached photo

Next steps are to remove the water block jigs, re-connect the loosened brackets for the remaining water lines (entrance window, optics tray), re-install the containment box and associated VCR connectors, video inspect, and leak check.

While continuing on the TM3 Rev 3 source tray installation, Chad noticed today that one of the steerer wires is broken and has come out of the thermocouple-style connecter block (see attached photos).

From drawing ITA2826 (Rev D02 attached) it is likely that this is for either the "upper" or "right side" steerer plate.

Today Chad finished making all connections to the new source tray, except for:

- steerers - broken wire, must be repaired at a later date

- IMG gauge - bracket required, old bracket thrown out with previous tray by accident, see ITA3301, must be installed at a later date

- entrance window - the window lines on module side were left blanked-off for leak checking, containment box must be removed in the next few days to complete steerer repairs so these lines were not connected to save work from having to be repeated

The containment box was replaced, and TM3 was moved to the CS for pump-down and leak checking (see other e-log).  David reported at 6:15 pm that the turbos were turned on at the CS and the vacuum looked normal (so far).

As stated above, Chad realized today that he had thrown out the old IMG gauge bracket with the previous source tray, and a new one must be manufactured.  The assembly is described by ITA3301.  Three components must be manufactured (ITA3302, ITA3303, and ITA2673).  A stock split-ring clamp for ITA2673 (MDC #716001) was graciously donated by Dimo Yosifov.  Grant Minor submitted a work order to the machine shop today for 2 each of ITA3302 and ITA3303.

Target Module 3 with no target, and a partially-installed Rev-3 source tray (no steerers connected, no IMG connected, entrance window lines blanked off on module side and not connected to containment box) was transported successfully to the Conditioning Station by Travis Cave and David Wang at around 4pm today.  David attached the vacuum system for start of pump-down to prepare for leak checking.

David Wang started a helium leak check on TM3 at the CS this morning.

David will create an e-log once his check is complete, but here are the preliminary results:

- target oven +/- OK

- mounting support plate OK

- ionizer tube heater +/- OK

- extraction electrode OK

A large leak was found in the heat shield circuit:

- The module pumped down with the leak rate stabilizing at 4.8xE-9 atm.cc/sec
- 60 psi helium was applied to the heat shield fitting on the right side of the Y-shaped connector (see attached sketch, each side of the "Y" is separated by a face to face metal contact only, i.e. no o-ring seal separates supply and return water)
- The maximum leak response was detected in 10 seconds, with base pressure rising from 2.0E-2 torr to 2.8xE-2 torr
- The helium was vented from the line by using an allen wrench to open the valve on the water quick-connect (see attached photo)
- The module continued pumping down, and after some time, the allen wrench was used again to open the valve, but air rushed inside, indicating vacuum was being drawn inside the water line through the leaking interface
- As the module continued to pump, this vacuum vent air rush inside the water line could not be reproduced by opening the valve again with the allen key

The module is currently pumping down at the Conditioning Station. We will leak check the remaining lines tomorrow with whatever base leak-rate we have achieved, and then move TM3 to the Hot Cell for further diagnosis.

Bevan Moss wrote:

chad fisher wrote: The heat shield circuit was pressurized with air (15-20 psi) and snoop applied to the water block joint and brazes. Bubbles formed at the joint between the two water blocks indicating that the leak is a c seal problem. Pictures attached but also on docushare Collection-11285.  Last night prior to this leak check the Heat Shield was pumped on and helium sprayed on the circuit. The pressure was 0.0 x 10^-4 Torr. The leak was traced to water block. response time was 2 seconds and went from 5.5 x 10^-8 to 1.6 x 10^-6 atm*cc/sec. There was also a response without spraying helium (drift from the nozzle). The results of this testing prompted the testing completed in Chads elog.

 Please find attached some schematics which help illustrate the setup for this heat-shield circuit leak check.

Today Maico measured the thickness of three indium-plated inconel c-seals, presumably Ultra-Seal P/N 50606.  These seals had NOT been in the Hot Cell.

Uncompressed (new) "thin" c-seal

OD .435"

ID .300"

Thickness .094"

Uncompressed (new) "thick" c-seal

OD .438"

ID .280"

Thickness .096"

Compressed (used) "thin" c-seal (Beginning of Life bench test compression only, not long term compression)

OD .438"

ID .294"

Thickness .079"

Maico commented that he measured the thickness of a few other compressed "thin" c-seals and they all had .079" thicknesses.

The seal space when the blocks are bolted together (shown on ITA2342 Rev F) should be .070" (steel insert CB) + .002" (recess on module side)  + .002" (recess on source-tray side) = .074"

Thus the compressed seals seem to have .079" - .074" = .005" spring-back.

C-seal drawings from Ultra-Seal and Garlock in chronological order from Guy Stanford's design file are attached.  Guy's whole design file is also attached for reference.

Yesterday, Grant entered the Ante Room with David Wang to visually inspect the two sets of failed c-seals from the Heat Shield circuit, to remove the second set from the Heat Shield water blocks, and to leak check the heat shield circuit with o-rings and a water-block-to-leak-detector fitting.

Some photos of the inspected seals are in the attached design review presentation.  Some unusual marks were observed on both sets of failed seals.

The heat shield lines and water block assembly leak checked successfully to the bottom of the leak rate range ("UNDER" ~1x10-9 atm cc / sec on the Varian 979) with a great flood of helium on all joints with no response anywhere.

Bevan Moss wrote:

Today Maico completed the new leak testing tool and blank off. He tested them with C-seals without springs and all was leak tight. He then inspected the crushed seals and noticed that there was an area that was crushed less on both seals (more prominent in one,  20130906_tm3sourtrayefurb_P1020975) and that this reduced crushed zone only appears on one side of the seal. This is similar to the failure seen on both sets of the heat shield lines. On the seal with the greatest change in crush zone the average crush zone was ~0.025 and the reduced crush zone was ~0.012". He inspected the leak tester and the blank off and determined they were within tolerance and that they had a total seal goove height of 0.074" which is nominal. New seals were selected and one had a visible dimple prior to crushing (20130906_tm3sourtrayefurb_P1020978)  and the other had some defects on the inside (20130906_tm3sourtrayefurb_P1020962). These defects were marked and crushed using the same leak testing tool and blank off. The seal with the dimple showed a reduced crush zone in the same area (20130906_tm3sourtrayefurb_P1030021) and the seal with the defects on the inside showed less or undetectable change in crush zone. Maico then inspected (20130906_tm3sourtrayefurb_P1020991) and crushed the "thicker seals" (more indium coating). When inserting the seals he noticed that they fit tight on the counter bore of the blank off. He then attempted to crush the seal to the point where the copper faces would touch (as design intent) but the seal locked. The gap between the 2 copper faces was measured to be ~0.001". This setup was leak tested and found to be leak tight. When inspecting the crushed thicker seals it was found that the material had actually been pushed sideways causing a lip to form around the crush zone (20130906_tm3sourtrayefurb_P1030003). Maico then fitted the wires for the testing of the module and the heat shield line.

 Hello all,

As an addendum to Bevan's e-Log:

Maico also prepared eight (8) new retainer spring windings out of the .025" diameter stainless welding wire for the next seals that we will attempt in the Hot Cell and Ante Room

Maico, Bevan, Keith and I had a discussion about the bolt torque related to the c-seal and water block compression:
- It was found that not much torque is required to compress the "standard" design "thin" c-seals (Ultra-Seal P/N 50606 .001 - .0015 thou indium plating) - basically hand tight only with an allen wrench
- By hand-tightening the bolts with an allen key until the faces of the blocks came together, and then measuring the torque with a torque wrench, Maico discovered that about 8 foot-lbs = 96 inch-lbs (or 5/8ths of a turn past finger tight) was required
- The c-seals are fully compressed when the faces of the blocks are contacting... any additional torque applied is only to pre-load the bolts to ensure that they do not come loose due to temperature cycling and mechanical vibration
- Chad's Hot Cell torque tool is nominally set to about 168 - 180 inch-lbs (or about 14 to 15 foot lbs) based on an e-mail update from him 3-June-2013 - this torque is normal chart torque for a 1/4"-28 UNF SAE Grade 8 bolt pre-load of 3,250 lbs
- see http://www.imperialsupplies.com/pdf/A_FastenerTorqueCharts.pdf
- This amount of bolt pre-load torque may not be required if the bolts are SAE Grade 5.  I recommend at this point that the minimum possible pre-load torque be applied to achieve the chart recommended pre-load for the grade of the bolt, which should be investigated
- This is to ensure that we do not over-stress the thread inserts on the water blocks... if these inserts are damaged, we will basically render the service chase unusable and ruin the module

Cheers,

Grant

I have compiled the e-mail comments from people involved in High-Power Conditioning of TM3 w/ the new Rev 3 source tray and NiO#2 since Bevan Moss' last log on October 2nd below:

On October 2nd at 6:35pm Friedhelm Ames wrote:
TM3 heaters have been tested already yesterday in the conditioning station. Current and voltage readings were in the normal operating range.
The heaters were kept on over the night and no changes except a slow improvement in the vacuum has been seen. Today the high voltage was started. A high bias current of up to 150 micro Amps at 10 kV and an oscillation of the current with a frequency of about 3/minutes has been seen. Going higher in voltage resulted in an erratic behavior. After a visual inspection in the target hall some water leakage on top of the module was detected. it could be traced to 2 of the water connectors. They have been replaced. After this the high voltage current was down to normal values and the oscillation disappeared. With the heaters off the voltage could be raised to 22 kV, after which sparking occurred.
Earlier with the heaters on the functioning of the source has been tested. Extracted beam could be seen on the Faraday cup and the dependance on both the anode voltage and the coils could be verified.
One of the steerer plates (bottom) appears to have a short. As there is some redundancy in the steering elements in the target station optics, I don't believe this will effect the beam delivery very much.
A full test of all the source systems is planned for tomorrow including a careful attempt to rise the high voltage to a higher value. For the acceleration of 10C we will need 20.4 kV.

On October 3rd at 9:32am Bevan Moss wrote:

FYI Module is holding 24kV with zero sparks as of yet  I will continue with updates as I go

On October 3rd at 10:18am Bevan Moss wrote:

Sparking started at ~28kV I have had to dial it back to 22kV to hold stable voltage. I will continue with updates.

On October 3rd at 11:17am Bevan Moss wrote:

Sparking has forced the voltage down to 20kV. Friedhelm has asked me to turn everything on and start ramping the heaters.

On October 3rd at 4:15pm Bevan Moss wrote:

The heaters have been brought online and the Bias raised to 25kV. It has been sitting there for the last 10 minutes with 1 spark during this time. All other PS are on and beam is being extracted.

On October 3rd at 4:37pm Bevan Moss wrote:

Sparking has just occurred and the Vacuum is now getting worse significantly faster. Voltage was lowered to 21KV where it has been stable for the last 5 minutes. I am going to turn off the HV leave the module overnight with the heaters on. Hopefully we will have good Vacuum again in the morning.

On October 4th at 2:20pm Bevan Moss wrote:

I have finished completing another complete leak test on the module and it is leak tight. Once the heaters were turned off the vacuum did improve it is currently sitting at 9.3 x 10^{^-7} Torr in the service cap, 5.71 x 10^{^-6} Torr in the diagnostics box and 4.0 x 10^{^-6} in in the secondary.  I have raise the bias voltage and it can hold 20kV for 5 minutes but did have 1 spark. At 22.5kV it appears to be doing some conditioning as the current is unstable. I would not recommend moving TM3 to ITE on Monday until we can establish that the module can hold 25kV for at least 1 hour preferably with the heaters on. I have left the turbo pumps running in case there is more testing you would like to do. If you choose to move to ITE on Monday please have someone stop the turbo pumps otherwise I must wait an hour before venting on Monday. The TCS laptop on my desk should not have the screen saver come on so TCS can be operated through it.

Here is the change in the TCS vacuum after heaters were turned off earlier today:

2:20 pm readings

TCS:IG1 (service cap) 9.3 x 10^{^-7} Torr

TCS:IG2 (diagnostics box) 5.71 x 10^{^-6} Torr

TCS:IG1S (secondary vacuum) 4.0 x 10^{^-6} Torr

7:17 pm readings

TCS:IG1 (service cap) 8.69 x 10^{^-7} Torr

TCS:IG2 (diagnostics box) 5.0 x 10^{^-6} Torr

TCS:IG1S (secondary vacuum) 3.38 x 10^{^-6} Torr

Grant Minor wrote this e-mail today:

***

Hello all,

During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke.

We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars.

Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?)

Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator.

Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment.

Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section.

Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them.

I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed.

Thank you all for your assistance,

Grant

***

A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

Grant Minor wrote:

Grant Minor wrote this e-mail today: *** Hello all, During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke. We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars. Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?) Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator. Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment. Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section. Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them. I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed. Thank you all for your assistance, Grant *** A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

E-mail from today:

Hello Dave, Jane, Franco, Michael, Don (controls + Franco), Remy,

We have lost the East-West bridge drive control functionality of the ISAC Target Hall crane, and unfortunately this very much prevents us from operating the ISAC facility.

The problem was originally thought to be a stuck bridge wheel but this was not actually the case.

Mike Smith from Norelco was on-site for most of today working with Michael Rowe in an attempt to diagnose the problem.

Mike believes the problem to be one of the following three items:
- input PLC board on the crane bridge
- output PLC board in the interface panel in B1 level Room 5 (ISAC remote crane control room)
- communication between the two boards (festooning hardware)

We are in a pretty dire situation without the ISAC target hall crane, and we lack the expertise within the Remote Handling group in PLC hardware to diagnose this on our own.  I need to ask for somebody's help from the controls group in diagnosing and repairing this problem.

I have notes from my telephone conversation with Mike Smith which I can discuss with a controls representative in the morning.  There is a pretty muddled history of the design and implementation of these controls but essentially Norelco outsourced the PLCs and festooning to a third party contractor, Seattle Crane in the US, which no longer exists anymore.  Neither Kone, Seattle Crane, or Norelco made appropriate as-built drawings of the wiring of the PLCs.  Mike Smith at Norelco had a stack of hand-written notes about the as-built configuration he made at the time of installation in the late 1990's but these notes have not been located yet.

The PLCs appear to be Allan Bradley but the modules are from approximately 1997 and we're not sure if spare units are available.

Mike Smith says the easiest thing to do at this point is install replacement PLC modules on the crane bridge and in the control room that have confirmed functionality.  If control is not restored then the problem is likely in the festooning.

Control of the crane bridge was temporarily attained by bypassing the controls (speed control / end limits, etc.) and sending drive signal directly to the drive motor controller in order to return the bridge to its home position at the access ladder.  This mode of operation is not safe as the crane has no speed reduction / limits to prevent driving the bridge into the walls at max speed.

Jane / Remy / Franco: please let me know who I should correspond with on the controls side and I will talk with them tomorrow morning.

Thanks,

Grant

E-mail correspondence on the Target Hall crane fault diagnosis between October 11th and 15th:

On 15/10/2013 3:34 PM, Grant Minor wrote:

Hello all,

I spoke with the Norelco tech Mike Smith this afternoon regarding the failed thermal sensor component / circuit.  He will call Kone on holiday Monday (their office is not closed for Canadian Thanksgiving) and determine if they have a replacement module.

I explained (to the best of my understanding) the diagnosis performed by Dave Morris / controls group and the test that was done to isolate the thermal sensor unit.  Unfortunately, Mike Smith's drawings from Seattle crane contain notes and revisions that don't match the drawing that Dave Morris provided me (Seattle Crane dwg E5 - 1/1 revision 1 02-10-99), and he was unable to confirm in his opinion whether the bypass performed by Dave Morris has correctly isolated the problem components.  In addition, he believes that the bypass he performed on Wednesday isolated portions of circuitry that are inconsistent with this thermal sensor circuit.

Mike Smith will return Tuesday morning to review the bypass and schematics.  He recommended not operating the crane until he has a chance to confirm this thermal sensor issue.

From the drawings I have in hand and from discussions with Dave Morris, the failed component might be the thermistor sets inside the motor housing which are in series with the suspect module, or the module itself.  My understanding is that the crane is designed with two pairs of motors for the bridge drive, and each pair is protected by one of these thermal modules (thus there are two modules total).  Both modules would be the same age and both might be susceptible to the same failure mode.

As it is unclear to me the exact nature of the drive system failure, and I have a recommendation from Norelco to hold on operation of the crane, I must unfortunately state that the crane should not be used and we will have to wait until Tuesday to gather more information.

I am not so concerned about temporary loss of the thermal protection circuitry, as we have not had an overheating scenario in those motors in the lifetime of the crane (to my knowledge).  I am more concerned about getting 100% confirmation that we have addressed the problem.  If we attempt to use the crane and experience another failure during a lift that prevents us from placing the load down safely with the crane stuck in position over a target station, then we would be in much deeper trouble.

My apologies for this situation.

Best regards and have a good weekend,

Grant


On 11/10/2013 1:28 PM, Grant Minor wrote:
Thanks Dave,

I've contacted Norelco, Mike Smith and I are planning a path forward for further diagnosis and repair.

The crane MUST NOT be used to carry a load until it is identified whether the module or the sensors have failed, the reason for the failure, and the functionality of the thermal sensors has been restored.

Without the thermal sensor circuit we lose protection from motor overheat, an event which could have much more serious failure implications to the crane.

Cheers,

Grant

On 11/10/2013 1:04 PM, David Morris wrote:
The fault has been identified in the Target Hall Crane as a failed motor temperature module, or motor temperature sensor, on the East-West motion. The status contact in the module was bypassed allowing crane motion. There was no fault with the controls.

Dave