Leak checked TM4 coil1 in SHC with Isaac . The leak detector only can pump down to 1.5xE-7 atm.cc/sec. The leak detector got the response once the helium valve is open, not even close to the leak area. The leak detector got a very big response when the helium probe was put close to the coil terminal block on target front side and spray helium for 2 clicks. The leak detector got a small  response when the probe was put close to water block area and spray helium for 2 clicks. The leak is located on the front target side coil line but not from water blocks. The leak is too big to be further pin pointed by this method. See attachment

 The containment box has been removed. Tomorrow the wiring harness will be installed.

September 18th

The wiring harness was completed by Maico and Travis today (see pics) and was installed into the target module. During installation the bracket went on with ease. The junction box appeared to be hung up on something but did engage eventually. The plug for the collimator is not quite long enough for the manipulators to install completely. I was only able to get it to just engage by sneaking the manipulator in from the shutter side and then pushing on it with a long rod. The wiring for the collimator is several inches too long but does not interfere with anything. The remote installation of the wiring to the IMG gauge was easy but there at least 1 foot of extra wire and with the thicker gauge it has a mind of its own. I was eventually able to wrap the wire under the table before so it will not interfere with anything. The module side junction box bracket was not clamped to the table I was able to do up one bolt but other one was blocked by the new steerer strain relief. There is no requirement for both to be done up.

September 19th

Travis completed and electrical test this morning and it was found that coil 1 touches coil 2 touch each other very slightly. By inserting a piece of cardboard between the closest water block and the lines we no longer in contact and passed the test. Once the cardboard was removed the lines remained separated but if touched slightly they would spring back into place and make contact. There was some debate as to if we should make a custom part to separate the line or if we should attempt to tweak the lines. I decided that it is known that a very small change in the position of the module will cause separation between the lines, that we should attempt to tweak the lines. I braced the lines with one manipulator to prevent loading the ceramic while Maico "tweaked" the line. The line was successfully moved to cause separation. Maico and I then completed a visual inspection of the remaining water blocks and noticed that the ground electrode cooling lines are estimated 1/8" away from one of the module side coil lines. All other places that can easily be inspected appear to have enough clearance.

Travis and Don then completed an electrical check again (https://elog.triumf.ca/TIS/RH-ISAC/457). This module still failed the test but as noted not due to continuity rather lower than infinite resistance from the coil to coil and positive coil to 60kV common. This indicates that there is dirty or failing insulation that is either from the fiber glass wrap or something within the service tray. From inspection from outside the hot cell the best guess as to where this is happening is just before to after the exit of the water lines from the service tray but there is no way to confirm this.

Although the module may be able to run for a period of time with these resistances the insulation may continue to break down and there is the risk that during a module move the lines may shift and touch again. This combined with the known HV issues of TM3 paints a dark picture of the service tray.

Friedhelm and Lia have been advised of these recent results and have decided to continue with the installation of the target and conditioning of the target. Should the module be able to operate and extract beam it will be installed in ITE. Friedhelm also came for a visual inspection of the module and agreed that the issue is on the module side and that the lines are too close to tell exactly where the issue may lie. The possibility of separating the the blocks with further insulation was discussed but there is limited area that this can be completed remotely. In addition to this there is no indication that further insulation/bracing will improve the problem and there is considerable risk that it could make the issue worse.

Installation of the containment box started this afternoon and is completed other than the tightening of the window VCR joints. 

This morning, Bevan and I leak checked window circuit on TM3 at SHC after containment box was installed. It took about 20minutes to pump down the circuit to  0.0xE-9atm.cc/sec. Bevan sprayed helium on both VCR joints and a final splash. No response had been found on leak detector.

 September 20th

Today was a trying day. When attempting to make the water line connections for the window the VCR seal fell off several times. Each time this happens the seal becomes stuck at the bottom of the containment box. While trying to replace the seal the nut slipped down the line to the bottom of the containment box. When trying to recover said nut with the pick tool it too fell to the bottom of the containment box. Due to the design of the containment box the only way to recover the pick is to remove the containment box (several other attempts were made with no luck). Before anyone asks no the pick cannot be left in the containment box. When attempting to remove the containment box one of the bolts for the shutter panel stripped. I then found the stud removing tool and have successfully removed the bolt. Isaac is now going to finish removing the containment box and that will be as far as we can get today. 

September 22nd 

- containment box removed
- dropped tool and seals removed from box
- containment box reinstalled
- window water lines done up
- target installed (with difficulties detailed below)

The threads for the middle tube heater connection (circuit D) appear to be damaged and the bolt in that location cannot be fully screwed in.  The bolt was removed and showed damage to the threads.

September 23rd

A leak check of the window line was completed (https://elog.triumf.ca/TIS/RH-ISAC/459) and several electrical checks of the coil with and without the target in place. The values for the electrical checks continued to fail the open circuit criteria, an elog on the testing will be reported tomorrow. Isaac and Maico removed the target so that the helicoil could be repaired. The helicoil was repaired by chasing the threads with a modified tap but it was noted during the repair that the bolt holding the line in place and come loose and was no longer clamping the tube heater and ceramic tight. The target was also checked electrically to confirm that the coil was not shorted to common. It showed a open circuit resistance with a ohmmeter.

It was then attempted to re-install the target but as the target was being lifted a cable for the tong motion schematic broke (A-11682). 

Currently the target is sitting in the hot cell with the manipulator still broken. We are attempting to find parts and instructions as to how to replace the cable.

 The right manipulator A-9885 Handle Cable Assembly has been repaired by Maico and myself. We repaired the manipulator following some verbal instructions from Chad on the modified repair strategy. This allows the repair to be completed without going into the hot cell. The instructions in the repair manual are very vague and can actually cause problems. For example it is best to pin the manipulator before completing any work. Other than that the work went smoothly with the exception that the counter weight hangs up on the manipulator occasionally and can cause the cable to come off of the pulley. Also it appears that the left manipulator has had the cable installed incorrectly as it rides on the housing of the wrist.

The containment box cover has had all  the bolts installed and all bolts were checked with a mirror. TM3 has been raised and is ready for the last electrical  test before  it goes to the conditioning station.

TM3 has had all of the water lines pressurized with Helium and the Helium Leak detector was connected to the turbo pumps. There were no leaks detected.

 Today Maico and Travis completed and electrical test to determine the right and left coil on TM4. The left coil is the negative side. From there Maico pumped on both lines for a leak test. The leak cart pumped down to its lower limits on the left line and there was no response when sprayed with helium. The right line would not pump below 2.5 x 10^-2 Torr and had a base leak rate of 3.3 x 10^-7 atm*cc/sec (this indicates a massive leak). Without spraying helium the leak cart responded and peaked at 1.9 x 10^-6 atm*cc/sec. Attached is a picture of the module indicating the line.

Grant Minor wrote this e-mail today:

***

Hello all,

During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke.

We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars.

Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?)

Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator.

Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment.

Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section.

Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them.

I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed.

Thank you all for your assistance,

Grant

***

A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

Grant Minor wrote:

Grant Minor wrote this e-mail today: *** Hello all, During our planned operation of transporting TM4 from a storage silo to the Hot Cell, we had a problem with the East-West bridge drive on the Target Hall crane.  The crane bridge is currently stuck in position over the silo area.  We were unable to complete the transport operation.  TM4 has been placed back down into the silo and the crane hook block has been disconnected from the module lifting yoke. We have been on the phone with Norelco this afternoon and have determined the likely cause of the problem.  There is a splice joint in the crane rails at this position, where a small gap exists between the rail sections.  If the crane bridge is stopped during operation such that the wheel comes to rests in this gap, the crane motors have insufficient torque for the wheel to overcome the gap and ride back onto the rail.  The Norelco tech Mike Smith remembers encountering this problem several years ago (before my time) but cannot remember how it was overcome.  The fix likely involved moving the wheel out of the gap with pry bars. Target Hall Survey Needed at or before 7:00 am, Wednesday Oct 9th (Lynne, Fiona, Danka, Max or Dano?) Mike Smith and possibly a few others from Norelco will come tomorrow morning at 7:00am, and require access to the Target Hall.  Since we have opened the Hot Cell in preparation for the module move, we will need a survey as early as possible to allow Mike and his crew to enter the hall without a suit and respirator. Most of the Remote Handling group will be off-site for a Division meeting, but Travis and Maico will stay for the day to accompany the Norelco guys and help with the problem assessment. Travis: Please take Mike to the control room when he arrives, one of the cameras is currently zoomed in on the wheel and problem rail section. Maico: Scaffolding may be required to get access to the crane rail.  Unfortunately we don't have an operational crane to transport a man-lift into the hall ;)  Please see what you can do to get this ready in case we need it.  Max's leak check might have to go on hold until we get this resolved.  Please take responsibility for ensuring correct Fall Protection procedures are followed.  This means that nobody should be walking up on the crane rail unless they are tied in with harnesses, or there is scaffolding directly under them. I can be reached on my cell phone at 778-319-4612.  The Division meeting is on UBC campus so I can return to TRIUMF fairly quickly if needed. Thank you all for your assistance, Grant *** A photo of the THall camera view of the bridge wheel at the rail splice is attached, as well as Mike Smith's crane inspection report from March 2013.

E-mail from today:

Hello Dave, Jane, Franco, Michael, Don (controls + Franco), Remy,

We have lost the East-West bridge drive control functionality of the ISAC Target Hall crane, and unfortunately this very much prevents us from operating the ISAC facility.

The problem was originally thought to be a stuck bridge wheel but this was not actually the case.

Mike Smith from Norelco was on-site for most of today working with Michael Rowe in an attempt to diagnose the problem.

Mike believes the problem to be one of the following three items:
- input PLC board on the crane bridge
- output PLC board in the interface panel in B1 level Room 5 (ISAC remote crane control room)
- communication between the two boards (festooning hardware)

We are in a pretty dire situation without the ISAC target hall crane, and we lack the expertise within the Remote Handling group in PLC hardware to diagnose this on our own.  I need to ask for somebody's help from the controls group in diagnosing and repairing this problem.

I have notes from my telephone conversation with Mike Smith which I can discuss with a controls representative in the morning.  There is a pretty muddled history of the design and implementation of these controls but essentially Norelco outsourced the PLCs and festooning to a third party contractor, Seattle Crane in the US, which no longer exists anymore.  Neither Kone, Seattle Crane, or Norelco made appropriate as-built drawings of the wiring of the PLCs.  Mike Smith at Norelco had a stack of hand-written notes about the as-built configuration he made at the time of installation in the late 1990's but these notes have not been located yet.

The PLCs appear to be Allan Bradley but the modules are from approximately 1997 and we're not sure if spare units are available.

Mike Smith says the easiest thing to do at this point is install replacement PLC modules on the crane bridge and in the control room that have confirmed functionality.  If control is not restored then the problem is likely in the festooning.

Control of the crane bridge was temporarily attained by bypassing the controls (speed control / end limits, etc.) and sending drive signal directly to the drive motor controller in order to return the bridge to its home position at the access ladder.  This mode of operation is not safe as the crane has no speed reduction / limits to prevent driving the bridge into the walls at max speed.

Jane / Remy / Franco: please let me know who I should correspond with on the controls side and I will talk with them tomorrow morning.

Thanks,

Grant

 Blocks 1-4 have been moved from ITW to ITE. Block 5 was moved to a storage area as it will be in the way of the harp repair. Danka took air samples and found 200 counts of alpha on the air samples but nothing on the swipes. It is recommended that resporators be worn while blocks are being lifted and lowered. She will check the monitor tomorrow to confirm it is behaving as it should.

E-mail correspondence on the Target Hall crane fault diagnosis between October 11th and 15th:

On 15/10/2013 3:34 PM, Grant Minor wrote:

Hello all,

I spoke with the Norelco tech Mike Smith this afternoon regarding the failed thermal sensor component / circuit.  He will call Kone on holiday Monday (their office is not closed for Canadian Thanksgiving) and determine if they have a replacement module.

I explained (to the best of my understanding) the diagnosis performed by Dave Morris / controls group and the test that was done to isolate the thermal sensor unit.  Unfortunately, Mike Smith's drawings from Seattle crane contain notes and revisions that don't match the drawing that Dave Morris provided me (Seattle Crane dwg E5 - 1/1 revision 1 02-10-99), and he was unable to confirm in his opinion whether the bypass performed by Dave Morris has correctly isolated the problem components.  In addition, he believes that the bypass he performed on Wednesday isolated portions of circuitry that are inconsistent with this thermal sensor circuit.

Mike Smith will return Tuesday morning to review the bypass and schematics.  He recommended not operating the crane until he has a chance to confirm this thermal sensor issue.

From the drawings I have in hand and from discussions with Dave Morris, the failed component might be the thermistor sets inside the motor housing which are in series with the suspect module, or the module itself.  My understanding is that the crane is designed with two pairs of motors for the bridge drive, and each pair is protected by one of these thermal modules (thus there are two modules total).  Both modules would be the same age and both might be susceptible to the same failure mode.

As it is unclear to me the exact nature of the drive system failure, and I have a recommendation from Norelco to hold on operation of the crane, I must unfortunately state that the crane should not be used and we will have to wait until Tuesday to gather more information.

I am not so concerned about temporary loss of the thermal protection circuitry, as we have not had an overheating scenario in those motors in the lifetime of the crane (to my knowledge).  I am more concerned about getting 100% confirmation that we have addressed the problem.  If we attempt to use the crane and experience another failure during a lift that prevents us from placing the load down safely with the crane stuck in position over a target station, then we would be in much deeper trouble.

My apologies for this situation.

Best regards and have a good weekend,

Grant


On 11/10/2013 1:28 PM, Grant Minor wrote:
Thanks Dave,

I've contacted Norelco, Mike Smith and I are planning a path forward for further diagnosis and repair.

The crane MUST NOT be used to carry a load until it is identified whether the module or the sensors have failed, the reason for the failure, and the functionality of the thermal sensors has been restored.

Without the thermal sensor circuit we lose protection from motor overheat, an event which could have much more serious failure implications to the crane.

Cheers,

Grant

On 11/10/2013 1:04 PM, David Morris wrote:
The fault has been identified in the Target Hall Crane as a failed motor temperature module, or motor temperature sensor, on the East-West motion. The status contact in the module was bypassed allowing crane motion. There was no fault with the controls.

Dave

The blocks above the pre-separator section of the ISAC beam lines were removed today in the Target Hall by Bevan Moss and Chad Fisher with Lynne LeMessurier present from RPG.

The two blocks in the layer below the top layer above the pre-separator were damaged on removal (see attached photos).  This was due to an attempt to lift them in the incorrect sequence caused by misinterpretation of the labeling / numbering system drawn on the top of the blocks.  The blocks are labelled for the replacement sequence, but not for the removal sequence.  An attempt was made to lift a block that had an overlap with an adjacent block that was not visible, causing damage to two blocks.  A note was added in black marker to the critical block to indicate that it should be removed last (shown in attached photo).  The block replacement sequence will be photographed and a procedure document will be written by Bevan, with a bound copy of the procedure provided in the Target Hall, in order to prevent such an incident from occurring again.  Broken pieces of the blocks were collected and placed in a 5-gallon pail.

The removal of all of the blocks caused the depression in the target station to fluctuate around the trip point (17 inches of water column), which caused the ventilation alarm to trip on and off constantly.  Operations contacted me to ask if this could be remedied.  Bevan closed the damper to ITW, which brought the depression to a low enough value so that the alarm could be defeated properly.  This was around 2pm.

Preliminary co-ordination of the ITE Harp 9 removal job was discussed by Scott Kellogg and Ron Kuramoto around lunch time.  Keith Ng (RH) and Scott searched for the blank-off that was used during the last Harp 9 repair (removed from B/L in March 2013) but they could not locate it.  I was later able to find the blank-off bagged in a grey rubbermaid bin at the bottom of the stairs in the target storage vault pit (under the SHC / NHC ventilation and filters).  The blank-off was found bagged and tagged as surveyed at 100 counts @ 0.5m on March 27, 2013.  I gathered Scott, Ron Kuramoto, and Danka Krsmanovic to co-ordinate the Harp 9 removal and blank-off installation, starting at around 2:30 pm.

For the Target Hall Harp work, Ron Kuramoto and Scott prepared the required tools and plastic coverings / bags.  I acted as crane operator.  Danka was the surveyor present.  Anders Mjos joined the group for photos and moral support.

Scott disconnected services from the Harp device, and prepared it for lift.  I called operations to confirm that the section of B/L had been vented.  Ron and Scott used the RH tools to disconnect the 2 vacuum flange bullet bolts.  Ron directed the crane with signals and I lowered the chain rigging to the Harp, it was connected and removed from the BL with the main hoist.  (I made the mistake of directing the rigging of the harp to the main 20T hoist, when in fact the 3T Aux hoist should have been used, as the main hoist can't reach to the vacuum / storage pit on the west side of the hall.)

Danka performed several swipes and measurements of the Harp after removal.  Alpha contamination was discovered on the bottom of the Harp flange.

Ron and Scott installed the blank-off flange, then I called operations to indicate that pumping should start.

Ron and Scott retrieved the vacuum test flask jig / support structure from the West storage pit and brought it down to the pre-separator pit.  I lowered the Harp into this flask jig, and Scott used the side-wall location bolts from the top of the harp assembly to fasten the flanges of the Harp and the jig.

I had trouble at first activating the Aux hoist but soon realized that it's necessary to press the Aux pendant "ON" button down for several seconds before the power actually turns on to this hoist.

We then removed the main hoist hook and attached the Aux hoist to the Harp / flask.  The assembly was transported to the West storage pit.

Some of the handrails had to be removed to facilitate initial movements of the Harp assembly.  These handrails should be replaced tomorrow morning.

Scott also submitted a Work Order for a back-up blank-off today, job number 40303.

The pre-separator blocks have NOT been replaced yet, we will do this tomorrow morning.

It was a busy day, thanks to all for their efforts and for staying a bit late to get the blank-off installed.

Best regards,

Grant

The containment box was removed from TM4 earlier this morning in preparation for removal of the C2 circuit water lines, their respective water blocks and conductor block.

The pre-separator-area blocks were replaced today by Bevan and Travis, with Grant present for photos and observation.

Photos are attached of each block replacement.

Some difficulty was encountered in replacing the "Layer #3" blocks (photo 9 attached).  An interference occurred between the outside of the west "Layer #3" block and an aluminum channel directly beside the rubber pad that supports the block.  The block was initially resting on this channel but slipped off, breaking off some concrete at the edge.   See photo 12 attached and notice crumbled concrete inside this channel around the area where the interference occurred. Photo 14 shows the damaged block edge.

It should be noted that when these two blocks are resting flat, there should be a gap between them (i.e. they should NOT be touching as shown in photo 9).

 Yesterday I visually inspected the paddle wheel and noticed that it was spinning freely and constant. As the signal was intermittent replaced the coil at the back of the wheel only. This does not require access to the high active water. With the coil replaced the signal looked strong. I asked operations to keep and eye on it and advise if the problem has been resolved. I also discussed with David this morning and he informed me that this is the same procedure he would have followed.

The C2 circuit (coil 2) water lines and blocks have been revoed from TM4.

Water block jig could not be used as the window water block connection was nowhere near fitting into the jig (IMG_1224 image below) and therefore caused an interference causing the rest of the blocks to not mate with the jig. A different tool that has never been used was implimented and sucessful (IMG_1227 below).

 Sorry about the picture numbering format...new format will be implimented next time.

 TM4 without a source tray was moved from the SHC to the TCS.  All went well.

 The roughing pump has been started.