The HV conditioning reached a maximum of 44kV today before a spark took out the canbus card. I will attach the data once I have had a chance to format it. 

 Argon venting has started. It will be left to vent over night. 

ESC was called out again to inspect the chiller as it was showing the same problem as before. This time when started one fan came on (this is not what happened last time) but the fan controlled by the VFD did not. It was eventually traced to a relay that has been replaced. The system was tested as best it could without a load and appears to be working. It will be difficult to tell until a heat load is in place if the fix has solved everything. 

 The roughing pumps have been started on the conditioning station. As it is late in the day for David he will make the turbo pump connections tomorrow.

TM3 has been moved from the south hot cell to the conditioning station. David and I completed the move. All went well.

   I have fallen behind on the HV testing that has been completed on TM3. Data can be found on docushare in the TM3 Rev 3 folder files are too large to attach. From the last elog entry the following tests have been completed

July 8 2013 - Long HV conditioning 50kV achieved before sparking caused PS to trip. Was unable to recover to the same voltage

July 9 2013 - Attempted to repeat previous test lost PS at 45kV. Voltage continued to degrade during the day.

July 10 2013 - Max voltage achieve 42.5kV discussed with Friedhelm and decided to run the EINZEL lens at 50% of the BIAS in case this is the source of the sparking

July 10 2013 - Max voltage 40.5kV with EINZEL lens. Communicated with Friedhelm and Lia. It was decided 40kV was good enough. Started developing standard test (see below for testing procedure) for Module so a with and without source tray 

July 12 2013 - Only one spark on route to 40kV. PS tripped after ~10 minutes at 40kV. Near the end of the day there wasn't enough time to repeat test.

July 15 2013 - Test repeated, could not hold 40kV. 39kV held for 1 hour with EINZEL lens off. Sparking was making it difficult to keep on.

July 15 2013 - TM3 was electrically disconnected from the conditioning station. The station ran at 60kV for 1 hour with zero sparks.

July 16 2013 - Test repeated with goal of 39kV intense sparking after 20 minutes caused the PS to trip. Once again the EINZEL lens PS was tripping and eliminated from use to make the test easier. With the failure of this test once again Friedhelm was once again consulted. He advised to stop testing as it appears to be causing damage. There is concern that the module is degrading. Lia has stated this does not affect the source tray and that a source tray will be inserted into TM3 and she accepts responsibility if the module does not hold voltage. 

Standard testing procedure for the bias was as follows. Please note that the EINZEL lens was supposed to follow the BIAS steps at 50%. This procedure was approved by Friedhelm prior to operating. This test was never successfully completed and we will not be able to complete a "apples to apples" test once the source tray is installed.

> Step to 5000 kV If no sparking after 1 minute

> Step to 10000 kV If no sparking after 1 minute

> Step to 15000 kV If no sparking after 1 minute

> Step to 17500 kV If no sparking after 1 minute

> Step to 20000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 22500 kV If no sparking for at least 1 minute after 5 minutes

> Step to 25000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 27500 kV If no sparking for at least 1 minute after 5 minutes

> Step to 30000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 31000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 32000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 33000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 34000 kV If no sparking for at least 1 minute after 5 minutes

> Step to 35000 kV If no sparking for at least 2 minute after 10 minutes

> Step to 35500 kV If no sparking for at least 2 minute after 10 minutes

> Step to 36000 kV If no sparking for at least 2 minute after 10 minutes

> Step to 36500 kV If no sparking for at least 2 minute after 10 minutes

> Step to 37000 kV If no sparking for at least 2 minute after 10 minutes

> Step to 37500 kV If no sparking for at least 5 minute after 10 minutes

> Step to 38000 kV If no sparking for at least 5 minute after 10 minutes

> Step to 38500 kV If no sparking for at least 5 minute after 10 minutes

> Step to 39000 kV If no sparking for at least 5 minute after 10 minutes

> Step to 39500 kV If no sparking for at least 5 minute after 10 minutes

> Step to 40000 kV Maximum 3 sparks per hour maximum test time 3 hours.

chad fisher wrote:

The heat shield circuit was pressurized with air (15-20 psi) and snoop applied to the water block joint and brazes. Bubbles formed at the joint between the two water blocks indicating that the leak is a c seal problem. Pictures attached but also on docushare Collection-11285.

 Last night prior to this leak check the Heat Shield was pumped on and helium sprayed on the circuit. The pressure was 0.0 x 10^-4 Torr. The leak was traced to water block. response time was 2 seconds and went from 5.5 x 10^-8 to 1.6 x 10^-6 atm*cc/sec. There was also a response without spraying helium (drift from the nozzle). The results of this testing prompted the testing completed in Chads elog.

 Today Chad entered the south hot cell on work permit 2013-08-29-3 to repair the leaking heat shield line. The heat shield line was removed from the module and the surfaces cleaned. New C seals and spring were inserted and the line re-attached. During this time Chad also installed the new vacuum gauge. Grant took video of this work and Keith some pictures. The video and pictures can be found on the M: drive (groups) in the RH folder. The total time taken was ~ 25 minutes. The line was then pumped on and the block was sprayed with helium. The pressure and leak rate started at 0.0 x 10^-4 Torr and 4.0 x 10^-8 atm*cc/Sec. When the helium tank was opened there was an immediate response with the leak rate climbing to 2.7 x 10^-7 atm*cc/Sec. It then stabilized at 2.0 x 10^-7 atm*cc/Sec and was sprayed with helium. Response time was ~2 seconds the leak rate climbed to ~2.3 x 10^-5 atm*cc/Sec the pressure also increased to ~3.0 x 10^-3. The leak detector cart was left attached for another 1.5 hours but still was not able to stabilize in the 10^-9atm*cc/Sec range. 

Bevan Moss wrote:

Today Chad entered the south hot cell on work permit 2013-08-29-3 to repair the leaking heat shield line. The heat shield line was removed from the module and the surfaces cleaned. New C seals and spring were inserted and the line re-attached. During this time Chad also installed the new vacuum gauge. Grant took video of this work and Keith some pictures. The video and pictures can be found on the M: drive (groups) in the RH folder. The total time taken was ~ 25 minutes. The line was then pumped on and the block was sprayed with helium. The pressure and leak rate started at 0.0 x 10^-4 Torr and 4.0 x 10^-8 atm*cc/Sec. When the helium tank was opened there was an immediate response with the leak rate climbing to 2.7 x 10^-7 atm*cc/Sec. It then stabilized at 2.0 x 10^-7 atm*cc/Sec and was sprayed with helium. Response time was ~2 seconds the leak rate climbed to ~2.3 x 10^-5 atm*cc/Sec the pressure also increased to ~3.0 x 10^-3. The leak detector cart was left attached for another 1.5 hours but still was not able to stabilize in the 10^-9atm*cc/Sec range.

 Also of note is that Chad received a full days dose and that when the plastic was surveyed there were ~150 counts found on it.

Bevan Moss wrote:

Bevan Moss wrote:  Today Chad entered the south hot cell on work permit 2013-08-29-3 to repair the leaking heat shield line. The heat shield line was removed from the module and the surfaces cleaned. New C seals and spring were inserted and the line re-attached. During this time Chad also installed the new vacuum gauge. Grant took video of this work and Keith some pictures. The video and pictures can be found on the M: drive (groups) in the RH folder. The total time taken was ~ 25 minutes. The line was then pumped on and the block was sprayed with helium. The pressure and leak rate started at 0.0 x 10^-4 Torr and 4.0 x 10^-8 atm*cc/Sec. When the helium tank was opened there was an immediate response with the leak rate climbing to 2.7 x 10^-7 atm*cc/Sec. It then stabilized at 2.0 x 10^-7 atm*cc/Sec and was sprayed with helium. Response time was ~2 seconds the leak rate climbed to ~2.3 x 10^-5 atm*cc/Sec the pressure also increased to ~3.0 x 10^-3. The leak detector cart was left attached for another 1.5 hours but still was not able to stabilize in the 10^-9atm*cc/Sec range.   Also of note is that Chad received a full days dose and that when the plastic was surveyed there were ~150 counts found on it.

Today Chad entered hot cell again and removed the HS module side and Junction Block Wiring Harness (module side). In addition to this he blanked off the module side HS line. It took Chad 16 minutes to perform these tasks and he received a dose of 0.20 mSv bringing his total to 0.84 over the last 2 days. After exiting the hot cell Chad completed a leak check on the module side. The line pumped down to the limits of the leak testing cart 0.0x10^-4 Torr and 0.0x10^-9 atm*cc/sec and there was no response. A video of Chad in the hot cell can be found in:

M:\remote handling\Photos\2013\2013_tm3_source_tray_refurb

This afternoon was spent surveying the anteroom and receiving teaching from Chad as to how to be a hot cell operator. There was little in the way of contamination ~150 counts near the HC door and ~50 counts on the surrounding floor. The wiring harness has a field of 900 uSv/hr but is suspected most of that is coming from the aluminum steerer bracket. 

 September 3rd

David Wang and myself lowered the leak testing cart into the Ante room and prepared the ante-room for the leak testing of TM3s line. David and Grant then entered the anteroom and inspected the old c-seal and leak tested the the heat shield line. The line pumped down to the lower limits of the leak detector and there was no response from the cart. A presentation with the results of the C-seal inspection results was completed by Grant Minor and email for a design review held on the 4th.

September 4th

David Wang and myself removed the leak detector and surveyed the anteroom and cart. The following results were found with the high spec gamma detector.

Leak Detector body and cart 0 

Leak Detector wheels 200 cpm

Floor off of plastic 300 cpm (there was contamination before)

Plastic in front of HC opening 2000 cpm

Working table after plastic was removed < 50 cpm

Tool box < 50 cpm

First tacky mat 2500 cpm

Second tacky mat 200 cpm

outside of door tacky mats 0 cpm

David then laid fresh plastic on the floor around the HC opening and covered the exposed floor as well.

Following this Travis and David measured the lengths of the wiring and attempted to separate the wiring harness from the Ultem block. It was found that the block was pinned and could not be separated, a new block will need to be made. The measured the following lengths:

Steerering - 26" to 27"

Collimator - 32" - 33"

PNG - 60"

 Today Maico completed the new leak testing tool and blank off. He tested them with C-seals without springs and all was leak tight. He then inspected the crushed seals and noticed that there was an area that was crushed less on both seals (more prominent in one,  20130906_tm3sourtrayefurb_P1020975) and that this reduced crushed zone only appears on one side of the seal. This is similar to the failure seen on both sets of the heat shield lines. On the seal with the greatest change in crush zone the average crush zone was ~0.025 and the reduced crush zone was ~0.012". He inspected the leak tester and the blank off and determined they were within tolerance and that they had a total seal goove height of 0.074" which is nominal. New seals were selected and one had a visible dimple prior to crushing (20130906_tm3sourtrayefurb_P1020978)  and the other had some defects on the inside (20130906_tm3sourtrayefurb_P1020962). These defects were marked and crushed using the same leak testing tool and blank off. The seal with the dimple showed a reduced crush zone in the same area (20130906_tm3sourtrayefurb_P1030021) and the seal with the defects on the inside showed less or undetectable change in crush zone. Maico then inspected (20130906_tm3sourtrayefurb_P1020991) and crushed the "thicker seals" (more indium coating). When inserting the seals he noticed that they fit tight on the counter bore of the blank off. He then attempted to crush the seal to the point where the copper faces would touch (as design intent) but the seal locked. The gap between the 2 copper faces was measured to be ~0.001". This setup was leak tested and found to be leak tight. When inspecting the crushed thicker seals it was found that the material had actually been pushed sideways causing a lip to form around the crush zone (20130906_tm3sourtrayefurb_P1030003). Maico then fitted the wires for the testing of the module and the heat shield line. 

 This e-log is to cover the work completed from September 9th until the 11th. 

September 9th

Today Maico completed the leak checking blank off and leak checking tool (to replace the one that was contaminated). This tool was then leak checked with C-seals and no springs at the machine shop, both halves were determined to be leak tight. 2 sets of small c-seals without divots were selected and used for leak checking the module side and the source tray side. The heat shield line was tested in the ante-room and was determined to leak. Upon further investigation it was determined that the bore of the seal gland was determined to be over sized (0.450"). Isaac then installed the blank off onto the module and pump down was started, the testing car would not stabilize so Isaac tried turning the screw more allowing for further pump down. The leak check was then completed and there was no response. The flow of helium for the target module was checked but was much lower than typical as the second valve was not fully opened. 

September 10th

With the belief that the larger bore in combination with the small c-seals was the cause of the leak Maico found and polished another block that also had a larger bore. Once polished, small c-seals without dimples were selected and tested with the machine shop leak testing cart. These seals were leak tight but upon investigation the crush zone in one section of the seals was significantly reduced. The smallest width of the crush zone was measured to be ~0.007". This result is concerning as it means that larger bore water blocks can be sealed with small c-seals but once installed there is no way to tell one block from the other. There is no data on the lifetime of this combination of seal and bore. 

Following this test a small c-seal with dimples was selected and checked in the large bore. This combination resulted in a significant leak. Following that test the large c-seals were crushed in the larger bore water block. The torque required to crush the seal and to have the faces touch was more than that of a small c-seal regardless of what size bore it was crushed in. However with the larger bore the large c-seals could be completely crushed allowing for the copper faces to touch (the total force on the faces is unknown). The crushed seal was then inspected, it had a nice uniform crush zone that was relatively large when compared to a properly crushed small c-seal. Another testing with the large seal and a 0.025" wire diameter spring was completed. It was found that the wire interferes with the the seal when it exits from the groove. This interference causes a localized increased crushed zone but does not reduce the crush elsewhere or prevent the copper faces from touching. At this point it was decided to repair the heat shield using large seals. Large seals with springs were tested on the actual HS line in the anteroom and yielded that same results. The line was then installed by Maico inside the HC. It pumped down to the lower limit of the leak testing cart and sprayed with helium, no leak was detected. The containment box was then put back on by Isaac and Grant in preparation for a module move.

September 11th

Today the containment box installation was completed and the module moved. Following that Maico and myself entered into the ante-room and retrieved the blank off block from the HC via the tool port. The large c-seal used in the test and the small ones used in the blank off were then collected and bagged. The large c-seal had typical measurements. The small c-seals used in the blank off did not have typical measurements. They had not been compressed as much and the crush zone was near impossible to see (if visible at all). From the measurements it is believed that the block was not sufficiently tightened. It also calls into question the validity of the leak check due to the combination of less helium and reduced crush. 

After these measurements the tools that were contaminated but were to be recovered were bagged and checked by safety. They have been moved to the jacks area where Maico will decontaminate them. The old wiring harness from TM3 was bagged and given to safety for storage in the cyclotron tunnel. The plastic and all of the garbage was then lifted, bagged, and removed from the ante room. Safety surveyed the anteroom following this no alpha contamination was found but there was 5000 counts on the floor of the ante room. The contamination may have existed prior to this work.

Many pictures of the cseals have been taken and have been put into \\trwindata\remote handling\Photos\2013\2013_tm3_source_tray_refurb. A report detailing all of the testing and results would be invaluable.  

 The containment box has been removed. Tomorrow the wiring harness will be installed.

September 18th

The wiring harness was completed by Maico and Travis today (see pics) and was installed into the target module. During installation the bracket went on with ease. The junction box appeared to be hung up on something but did engage eventually. The plug for the collimator is not quite long enough for the manipulators to install completely. I was only able to get it to just engage by sneaking the manipulator in from the shutter side and then pushing on it with a long rod. The wiring for the collimator is several inches too long but does not interfere with anything. The remote installation of the wiring to the IMG gauge was easy but there at least 1 foot of extra wire and with the thicker gauge it has a mind of its own. I was eventually able to wrap the wire under the table before so it will not interfere with anything. The module side junction box bracket was not clamped to the table I was able to do up one bolt but other one was blocked by the new steerer strain relief. There is no requirement for both to be done up.

September 19th

Travis completed and electrical test this morning and it was found that coil 1 touches coil 2 touch each other very slightly. By inserting a piece of cardboard between the closest water block and the lines we no longer in contact and passed the test. Once the cardboard was removed the lines remained separated but if touched slightly they would spring back into place and make contact. There was some debate as to if we should make a custom part to separate the line or if we should attempt to tweak the lines. I decided that it is known that a very small change in the position of the module will cause separation between the lines, that we should attempt to tweak the lines. I braced the lines with one manipulator to prevent loading the ceramic while Maico "tweaked" the line. The line was successfully moved to cause separation. Maico and I then completed a visual inspection of the remaining water blocks and noticed that the ground electrode cooling lines are estimated 1/8" away from one of the module side coil lines. All other places that can easily be inspected appear to have enough clearance.

Travis and Don then completed an electrical check again (https://elog.triumf.ca/TIS/RH-ISAC/457). This module still failed the test but as noted not due to continuity rather lower than infinite resistance from the coil to coil and positive coil to 60kV common. This indicates that there is dirty or failing insulation that is either from the fiber glass wrap or something within the service tray. From inspection from outside the hot cell the best guess as to where this is happening is just before to after the exit of the water lines from the service tray but there is no way to confirm this.

Although the module may be able to run for a period of time with these resistances the insulation may continue to break down and there is the risk that during a module move the lines may shift and touch again. This combined with the known HV issues of TM3 paints a dark picture of the service tray.

Friedhelm and Lia have been advised of these recent results and have decided to continue with the installation of the target and conditioning of the target. Should the module be able to operate and extract beam it will be installed in ITE. Friedhelm also came for a visual inspection of the module and agreed that the issue is on the module side and that the lines are too close to tell exactly where the issue may lie. The possibility of separating the the blocks with further insulation was discussed but there is limited area that this can be completed remotely. In addition to this there is no indication that further insulation/bracing will improve the problem and there is considerable risk that it could make the issue worse.

Installation of the containment box started this afternoon and is completed other than the tightening of the window VCR joints. 

 September 20th

Today was a trying day. When attempting to make the water line connections for the window the VCR seal fell off several times. Each time this happens the seal becomes stuck at the bottom of the containment box. While trying to replace the seal the nut slipped down the line to the bottom of the containment box. When trying to recover said nut with the pick tool it too fell to the bottom of the containment box. Due to the design of the containment box the only way to recover the pick is to remove the containment box (several other attempts were made with no luck). Before anyone asks no the pick cannot be left in the containment box. When attempting to remove the containment box one of the bolts for the shutter panel stripped. I then found the stud removing tool and have successfully removed the bolt. Isaac is now going to finish removing the containment box and that will be as far as we can get today. 

September 22nd 

- containment box removed
- dropped tool and seals removed from box
- containment box reinstalled
- window water lines done up
- target installed (with difficulties detailed below)

The threads for the middle tube heater connection (circuit D) appear to be damaged and the bolt in that location cannot be fully screwed in.  The bolt was removed and showed damage to the threads.

September 23rd

A leak check of the window line was completed (https://elog.triumf.ca/TIS/RH-ISAC/459) and several electrical checks of the coil with and without the target in place. The values for the electrical checks continued to fail the open circuit criteria, an elog on the testing will be reported tomorrow. Isaac and Maico removed the target so that the helicoil could be repaired. The helicoil was repaired by chasing the threads with a modified tap but it was noted during the repair that the bolt holding the line in place and come loose and was no longer clamping the tube heater and ceramic tight. The target was also checked electrically to confirm that the coil was not shorted to common. It showed a open circuit resistance with a ohmmeter.

It was then attempted to re-install the target but as the target was being lifted a cable for the tong motion schematic broke (A-11682). 

Currently the target is sitting in the hot cell with the manipulator still broken. We are attempting to find parts and instructions as to how to replace the cable.

 The right manipulator A-9885 Handle Cable Assembly has been repaired by Maico and myself. We repaired the manipulator following some verbal instructions from Chad on the modified repair strategy. This allows the repair to be completed without going into the hot cell. The instructions in the repair manual are very vague and can actually cause problems. For example it is best to pin the manipulator before completing any work. Other than that the work went smoothly with the exception that the counter weight hangs up on the manipulator occasionally and can cause the cable to come off of the pulley. Also it appears that the left manipulator has had the cable installed incorrectly as it rides on the housing of the wrist.

September 24

 After the manipulator repair Maico installed the NiO#2 onto the target module. Once connected I completed a leak test on the HS line with Isaac. The leak testing cart would not pump down originally (something wrong with the cart) but eventually it did start to work. The cart pumped to its lower limits and did not respond even with the module was flooded with helium. A electrical test was then conducted by myself and and Don Jackson (see attached). During the standard test the Target to Anode and to 60kV common read lower than expected at ~47Momh @ 1kV. Also of note during the conductance check the measurement from c-d was low at 1.82 mohm.

3  additional tests were completed 2 of which were coil to common (both sides). The resisance was measure at 47 ohm with the megger and 1.32 Kohm with the uohmmeter. The coil lines lines were also measured to get and indication of the resistance of a copper line. These lines were measured while they were not jumpered both the coil + and coil - measured 5.73 mohm.

A meeting with Lia and Friedhelm was held and and the decision was to move the module and test it in the conditioning station. Fingers crossed that the module will still work.

Sept 25th

Maico finished installing the cover today for the module to be moved. Other elogs cover the other activities for today.

maicodallavalle wrote:

TM3 has had all of the water lines pressurized with Helium and the Helium Leak detector was connected to the turbo pumps. There were no leaks detected.

 While pressurizing the lines it was found some of the lines had water in them. It is unknown where this water came from. Perhaps David connected the conditioning station last time it was in? Lines that were free of water were A, B, C, and window The leak detector pumped down to 4.0 x 10^-3 Torr and the leak rate was 1.4 x 10^-9 atm-cc/sec, there was no response during the leak test and the leak rate continued to decrease during the test.

Bevan Moss wrote:

September 24  After the manipulator repair Maico installed the NiO#2 onto the target module. Once connected I completed a leak test on the HS line with Isaac. The leak testing cart would not pump down originally (something wrong with the cart) but eventually it did start to work. The cart pumped to its lower limits and did not respond even with the module was flooded with helium. A electrical test was then conducted by myself and and Don Jackson (see attached). During the standard test the Target to Anode and to 60kV common read lower than expected at ~47Momh @ 1kV. Also of note during the conductance check the measurement from c-d was low at 1.82 mohm. 3  additional tests were completed 2 of which were coil to common (both sides). The resisance was measure at 47 ohm with the megger and 1.32 Kohm with the uohmmeter. The coil lines lines were also measured to get and indication of the resistance of a copper line. These lines were measured while they were not jumpered both the coil + and coil - measured 5.73 mohm. A meeting with Lia and Friedhelm was held and and the decision was to move the module and test it in the conditioning station. Fingers crossed that the module will still work. Sept 25th Maico finished installing the cover today for the module to be moved. Other elogs cover the other activities for today.

 Also of note the turbo pumps were started on the conditioning station around 1:30. Don and Travis completed and electrical test, see other elogs. When attempting to start IG1 the controller indicated there was no filament, this gauge only has 1 to start. It will have to be replaced when the module is next vented. In addition to this it appear the PNG1 controller has failed again. I have emailed Dimo to see if we can get one to test. The img100 is new and different cables have been used. All 4 options for connections were also tested in case of a wiring mistake.