Found spark signs on TM3's containment on the side of the water blocks. They may be OLD or/and NEW sparking signs, but this in not determined at this moment. Please see picture: first attachment. First and second attachment show the profiles of water block and HV lines.

A video inspection of the lower service tray was performed this morning. Camera was inserted upwards into the service tray/service chase from the containment box. One potential spark mark was identified as well as epoxy not flush with insulator. Fiberglass does not extend into service tray/chase much and does not look, from initial inspection that it will be too difficult to remove.

Video can be found in docushare Document-142295

containment box was re-installed onto TM3 on Monday, Left manipulator may be showing signs of eminent handle or tong cable failure.

 TM#3 has been moved from the north east silo to the south hot cell.

the containment box has been removed from TM3.

chad fisher wrote:

A video inspection of the lower service tray was performed this morning. Camera was inserted upwards into the service tray/service chase from the containment box. One potential spark mark was identified as well as epoxy not flush with insulator. Fiberglass does not extend into service tray/chase much and does not look, from initial inspection that it will be too difficult to remove. Video can be found in docushare Document-142295

 The attached picture is from the video: showing the fiber glass inside the Service Tray.

Please see the first two pictures of the HV test at atmosphere done on April 19.

First picture shows a big spark between the Extraction Electrode mounting plate holder (at HV) to the Ground Electrode. Please note minimum gap between these electrodes is 7.8mm.

However the biggest visible spark was noticed at the bottom screw hole. (It is difficult to determine if at the 7.8mm gap were concomitant sparks occurring.)This spark started at 22kV and became stable above 24kV.

On April 20, 2017 Chad inspected this area with the Hot Cell camera and found a spark mark on the Extraction Electrode holder plate at the bottom screw hole. See the third picture.

Again it is very difficult with the existing Hot Cell cameras to inspect the areas where the gap is small (at 7.8mm gap between the conical surfaces of the Ground Electrode and Extraction Electrode holder)

In the second picture there are 2 small sparks: one on the top and another one on the right side at the lines wrapped in Fiber Glass. These sparks started at 20kV.

On April 21 the HV test was repeated with the Containment box ON and the sparks occurred again, at the same locations and at the same voltage.

Please see TM3's Updated Schedule (the current schedule) at the SHC and TCS.

For the record I am uploading TM3's Initial Schedule at the SHC and TCS.

Today a HV test at atmosphere was performed for Each Line(s), while floating the other lines.

Please note the Einzel Lens line (Y) was grounded.

Tests performed:

1. Applied up to 29kV to "(A) OVEN -"

Sparks occurred at 29kV at the Fiber Glass inside the Service Tray; and at the right side at the lines wrapped in Fiber Glass.

2. Applied up to 29kV to "(B) OVEN +"

Sparks occurred at 29kV at the Fiber Glass inside the Service Tray.

3. Applied up to 6-7kV to "(C) TUBE -"

Could not identify the sparks; it was a continuous discharge.

4. Applied up to 6-7kV to "(D) TUBE +"

Sparks/discharges occurred at 6-7kV at the Fiber Glass on the right side at the lines wrapped in Fiber Glass. We had to turn OFF the lights in the Hot Cell to identify the small sparks.

5. Applied up to 4kV to "(P-Q) COIL +"

Discharge occurred at 4kV. Could not identify it.

6. Applied up to 29kV to "(R-U) COIL -"

Sparks occurred around 27-29kV at the Fiber Glass inside the Service Tray; and at the right side at the lines wrapped in Fiber Glass.

Tom saw a spark at 27kV between the Extraction Electrode holder and the Ground Electrode.

Saw also a spark at the Ring Alumina insulator in the Service Tray, located on top of the second comb insulator with Epoxi.

7. Applied up to 27kV to "(E) Cavity or MSP, (G) Gas, (J) Heat Shield" - all these are connected together at the BIAS voltage.

Sparks occurred at 27kV at the Fiber Glass inside the Service Tray.

8. Applied up to 23kV to "(I) EE" - this is for the Extraction Electrode.

Sparks occurred at 23kV again at the same spot near the bottom screw hole, between the Extraction Electrode holder and the Ground Electrode.

9. Applied up to 27kV to "(F) Plasma"- this is for the Anode.

Sparks occurred at 26-27kV again at the same spot near the bottom screw hole, between the Extraction Electrode holder and the Ground Electrode.

Please see attachment for TM3 May 1-15, 2017 Tentative Schedule at the SHC and TCS.

Fiberglass, capton tape and tie straps have been removed from TM3.

RU( coil 1), F (anode), and 60kv bias(HS, Gasline, and MSP)  are shorted together. Other lines are good.  They are measured between  individual line each other  @500V. The megga showed OL. All of  them to ground are OL @500V as well. EL is grounded during check.

Today a HV test at atmosphere was performed for Each Line(s), while floating the other lines. THIS TEST AT ATMOSPHERE IS THE FIRST ONE AFTER THE FIBER GLASS WAS REMOVED.

Please note:

a). the Einzel Lens line (Y) was grounded.

b). after the fiber glass was removed the "R-U COIL" line and the "(F) Plasma" (which is the Anode line) were shorted to the BIAS voltage.

Tests performed:

1. Applied up to 32kV to "(A) OVEN -"

Sparks occurred at 32kV between the Extraction Electrode holder and the Ground Electrode.

2. Applied up to 33kV to "(B) OVEN +"

Sparks occurred at 33kV but we could not identify the spark location.

3. Applied up to 31kV to "(C) TUBE -"

Sparks occurred at 31kV between the Extraction Electrode holder and the Ground Electrode, at the top and the bottom.

4. Applied up to 30kV to "(D) TUBE +"

Sparks occurred at 30kV between the Extraction Electrode holder and the Ground Electrode, 2 spots at the bottom.

5. Applied up to 30kV to "(P-Q) COIL +"

Sparks occurred at 30kV between the Extraction Electrode holder and the Ground Electrode, all around.

6. Applied up to 30kV to "(R-U) COIL -", to "(F) Plasma" the Anode, to "(E) Cavity" or MSP, "(G) Gas", and  "(J) Heat Shield" - all these are "connected" together at the BIAS voltage.

Sparks occurred around 29-30kV at the Extraction Electrode holder and the Ground Electrode, at the bottom.

8. Applied up to 23kV to "(I) EE" - this is for the Extraction Electrode.

Sparks occurred at 23kV again at the same spot near the bottom, between the Extraction Electrode holder and the Ground Electrode.

During TM3 roughing down yesterday afternoon,The vacuum gauges CG1 and CG1S stopped at 1.20 torr and refused to drop down. Leak detector was used to found where the problem was. LD was in rough detecting mode and showed 2.7xE-6 atm.cc/sec as base leak rate.  Spray medium amount of helium to TM3 service top, and found this leak was on round panel of TM3 side wall. LR climbed to E-4 atm.cc/sec range on helium spraying. In my vie,One 10# socket head screw was not tightened properly after cameras job and caused this.  The TCS was vented. The panel was took off and reinstalled. O-ring on the panel  was replaced by a new one as well. Then TCS was pumped down again. The leak disappeared .  All turbo pumps on TCS and TM3 were started successfully later.

Today TM3 was conditioned up to 50kV. At 50kV sparking started and to go higher in voltage it needs more conditioning time. Lowered it to 49kV for the night.

  TH crane is used for transferring five shielding blocks(30000lbs) today. All brakes work fine. Mostly, I do not need crane slow up motion for this job so during 1 hour moving, no rattle sound and burning smell has been found.  I think TH crane is good to use as we decided. 

Sparking was found this morning inside HV cover on TCS in TH while TM3 was HV conditioning at TCS. The TCS bias was set to 50KV when I found this spark. It sparks between HV cover front side wall to a KF 25 clamp on HS water terminal. The KF 25clamp is used in this situation on HS water line top because the HS line is opened inside module, and we need a vacuum seal on the top of this line.  Actually, The clap is much smaller than the corona ring on HS terminal, and the corona ring is more close to side wall. But there is a 90 degree straight corner on this clam. So the spark happened between the side wall and this 90degree corner on the clamp. We can not change the clap now since it will break vacuum. So, we decide to do HV test without HV cover installed onTM3 at TCS. A work permit is issued for  this. Lock out procedure: Document-70039 is clarified in WP.

Today TM3 HV conditioning went up to 65kV. I'll leave it here over night. ISAC OPS have instructions to keep an alarm on the HV, and ion gauge and turbo pump (TP2 tripped 3 times due to sparking).

Today the voltage was raised from 65kV to 65.7kV. After about 10min at this level TM3 started current limiting. TM3 was not sparking when this happened.

ISAC OPS turned it OFF and then tried to ramp it back ON, but could not do it due to the current limiting.

I turned OFF/ON the PS (locally) 20ming ago and and then tried to get it back from the Control Room, but around 4-5kV it started current limiting.

Turned everything OFF on EPICS for the time being. We'll investigate on Monday.

Over the night the sparking rate at 65kV was from 6 to 2 sparks per hour. It tripped OFF once over the night: backing valves closed, tripping the turbos, tripping IG1 and turning OFF the BIAS. At that time ISAC OPS got it back smoothly.

Aurelia and I investigated this current limit issue May 8th. First, By separating TM3 from TCS. we found the problem is on TCS side. We did a Megga test on TM3 as well. TM3 is good at 1KV megga test. After that, The insulators between TCS to ground is test. they are good which could hold 1G ohm at 1000V megga test. So the problem should be on power supply side  including power supply itself ,cables and transformer. Later, Tom and John found the TCS bias HV cable from power supply to HV chase top in faraday cage is broken down. It can not hold 5KV on HV test now. This cable should hold 100KV. But not this time. Tom also found  the adapter on Faraday cage could be designed in a better way. The broken cable was removed yesterday. John and Tom will order an exactly same cable from Glassmen.