chad fisher wrote:

A video inspection of the lower service tray was performed this morning. Camera was inserted upwards into the service tray/service chase from the containment box. One potential spark mark was identified as well as epoxy not flush with insulator. Fiberglass does not extend into service tray/chase much and does not look, from initial inspection that it will be too difficult to remove. Video can be found in docushare Document-142295

 The attached picture is from the video: showing the fiber glass inside the Service Tray.

Please see the first two pictures of the HV test at atmosphere done on April 19.

First picture shows a big spark between the Extraction Electrode mounting plate holder (at HV) to the Ground Electrode. Please note minimum gap between these electrodes is 7.8mm.

However the biggest visible spark was noticed at the bottom screw hole. (It is difficult to determine if at the 7.8mm gap were concomitant sparks occurring.)This spark started at 22kV and became stable above 24kV.

On April 20, 2017 Chad inspected this area with the Hot Cell camera and found a spark mark on the Extraction Electrode holder plate at the bottom screw hole. See the third picture.

Again it is very difficult with the existing Hot Cell cameras to inspect the areas where the gap is small (at 7.8mm gap between the conical surfaces of the Ground Electrode and Extraction Electrode holder)

In the second picture there are 2 small sparks: one on the top and another one on the right side at the lines wrapped in Fiber Glass. These sparks started at 20kV.

On April 21 the HV test was repeated with the Containment box ON and the sparks occurred again, at the same locations and at the same voltage.

Please see TM3's Updated Schedule (the current schedule) at the SHC and TCS.

For the record I am uploading TM3's Initial Schedule at the SHC and TCS.

Today a HV test at atmosphere was performed for Each Line(s), while floating the other lines.

Please note the Einzel Lens line (Y) was grounded.

Tests performed:

1. Applied up to 29kV to "(A) OVEN -"

Sparks occurred at 29kV at the Fiber Glass inside the Service Tray; and at the right side at the lines wrapped in Fiber Glass.

2. Applied up to 29kV to "(B) OVEN +"

Sparks occurred at 29kV at the Fiber Glass inside the Service Tray.

3. Applied up to 6-7kV to "(C) TUBE -"

Could not identify the sparks; it was a continuous discharge.

4. Applied up to 6-7kV to "(D) TUBE +"

Sparks/discharges occurred at 6-7kV at the Fiber Glass on the right side at the lines wrapped in Fiber Glass. We had to turn OFF the lights in the Hot Cell to identify the small sparks.

5. Applied up to 4kV to "(P-Q) COIL +"

Discharge occurred at 4kV. Could not identify it.

6. Applied up to 29kV to "(R-U) COIL -"

Sparks occurred around 27-29kV at the Fiber Glass inside the Service Tray; and at the right side at the lines wrapped in Fiber Glass.

Tom saw a spark at 27kV between the Extraction Electrode holder and the Ground Electrode.

Saw also a spark at the Ring Alumina insulator in the Service Tray, located on top of the second comb insulator with Epoxi.

7. Applied up to 27kV to "(E) Cavity or MSP, (G) Gas, (J) Heat Shield" - all these are connected together at the BIAS voltage.

Sparks occurred at 27kV at the Fiber Glass inside the Service Tray.

8. Applied up to 23kV to "(I) EE" - this is for the Extraction Electrode.

Sparks occurred at 23kV again at the same spot near the bottom screw hole, between the Extraction Electrode holder and the Ground Electrode.

9. Applied up to 27kV to "(F) Plasma"- this is for the Anode.

Sparks occurred at 26-27kV again at the same spot near the bottom screw hole, between the Extraction Electrode holder and the Ground Electrode.

Please see attachment for TM3 May 1-15, 2017 Tentative Schedule at the SHC and TCS.

Fiberglass, capton tape and tie straps have been removed from TM3.

RU( coil 1), F (anode), and 60kv bias(HS, Gasline, and MSP)  are shorted together. Other lines are good.  They are measured between  individual line each other  @500V. The megga showed OL. All of  them to ground are OL @500V as well. EL is grounded during check.

Today a HV test at atmosphere was performed for Each Line(s), while floating the other lines. THIS TEST AT ATMOSPHERE IS THE FIRST ONE AFTER THE FIBER GLASS WAS REMOVED.

Please note:

a). the Einzel Lens line (Y) was grounded.

b). after the fiber glass was removed the "R-U COIL" line and the "(F) Plasma" (which is the Anode line) were shorted to the BIAS voltage.

Tests performed:

1. Applied up to 32kV to "(A) OVEN -"

Sparks occurred at 32kV between the Extraction Electrode holder and the Ground Electrode.

2. Applied up to 33kV to "(B) OVEN +"

Sparks occurred at 33kV but we could not identify the spark location.

3. Applied up to 31kV to "(C) TUBE -"

Sparks occurred at 31kV between the Extraction Electrode holder and the Ground Electrode, at the top and the bottom.

4. Applied up to 30kV to "(D) TUBE +"

Sparks occurred at 30kV between the Extraction Electrode holder and the Ground Electrode, 2 spots at the bottom.

5. Applied up to 30kV to "(P-Q) COIL +"

Sparks occurred at 30kV between the Extraction Electrode holder and the Ground Electrode, all around.

6. Applied up to 30kV to "(R-U) COIL -", to "(F) Plasma" the Anode, to "(E) Cavity" or MSP, "(G) Gas", and  "(J) Heat Shield" - all these are "connected" together at the BIAS voltage.

Sparks occurred around 29-30kV at the Extraction Electrode holder and the Ground Electrode, at the bottom.

8. Applied up to 23kV to "(I) EE" - this is for the Extraction Electrode.

Sparks occurred at 23kV again at the same spot near the bottom, between the Extraction Electrode holder and the Ground Electrode.

During TM3 roughing down yesterday afternoon,The vacuum gauges CG1 and CG1S stopped at 1.20 torr and refused to drop down. Leak detector was used to found where the problem was. LD was in rough detecting mode and showed 2.7xE-6 atm.cc/sec as base leak rate.  Spray medium amount of helium to TM3 service top, and found this leak was on round panel of TM3 side wall. LR climbed to E-4 atm.cc/sec range on helium spraying. In my vie,One 10# socket head screw was not tightened properly after cameras job and caused this.  The TCS was vented. The panel was took off and reinstalled. O-ring on the panel  was replaced by a new one as well. Then TCS was pumped down again. The leak disappeared .  All turbo pumps on TCS and TM3 were started successfully later.

Today TM3 was conditioned up to 50kV. At 50kV sparking started and to go higher in voltage it needs more conditioning time. Lowered it to 49kV for the night.

  TH crane is used for transferring five shielding blocks(30000lbs) today. All brakes work fine. Mostly, I do not need crane slow up motion for this job so during 1 hour moving, no rattle sound and burning smell has been found.  I think TH crane is good to use as we decided. 

Sparking was found this morning inside HV cover on TCS in TH while TM3 was HV conditioning at TCS. The TCS bias was set to 50KV when I found this spark. It sparks between HV cover front side wall to a KF 25 clamp on HS water terminal. The KF 25clamp is used in this situation on HS water line top because the HS line is opened inside module, and we need a vacuum seal on the top of this line.  Actually, The clap is much smaller than the corona ring on HS terminal, and the corona ring is more close to side wall. But there is a 90 degree straight corner on this clam. So the spark happened between the side wall and this 90degree corner on the clamp. We can not change the clap now since it will break vacuum. So, we decide to do HV test without HV cover installed onTM3 at TCS. A work permit is issued for  this. Lock out procedure: Document-70039 is clarified in WP.

Today TM3 HV conditioning went up to 65kV. I'll leave it here over night. ISAC OPS have instructions to keep an alarm on the HV, and ion gauge and turbo pump (TP2 tripped 3 times due to sparking).

Today the voltage was raised from 65kV to 65.7kV. After about 10min at this level TM3 started current limiting. TM3 was not sparking when this happened.

ISAC OPS turned it OFF and then tried to ramp it back ON, but could not do it due to the current limiting.

I turned OFF/ON the PS (locally) 20ming ago and and then tried to get it back from the Control Room, but around 4-5kV it started current limiting.

Turned everything OFF on EPICS for the time being. We'll investigate on Monday.

Over the night the sparking rate at 65kV was from 6 to 2 sparks per hour. It tripped OFF once over the night: backing valves closed, tripping the turbos, tripping IG1 and turning OFF the BIAS. At that time ISAC OPS got it back smoothly.

Aurelia and I investigated this current limit issue May 8th. First, By separating TM3 from TCS. we found the problem is on TCS side. We did a Megga test on TM3 as well. TM3 is good at 1KV megga test. After that, The insulators between TCS to ground is test. they are good which could hold 1G ohm at 1000V megga test. So the problem should be on power supply side  including power supply itself ,cables and transformer. Later, Tom and John found the TCS bias HV cable from power supply to HV chase top in faraday cage is broken down. It can not hold 5KV on HV test now. This cable should hold 100KV. But not this time. Tom also found  the adapter on Faraday cage could be designed in a better way. The broken cable was removed yesterday. John and Tom will order an exactly same cable from Glassmen.

TM3 service tray side panel is cleaned and installed back to TM3 this morning. I visually inspected service tray insulators. Nothing is changed comparing to inspection before.Nine screws are installed on  panel to hold it onto TM3.

 TM#3 has been moved from the north east silo to the conditioning station.

50 kV stable with less than 1 spark/h during the weekend by RIB obs.

Today 57 kV reached, but observed sparking on the resistors of the new HV cable. The cable will be removed to take out the resistors, before reinstalling it.

The target hall was locked out, but has now been reopened. This was to avoid sparking to the HV cover.

Anders Mjos wrote:

50 kV stable with less than 1 spark/h during the weekend by RIB obs. Today 57 kV reached, but observed sparking on the resistors of the new HV cable. The cable will be removed to take out the resistors, before reinstalling it. The target hall was locked out, but has now been reopened. This was to avoid sparking to the HV cover.

 EL was brought to 27.5kV.

Conditioning continues today with resistors removed from the HV cable

Tuesday, July 18, 2017, 16:53: BIAS / EZL 50 / 25 kV - no sparks

 Tuesday, July 18, 2017, 16:56: BIAS / EZL 55 / 27.5 kV - one spark

TM3 remained at 55 kV over night. Turbo pumps tripped twice and was restarted and HV recovered by RIB ops. Last recovery brought EZL to 16 kV (not 27.5 kV)

About 25 sparks over 15 h. No sparks for the last 3 h

Wednesday, July 19, 2017, 08:12: EZL to 27.5 kV

Wednesday, July 19, 2017, 08:38: BIAS to 57 kV

Wednesday, July 19, 2017, 08:46: BIAS to 57.5 kV

Wednesday, July 19, 2017, 09:05: BIAS to 59 kV

Wednesday, July 19, 2017, 09:19: BIAS to 60 kV

Wednesday, July 19, 2017, 09:28: EZL to 30 kV

Wednesday, July 19, 2017, 12:25: Heavy sparking. Lowered BIAS to 30 kV

Wednesday, July 19, 2017, 12:37: Increased BIAS again towards 50 kV, but heavy sparking again between 45 and 50 kV. Lowered back to 30 kV

Wednesday, July 19, 2017, 12:40: BIAS at 46 kV

Wednesday, July 19, 2017, 12:42: BIAS at 47 kV

Wednesday, July 19, 2017, 12:44: BIAS at 48 kV

Wednesday, July 19, 2017, 12:48: BIAS at 50 kV

Wednesday, July 19, 2017, 14:50: BIAS at 55 kV after ~2h without sparking at 50 kV

Wednesday, July 19, 2017, 15:12: Sparking. Lowered to 40 kV

Wednesday, July 19, 2017, 16:39: BIAS at 55 kV

Friday, July 21, 2017

Observed top of module inside target hall during HV sparks using crane cameras. No sparks observed, but some faint sound could be heard. Module still sparks at 60 and 55 kV. Have to lower to 40 kV to stop sparking.

Friday, July 21, 2017, 13:25: BIAS / EZL at 50 / 25 kV for without sparking for the last 90 min