TM#4 has been moved from the south hot cell to the conditioning station.

TM4 has been connected at TCS for HV test. TCS water system and vacuum pumps are switched on.

TM4 in conditioning station with no source tray, extraction electrode set to HV bias. Brought up to 30kV over the weekend by ISAC ops with no issues.

BIAS=30kV

EL=15kV

IG1=9.4e-7T

Monday, June 10, 2019, 08:21 : At 32.4kV on bias, EL at 15kV, bias sparked to EL. Every bump of 100V on bias is accompanied by a small vacuum jump and often by a small current increase from ~60uA to 120uA.

* Conditioning station read back and set values are off by 2kV but archive values show this has always been the case. Which to trust? *

Monday, June 10, 2019, 08:54 : Spark while sitting at 38kV on bias, 19kV on EL. Seems to be a spark to EL.  Sitting at 38kV for ~10mins

Monday, June 10, 2019, 09:32 : Sat at 40kV for ~10mins or so, bumping up again.

Monday, June 10, 2019, 09:38 : Spark to EL while sitting at BIAS / EL = 42kV / 20.5kV

Monday, June 10, 2019, 13:57 : Ramped to BIAS / EL = 43 / 21.2. No problems other than what seems to be the usual conditioning station stuff.

 Monday, June 10, 2019, 14:17  : Couple sparks on the way to 48kV, seems to be to the EL.

Monday, June 10, 2019, 14:36 : Up to BIAS/EL = 51/25.5 with no big problems. Will stay here for a couple mins

Ramped down the bias so David could remove the HV cover and lock out the target hall.

Ramped back up to BIAS / EL = 50kV / 26kV quickly with no sparks.

Monday, June 10, 2019, 15:35 : BIAS / EL = 52kV / 26kV

Monday, June 10, 2019, 15:45 : BIAS / EL = 54kV / 28.5kV. It seems that keeping the EL at 1/2 the bias voltage is right on the edge of what is acceptable - the HV seems to be sparking to the EL instead of to ground (either that or the EM interference from the sparks it producing a reaction in the EL...) I will run the EL higher and see if that helps.

Monday, June 10, 2019, 15:57 : Sat at 54kV for > 10mins, will continue up

Monday, June 10, 2019, 16:11 : At BIAS / EL = 55.8kV / 29kV got some sparks, some to EL and some to ground. Back at 55kV.

Monday, June 10, 2019, 16:24 : Sitting at 56kV for a couple mins

Monday, June 10, 2019, 16:37 : Sitting at 56.3kV bias, a spark tripped off the IG1S, which tripped the turbo and shut down all the HV. Back to zero!

Monday, June 10, 2019, 16:59 : TM4 back up to BIAS / EL = 56.1kV / 29.5kV

Monday, June 10, 2019, 17:14 : After 57kV on the bias, things seem to be getting sketchy. The EL voltage and current are a bit jumpy and stepping that voltage up seems to produce a slight spike in the bias current.

Monday, June 10, 2019, 17:31 : At BIAS / EL = 58 / 30.3, looks ok.

Monday, June 10, 2019, 18:11 : Sitting at 59.5kV, could probably push further but will sit here for ~30mins and make sure it is relatively stable. Note 59.5kV set point is really 56.9kV on readout.

Monday, June 10, 2019, 18:18 : some current inconsistencies (not really sparks probably corona discharges...) so went down to 59kV

Monday, June 10, 2019, 18:23 : still getting some sparks at 58kV, going down to 57.5.

Monday, June 10, 2019, 18:37 : went back up to 58, spent 10 mins there but was getting these corona-discharge-type-sparks where the vacuum responds and current is drawn but voltage is maintained. Went down to 57kV, still getting them....

Monday, June 10, 2019, 18:40 : Going down to 56 to see if they stop....

Monday, June 10, 2019, 18:42 : Trying 55 kV now... see attached screenshot for example of sparks

Monday, June 10, 2019, 18:48 : No luck at 55 or 54kV. Going down to BIAS / EL = 50 / 27 for a while to see if this resolves

Monday, June 10, 2019, 19:20 : 30 mins at 50kV, a couple little sparks, but no big deal. Ramping up to 55 and let's see if it is ok.

Monday, June 10, 2019, 19:30 : 55 still not working, going down to 54

Monday, June 10, 2019, 20:16 : Held bias at 54kV for about an hour - dropped it down once to alleviate some sparking. Leaving it at 50kV overnight, ops have instructions to keep an eye on it and shut it off if anything seems strange.

The conditioning station was ramped down overnight and David ramped it back up this morning. Currently trying to get it to sit stable at something higher than 55kV.

Wednesday, June 12, 2019, 08:11 : Sitting at BIAS / EL = 53.9 / 28, 2 sparks to ground, one turned off TP2

Wednesday, June 12, 2019, 08:53 : Been at BIAS / EL = 56 / 28 for >10mins with no real sparks. Will continue up.

Wednesday, June 12, 2019, 08:54 : Spark to EL at 56.1

Wednesday, June 12, 2019, 09:34: Sitting at 55kV for 40 mins. 2 sparks to ground, 1 spark to the EL.

Wednesday, June 12, 2019, 09:49 : Leaving it at 56.6 for a while

Wednesday, June 12, 2019, 09:56 : Either there was a spark to IG1S or there is a vacuum limit that does not accept a 1.2e-6T spike. Everything turned off, but everything seems to still be working.

 TM#4 has been moved from the conditioning station to the south hot cell. The module has no source tray.

I brought up TCS bias to 57.5 KV on TM4 without source tray at TCS. It was kept at this level about 1.0 hours. Some intermediate sparks. No spark trip on devices. 57.5KV is very close to the HV up limit of this module.

New source tray was installed on TM4 yesterday. LD was used to pump down all water lines. Tube heater +D circuit has a big leak  around water block area. At base LR5.5 XE-5 atm.cc/sec, base pressure 170 mtorr, When Chad  moved this helium probe close to  water block area, The leak rate on LD went to maximum level right away. When the probe was moved out of water block area, the LR went back to base level. The test repeated again with same result. Also, Before helium spray test, I applied 80 psi air into the circuit. The air pressure can not be held in the circuit. The pressure drop on the line is quick. This shows a big leak on  D line as well. The rest water lines are fine. I pumped all of them down to LD bottom scale without problem. Chad and I helium sprayed window and MSP water blocks. No leak has been found.  The only line we have problem is the D line.

D line is still leak after repair and installation. We suspected the c-seals issue on this module. So, all water lines have been sprayed with helium for leak check.  Found "B" line target oven negative  is also leak. The rest line are good on helium spray leak check.  The further investigate on thi issue is taken place now.

Ramping up TM4 with new source tray in. Current condition: new source tray but no water running in lines B, D or window. Multipin on the source tray grounded but no lines running into the service tray. Vacuum at 1e-6T.

Small sparks at each BIAS voltage increase starting at 30kV. First real spark around 38kV.

Going up in more than 100V steps causes spikes that take out TP2.

Monday, July 08, 2019, 14:54 : TP2 sparked off at BIAS/EZL = 44kV/25.1kV after 10mins at this point... will leave it here for a couple hours

Monday, July 08, 2019, 16:44 : 6 sparks in last 2hrs, some very strange vacuum response in which the pressure rose steadily and stayed high for 30mins before coming back to normal.

Monday, July 08, 2019, 17:25 : After 30mins at 45kV, started sparking, so will ramp down to 40kV and leave it overnight

Left TM4 at 40kV overnight with no previous heating. There was only one real spark which turned off TP2 but ops turned it back on. Ramping up again.

Tuesday, July 09, 2019, 09:13 : At 45kV the module is sparking on the top pretty regularly. Turned down and David will remove cover and lock out target hall.

Tuesday, July 09, 2019, 16:30 : Went up to 45kV pretty easily now.

Tuesday, July 09, 2019, 16:32 : Everything tripped at 46.5kV. Looks to be a spark outside the vacuum. TM4 sparking much more outside the vacuum than last checks done without the source tray...

Tuesday, July 09, 2019, 16:43 : Everything easy up to 46.8kV, then sparking a bit. Most of the sparks are outside the vacuum. Sitting at BIAS / EZL = 47.1kV / 28kV for a couple mins before ramping down.

Tuesday, July 09, 2019, 16:57 : 10 mins at 47.1kV with not problems. Continuing up still looks like it is really sparking a lot outside the vacuum.

David raised the bias to 49.5kV. Module is still sparking mainly on top. Humidity is super high.

Wednesday, July 10, 2019, 09:50 : sitting at 49.7kV for a coupe mins, big spark in vacuum took everything down.

Wednesday, July 10, 2019, 10:16 : Bias at 50kV drawing 0.1mA but looking pretty jumpy. Left it here for ~5 mins then had to turn down bias for TH work.

Wednesday, July 10, 2019, 13:33 : Back at 50kV on bias with no problems.

Wednesday, July 10, 2019, 13:46 : 10 mins at 50kV, will try to go a bit higher

Wednesday, July 10, 2019, 14:06 : Everything sparked off while sitting at 50.6kV. This seems a bit futile....

Wednesday, July 10, 2019, 14:15 : Back at 50.1kV with no problems

 Wednesday, July 10, 2019, 14:33 : Up to 51kV

Wednesday, July 10, 2019, 15:20 : Difficult to go above 51.4. Turning up EZL since there were a couple sparks to it. -> Putting this up causes significant sparks between HV bias and EZL in vacuum at 5kV on the EZL, 51.1kV on bias. Leaving EZL @ 5kV

Wednesday, July 10, 2019, 16:04 : Around 52.2 sparking on the top of the module

Wednesday, July 10, 2019, 16:36 : 53kV

Wednesday, July 10, 2019, 17:16 : up to 54kV without much trouble

Wednesday, July 10, 2019, 17:53 : Sitting at 55kV for about 4mins when a spark tripped off everything. Will try to bring it back to sit stable at 55kV tomorrow

Thursday, July 11, 2019, 10:28 : TM4 up to 54.9kV without too many problems, EZL at 25kV. Will sit here for a little while

Thursday, July 11, 2019, 10:48 : 20mins at 54.9kV, going to 55kV

Thursday, July 11, 2019, 11:14 : 25 mins at 55kV and everything sparked off

Thursday, July 11, 2019, 11:26 : Back up to BIAS / EZL = 54.8kV / 26kV. Checked that the TGHT and TBHT power supplies have not been killed by sparks, everything looks fine.

Thursday, July 11, 2019, 12:56 : Sat at 49.6kV for 1.5hrs with one spark.

Thursday, July 11, 2019, 14:02 : Been sitting at 55kV for 2 hrs now with one spark.

Friday, July 12, 2019, 08:09 : Back up to 52kV easily, then had to reduce step size to 100V to keep going

Friday, July 12, 2019, 08:47 : 56kV

everything sparked off at 56.5 or so

Yesterday, TM4 B line was supplied with water at TCS for testing.  TCS: IG1 and IG2 had big response on it. See attachment. The water was disconnected right away, and line was purged. Vacuum recovered after overnight pump.

 TM4 has been moved from the conditioning station to the south west silo. Module was placed with the turbo pump facing south which puts the windows facing north and south and the shutter facing west. 

TM4 has been moved from the South West silo to the South Hot Cell. New source tray, no target.

Tm4 has been moved from SHc to TCS without containment box. It is a new source tray on TM4.  A layer of plastic sheet was put on TH floor on module pass route during the move.

Electrical check on TM4 with spare test target. PQ is still shorted to RU. But, The short between 60KV bias and PQRU which we found last time is gone.  Target to PQRU at 250V mega measurement: 22.08 M. All conductivity between ABCD are about 5.0 m ohm which are good. All other ohm meter and Mega measurements are OL. Helium spray leak check on HS target VCR fittings is passed. Two blank offs are installed on source tray window cool lines. They passed  leak checked at SHC as well. 

Original Problem:

Leaks on the B and D copper blocks, probably due to corrosion between the blocks and the inserts. Final solution, after many tries, was to make an indium sheet the size of the copper blocks and use that to seal the entire surface.

So far offline tests seem to show these seals working, though long term tests are needed to say for sure.

New Problem:

TM4 was in the conditioning station, running water and under vacuum. 50A on the target and tube heaters from Friday evening to Saturday afternoon.

Sunday morning 9:30am target heating increased to 200A, 100A on the tube heater, ramp finishing around 1:30pm.

At around 3pm, the vacuum stops dropping and begins to climb very slowly, continuing until Monday morning (see first attachment).

Tests on the individual water lines show an immediate vacuum response only to the EE lines, the other lines show no response (note the window line is blanked off since the containment box is not on).

With water in the EE lines, the vacuum sits around 1e-6 near the target.

Testing:

Ramped target to 400A and tube to 200A. Sitting there for several hours shows no change in the vacuum response, though there is a variation that corresponds to the chiller turning on and off (see second attachment).

Ramped target to 450A and tube stayed at 200A. Vacuum staying in the low 10^-6 range, seems to be entirely due to EE leak, other seals are holding well.

TM4 has been moved from the Conditioning Station to the South Hot Cell. No target.

TM4 has developed 4 leaks so far:

1. Line B - seems to be solved with an indium gasket (some testing already done)
2. Line D - seems to be solved with an indium gasket (some testing already done)
3. EE - indium gasket installed, preliminary tests are good
4. Window cooling line - David found with 70 psi helium, leak detector response went from 3E-7 atm.cc/sec to 4E-4torr atm.cc/sec in 1 min and stayed there. When water was put through the circuit, there was an initial vacuum increase to about 1E-6, then a couple hours later it began to fall and has been falling for ~14hrs, now in the 7E-7 range for IG1.

Testing indium seals on lines B, D, EE. No repair has been done for window line. Ramping heaters to TGHT=450A, TBHT=200A. Everything behaving normally.

Wednesday, ramped up BIAS, EZL and EE to 36kV, 17kV, 1.5kV. Sparking at about 1 spark per 20 - 30 mins. Bias current somewhere between 66uA and 78uA, varying with the chiller ON/OFF.

Thursday morning around 2am everything tripped off. No spark visible on strip tool, seems this is from an increase in the window water leak. Vacuum recovered on its own and heaters were ramped back up.