TMSRP2 has been struggling since the power bump occurred Nov 18, 2020

Monday, November 23, 2020, 09:48: Open TMS:PV2.

Monday, November 23, 2020, 09:51: Close TMS:PV2. CG2B @ 0.515T.

Monday, November 23, 2020, 09:52: TMS:RP2 off.

After disconnecting the backing line to RP2, a lot of water poured out of the line. May have back tracked from the vapour off of RP1 and the leaks on the chamber side. Both are on the same line.

Monday, November 23, 2020, 13:06: TMS:RP2 swapped w/ old Tri-Scroll pump. RP2 on.

Monday, November 23, 2020, 13:23: TMS:TP6 & TMS:TP2 on.

Monday, November 23, 2020, 13:29: Switch over FL1  to FL2 on IG6 Filament. IG6 on.

Monday, November 30, 2020, 10:12: CG2B @ 0.131T. TP6 & TP2 off. IG6 off. Close PV2 & shut off RP2.

Monday, November 30, 2020, 10:31: RP2 large Tri-scroll pump swapped back out for the small Varian RP.

Monday, November 30, 2020, 10:32: RP2 on.

Monday, December 07, 2020, 11:06: Turbo pump TP6 on.

Monday, December 07, 2020, 11:11: Turbo pump TP6 up to speed. TP2 on.

Monday, December 07, 2020, 11:14: IG6 on.

Tuesday, December 01, 2020, 11:56 :

TSG:BIAS unplugged and controls pigtail (made by Ray) added in order to connect controls cable to TCS:BIAS, which is now installed in the same rack.

Basic controls testing done without turning on the power supply, controls max values scaled to 75kV, 5mA.

In process of grounding all features inside the faraday cage. New grounding will be marked with a label for easy identification later.

NOTE: we should get together in January to discuss test stand wiring, since I discovered another burned up connection (see next elog entry, #823).

Tuesday, December 01, 2020, 16:47 :

Grounding of all feedthroughs on the service cap complete and continuity checked.

Removed gas line with Fernando's approval, to make more space for our feedthrough.

Awaiting nylon insulator to proceed.

Between feedthroughs F (magnet coil) and B (tube heater negative), I found a jumper with two resistors that has been burned up. I am not sure when this happened, but it would be good to review the overall setup of the service cap wiring in the new year.

The cable connected to feedthrough Q where the multipin sits also looks burned.

Wednesday, December 02, 2020, 08:31 :

Added copper wire connection to multipin screw terminals in HV cage to make sure they are grounded.

Attached grounding rod to back of HV cage. Not ideal, but couldn't find a better place without making a extension to the cable. Will see what John Chow says. EDIT: John says it is ok for the tests, not for permanent.

Vented test stand to swap window positions. Swapped rightmost window on the door with two windows with window on the door with only one window. Replaced one oring, that on the single window. Cleaned the inside surfaces of both windows with methanol before re-installing. Roughing down just to make sure there are no major issues, at 0.313T and falling. EDIT: down to 0.015T overnight.

All breakers on HV cage panel are OFF.

Yesterday, December 1st, I successfully installed and tested the gas system inside the faraday cage.

The changes performed were as follows:

• Remove He gas tank (to make room for new tank)
• Remove calibrated leak to connect new gas line
• Remove convectron gauge to install a high range dial manometer.

The tests were done manually and consisted as follows:

• Gas lines roughed by opening SV1.
• Tank regulator set to deliver 1 bar
• SV4 opened to allow the gas to enter the lines
• Dial manometer verified to  match the pressure set by the regulator
• Alternatively roughed and pressurized the lines by opening/closing SV1/SV4
• Tank lines emptied with SV1 and SV4 open simultaneously.

The test involve the lines up to SV5. The gas line portion from SV5 to the top of the HV cage was not included as it was removed for TM3 HV tests (se entry #822)

When the HV tests are concluded I will install a new line that will include a gauge.

Thursday, December 03, 2020, 15:54 :

Got sign off from John Chow to run the system, though the nylon insulator was not ready when he came by. But we went over everything together.

Installed blue nylon insulator with all vacuum surfaces cleaned at least with methanol. No NPT fitting or tube installed, just testing service cap.

Thursday, December 03, 2020, 17:31 :

Tested voltage on the blue nylon insulator up to 75kV, left it at 75 for ~30mins. No sparking, no current drawn except occasionally when changing voltage and only ~1uA then.

Cleaned (but not very well...) the NPT fitting, copper tube, alligator clip and section of copper wire and installed these in the vacuum chamber. Will pump down overnight for testing tomorrow.

See attached photos for insulator setup and wire inside vacuum chamber.

Friday, December 04, 2020, 08:52 :

Test stand roughed down to 29mT overnight. Started turbos for testing later.

Friday, December 04, 2020, 13:34

Vacuum is at 3.6e-6 Torr and possibly still dropping. Starting HV tests with just a tube and wire hanging down from the feedthrough. Turned on power supply and started raising voltage.

Friday, December 04, 2020, 15:13

Achieved 75 kV. No sparking at any point along the way. Vacuum is around 3.1e-6 Torr. Will hold for about 30 minutes to confirm.

Friday, December 04, 2020, 16:07

Held 75 kV for ~35 minutes. Ramped down and shut off power supply. Vacuum is at 2.7e-6 Torr.

The copper tubes to be used in the TM3 test piece may have flux inside them from soldering the end caps on. There are holes along the length for outgassing, and we will try to bake them out in the test stand before installing them. There is still a dummy bake out target installed there, see the photo for how the tubes are loaded in. TGHT ramping started around 3pm.

I had to remove the grounding connections from the TGHT feedthroughs for this, so I redid them such that the tube is attached to the EZL. This target has no tube, so the connection between the tube and target is just through the 100Ohm resistor.

We must re-do the grounding before proceeding with HV tests.

Wednesday, December 09, 2020, 19:38 :

Target heater ramp down started. See attachment for vacuum progression during bake out.

Test piece installed with Sam, Aaron and Alexander. Ended up shimming the support stand (middle foot) since it seemed to be bowing a bit and distorting the gap between the containment box stand-in piece and the service tray piece. We should measure the gaps when testing is done with the same configuration. Installation was not too hard, installed the whole thing as one piece. Had to remove the middle foot on the long side to get it through the door but otherwise no problems.

Test stand re-wired for HV testing. Wire hooked up to test piece by pinching between two tubes. Verified GND and HV continuity.

Monday, December 14, 2020, 11:30

Switched off all breakers in HV cage. Started HV testing of TM3 prototype.

Monday, December 14, 2020, 12:58

Possible first spark at 18 kV.

Monday, December 14, 2020, 14:51

Nothing interesting since then, up to 33.5 kV now.

Monday, December 14, 2020, 14:56

Possible spark at 34.5 kV. 35.5 kV, 36.5, 37.5, 38.5, 39, 39.5, 40.5, 41, 41.5, 42.5, 43 (bigger current spike), 43.5, 44 (vacuum jumped for a bit), 44.5, 45 (voltage drop and vacuum jump).

Monday, December 14, 2020, 16:14

Holding at 45 kV for 10 minutes. Each of these previous "sparks" occurred just after raising voltage. Will decrease step size to 250 V from 500 V if these get more significant.

Monday, December 14, 2020, 16:19

45.5 kV. Went to observe possible spark while Carla raised voltage. Saw small flash towards bottom left, unsure of depth, but perhaps farther to the back.

46 kV. Sparking maybe in the same location. Carla reduced voltage.

Monday, December 14, 2020, 16:47

47 kV. Sort of unstable every time voltage is raised. Decreasing step size to 250 V.

Monday, December 14, 2020, 17:11

48.75 kV. Decreased current limit to 0.005 mA a couple steps previously to see effects. Seems like the current bumps at the start of each voltage raise have dropped from 0.010 mA to 0.005 mA. This seems to indicate that the current limiting is working and is controlled by EPICS. Pierre thinks the prototype must be pretty clean since the testing has been pretty smooth up to this point.

Monday, December 14, 2020, 17:29

49.75 kV. Heard what sounded like an air piston in the background and also saw a large drop in voltage. Voltage dropped in a similar manner a couple minutes after but without the sound. No effect on the vacuum though.

Monday, December 14, 2020, 17:45

50 kV. Held for 8 minutes and then a sound again occurred that sounded like some sort of pneumatic system; along with this a total drop in voltage, but no change in vacuum Otherwise completely stable. Will hold for another 10 minutes or so and ramp down.

Monday, December 14, 2020, 17:53

50 kV. Stable. Starting ramp down.

Monday, December 14, 2020, 18:11

Voltage ramped down. Vacuum has been at around 1.13e-6 Torr throughout this. Shut off power supply. Dumped data to TM3HVtest-20201214-Prototype_2, and earlier I dumped to TM3HVtest-20201214-Prototype. GoPro is taped to vacuum box window.

Tuesday, December 15, 2020, 11:15

Vacuum at 1.0e-6 Torr. Turned bias power supply on, starting HV ramp at 500 V increments, increasing every 30-50 seconds seconds or so.

Tuesday, December 15, 2020, 12:47

Reached 50 kV. Not a single dip in voltage, significant current bump, or change in vacuum up to this point. Will hold for 10 minutes before continuing.

Tuesday, December 15, 2020, 12:57

Continuing voltage ramp. Decreasing increments to 250 V as this is uncharted territory, and that rate of increase seemed safe yesterday. Will also hold at each voltage for about 2 minutes after no signs of activity.

Tuesday, December 15, 2020, 13:04

Small dip in voltage and wiggle in current at 50.75 kV, no change in vacuum.

Spark at 51.25 kV with change in vacuum; reduced voltage temporarily. Did the same at 51.75 kV. Did the same at 52 kV, but let it sort itself out this time since it didn't seem too violent.

Actual spark at 52.25 kV with big voltage drop, and I saw it happen. On the vacuum gap side, somewhere in the flat plane and close to the middle. Another spark after about 3 minutes of holding. Just missed recording it but I saw it on the feed of the GoPro; maybe in a similar location as the previous. Another 1 minute later, didn't see it this time. The last two had no effect on vacuum.

Tuesday, December 15, 2020, 13:31

5 minutes since last spark. Raising voltage again, this time recording with GoPro. Spark captured at 53 kV. No change in vacuum... until a minute later.

Tuesday, December 15, 2020, 14:46

55 kV. Each time voltage is raised, it takes a couple seconds to reach a stable voltage. During this time the current limit of 0.005 mA seems to be drawn. Will reduce current limit to 0.001 mA to see effect. Dumped StripTool data at 55.25 kV.

Recorded spark at 56 kV. Voltage took longer to recover as a result of the reduced current limit.

58.5 kV. Effect of reducing current limit seems to make it take longer to reach a stable voltage. I think this is a good thing as it should smooth out the conditioning. Sparked here twice, reduced voltage for a bit before bringing back up again.

Tuesday, December 15, 2020, 16:23

60 kV. Sparked twice. Carla observed some: a couple small ones on the face of the insulator as the voltage was recovering from the first spark followed by a large one that went all the way across. Only the larger events seem to be captured by the GoPro.

Tuesday, December 15, 2020, 16:47

Finished holding at 60 kV to 20 minutes with no sparks other than the other two previously mentioned. Resuming voltage increases.

Tuesday, December 15, 2020, 17:02

61.5 kV. Small spark but not full breakdown, then a large spark that cut voltage to 0 V. Voltage dropped relatively slowly to about 48 kV. Lowered the voltage limit to 44 kV. During this entire time, 0.001 mA of current was being drawn. Carla started raising voltage while increasing the current limit as needed to reach the set voltage.

Tuesday, December 15, 2020, 17:44

Back at 60 kV and holding. Drawing 0.005 mA to 0.006 mA though. This is different from earlier when it was holding 60 kV with no current draw.

Tuesday, December 15, 2020, 19:02

Held 60 kV for an hour with current staying constant around 0.005 mA. Ramped down (first ramped down current to see effect) and then ramped down voltage. Shut off power supply.

Wednesday, December 16, 2020, 10:30 (Carla) :

Started ramping up about 45 mins ago, now at 50kV with no issues. No sparking, occasional current draw to get to the next voltage step. Test piece is no longer drawing any current, as opposed to yesterday when it was drawing around 4uA at this point. Trace left on insulator must have partially evaporated in the vacuum overnight.

Wednesday, December 16, 2020, 10:34 :

Starting to draw some current at 50.5kV. Drawing 4uA at 55kV.

Wednesday, December 16, 2020, 10:51 :

Back up to 60kV, no sparks so far and no issues, though it is still drawing 5uA of current, as it was yesterday.

Wednesday, December 16, 2020, 12:28 (Alexander)

No sparks at 60 kV for 1.5 hours. Starting voltage increases again at 250 V increments. Currently limiting current to 6 uA. Current draw is 4 uA.

Wednesday, December 16, 2020, 13:20

63 kV. Appears current limited, increased current limit to 7 uA. First spark was at 62.75 kV.

63.25 kV. Seems current limited again but eventually reached a stable voltage after a minute. Increased current limit to 8 uA for next voltage increase.

63.5 kV. Spark.

63.75 kV. Unstable trying to reach voltage, appears current limited. Small spark followed by a big one on the rise of the previous. Second one activated something pneumatic nearby. Reduced voltage by 250 V. Looked at vacuum system but nothing abnormal.

63.5 kV. Spark, not recorded on GoPro. Stable after 5 minutes. Continuing.

63.75 kV. No longer current limited and reached voltage stably. Dumped data.

Wednesday, December 16, 2020, 14:02

64.5 kV. May be current limited again, took over a minute to reach stable voltage.

Note: None of the sparks recorded today so far have produced a response in the vacuum, which is holding steady at 1e-6 Torr.

65 kV. Smooth transition to it. Drawing 5 uA.

65.25 kV. Spark after voltage instability and current draw at 8 uA limit. Moved the Faraday cup, which made a sound. This is the sound I heard earlier. Lower current draw after the spark, back at 5 uA.

Wednesday, December 16, 2020, 15:20

Stable at 65.25 for an hour, no sparks.

Spark at 66 kV. Current limited and unstable voltage for about 2 minutes afterwards, then stable with current back to 6 uA.

Spark at 66.5 kV after some voltage instability and current limiting. Afterwards current and voltage returned to stability.

67 kV, two sparks, one right after the other. The second produced a bump in the vacuum. Reduced voltage. Brought back up easily; current stable at 5 uA.

67.25 kV. Spark. No instability preceding it for a good 20 seconds.

67.5 kV. Spark. Instability before, none after.

Wednesday, December 16, 2020, 16:08

68 kV. Spark. Instability before, none after. Vacuum dropped about 20 seconds after recovering. Current also dropped to 4 mA, at about the same time the vacuum dropped. Dropping current limit to 7 uA.

68.25 kV. Vacuum jumped back up to earlier values and then an eventual spark after some instability. Returned to stability after spark. Current now at 5 uA.

68.75 kV. Spark.

69 kV. Carla went to look inside. Saw some small sparks towards the back between the HV sheath and GND sheath in the small gap side. Very little registered in EPICS, though. Just a couple jumps in current and small dips in voltage. Not visible in GoPro footage.

Wednesday, December 16, 2020, 16:38

69.25 kV. Two sparks. Stable after second. Third spark. Reviewed GoPro footage because third spark was suspicious. Appears on video but not near or on the prototype. Could have been the wire. Dumped data.

69.75 kV. Some instability paired with a change in vacuum for the same duration.

70 kV. Carla went to look. Nothing visible on the StripTool but Carla said she saw small sparks towards the back. Going to leave here for 20 minutes before raising again.

Wednesday, December 16, 2020, 17:12

Held for 20 minutes at 70 kV with no action. Starting voltage increases again.

70.25 kV. Two and a half minutes of voltage instability, raised vacuum, and limited current. All of them stopped and started at the same time. Seems like conditioning to me.

70.5 kV. Carla saw two very faint sparks towards the back on the small gap side. Little appeared in EPICS. Spark followed maybe a minute and a half later. 2nd spark four minutes after that. Before and between those sparks the current was generally at 5 uA. About a minute after the 2nd, the current dropped to 4 uA.

70.75 kV. Short conditioning (response from all 3 monitors) period of about 10 seconds.

71 kV. 2 sparks one minute apart with some conditioning in between and after.

71.5 kV. Spark followed by 2 minutes of conditioning.

71.75 kV. One minute of conditioning.

Wednesday, December 16, 2020, 18:06

72.25 kV. Spark followed by a minute and a half of conditioning.

72.75. About a minute of conditioning.

73 kV. Instant spark followed by about a minute of conditioning.

73.25 kV. 30 seconds of conditioning then a spark. 1 minute of conditioning, then another spark, and conditioning done right after. Second one sounded like it activated the Faraday cup.

73.75 kV. Sporadic and weak conditioning? A few blips in voltage and current over about a minute and a half.

74 kV. Similar to previous but even less significant.

74.5 kV. Two minutes of conditioning.

Wednesday, December 16, 2020, 18:52

75 kV. 30 seconds of conditioning. Spark. 2.5 minutes of conditioning. 2nd spark. 10 seconds. 3rd spark; lowering voltage by 250 V for stability to capture footage of previous 3 sparks. Stable here for 5 minutes.

Wednesday, December 16, 2020, 19:01

75 kV. Nothing happening. Gonna leave for 30 minutes.

Wednesday, December 16, 2020, 19:31

75 kV for 30 minutes. No sparks or any other anomalies. Vacuum has been around 9.5e-7 Torr. Current hovering around 5 uA. Starting voltage ramp down.

Wednesday, December 16, 2020, 19:43

Ramped down. Nothing interesting. Turned power supply off in EPICS. Still on physically such that ramp up tomorrow can start remotely. Dumped data.

Ramping up to 75kV. Started drawing current at 63.5kV this time.

Now at 75kV, drawing 5uA of current. Ramp up easy.

Thursday, December 17, 2020, 13:02

Been at 75 kV since about 8:45. A single spark at 11:58.

Thursday, December 17, 2020, 18:10

75 kV. No sparks since that last one 6 hours ago. Current has been slowly decreasing over time, now sitting around 2 uA. Was about 5 uA around the time of the spark. Starting ramp down. Dumped data.

Shutting down the ISAC TS for the planned outage on Dec 22.

Monday, December 21, 2020, 07:50: Ion gauges & turbo pumps off.

Monday, December 21, 2020, 07:51: Confirm wire scanner shutoff.

Monday, December 21, 2020, 07:55: Waterlines purged.

Monday, December 21, 2020, 07:57: Backing valves TMS:BV6 & TMS:BV2 off. PV1 & PV2 off. Roughing pumps off.

Restarting TS after holidays & planned power outage.

Monday, January 04, 2021, 08:17: All epics controls in a 'white out' state.

Monday, January 04, 2021, 08:20: Rebooted PLC / IOC. Controls returned.

Monday, January 04, 2021, 08:22: Strip tools started / epics response ok.

Monday, January 04, 2021, 08:23: Switch on TMS:RP2 & TSG:RP1

Monday, January 04, 2021, 08:24: Open PV1 & PV2.

Monday, January 04, 2021, 08:26: Optics, steering, and archivers still in a non-responsive / 'white-out' state.

IOC rebooted on January 6. Controls returned to normal.

I have rewired the test stand (service cap wiring only, not power supply wiring) to anode bias mode as shown in the diagram. This was not how it was wired before, there are several things I have removed that appear to be newer additions. I also removed the burned cable between TBHT(-) and COIL(-), and between the multipin feedthrough and ground (probably related to the other burned cable).

I tried to measure the resistances across the feedthroughs (as opposed to across the resistors themselves) but they all seem wrong. I get 27Ohms across the TGHT and TBHT circuits instead of 100Ohms. Between the TGHT and TBHT I get 100Ohms as expected. Across the coil circuit I get 6Ohms instead of 100Ohms. Probably only the 6Ohms really needs to be fixed, but if the resistors are damaged, then the varistors might be as well.

Fernando Maldonado wrote:

Yesterday, December 1st, I successfully installed and tested the gas system inside the faraday cage. The changes performed were as follows: Remove He gas tank (to make room for new tank) Remove calibrated leak to connect new gas line Remove convectron gauge to install a high range dial manometer.   The tests were done manually and consisted as follows: Gas lines roughed by opening SV1. Tank regulator set to deliver 1 bar SV4 opened to allow the gas to enter the lines Dial manometer verified to  match the pressure set by the regulator Alternatively roughed and pressurized the lines by opening/closing SV1/SV4 Tank lines emptied with SV1 and SV4 open simultaneously.   The test involve the lines up to SV5. The gas line portion from SV5 to the top of the HV cage was not included as it was removed for TM3 HV tests (se entry #822) When the HV tests are concluded I will install a new line that will include a gauge.

 Yesterday, January 13th I added back the gas line from SV5 to the service cap.

I temporarily added a gauge in between to measure the gas pressure. After successful tests I removed it from this location.

Finally, I added the gauge on the service cap, which its desired location (see attached picture). The tests corroborate no pressure losses from the gas bottle up to this location.

Fernando Maldonado wrote:

Wednesday, October 30, 2019, 14:49: John C. changed the polarity of the anode power supply to negative, and Fernando M. rewired the cabling on the service cap.

 Wednesday, January 20, 2021, 11:26: John Chow verified that the power supply was in negative mode.

Just as side note, I think the PS was in negative mode since the quoted entry from October 30 2019. Given that the SiC/FEBIAD was supposed to run in cathode mode, the test stand was rewired to this. The only rewiring after this was for the P2N.

On Tuesday the hard good vacuum level was changed from 7e-6 T to 1e-5 T to allow larger leak rates when testing FEBIAD.

Also this new level should allow faster heating rates (be mindful and cautious)

Work done as per WP C2021-01-19-9