Friedhelm Ames commented to me that there appears to be a short between High Voltage and Ground at TM3 at ITE.

Friedhelm, Anders and I initiated a survey by Andrew from RPG, and entered the Target Hall at approximately 5pm to visually assess the HV cover at the MAA.  There was no clear indication of a problem.  We opted to wait for David to return as we weren't sure exactly what we were looking for.

Friedhelm and Anders confirmed in the Electrical Room that the short is in the Target Hall.

 The HV enclosure and module connections were inspected by David and Anders on Wednesday 19th in an attempt to find any HV problems causing the high leakage current. One cable was cable tied to a copper conductor in the chase. Some small pieces of loose material were removed. No obvious problems could be detected.

Interlock key  are returned back to B2 level.TM2 is ready for HV conditioning and heating up at TCS.

ISAC operators attempted HV conditioning over the weekend.

https://web.accel.triumf.ca/isac_elog/Elog_show_result.pl?inid=218447&style=entry

 It was attempted to HV condition TM2 to 55kV and leave it operating there for an hour to demonstrate stability. Unfortunately at 52.65kV a spark destroyed the canbus controller. This controller had shown issues in the past but seemed to magically fixed itself. I will contact Hubert for another card. Attached  is is a screen shot of the stip tools. 

 The HV conditioning reached a maximum of 44kV today before a spark took out the canbus card. I will attach the data once I have had a chance to format it. 

 The first attempt at HV conditioning of TM3 with the new source tray and NiO#2 target installed. Conditioning so far has been very slow, with the current increasing quickly to a maximum of 400uA. Voltage in these cases was lowered until the current was below 200uA. Attached is the raw data. It appears the source tray and target are conditioning just there is either something burning off or there is localized poor vacuum.

Bevan Moss wrote:

The first attempt at HV conditioning of TM3 with the new source tray and NiO#2 target installed. Conditioning so far has been very slow, with the current increasing quickly to a maximum of 400uA. Voltage in these cases was lowered until the current was below 200uA. Attached is the raw data. It appears the source tray and target are conditioning just there is either something burning off or there is localized poor vacuum.

 Attached screen shot. Vacuum was around 8.3 X 10^-6 on IG2 (which is in the diagnostics box)

 HV conditioning of TM2 continued today. At ~53.5kV the module started to spark externally to the HV cover. This is the first time the cover has been tested to this high of a voltage due to the limitations of the modules. As such for the conditioning station to achieve 60kV while keeping the target hall open of personnel the cover and cage will need to be modified. To continue with HV testing of TM2 the cover was removed and the target hall locked out as per the procedure covered in Document 70039 "ISAC Conditioning Station: First Lock-Out and High-Voltage "ON". With the cover off the module was able to achieve ~56.7kV without the Einzel lens on. Eliminate the Einzel lens as a potential source of sparking it was brought to 25kV while the module continued conditioning. Maximum voltage achieve under this configuration was 57.1kV. Increments are now 50V steps. The Canbus continues to be very sensitive to any sparking and is slowing the process down considerably. It does however appear that the module is continuing to condition. I will continue tomorrow after the module moves to attempt to reach 60kV and hold it for 1 hour.

 TM4 with Ta#42 was HV conditioned to 35kV with the optics tray at 17.5kV. There was some vacuum response while ramping but well within normal. Attached is a screen shot and the associated data.

 The TiC#4 target that is on TM3 was conditioned to 25kV today with the EINZEL lens at 12.5kV. There was one spark shortly after reaching 25kV that reset the bias PS and another one  after the module sat at 25kV for ~30 minutes. Attached is the strip tool data and screen shot. Prior to starting the conditioning the power on the chiller needed to by cycled as the VFD on the fans had tripped due to a recurring inrush problem.

 High voltage testing was started on TM2 with no source tray today. The maximum stable for 1 minute voltage reached was 47.5kV but the module is still conditioning. With every step up in voltage the current goes up a large amount and slowly bleeds down. Large sparks disrupts the connection of the CANBUS to the PS which is irritating but the module can be brought up to the equivalent voltage in less than 30 seconds. Attached is a screen shot of most of the strip tool. I have also attached all of the dumped data. The module will be left under vacuum with the water running over the weekend and HV testing will continue on Monday. 

 The Glory of TM2 continues 100V at a time. Today I brought the module to 46kV and left it there for 1 hour. During this time there was only 1 spark. I then continues at up to 53kV. At this voltage the canbus continues to be temperamental when the module sparks. The module is still conditioning and I will continue more tomorrow. Attached is the dumped data and a screen shot of the conditioning from 46kV to 53kV.

Tuesday, September 12, 2016, 10:00: RGA on

Tuesday, September 13, 2016, 13:52: Source on

Tuesday, September 13, 2016, 13:55: BIAS / EL at 10 / 5 kV

Tuesday, September 13, 2016, 13:59: BIAS / EL at 18 / 9 kV

Tuesday, September 13, 2016, 14:06: BIAS / EL at 20 / 10 kV. Minor sparking

Tuesday, September 13, 2016, 14:25: Sparking at about 1 spark/min for the last 10 min

Tuesday, September 13, 2016, 15:38: ~10 sparks for the last 60 min.

Tuesday, September 13, 2016, 15:41: BIAS / EL 22 / 11 kV

Tuesday, September 13, 2016, 15:57: Heavy sparking at 24 kV. Lowered to 21 kV. EL at 12.5 kV

Tuesday, September 13, 2016, 16:00: BIAS / EL at 22 / 12.5 kV

Tuesday, September 13, 2016, 16:05: Trip on resistivity caused by spark

Tuesday, September 13, 2016, 16:07: BIAS / EL at 10 / 5 kV

Tuesday, September 13, 2016, 16:14: BIAS / EL at 20 / 10 kV

Tuesday, September 13, 2016, 16:25: BIAS / EL at 22 / 11 kV

Tuesday, September 13, 2016, 17:00: Sparking at about 1 spark/min for the last 30 min

Tuesday, September 13, 2016, 17:02: Source off.

Wednesday, September 14, 2016, 08:22: BIAS / EL at 20 / 10 kV

Wednesday, September 14, 2016, 08:33: BIAS / EL at 22 / 11 kV

Wednesday, September 14, 2016, 08:35: Trip on resistivity caused by spark

Wednesday, September 14, 2016, 08:38: BIAS / EL at 20 / 10 kV

Wednesday, September 14, 2016, 08:41: BIAS / EL at 21.5 / 11 kV

Wednesday, September 14, 2016, 08:45: BIAS / EL at 22 / 11 kV

Wednesday, September 14, 2016, 09:06: Sparking at about 1 spark/min for the last 20 min

Wednesday, September 14, 2016, 10:17: Sparking rate somewhat reduced. BIAS / EL at 23 / 11 kV

Wednesday, September 14, 2016, 10:23: BIAS / EL at 24 / 12 kV

Wednesday, September 14, 2016, 10:31: BIAS / EL at 25 / 12 kV.

Wednesday, September 14, 2016, 10:32: Trip on resistivity caused by spark

Wednesday, September 14, 2016, 10:36: BIAS / EL at 20 / 10 kV

Wednesday, September 14, 2016, 10:40: BIAS / EL at 23 / 12 kV

Wednesday, September 14, 2016, 10:51: BIAS / EL at 24 / 12 kV

Wednesday, September 14, 2016, 10:57: BIAS / EL at 25 / 12 kV

Wednesday, September 14, 2016, 11:01: Heavy sparking at 25 kV (>15 /min). Trip on resistivity caused by spark

Wednesday, September 14, 2016, 11:10: RGA off.

Water cooling is performed manually with CP1. RGA is ON since yesterday. EE is not connected and in conditioning mode.

Friday, August 12, 2016, 10:43: BIAS at 10 kV, EL at 0.5 kV. IG1 at 1.8E-6 T

 Friday, August 12, 2016, 11:04: BIAS at 20 kV

Friday, August 12, 2016, 11:10: Sparking at 21 kV

Friday, August 12, 2016, 11:29: BIAS at 22 kV. Many small sparks/current excursions.

Friday, August 12, 2016, 11:32 : BIAS at 23 kV

Friday, August 12, 2016, 11:37: BIAS at 24 kV

Friday, August 12, 2016, 11:43: BIAS at 25 kV. Heavy sparking. Lowered to 24 kV to reduce spark rate.

Friday, August 12, 2016, 11:52: Increased sparking rate. Lowered to 23 kV.

Friday, August 12, 2016, 12:05: Lowered to 20 kV

Friday, August 12, 2016, 12:21: Back to 23 kV

Friday, August 12, 2016, 12:28: Heavy sparking between 23 and 24 kV. Lowered to 22 kV

Friday, August 12, 2016, 12:29: More sparking. Lowered to 21 kV.

Friday, August 12, 2016, 13:02: Increased back up to ~25 kV while Aurelia was listening for sparks in the Target Hall.

Friday, August 12, 2016, 13:03: EL to 10 kV

Friday, August 12, 2016, 13:06: BIAS / EL to 23 / 12.5 kV

Friday, August 12, 2016, 13:34: Repeated sparking at 24.5 kV. Ramping down

Friday, August 12, 2016, 15:33: Stopped water package. RGA off

 A HV test was conducted yesterday on TM3. There was no target and all items were tied to common or ground. The conductivity of the water ranged from 7.1 to 7.5 Mohm. 40kV @ ~600uA without continuous sparking was achieved. The Strip tool was run but something went wrong with the export and the vacuum gauge data is missing. I have attached the file for reference and the pulled data from the achiver (only logs every 300 seconds). RGA scans were taken periodically during the test nothing unexpected was measured but we also do not know what normal looks like. The RGA application is available at http://www.thinksrs.com/downloads/soft.htm. Other items of note include significant sparks disrupts the communication between the computer and the RGA, the PNG1 gauge and its controller, and TP2 local readout screen. 

Bevan Moss wrote:

A HV test was conducted yesterday on TM3. There was no target and all items were tied to common or ground. The conductivity of the water ranged from 7.1 to 7.5 Mohm. 40kV @ ~600uA without continuous sparking was achieved. The Strip tool was run but something went wrong with the export and the vacuum gauge data is missing. I have attached the file for reference and the pulled data from the achiver (only logs every 300 seconds). RGA scans were taken periodically during the test nothing unexpected was measured but we also do not know what normal looks like. The RGA application is available at http://www.thinksrs.com/downloads/soft.htm. Other items of note include significant sparks disrupts the communication between the computer and the RGA, the PNG1 gauge and its controller, and TP2 local readout screen.

 Data has been put into and excel spread sheet

 Monday, July 25, 2016, 08:14: Enabled staging for chiller

Monday, July 25, 2016, 08:33: Disabled staging. Fault 2. Staging does not work at this point.

Monday, July 25, 2016, 08:34: Running CP1 manually to cool down water.

Monday, July 25, 2016, 09:15: Bias current limiting at 150 V. Investigating short.

Monday, July 25, 2016, 09:51: EE cable was shorting the BIAS on the TCS side.

Monday, July 25, 2016, 09:57: BIAS/ / EL at 20 / 10 kV

Monday, July 25, 2016, 10:08: Sparking at 24 kV. Lowered to 22 kV

Monday, July 25, 2016, 10:44: BIAS at 25 kV

 Monday, July 25, 2016, 11:00: Sparking at 25.5 kV. Lowered to 25 kV

Monday, July 25, 2016, 12:03: 25.5 kV. No sparking for 10 min. Lower to 20 kV for lunch break.

Monday, July 25, 2016, 12:51: 26 kV

Monday, July 25, 2016, 13:00: Sparking at 27.3 kV. Lowered to 27 kV. EL at 14 kV

Monday, July 25, 2016, 13:04: More sparking. Lowered to 25 kV

Monday, July 25, 2016, 13:39: 25 kV

Monday, July 25, 2016, 14:21: A spark tripped the PS on water resistivity

Monday, July 25, 2016, 14:31: 25 kV

Monday, July 25, 2016, 15:34: ~20 sparks for the last 60 min

Monday, July 25, 2016, 16:28: ~10 sparks for the last 60 min. BIAS an EL off

Monday, July 25, 2016, 16:36: Stopped water cooling system.

Tuesday, July 26, 2016, 08:20: Started water cooling system.

Tuesday, July 26, 2016, 08:24: Ran CP1 manually for 3 min. Enabled staging.

Tuesday, July 26, 2016, 08:31: BIAS / EL at 25 / 0 kV

Tuesday, July 26, 2016, 08:39: EL float to 300 V. Set to 500 V

Tuesday, July 26, 2016, 08:45: BIAS to 25.5 kV

Tuesday, July 26, 2016, 09:07: 26 kV

Tuesday, July 26, 2016, 09:12: Lost contact with EPICS. Scheduled maintenance

Tuesday, July 26, 2016, 10:26: 6 sparks for the last 60 min

Tuesday, July 26, 2016, 10:33: BIAS to 27 kV

Tuesday, July 26, 2016, 10:43: Sparking at 27.8 kV. Lowered to 23 kV

Tuesday, July 26, 2016, 10:51: BIAS back at 27 kV. A spark tripped the BIAS on water resistivity

Tuesday, July 26, 2016, 11:08: BIAS at 27 kV

Tuesday, July 26, 2016, 11:22: BIAS at 28 kV

Tuesday, July 26, 2016, 11:42: BIAS at 27 kV after trip due to water resistivity (spark)

Tuesday, July 26, 2016, 12:00: BIAS / EL off for lunch break

Tuesday, July 26, 2016, 14:10: EL to 10 kV

Tuesday, July 26, 2016, 14:14: El to 20 kV

Tuesday, July 26, 2016, 14:20: EL to 25 kV

Tuesday, July 26, 2016, 14:27: First spark at 27.6 kV

Tuesday, July 26, 2016, 14:29: EL to 28 kV (BIAS at 0 kV)

Tuesday, July 26, 2016, 14:43: RGA on

Tuesday, July 26, 2016, 14:49: No sparking for 20 min

Tuesday, July 26, 2016, 16:02: No sparks for 90 min.

Tuesday, July 26, 2016, 16:06: PS and water off.

Wednesday, July 27, 2016, 09:14: BIAS at 24 kV

Wednesday, July 27, 2016, 09:19: Lowered to 20 kV due to sparking

Wednesday, July 27, 2016, 09:28: BIAS at 22 kV

Wednesday, July 27, 2016, 15:30: BIAS and water off

Thursday, July 28, 2016, 08:43: RGA off

 HV was brought up to 70 kV for 1 h without sparks to test the new water installation.

While under vacuum Travis and I completed diagnostics of the HV chase. Majority of the lines showed OL at 1kV but there were the following "failures"

A to P @ 250V 716kOhm
A to Q @ 1kV 25.18MOhm
B to C with ohm meter 8Mohm @250V 830kOhm
B to D @ 1kV 15.57MOhm
C to D @1kV 16.18 MOhm
P to Q @1kV 1.069 GOhm
G to J with ohm meter dead short.

Ideally all of these values would be OL and they may improve or get worse when the source tray is installed depending on where the electrical contact is. I need to discuss with Friedhelm but I believe that as these values the module can still operate. Without changing the HV chase there is very little chance of improving these values.