An annual electrical and mechanical inspection was performed on the SMP today by Adam Newsome, Travis Cave, Eric Lahe, Maico Dalla Valle, Antoniette Yap, and Jason Zhang as per Work Permit I2023-06-21-4. The following items were checked:

• Check condition of wiring for physical/radiation/UV damage
  • OK, no signs of significant damage
• Check for cable tray debris or damage
  • There were multiple items in the cable tray which do not belong there - misc. pieces of wood and metal... these items were not disposed of but should be in the near future
• Check inside control panel: components and wiring, labeling
  • Labeling intact, components in good condition
  • Misc. random wires pulled to check... one ground wire was loose - it was tightened
• Verify camera views
  • This was not inspected in detail but signal status was checked, and is checked prior to remote operation
• Check connectors for damage
  • All connectors OK
• Check pendant for damage and verify labeling intact
  • Pendant OK
• Lid open/close test
  • Lid was opened and close multiple times - smooth operation, no concern
  • Lid close logic was checked - OK
• Vessel rotation and limit switch check
  • Both CW and CCW limits were checked, functioning
  • No concern on rotation - chain drive, rotary limit switch, and cable reel all functioning normally
• Lubricate chains
  • Not done - should be done next year
• Inspect tensioners
  • Done - OK
• Tighten fasteners
  • Not all fasteners were checked - washers will be replaced soon so this was not deemed necessary
  • Approximately 50 washers are missing for socket head screws in slotted parts. Not safety critical but they should be replaced soon.
• Visual inspection
  • All OK

Overall the inspection was successful.

Bearings for the main storage vault door were lubricated on 2023-07-28. It is unknown when this was last done. It is suggested that this be a recurring maintenance task every two years.

The bearings on the mini storage vault were lubricated. It is suspected that this has never been done. It is suggested that this be performed every two years.

A draft procedure will be created for this job, but at a high level, this process involves:

• Opening the door fully
• Disconnection of the motor, limit switches, and cable management system
• Attachment of a lifting magnet to the door itself
• Raising the door
• Lubricating bearings
• Reversing the process to re-assemble

The entire job took less than 1 hour to complete.

In an effort to clean up the camera installation on top of the ISAC target hall crane to resolve a fault, the position measuring tape and its associated camera were removed from the target hall crane on July 31, 2023.

Still to do: clean up power bar installation.

 The cleanup of the ISAC Target Hall crane and installation of the new SICK laser which replaced the IBEO laser for the EW crane position was completed on Aug 03 2023.

The clean up tasks which were completed: (Removal of tape viewing camera, Removal of tape, Installation of extension cord, Consolidation of power supplies for the lasers and removal of old laser) 

Currently power bar is still present to act as an extension cable. This will be removed when 2m extension cord is made and put in next target exchange cycle.
   

The installation and testing of new SICK Laser:

1. Calibration points were taken at the 4 corner limits of the crane. North South and East West positions were recorded.

2. Crane was parked at the ladder access position and mounting bracket was machined and installed.

3. A new cable was made and installed for the SICK laser. Continuity test was performed from the control room and crane to ensure pinout of the laser worked with existing serial communication setup in the control room PC.

4. The feedback values from the new SICK laser and old IBEO laser were compared to determine that a calibration offset of 0.06m was needed in the Laser Position Feedback program.

5. Laser Position Feedback program was re-compiled to communicate with 2x SICK lasers with the calibration offset value included in the EW feedback position. (Laser Crane PositionDlg.cpp)

6. Calibration points were taken at the 4 corner limits of the crane again with the new Laser. North South and East West positions were recorded.

There was less than 0.1% variance between the new laser and old laser which is well within the tolerance of the lasers themselves. It can be determined that it was properly calibrated.

HALCW system O2 sensor service and calibration were done in Sep 26th. electrolyte and membrane on rose mount sensor 499ADO were refilled and replaced. Sensor was zero and full scale calibrated. The sensor works fine after the job.

ITH:MP1 is replaced this morning due to its poor performance started yesterday afternoon. The general field in pit is about 260usv/hr. the contact field on pump is 2.0 msv/hr. New pump works good. Lubricant in pump will be drained next week. then motor will be removed. The pump body will be bagged and stored in silo area.

 All 20 targets in the vault were measured. The target spread sheet updated. See attached PDF for details.

The following four pails were moved from the main storage vault to the mini storage vault, to stage for shipping in Jan. 2024:

1. Pail 260 (SiC#41), Tray 5B
2. Pail 271 (SiC#42), Tray 2B
3. Pail 246 (Nb#10), Tray 6C
4. Pail 265 (UCx#36), Tray 2A

There is current one open slot in the mini storage vault which is planned for the Ta#58 target (formerly in Pail 211, to be repacked in Dec. 2023).

Target Index spreadsheet updated.

See attached printout with mini storage status.

During operation today, it was observed that the main vault door had issues opening.. typically when the door was around 80-90% opened, it would stop motion. Note that this issue has allegedly been present since early 2023. It appeared that the motor was still spinning but the clutch was slipping. This issue was consistent and observed multiple times when trays 2, 5, and 6 were engaged (note these are all on the same side of the vault). It was overcome by backing the door up slightly then trying again. The door close function worked with no issue.

It is suspected that there is a problem with the clutch or potentially a piece of debris on the track. [no longer true after 2023-11-17 inspection]

Edit [2023-11-17]: Maico Dalla Valle and Adam Newsome inspected the vault on 2023-11-17. It was very clear that the door, when opening, is rubbing up against the rails. Specifically, the side rollers located on each side of the vault (red housing). There are clear wear marks on each side. The skew is obvious because on one side, the wear marks begin immediately and stop halfway through travel, whereas the other side shows the reverse wear pattern. It is therefore suspected that the skew has become so significant that the friction is too high towards the end of travel which causes the clutch to slip. The travel rails were cleaned (significant grime buildup) and the offending side was filed down slightly (a sharp burr was present from the wear). It is not advised to adjust the clutch. Next steps: open and close the vault with no trays engaged to see if the issue persists (perhaps weight distribution was the issue), and then try different combinations of weight distribution to try to replicate the issue. After this, measure separation distance between guide rails at different points and try to identify the source of the skew issue and rectify it.
 

Edit [2023-11-20]: Maico and Adam performed further investigation by opening and closing the vault door under 3 different configurations: no trays engaged, 2 trays engaged on West side, 2 trays engaged on East side. The previously observed stopping issue was not observed this time under multiple cycles with each configuration. It is noted that the door appears to slow down slightly and experience a bit of a "chugging along" motion towards the end of travel when opening. Based on these observations, it is concluded that the cleaning and filing helped enough to allow the door to fully travel open/close without getting stuck. The root cause of the issue still remains to be fixed, but for now there is no major concern. Before end of 2023, measurements and planning will be performed to develop a proper repair for the root cause. It is expected that either late 2023 or early 2024, the repair will be attempted.

Edit [2023-11-22]: during a routine spent target exchange, the issue was not observed when opening tray number 2. 

Edit [2023-11-24]: a dial indicator was set up on the vault on the right side (looking down from above), mounted on the traveling part with the tip touching the black structural member where the wear is observed. The purpose is to determine how much out of alignment the door is with respect to the structural assembly. The following was observed: approximately halfway through the travel, the indicator read approximately 185 thou of travel. This corresponds to when the wear marks start to occur and where the small red enclosure around the roller wheel interferes with the structural member (causing the wear strip). At this point, as expected, the indicator no longer changes because the roller wheel has bumped up against the structural member. This implies the gap between the structural member and the roller wheel, when the door is closed, is approx. 185 thou on the right side (and 0 on the left side). These values reverse around the midpoint of travel. It is suspected that because of this, and because there is a small observable gap between the door and the vault when it is fully close, the right and left side of the door are not parallel with respect to each other. In other words, adjusting the angle so that the starting gap is roughly 92.5 thou on each side should resolve the issue. Measurements to confirm parallel-ness between each side will be performed, and then an adjustment will be made to create even gaps between each side at the starting point. After this, a dial indicator will be set up again to confirm that, when opening the door, the gap is maintained throughout the duration of travel. Small deviations are acceptable, but in general what should be observed is a roughly constant gap, and no interference (i.e. if the gap begins at 92.5 thou, it should not drop to 0 or not exceed 185 thou, but should hover somewhere on the order of 20 to 140 thou, for example).

 

Edit [2023-11-24, 2]: a dial indicator was set up on the left side of the vault in a similar manner as described above. In this case, the indicator's reading remained roughly constant during the start of travel (starting with the door closed and opening it up), with only up to around 10-20 thou variation at first slowly, followed by a slow increase to up to approximately 135 thou. This is similar to the 185 thou observed on the other side. One suggested means of resolving the issue is: raise the entire door slightly with overhead crane, translate it sideways approximately half of door's lateral travel distance (i.e. around 90 thou) and then lowering it back down. This should, in theory, be acceptable. It may not resolve the issue of whether the door is rotated, but should give enough clearance on both sides throughout the entire range of motion to be deemed an acceptable fix. Other opinions will be gathered on the subject to determine if this could work. 

 3 light bulbs replaced in the ISAC South Hot Cell - North West light, South west light, and South East light bulbs. Northeast light currently still working.

The South East light is not working after bulb replacement. This bulb was tested using the wires from the Northeast light station and the bulb is working properly. Electrical services is being contacted to troubleshoot the cable issues. 

It is reported that the crane's hook rotation is not working currently. It was working yesterday (Nov 22). Investigation to follow, and this e-log will be updated.

Edit [2023-11-23]: the issue was a result of a tripped circuit breaker. Perhaps the motor became overloaded. It is advised to monitor the rotation mechanism over the new few operations, and rotate the hook CW and CCW prior to performing any critical operation to confirm it is working.

Broken tong cable on SW manipulator (serial #8106) of SHC was replaced today.

Following ISAC Hot Cell entry procedure, Document 157554, SHC was entered and wrist and tong cable were replaced with a pre-run/assembled wrist, cable and tong. It is standard to try and pre-make/load a new cable into a spare tong assembly to reduce time spent in the HC. This marks the first time a wrist has been replaced. Rationale for replacing the wrist at this time is that internal spacer A-14118 which the ting cable rides over has had grooves worn into it. This is a result of regular wear and tear but could be causing premature wear on the tong cable. Replacement is a preventative measure. This is the original wrist and spacer from the original install in 1998. The wrist that was removed will be rebuilt and swapped out with the original one on the SE manipulator (serial #7986) at a later date.

Parts replaced today are: New - Bottom half of wrist joint (CRL-13950), New - Tong cable (CRL-14230) and used tong assembly (CRL-18062)

Procedure took about 40 minutes. Dose received was ~0.046 uSv

TM3 was moved from the Silo to South Hot Cell.

Pumped down the gas lines in ITE on Tuesday Nov. 28 2023 and opened the CF4 bottle valves. Missed the small section between the regulator and SV2 though, so will probably redo it.

Pumped down again on Dec. 12 2023. Was taking a long time so might not be as good as earlier. Maybe there is a leak in there...

TM3 has been moved from the target conditioning station to the south hot cell.

TM3 was moved from the SHC to the TCS (no target installed).

Tuesday, January 02, 2024, 15:58 :

Started HV ramp up on TM3 service tray about 3pm today. Had to bypass TCS:BIAS interlocks to make it work - there are some flow restrictions on the water lines that are causing the flow interlock to periodically trigger. The water flow it not very important for this test as long as the water is moving. Turned out that it was also conveninent to have all the interlocks bypassed because sparks tripped off the IG1 gauge and this did not cause the HV bias supply to turn off. Should use same setup next time.

Some sparking around 35-44kV, tripped off turbos/IG1/IG1S 3 times and so decided to ramp down.

Tried again while standing outside TH door to hear if sparking is outside the vacuum. Went to 48kV very slowy with multiple trips of the turbos, mostly TP2. Lots of sparks but easy to ramp back up to previous high point, so likely just a painful conditioning process. Current draw 167uA at 48kV. No sparks audible outside door, and vacuum response looks too large to be electrical noise.

Continued on Wednesday January 03 - went up to 53kV in the same way as before, lots of TP2 trips making its current a bit unstable... data was dumped as TM3_servicetray_conditioning_jan32023. Tried limiting the current on the power supply to very near the drain current to see if that reduces the number of trips. Didn't notice a significant effect.

Thursday, January 04, 2024, 09:57 :

Restarting conditioning, starting at 180uA at 53kV. Got up to around 60kV with many trips of the turbos, and one restart of the IG1 controller needed. Added a grounding hook to the vacuum controller rack to see if that helps reduce the trips. (doesnt appear to have).

Friday, January 05, 2024, 09:51 :

Continuing conditioning. Starting by sitting at 50kV for 10mins. No sparks observed during this time.

Even though we got to 60kV last time, onset of conditioning started at 58.6kV with a trip at 58.9kV... got up to 70kV.

Got to 72kV on Monday January 08 2024.

On Monday Jan. 15, tried to improve the grounding on the vacuum services rack to reduce sparking through the turbo controllers. Added two welding cables from the vacuum rack ground busbar to the copper grounding strip on the ground. If the spark is travelling from the faraday cage to the rack through conduit or grounding cables, maybe this will help. If it is coming through the turbo cable then it likely won't. Tried to add grounding straps on the turbo controller chassis (there is a brass screw which I think is for this purpose) but couldn't get either of the screws out with any resonable force. They undo a bit then get stuck...

 Wednesday, January 17, 2024, 20:39:

David vented TM3 with N, then opened the service cap to install new insulators. We pumped back down and Alexander and I ramped back up today - instabilities started around 55kV and we were back to slow conditioning in the high 50s. Got up to a set point of 65kV. Ray did some work on the TCS bias supply and now neither the set point or readback is correct. The PSU was reading 51.5kV at set point 50kV and 60kV at set point 58kV. Will continue later...

Wednesday, January 31, 2024, 16:23

Ramped up again, some slight instabiities around 56kV but could still go up pretty quick until about 65kV. Since Ray's testing of the TCS:BIAS supply both the readback and the set point are off from the powersupply display. Ran out of time and ramped down at set point 70.5.

Tuesday, February 06, 2024, 14:47

Testing again after venting the module with air. First tiny spark at 48, then some vacuum reactions starting at 49 but still no issues. Bit more reactive around 53.

 stopped at 69.4

Wednesday, February 07, 2024, 15:28 :

Ramped easily to 60kV set point and let it sit there. Didn't spark at all then after 100mins, sparked TP2 off. Otherwise first sparking happened at set point 67.

Found at limit at around 72.5 on the PSU front output, set point 70.6. think it comes from EPICS. Will have to test the power supply.... stopped there.

Thursday, February 08, 2024, 15:03 :

Long term testing. ramped back up, first spark at set point 65.

Let it sit for 2hrs 20mins. Something like 6 sparks during this time. Current also dropped significantly from 0.32A to 0.27A.

EDIT 2024-11-25, Alexander Shkuratoff:

All the StripTool data related to the above tests are on the server as the following below. Those without "Alexander" in the name were dumped by Carla.

	TM3_servicetray_conditioning_Jan32023	2024-01-03 17:29	807K
	20240103_TM3_Service_Tray_TCS_Alexander	2024-01-03 17:31	1.2M
	20240104_TM3_Service_Tray_TCS_Alexander	2024-01-04 16:58	238K
	20240104_TM3_Service_Tray_TCS_Alexander_2.csv	2024-01-05 07:00	13M
	20240104_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 12:21	81
	20240105_TM3_Service_Tray_TCS_Alexander.csv	2024-01-05 16:51	13M
	20240105_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-05 16:52	2.7M
	TM3_servicetray_conditioning_Jan52023	2024-01-05 21:24	8.8M
	20240108_TM3_Service_Tray_TCS_Alexander_1sec.csv	2024-01-08 16:56	4.7M
	20240108_TM3_Service_Tray_TCS_Alexander.csv	2024-01-08 16:56	7.1M

	TM3_servicetray_conditioning_Jan172024	2024-01-17 13:29	257K
	20240117_TM3_Service_Tray_TCS_Alexander.csv	2024-01-17 20:11	4.0M

	TM3_servicetray_conditioning_Jan312024	2024-01-31 17:24	2.2M
	20240131TM3_Service_Tray_TCS_Alexander.csv	2024-01-31 17:55	4.5M
	20240206_TM3_Service_Tray_TCS_Alexander.csv	2024-02-06 17:58	2.6M
	TM3_servicetray_conditioning_Feb62024	2024-02-06 19:29	4.6M
	TM3_servicetray_conditioning_Feb82024	2024-02-08 17:55	3.1M

 And the StripTool settings:

Alexander_TCS_HV_Conditioning.stp

2024-01-17 19:57

3.7K

TM2 was moved from ITW to the South Hot Cell with Ta#66 target.