Yesterday Grant Minor (me), Maico Dalla Valle, Dan Louie, and Arthur Leung worked with Operations to perform the Cyclotron Lid-Up task for the 2014 winter maintenance.

Work Permit 2014-01-22-2 was issued for this job.

In general, the lid up operation was rough at the beginning.  The system began to trip-out within about 500 counts on the mechanical counter, just when the jacks started taking the load, then continued to trip within the first ~ 3 inches of travel.  Multiple station jogs, complete lowerings, and complete raisings were required to get beyond the rough spot. 

I tried as best as possible to record the mechanical counters for each trip event, as well as confirm the master synchro count and culprit station with Arthur.  The data I collected is summarized below:

Start of Operation: All stations were indexed with yoke gaps within .005" and .010" of the .250" nominal shim stock (see previous e-logs).  Mechanical counters were all set to 100 counts (each count is approximately .001").

1. First lid up attempt tripped at less than 1" of travel.  Station 9 was found to be lagging adjacent stations.  It as jogged in local mode as shown:

2. Lid was raised again and stalled.  Station 11 was found to be lagging and was jogged:

3. Lid was raised again and stalled.  The decision was made to lower completely and raise again.  All station counters were recorded before lowering:

4. Lid stalled on the way down.  Station 9 was found to be lagging and was jogged:

5. Lid was lowered completely and then raised again.  It tripped at Station 9.  It was examined and found not to be lagging a great deal.

6. System was reset and lid was raised further.  It continued to trip.  Station 9 was examined and not found to be out.

7. Lid was lowered completely and then raised again.  It tripped again, and Station 9 was found to be out.  It was jogged with some intentional overshoot.

8. Lid was raised again and tripped.  The decision was made to lower lid completely.  Some counter values were recorded when the lid had reached the lower limit, for comparison to the starting reference.

9. Lid was raised again and traveled more smoothly until tripping at around 850 counts.  6 counter values were recorded.  Station 9 was found to be lagging and was jogged:

10. Lid was raised again and tripped.  Station 9 was found to be lagging and was jogged:

11. Lid was raised again and tripped.  Station 9 was found to lag very slightly (but within the normal variation band), and was jogged:

12. Lid was raised further and tripped again.  Four of the counters recorded and the group broke for lunch (approximately 1 hr).  The lid was left at this position over lunch.

13. After lunch, the lid was raised up to about 1090 counts with 4 trips.  Counters were read as below:

14.  Lid was raised but tripped almost immediately.  Decision was made to lower again.  Counters at bottom limit were read again:

15. Lid was raised with 3 trips.  Station 9 was found to be lagging and was jogged:

16. Lid was lowered completely, then raised.  It went up fairly smoothly and tripped out at about 1350 counts.  All counters were recorded:

17.  A few more attempts were made to raise the lid with a few more trips.  The system traveled in longer bursts and more smoothly between trips.  Counters were recorded and stations 9 and 11 were jogged:

18. System was reset and lid was raised again with a few more trips close together.  The decision was made to lower the lid completely and raise again.  Master synchro was recorded at 2285 before lowering.

19. Lid was lowered completely and raised fairly smoothly up to a master count of 2363 before tripping out on stations 1 & 9.

20. A few attempts were made to lower the lid slightly (not all the way) and then raise again.  This did not seem to reduce the frequency of trips.  The system tripped out around 2820 and all counters were recorded before taking a short break to allow the contactors to cool down:

21. After the break, there was a raise and lower cycle within about 3600 counts that I was not present for.  When I had returned the system was still rising in small bursts and tripping out on station 9 mostly.  Over a span of 5 trips, Station 9 was jogged 5 times by between 20 and 50 counts, with 20-count overshoot.

22. The master synchro passed 3600 counts and then the system tripped.  Stations 9 and 10 were recorded and Station 9 was jogged:

23.  The master passed 3800 counts, and then they system went up smoothly for the first time by about 1500 counts before stopping. Some counters were recorded:

24.  Lid was raised again and stopped around 6350.  The group paused to check the tank seal and all counters were recorded:

25. The lid was raised further with 2 trips at station 9.  Stations 9 and 10 were checked and station 9 was jogged:

26. Lid was raised again with 2 trips initially and then smooth travel up to a Master count of 9600.  After 3 more trips and resets, I began recording just the station trips and master counts at each trip.  No jogs were required during this period.

27. Jacks 1 & 5 tripped at 10,200 counts on the master.

28. Jacks 1 & 9 tripped at 11,500 counts on the master.

29. Jack 9 tripped at 13,400 counts on the master.  Several trips occurred immediately after this.  Counter readings were taken and station 1 was jogged:

30. Lid was raised again with trips at station 1, 5, 7 & 9.  Master synchro count was 15,600.

31. Lid was raised again with trip at stations 1 & 7.  Master synchro count was 18,800.

32. Lid was raised again with trips at stations 5, 7, and 9.  Master synchro count was 21,650.

33. Lid was raised again with trips at stations 1, 3, 5, 7, 9, 11.  Master synchro count was 23,360.

34. Lid was raised again with trips at stations 1, 5, 7, 9. Master synchro count was 23,700.

35. Lid was raised again with trip at station 9.  Master synchro count was 25,420.

36. Lid was raised again with trip at station 7. Master synchro count was 27,700.

37. Lid was raised again with trip at station 5.  Master count was 27,700.

38. Lid was raised again with trip at station 7. Master count was 30,300.

39. Lid was raised again with trip at stations 1 & 3. Master count was 33,650.

40. Lid was raised with trip at station 7.  Master count was 37,400.

41.  A moaning / groaning noise was heard from the structure around 38,000 counts.  This noise was also observed during the 2013 lid-up.

42. Lid was raised with trip at station 7. Master count was 39,722.

43. Lid was raised with trip at station 1.  Master count was 41,600.

44. Lid was raised with trip at station 9.  Master count was 42,857.

45.  All counters were read and selected stations were jogged:

46. Lid was raised but tripped at station1.  Station one count was found to be behind and was jogged from 42,804 to 42,880.

47. Lid was raised until limit switches were contacted.  All final counter readings were recorded:

***

Photos of the limit switches and the reference "stick" relative to the roof beam are attached.  The stick appears to be between 1/16" and 1/8" clear of the roof beams.

Lid was raised today (2nd time during 2015 shutdown) for additional inflector work.

Dan and Maico report that the lid went up smoothly and no jogs were required.  The operation took about 30 minutes.

- Grant

The Cyclotron Lid Down was successfully completed today on Work Permit C2014-03-03-6.

The system only tripped twice on the way down.  Both trips were at Station 9, at the following Master Syncro readings: 43194, 39198.

Oil was observed on the Jack #1 platform, but it was determined that it was coming from the crane when it's parked in the beam-on position, and not actually from the elevating system.

The final counter readings after the lower limit was reached were recorded:

The Master Syncro was recorded at 28.  All station yoke gaps were very loose on the .25" shim, which makes sense, as they are about 70 thou lower than the starting position before lid-up this year (100 counts at each station).

The system was NOT re-indexed.  The decision was made to wait until lid-up next year to re-index.  The lit should be raised until the Master is at about 100 counts, then the yoke gaps checked and adjusted as required.

Regards,

Grant

Remote Handing is ready for January, 8 2018 Cyclotron lid up. Elevating system service has been completed.

A Cyclotron Lid raise was attempted on Monday, January 8 at 12:00, however this attempt was aborted after lifting the lid several millimeters due the motor
brake on Station #9 that would not release properly. After troubleshooting the elevating system for several hours, motor brake wires were disconnected from
the motor terminal block at the station and pulled through the vertical conduit down the B2 level below the Cyclotron for further inspection. The inspection
showed severe damage to the wire insulation and conductor over an 18" interval, which corresponded to the position directly across the mid-plane of the
cyclotron. New wires were spliced to the existing wires at the bottom of the vertical conduit section, which were run through the conduit and reconnected to
the motor terminal block.

The motor drive wires on Station #9 were also removed for inspection, which revealed major damage to the wire insulation over the same 18" interval;
however, the conductor remained intact and was covered in electrical tape and re-run through the conduit vertical and reconnected to the terminal block. The
elevating system was then tested and the motor brake on Station #9 released as expected.

The Cyclotron Lid was successfully raised on Monday, January 8 at 20:00. The initial station readings are shown below:

00075 - Master
00045 - Station #1
00044 - Station #2
00042 - Station #3
00045 - Station #4
00044 - Station #5 
00044 - Station #6
00045 - Station #7
00044 - Station #8
00044 - Station #9
00047 - Station #10
00048 - Station #11
00045 - Station #12

The final lid-up station readings are shown below:

47422 - Master
- Station #1
- Station #2
- Station #3
- Station #4
- Station #5 
- Station #6
- Station #7
- Station #8
- Station #9
- Station #10
- Station #11
- Station #12


The Cyclotron Lid was lowered on Tuesday, February 20, between 11:30 and 12:00. The final station readings are shown below:

00081 - Master
00086 - Station #1
00089 - Station #2
00054 - Station #3
00016 - Station #4
00087 - Station #5
00087 - Station #6
00085 - Station #7
00083 - Station #8
00086 - Station #9
00087 - Station #10
00089 - Station #11
00088 - Station #12

While lowering the lid, the following observations were made:

- The left High Field gear reducer on Station #1 made noise intermittently during the middle of the lid lowering operation
- The right High Field gear reducer on Station #2 made noise intermittently during the end of the lid lowering operation
- The right High Field gear reducer on Station #11 made noise intermittently during the beginning of the lid lowering operation

 We removed the left bushing that is around the air cylinder on the festooning rail, right outside the cyclotron. It was cracked. Will need to replace next year when the vault door opens again.

The copper blockers have been removed from the cyclotron and placed into the coffins.

Copper Blockers have been removed from the cyclotron and one ladder has been installed.

An issue was encountered on 2024-02-23 when attempting to operate the lift trolley.

The following symptoms were observed:
- Upon connecting the trolley's air and power/signals cable, nothing out of the ordinary was observed
- When the Lift Trolley program was selected, however, multiple (seemingly) random outputs on the trolley toggled rapidly without stopping
- On the operator display, a fault message was displayed indicating a power supply issue, and trolley off-rail (the trolley off-rail one being likely unrelated, and as a cascading result of the actual issue)
- It was noted that the output toggling issue persisted even if the trolley itself was disconnected from the "red box" (the controls box with Opto22 output modules)

The following troubleshooting steps were performed:
- Try selecting a different program. Result: the issue persisted regardless of the selected program
- Try with trolley disconnected, but program selected. Result: the issue persisted (indicator lights on modules in red box toggled).
- Try with a different trolley (upper resonator trolley). Result: the issue persisted.
- Try on the mockup, rather than the bridge. Result: the issue persisted.
- Try on the mockup, but with the main festooning cable system for the bridge isolated (disconnected in the control room at the junction box). Result: the issue persisted.
- Try the inverse of the above. Result: the issue persisted.

Various other things were checked:
- Inspected wiring and connectors inside the red box
- Inspected some wiring and connectors inside control room


After fairly extensive troubleshooting, it was concluded that the issue was:
a) in-common with all trolleys (i.e. upstream of trolleys themselves),
b) related to the program, likely during some kind of initialization routine,
c) related to a power supply issue, and
d) not likely to be a result of damaged wiring/connectors.

When the troubleshooting was focused on the control room, it was suspected that the CAMAC crate modules were not properly being powered (no indicator lights on, and also the CAMAC IO message on the operator display was not on... although this was not immediately noticed since, well, it wasn't on). Upon probing the CAMAC power supply unit, 5 of 6 voltages were correctly measured, but the +6V supply was incorrectly reading as approx. 1.9V. Therefore, it is suspected that the CAMAC power supply failure is the root cause of the issue. The theory is: because no inputs are available to the controls software, upon selecting a new program, the software is looking in a loop for inputs that will never be present, and it repeatedly performs some sort of initialization routine in which outputs are turned on (or otherwise just behaves erratically) because it was not designed to handle such a fault. It is unknown at this time why the CAMAC power supply failed, but simply age is a strong suspect.

The power supply model is Kinetic Systems 1500-320.

Controls Group (Erwin Klassen) was contacted and they confirmed the issue, and suggested that they can help replace the power supply with one of the spares they have in stock. This is planned for today (2024-02-26). This e-log will be updated after the replacement to confirm whether the issue was resolved.

Other recommendations:

• Attempt to repair the failed supply (check if simple repair is possible first, then attempt to install a custom 6V regulator in place)
• Request quote for procuring a spare power supply (in progress)
• Accelerate the cyclotron controls overhaul project, O503 (in progress) - potentially fully integrate the new control system during 2024

Edit [2024-02-26 afternoon]: the issue has been resolved by replacing the CAMAC power supply, as expected. Preliminary checks show that everything is working as normal. Controls group has also provided us with a power status monitoring card to easily monitor the 6 different voltages, and be able to probe them. This issue is considered resolved.

The harness, carabineer, and shock packed used during the ISAC hot cell swipe have been checked seperately by Mike and he has found that the carabineer was active (feild of ~3 uSv/h on contact) not contaminated. All other components are clean and non active.

The assumption being that this carabineer most have come from the cyclotron vault.

Tool Trolley:

The connector faceplate was replaced with a new version to allow for more space for service, and consolidate video lines into a single 9-pin CPC connector. A model and drawing were created and are part of the overall tool trolley assembly (to be added to PDM later): TRH1394

Air tubing that previously ran to the connector faceplate on the tool trolley was replaced as it was very old. New tubing is in 3 different colours for easy identification: blue, black, green.

Fittings were replaced with more modern push-connect style (1/4" tube).

The shield door on the B2 level of the RH building has been closed.

Fall equipment from the lockers in the RH shop have been checked for contamination and the lockers themselves have been swiped. All items checked in the target hall cave.

No contamination found.

Upon discussion with Dan it was mentioned the harness that I used was not used in the vault, and its origin is of question at this point.

 Center post bearing on service bridge inspected, turned freely by hand and regreased with Molub-Alloy 777-1 ES

 All 4 Under rail wheels serviced and cleaned. All Timken bearings repacked with grease. No defects in any bearing set. 4 spare Timken 02820 bearings in RH shop.

 Motor/Gearbox unit removed from carrier and fully inspected and serviced. Oil changed on May 29th using Mobil 600 W Super Cylinder Oil. No signs of wear in internal gears, recommend oil service roughly every 5 years after shutdown.

Serviced the Camera trolley this morning as per Mike Gallops service outline.

Performed continuity test at Cyclotron center post with RH and Operations group members using the service bridge, 
tool trolley, and probe attachment on 2017-03-03.

Lower tank:
Quad #1: All six plates PASSED.
Quad #2: All eight plates PASSED.
Quad #3: All six plates PASSED.
Quad #4: All eight plates PASSED.

Upper tank:
Quad #1: Had some repeat-ability challenges with all six screw-heads/wires. Retest required.
Quad #2: All eight plates PASSED.
Quad #3: All six screw-heads PASSED.
Quad #4: All eight plates FAILED. Retest required.

Performed extra continuity test at Cyclotron center post with RH and Operations on 2017-03-06: achieved a 
successful test on all upper quad #1 and #4 screws/plates. One unconfirmed explanation for the successful retest 
is that the multimeter cable was firmly grounded against the computer monitor rack in the RH service bridge 
control room (see attached picture), which may not have been the case during the initial test.

Noticed oil leaking out of reduction box, scan and swipe, disconnect e motor and all related cables from bridge, remove motor and reduction box assembly with over head crane, disassemble reduction box, grease broken down and separated likely cause of leak. Inspect bearings and internals for wear -all good

re assemble with new grease and gaskets (we added two grease nipples to the bearing housing to aid lubrication), test e motor on its own- good. Mount motor and box as unit adjust chain for tension and final test - good

 All the bearings on both the upper and lower wheel were replace to see if it will improve the accuracy of the orbital position reading. See drawing SK-2489 for list of bearings used.

The encoder was also replaced to match the style used elsewhere.