When one of the largest Cray Super Computers was recently installed in Europe, Riello UPS was awarded the crucial task of guaranteeing its power protection.
At a total project cost of £113m, it is one of the largest and most ambitious high-performance computing (HPC) initiatives in Europe. Equivalent in performance to 14,000 desktop PCs, it is three times faster than the EPCC’s existing super computer (the HPCX), with a memory of 35,000 GB and disk storage in the region of 700,000 GB.
The computer is critical to UK scientific research and is so powerful it has a massive power/cooling load density of 6.5kW per square metre. Resilience, availability and serviceability are the crucial elements Riello has had to deliver - and to an extremely tight deadline! HECToR (High End Computing Terascale Resource) is housed at the University of Edinburgh’s Advanced Computing Facility (ACF) in Midlothian, Scotland and directed and operated by EPCC (Edinburgh Parallel Computing Centre). Built by super computer specialists Cray Incorporated, HECToR will benefit academic research across the whole of the UK and beyond.
Professor Arthur Trew, director of EPCC, explains why HECToR is critical for UK scientists to compete on the world stage: “Traditional progress in science has been made through theory and experiment, but an increasing range of problems now require to be simulated computationally. Examples range from climate modelling; understanding sub-nuclear particles to the evolution of the Universe.”
Scientists will be able to access HECToR remotely but due to the nature of academic research, no-one quite knows when peak times will be; hence the need for high resilience and availability. Working with Crown House Technologies, who were responsible for the overall installation, the total project size for Riello was 3.4MVA with uninterruptible power supplies installed to support the super computer and the essential components of the air conditioning systems within the facility.
The Riello Master Plus UPS module was chosen and two N+1 parallel UPS systems installed; four 800kVA UPS modules to power the super computer and two 100kVA UPS modules for the critical components of the vital air conditioning system.
“Master Plus is the ideal choice for data centre installations and we are proud to have been chosen for this particular project” commented Robin Koffler, general manager for Riello UPS Ltd. “This is a key installation for Riello UPS and is similar in size to projects we are running across Europe.”
Resilient System Design
In terms of power continuity, the N+1 parallel configuration provides a high level of resilience. It allows one UPS module to be removed from the system (either automatically through a fault condition or manually for routine maintenance) without disrupting overall system integrity. Synchronisation of the UPS module outputs and load sharing is achieved through the use of a closed-loop communications flow. Each UPS module is connected to the other using a parallel card and communications cable in what is termed a decentralised architecture, more commonly referred to as a master-slave configuration. The first UPS to be switched on assumes the role of master and the remaining modules slaves. If the master module is removed one of the slaves automatically assumes this role and load sharing and synchronisation are maintained.
Simultaneous Power Sharing
For the super computer, the four 800kVA UPS modules are being supplied power from two separate 11kV substation transformers. The configuration allows power to be drawn from each transformer simultaneously whilst relying on the sophisticated control and communications technology of the Master Plus UPS to ensure a clean, stable and synchronised output is maintained at all times. Even if one of the transformers fails, the overall system design is resilient as all the UPS modules can be powered from the remaining transformer without introducing an upstream overload or fault condition.
Each UPS has been installed with its own battery set with an overall runtime of fifteen minutes (if all UPS modules are powered) and ten minutes should one be removed. For such a large installation the batteries contribute a significant portion of the overall weight (over 20 tonnes) and were built into cladded battery stands.
“We have enough battery power to ride through most interruptions in supply but the total extended runtime gives enough time to perform a controlled and safe shutdown, if required, without damaging the computer or any applications or data stored on it.” Comments Lawrence Valentine, operations manager at Crown House Technologies.
UPS Installation and Interlocking
The UPS were manufactured from the standard Riello Master Plus range with additional accessories chosen to enhance the security of the system and allow for top-entry power connections and the required tamper-proof switch-gear interlocking. Each UPS module had an additional ‘Barduct top entry’ cabinet – only 400mm wide to allow connection to both the supply and load power distribution Barduct system, which replaces multiple runs of cable along with their associated cable trays.
They were also configured to integrate fully with the operational interlocking and monitoring schemes required by the project to ensure optimum system safety and integrity.
For additional security a wrap-around emergency external bypass employing Castell interlocking was installed for the entire UPS system. The size of the computer load (currently 1.2MW, rising to 1.8MW during 2009) dictated that a dual input supply be implemented to further enhance resilience and ensure no single point of failure exists.
Jason Yates, technical services manager, Riello UPS explains: “In this case we were able to split input power between two substation transformers – something not always possible with competitor’s UPS due to the output synchronisation problems that can arise. The overall resilience of the power continuity plan installed at the University is extremely high as single points of failure have been removed.”
Crown House Technologies designed and built extensions to the facility in which HECToR and the UPS are housed. It was an energetic and hectic time for the Dartford-based building services company as Valentine explains: “Time was not on our side. In January 2007 we were awarded the contract with the computer being delivered in July! We had to design, build, test and fully commission the facility before that deadline.”
The building project did go to plan and the deadline was met without a hitch but it was unusual in the sense that half the facility was already in place. Following a refurbishment by Crown House in 2004, the computer room was there as a bare shell. But it was not large enough to house all of the driving mechanisms for the new computer so a new facility had to be built to house the UPS, air conditioning and other essential equipment.
Says Valentine: “The plant room is actually one and a half times larger than the computer room. But that is because of the scale of the equipment that has to be contained in there to handle the exceptionally high density power and cooling loads. The UPS systems themselves take up around 25% of the plant room floor space.”
Protecting such a valuable and powerful computer means that cooling requirements are critical and the electrical supply to the air conditioning must also be protected. Riello has installed two 100kVA UPS to support the critical plant room machinery that runs the air-conditioning system.
Valentine explains: “With load densities of 6.5kW per square metre, not many seconds without cooling would cause serious damage to the computer equipment. However, in conjunction with the UoE, we designed an innovative system that allows sufficient time for a controlled safe shut down of the computer without the need for masses of redundancy equipment. This reduced the high capital expenditure for redundancy equipment which is very rarely called upon without compromising the resilience, availability and serviceability requirements.” Valentine adds: “For future flexibility we designed systems that can accommodate the high density cooling loads via water or air side systems and the UPS supports the critical M & E equipment for both scenarios. The system is currently operating on air-con only with a capacity for 500 air changes per hour through the computer room.”
TeleNetGuard Remote Service
An installation of this size and magnitude requires a cast-iron monitoring and maintenance plan. The UPS installation is monitored remotely via TeleNetGuard and even here Riello UPS had to install a specially configured telecoms switch. TeleNetGuard is a Riello UPS software package that allows 24/7 remote monitoring via the Riello Service Centre. Technicians can view the status of the system at any point and be given prior warning of any impending condition. This takes place on a 24/7 basis with response requests issued automatically to field service personnel via SMS text, email and voice messages. Riello’s Yates explains: “We’ve had to build resilience and redundancy into the monitoring system as well as the UPS itself. We’ve done this by ensuring all of the UPS modules in the parallel configuration have their own remote monitoring communication links rather than relying on one overall UPS system connection. In this way, if one link fails we still have the other three to rely on. We often know of a fault or potential fault before it develops and can do something about it either remotely or by sending an engineer to site; further peace of mind for the customer.”
Working on the HECToR project has enabled Riello to fully demonstrate skills, expertise and customer service and to fully substantiate its leading-edge UPS technology. Valentine closes with “I would like to send a special thanks to Riello for the technical & delivery support offered throughout the project which significantly contributed to the successful hand over of the facility on time and to budget.”