We speak to a prestigious industry podcast about the importance of UPS maintenance for data centres and why engineers need to be fully certified.
Founded in 2004, Workspace Technology Limited has quickly established itself as one of the UK’s leading data centre design and build specialists.
Through its popular Data Centre Dave podcast series, the company helps share the knowledge of experts from across the data centre sector.
In this edition, Riello UPS’s Technical Services Manager Jason Yates and Service Sales Manager Emma Platt sit down with Dave to discuss the importance of UPS maintenance for data centres.
The near 15-minute chat covers several subjects, including how often a UPS should be serviced and what sort of things a preventive maintenance visit (PMV) should cover.
Jason goes on to explain why data centre owners should always opt for UPS specialists rather than rely on their in-house engineers or facilities management companies. He shares an example where one data centre was left counting the cost – into tens of millions of pounds!
Listen to the podcast using the player below or click here.
You’ll also find a transcription of the chat exploring the importance of UPS maintenance for data centres below.
Data Centre Dave’s Podcast With Riello UPS
Data Centre Dave: Hi I’m Data Centre Dave, and today we’re joined by two of our industry experts, Emma Platt and Jason Yates from Riello UPS, who join me to discuss all things UPS related when it comes to data centres.
Thanks for joining us today. So, you and me both know that a fully-functioning UPS can be all the difference between data centre downtime. Jason, maybe you could tell the listeners the purpose of a UPS in a data centre?
Jason Yates: Basically a UPS system (or uninterruptible power supply) is a system that works in conjunction with batteries, constantly producing power for the data centre.
Now in the event of a mains failure, when the power to the data centre is either interrupted momentarily or for a long period, or if there’s anything such as noise on the electricity supply, basically the UPS continues and ensures the critical system has uninterrupted power running to it before the generators kick in, when the generators can then take over the full load.
Alternatively, it’s until the batteries are fully depleted and everything’s switched off. Basically it’s a backup system for data centres to ensure they have a continuous supply of power at all times.
Data Centre Dave: That’s great, so hopefully all our listeners – if you didn’t know what a UPS is – definitely should know by now. OK, let’s turn to the maintenance requirements. Emma, how often should a UPS be serviced and what does that involve?
Emma Platt: A UPS is like any other critical equipment, so it should be serviced and maintained at least every 12 months.
Some sites may need their UPS servicing more frequently, particularly if its in a harsh environment in terms of temperature (both high and low), which increases the speed at which components deteriorate, or you’re looking at high levels of dust, dirt, debris in the atmosphere causing increased wear and tear, restricted airflow, or overheating.
You’ve also got corrosive chemicals causing contacts or terminal connections to fail, harsh electrical loads can again deteriorate components.
When you service a UPS, some of the key checks are thermal imaging to check the batteries, capacitors and connections. This highlights areas of overheating which could lead to impending failures.
You can also look at battery performance to ensure the batteries can support the critical load. The other thing we tend to look at is the UPS’s internal history, data file checks to look for potential issues.
Just a couple more, we tend to check the internal connections and all components for any possible signs of problems. Finally, we check the age of the components with a finite lifespan, like the capacitors or fans.
In-House Or Specialists?
Data Centre Dave: Well it sounds complicated! I’ve recently seen a trend with businesses who choose either to not service their UPS systems, or to do so using their in-house engineers or estates teams. Jason, as a Technical Manager, what’s your view of this?
Jason: Firstly and most importantly, just building on some of the points Emma’s just mentioned there, like any other piece of critical equipment, ideally if the UPS needs to be serviced throughout its lifespan – whether that’s 12 monthly, 6 monthly, whichever depending on the environment – it’s imperative that it is serviced.
It’s not about the UPS company making money, it’s about ensuring the UPS will be available and able to support the critical load if and when an issue does occur, because it’s like anything, unfortunately it’s only when something’s tested in anger do you find out whether it works or not.
Now obviously, you’ll have your in-house engineers carrying out their daily checks, walking around the UPS, and to be honest I encourage that sort of thing.
I don’t want people to be afraid of the UPS, I want the engineers touching it, going through the display menus, understanding it a little bit just so they know what their UPS system is actually doing.
This helps when there is a problem and you’re talking to them over the telephone, they’re confident enough to look at the display panel and take readings that helps us to solve the problem.
The main problem is when they push their own in-house engineers, or bring in a third-party facilities management company, who will sadly put money before customer service and simply take on a contract where they can’t adequately support that product in the field.
Unfortunately, a UPS system does contain hazardous AC voltages, hazardous DC voltages, it stores a lot of energy, and if you’re not trained to work them and how to discharge them properly, sadly you could end up with a fatality on your hands and in that case, that’s the last thing any company would want.
Now in addition to this, the battery systems are a key element in any UPS system. What many people don’t understand is that the DC systems within a UPS can be far more dangerous than the AC systems that are feeding it, they see it as a big bank of car batteries but it’s more than that it’s a high voltage system that can kill just as much as the AC voltage.
As far as I’m concerned, unless you’re trained and approved with that particular UPS, and have access to the various software, spare parts, firmware, and most importantly support from the manufacturer directly, then companies who take these contracts on or management who effectively push their engineers into performing these roles for which they aren’t adequately trained, it’s unfair on them as far as I’m concerned.
They don’t have the support, they don’t have the access, they’re actually endangering their own life. In the worst case, someone’s going to get hurt, the data centre’s going to lose power, and the customer’s going to get very upset.
Any work of this kind must be performed by a trained and fully-certified UPS engineer and that’s the way it always should be otherwise the consequences can be devastating not just in terms of business continuity but in terms of danger to life.
A Mistake Costing “Tens Of Millions”…
Data Centre Dave: I definitely agree. One of the points touched upon is that using a data centre specialist provides the reassurance that your downtime will be prevented and obviously the safety of employees as well. Could you share any examples where a company has gone down the line of a non-specialist approach and suffered any consequences for that?
Jason: Well, there was an incident I had probably two years ago – this is one of several – whereby it was a multiple parallel system, five UPSs in parallel, a total load of about 600 kW, in an N+2. So on paper, a perfectly resilient UPS system.
In the early hours of the morning 1am or 2am, they had a fault with one of the machines, the machine safely disconnected from the UPS system and the remaining machines picked up the load, so everything was good.
They then called in their facilities management company and this guy clearly didn’t know what he was doing. Now instead of this guy stopping, he carried on, and eventually he ended up shutting down the entire system, damaging several other of the UPS in the process.
As you can appreciate, that isn’t going to end well and I believe the total cost for the 48 hours the data centre was not fully up to speed because some of the other equipment was damaged too, ran into tens of millions of pounds. All for the sake of trying to save a few pounds rather than bringing in a professional who knows what they’re doing.
Data Centre Dave: Wow, tens of millions… I hope none of our listeners have experienced anything like that! So Jason, what sort of training does a UPS engineer need to go through to be qualified?
Jason: When we take on a new engineer, we’re always looking for the basics. So they must have an electrical background. We find, for instance, it’s easier to train an electrician up to be a UPS engineer rather than the other way around.
It depends on their background and previous experience. Some people we bring in will already be well-versed with Riello UPS systems, or with other manufacturers’ UPS systems so they already know about UPS.
A typical engineer will take anywhere between 6-8 weeks, now obviously the key element is bringing them up to speed with the company, we’ll go through all the health and safety aspects, the AC and DC isolation, we’d be covering things like UPS installation, commissioning, the maintenance, the repair, the diagnostics.
We’ll go through the various software, as obviously many modern UPS are software-driven, and that includes firmware and the monitoring systems that are used with a UPS. Then there’s customer service as they’re the face of the company and talking to people when they go onsite.
Prior to them being let loose into the world of UPS, they’ll shadow with a more experienced engineer for up to two, three, even four months possibly depending on how they’re getting on. And that enables us to get some feedback from our own engineers, how they are getting on, if there’s any gap analysis we need to carry out we can do so.
Eventually once they’re in the field, we have a fully-fitted training room in-house, which if they’re ever feeling a bit rusty or unsure, they’re welcome to come in at any time and basically we go through it, so they’ve always got that background, they’ve always got that technical support behind them, that contact with the manufacturer so they can do everything right under the guidelines of the manufacturer.
Otherwise, when things do go wrong, a lot of end-users will turn around and say “well what does the manufacturer say about that?” and if you’ve contradicted what the manufacturer says you’re opening yourself up to all kinds of trouble sadly.
In reality it takes between 12-18 months, even up to two years, for an engineer to be fully competent because there’s so many variations and UPS covers such a wide industry.
Data Centre Dave: It’s interesting as I guess there are some companies that think they can just get their staff to service the UPS just because they can, say, service the office AC units onsite. But as you say, it’s a specialist skill and one that takes time to learn.
Jason: Correct. With any data centre, while it might not be a heavy industrial environment, it’s definitely a heavy electrical environment, and they are exposed to, for example, open battery rooms where there’s hazardous voltages, chemicals from the batteries, operating processes, switching processes, the electrical energy within the UPS which again is an issue, the installations, you’ve got on-site generation, AC units…
There are many things inside the data centre, and to be honest with you, you can’t be a master of them all. That’s why you’ve got to bring in the specialist people in order for them to carry out that work.
Again for data centre owners to push their own in-house engineers and to put them into situations whereby they aren’t trained or suitable to perform that role, I think that’s completely unfair and dangerous.
Data Centre Dave: Absolutely. Data centre maintenance and management as we know is a specialist area and one wrong move, as you’ve said, the whole data centre can come crashing down. Emma and Jason, it’s been great chatting to you and I hope our listeners have learned the importance of using a specialist company for their UPS maintenance.
Emma: Yeah, hopefully! Thanks Data Centre Dave.
Data Centre Dave: Thanks to both of you. Well folks, that brings us to the end of today’s podcast.