Datacenter Backup Diesel Generators Winter Maintenance

 

 

Backup generators are vital for proper datacenter operation. Regular maintenance procedures are mandatory to ensure that your units will start smoothly and supply necessary power to your facilities.

 

The probability for power outages is higher during the winter season, due to possible overloading of the power grid in very cold days or after snow/wind storms and other natural disasters. Furthermore, internal combustion engines need special care at low ambient temperatures, especially during start up and warm up. Therefore in TSBG hosting ltd. we plan diesel generators maintenance just before the winter. October and November are the months, best suited for that.  

 

Our team would like to share what you have to pay attention to and take care of to ensure that you are ready for the winter.

 

OIL CHANGE

 

Oil filter change is a regular maintenance procedure for any internal combustion engine. Generators are driven by diesel engines, on which you need to replace oil at regular intervals, as specified by the engine manufacturer. Usually this interval is 200-300 hours of operation. All our datacenters are designed with redundant power supply systems and grid power outages are rare. Therefore, the generators are not operating more than 50 hours per year. Anyway, the oil ages, even without engine being in operation. Changing oil at least once per year is recommended. Changing at longer intervals is acceptable only if a probe is taken and sent to a lab for evaluation. In any case, the price for changing oil is relatively low, compared to overall datacenter maintenance costs and the maintaining of good oil parameters by regular oil changes will ensure longer and trouble free operation of the engine and you will ensure that your generator will remain operational for many years ahead.

 

When replacing oil don’t forget to also change the oil filter and fuel filter as this is a recommended good practice. The air filter needs to be inspected and replaced if dirty or damaged. When the engine is not running the air filter is not collecting dust and it is acceptable to replace it at longer intervals. Please have in mind that the air filter conditions depend not only on the running hours of the engine but also on the environment. If it is too dusty or the filter is overheated or there was a leakage of water, oil or coolant over the filter, it may become compromised and prevent normal air flow or lose its filter capabilities.

 

If the air filter is blocked and can not provide necessary air flow the engine output power will be reduced and eventually the engine may not start. If the filter can not provide filtering of the engine incoming air, that may result in engine's damage.

 

Before the oil and filter replacement it is good to let the generator run for 15 minutes. That will warm up the oil and make it easier and faster to drain it from the engine. Bigger engines need more oil and draining itl sometimes takes a long time, especially when it is very cold.

 

Shut down the generator and block the automatic start. Consult with generator manual. The best way is to block the generator controls and disconnect the battery. Accidental generator start, during oil replacement may cause injuries and damage the engine.

 

Please pay attention to the environment. Always use an oil drain pan with sufficient capacity. Store used oil in plastic or steel containers and return it to a local recycling center. Avoid any leakages or spills.

 

 

In case of an oil spill do the following (be prepared with necessary materials before staring oil change):

 

• Sprinkle sawdust, clay kitty litter (not the clumping kind), coconut husks, or a commercial oil-absorbing product on the stain. If you have none of these products on hand, set paper towels to soak up any wet oil.
• Allow the absorbent materials to work on the oil for 24 to 48 hours. Then, sweep up the debris with a broom and dustpan.
• Repeat with fresh absorbent materials as needed to remove any remaining oil.
• Dispose debris properly-consult local recycling center.
• Clean the area with water and liquid soap.

 

Protect your eyes, skin and clothes.

 

Attention! The hot engine oil can burn your eyes and skin. Don’t attempt oil replacement if the generator was operational for a long time. Wait for the oil to cool down to 35 degrees C (95 degrees F).

 

After the oil is drained, close the drain opening and dismount the oil filter. Please note that there is a small amount of oil remained in the filter. Follow manufacturers manual how to remove and replace the oil filter. In most cases you will need a special tool.

 

 

Before fitting the new oil filter don’t forget to fill it with fresh oil. That will make engine start easier and minimize the time engine is running without oil during start up and filling of oil system.

 

Dispose old oil filter properly. Consult the local oil recycling center.

 

Fill the engine with fresh oil with grade and type as specified by manufacturer. Always use high quality oil. Quality of the engine oil is critical for engine life, cold start capabilities and even fuel consumption.

 

If the manufacturer recommends oil with different viscosity, select the one that best matches your local environment conditions during the period in between oil changes. An example chart is pictured below, but always use manufacturer maintenance manual.

 

 

If the generator is equipped with a canopy filling the engine oil may be difficult.

 

After the oil filter replacement, proceed with the fuel filter replacement. Usually the fuel filter is fitted by a bolt and no special tool is required. Consult engine maintenance manual. Please note that the old filter has some amount of fuel in it. Before fitting the new filter, fill it with fuel. That will ease engine start. If you need to detach fuel hoses, depending on the clamps, you may need a special clamp tool like the one shown on the picture

 

 

 

 

Attention! The engine fuels are highly flammable. Do not smoke and/or use fire and heating tools when working with fuels. Always protect your eyes, skin and clothes. Avoid fuel spill.

 

 

 

After replacing the fuel filter inspect and if necessary clean or replace the air filter. Small amounts of dirt can be sprayed with compressed air.

 

Fuel hoses and fuel system inspection

 

The rubber or plastic which fuel hoses are made out of ages with time. Carefully inspect all hoses from fuel tank to fuel pump, fuel filter and engine assembly. Sometimes small cracks appear in the fuel supply hoses. The cracks let air enter in the fuel hoses thus preventing necessary fuel flow. The result is that the engine operates normal when iddle, but when the generator is loaded the fuel flow is not enough to supply the necessary amount of fuel to the engine and the generator starts acting up as it is overloaded with uneven operation and drop in supply voltage. In fact the generator is unusable and diagnosing the problem is not easy and often incorrect.

 

If you find any problem with the fuel hose, replace all of them. The cost is minor and may save you a lot of troubles. Always use high quality hoses. Please note that sometimes the fuel hoses are from different manufacturers when it comes to those for the engine and the generator and one type can look OK but another type may be compromised.

 

 Don’t forget to inspect and replace the additional fuel filter between fuel tank and the engine, if such additional filter is installed.

 

CHECK COOLANT

Engine coolant is a mixture of ethylene-glycol and water. Mixture ratio determines the freezing temperature of the coolant. It is a must to ensure that the freezing temperature of the coolant is lower than the lowest expected air temperature during the winter. Winter temperatures depend on geographic location but may vary a lot from year to year. Usually, the generator assembly contains a coolant heater to keep the engine warm and ease quick starting. Freezing of the coolant will surely cause severe damage of the engine. Therefore to be on the safe side, in any case keep freezing point of the coolant lower than -20 degrees C.

 

You can measure the freezing temperature with a simple and very useful tool, which we recommend to be added to the maintenance toolbox of every site, which has a backup diesel generators or any other device, driven by an internal combustion engine with a liquid cooling system.

 

        

 

Check freezing temperature of the coolant every autumn, before winter. During the test, check coolant color and transparency. If the coolant is dirty or unclear, drain the coolant and flush the system several times. Always use distilled water.  The color of healthy engine coolant is green (for ethylene glycol) or orange (for Dexcool). A rusty color indicates that the rust inhibitor in the coolant has broken down and it can no longer control rust and scale buildup. A milky color indicates the presence of oil in the system. Orange coolant normally last longer than the green one.

 

Fill new coolant. After refilling, let the engine operate for 20 minutes and check freezing temperature again as some water, which has remained in the system may increase the freezing temperature, above acceptable limits.

 

Record the last coolant replacement date and plan coolant replacement every 3 years for blue and 5 years for red coolant. Don’t mix different types of antifreeze.

 

CHECK BATTERY

 

Engine battery is required to provide necessary current to start the engine. If not maintained properly or if the charging system is faulty, the battery may be damaged irreversibly. Even well maintained and properly charged, the battery capacity will reduce substantially after 5 years of operation and needs to be replaced. During the winter season, when cold, the diesel engine start is more difficult, due to the nature of the ignition process. At the same time battery capacity is reduced and can’t drive the necessary current to the engine starter motor, resulting in lower revolutions per minute.

 

The battery health is vital for the proper operation of the generator. It doesn’t matter how well your generator assembly is maintained if you have a faulty battery the engine will not start when expected.

 

First – check electrolyte level if you don’t have maintenance free battery. If necessary fill with distilled water.

 

Measuring battery voltage can only indicate the level of charging but can not determine battery capacity. You need to test the battery under load to evaluate its state. The best way to perform this is with a special battery tester tool as shown below:

 

 

The tool has a powerful resistor inside and can measure a battery voltage drop when a high current applied. This tool is not expensive and it is a valuable measuring device for every battery plant. For more advance measurements it is recommended to use an electronic device.

 

If the measurement shows that the battery is OK inspect the electrodes. Clean and apply a thin layer of vaseline, which will prevent corrosion. Also check battery compartment for electrolyte spill and rust. If necessary clean, prime and paint.

 

Replace battery if the measurement shows weak battery or if it is more than 5 years old. Install maintenance free battery with the same capacity. If the battery is older than 3 year make battery test every month or two. This will save you a lot of troubles.

 

 

CHECK COOLANT HEATER

Don't forget to check and if necessary adjust the coolant heater. Generator assemblies are equipped with a coolant heater, which keeps the engine warm and eases starting. During the winter it is very important to have this operational. Raplace faulty heater if necessary.

FILL FUEL TANK

Fill fuel tank to the top. Having enough fuel will guarantee operation of the unit for as long as possible. Revise your contracts for fuel delivery on the site.Larger generators require grand amount of fuel.

CHECK BELTS

Check all belts for cracks. Check tension. If necessary replace old belts.

TEST THE GENERATOR UNDER LOAD

At the end start the generator under load for at least 1 hour. During that period check the engine temperature, measure supplied voltage, check for alarms, strange noises and other abnormalities.

 

 

 

 

 

 

 

 

 

 

Temperature monitoring in the datacenter

 

Monitoring of the environment parameters in a datacenter is very important for normal operation. Moreover, having a clear picture of the current values, trends and deviations will ensure system uptime, optimize performance and reduce operational costs.

 

Properly setting alarm triggers will send early alarm notices and give operational staff more time to eliminate faults and manage repairs.

 

 There are no doubts that monitoring systems must be adapted to the specific design and purpose of the infrastructure, equipment sensitivity, air conditioning system redundancy etc.

 

E.g. the monitoring of high power density server racks in hot/cold aisle configuration is different from the one in the equipment container, exposed to the elements at a remote location.

 

Last, but not least, the monitoring system design and operation falls outside the core tasks of network engineers and administrators, it often requires different and specific skills and malfunctioning or improper functioning often has no direct impact on operations - especially in early stages. It is therefore often underestimated and pushed aside in fovor of more urgent problems until disaster strikes.

 

In order to properly design, commission, operate and maintain our critical infrastructure monitoring systems we need to identify the key parameters and characteristics such systems must have.

 

In this article we will try to summarize all aspects we need to consider when solving this task-both general and specific ones. Your project may differ, but our solution is flexible enough to answer most needs, scopes and sizes.

 

What are the needs

 

Let’s see what problems we must address with our monitoring system.

 

-to be easy to install in diverse environments;

 

-to not disturb operation of other systems;

 

-to operate on 24/7/365 basis;

 

-to be in a redundant configuration;

 

-to keep history data to a predefined period of time;

 

-to be scalable and adjustable;

 

-to generate and clear alarm alerts via different media and at predefined thresholds with predefined severity levels;

 

-to be easy to integrate it in other management and monitoring systems;

 

-to stay within reasonable budget limits;

 

-to occupy a small space in the rack and in the datacenter and to have a lower power consumption;

 

-to not interfere with other systems;

 

-to be able to express the data in different formats for preselected periods of time and to be easy to switch between general view and detailed view;

 

-to be able to backup configuration and stored data;

 

-to easily add, delete and modify configuration, following current changes and demands;

 

And lastly – to comply with the applicable standards

 

 

 

Talking about standards, it is important to mention ASHRAE 90.4 was finalized and went into effect in 2016 and recently was replaced by ASHRAE 90.4-2019.

 

ASHRAE 90.4 standard establishes the minimum energy efficiency requirements of data centers for design, construction, operation and maintenance and utilization of on-site or off-site renewable energy resources. The standard also stipulates means for temperature monitoring, requiring minimum 3 sensors at the inlet and outlet of every rack-one at the top, middle and bottom section. That means that the monitoring system must maintain minimum six sensors per equipment rack.

 

The solution

 

There are many “turn-key” and ready to use monitoring solutions on the market. You can find information by searching the Internet or talking to your datacenter construction/maintenance contractor. In our opinion almost all of them are either not answering all problems, as listed above, or are too expensive, or require additional work/upgrade to be done in order to achieve full functionality. E.g. there are nice monitoring systems delivered with WEB interface only. They have SNMP interface but need extra work to integrate them in the operator’s existing SNMP monitoring system or develop a complete new one, dedicated to environment monitoring only.

 

Below I would like to show how you can create and integrate an excellent environment monitoring system within a very small budget.

 

The components

 

We will need good, reliable and cheap sensors, easily addressable, with compact size and simple and easy deployable cabling.

 

We will need a server, capable of retrieving information, storing, visualizing and analyzing it and triggering alarms.

 

We will need a converter, capable of converting sensor’s data into format, readable by the server.

 

Sensors

For this project we have selected addressable 1-wire sensors. There are many options here, but our team prefers the DS18B20 sensor as shown on the picture below. It is water and dust esistant with index of protection IP 67. The sensor has a stainless steel protective cap with a diameter of just 6mm and length 50mm and comes with 1 m pre-terminated cable

The DS18B20 provides 9-bit to 12-bit Celsius temperature measurements and communicates over 1-Wire bus that, by definition requires only one data line and ground for communication. The operating temperature range is between -55°C to +125°C and is accurate to ±0.5°C over the range of -10°C to +85°C-more than enough for our datacenter monitoring requirements. It is important to mention that, for temperatures below 100°C the sensor can derive power directly from the data line (“parasite power”), eliminating the need for an external power supply. Although we don’t use this functionality, it might be useful in some particular applications.

 

Each sensor has a unique 64-bit serial code, which allows multiple sensors to function on the same 1-Wire bus. Thus, it is simple to use one controller to control many sensors distributed over a large area. The sensor converts temperature to 9, 10, 11 or 12-bit digital word in maximum 750ms. This corresponds to increments of 0.5 °C, 0.25 °C, 0.125 °C and 0.0625 °C respectively.

 

What makes it an even better solution for our needs is the price of less than 3 Euro. Please note that the price depends on the number of pieces ordered. As we will need a minimum of 6 sensors per rack, bulk orders are preferred.

 

And of course, you may use different 1-wire sensors or the same sensor in different housing, depending on your particular needs and selected installation method.

 

Now onto how to mechanically install the sensor into the rack. There are many options-from simply placing it over a tray or equipment top, binding to 19” railing with cable ties, sticking it to the rack with a hot glue gun, silicone, adhesive tape or other.

 

We do it with a PG 7 cable gland, attached to a stainless steel bracket with appropriate holes, depending on the rack type and size. You can order several types of brackets for 19” racks, 21” racks, wall mount, ceiling mount etc. The bracket must have an opening for PG 7 cable gland –typically 13 mm in diameter. The sensor is housed into the cable gland and cable may be attached to 1-Wire bus using  3 pcs. 3 poles Wago splicing connectors for 0,75-2,5 mm2 stranded wires.

For 1-Wire bus we use  3 x 0,75 mm2 cable with stranded wires, preferably with halogen-free insulation.

Now we have a perfect sensor and good means to install it in the rack or elsewhere in the datacenter. I would like to mention that this sensor is water/dust proof and UV resistant, due to the stainless-steel housing and is appropriate for outdoor use as well. At all our facilities we use DS18B20 sensors to monitor ambient temperature as well.

 

But a good sensor is not enough to have a fully functional monitoring system. We need a server to collect, store, retrieve, visualize, process all the data coming from the sensors. We need to trigger alarms and set minimum and maximum values. Unfortunately WEB interface of some ready-made systems is not sufficient for achieving that.

 

What options do we have-our team focused on industrial control systems (ICS), including SCADA (Supervisory Control And Data Acquisition) System or SNMP (Simple Network Management Protocol) systems.

 

ICS are typically used in industries such as electric, water, waste water, oil and natural gas, transportation, manufacturing.

 

SNMP is a general purpose protocol which is widely used by the IT community to manage and monitor network equipment and is part of TCP/IP stack. The communication is generally between a management station and an agent-remote device. In broad terms SNMP consists of:

 

- at least one, but generally many devices that contain an SNMP application component that exposes elements of its configuration or operation for management through the SNMP protocol. These devices are referred as agents.

 

-at least one, but potentially more SNMP nodes that are capable of initiating management commands and/or receiving monitoring notifications from other devices-These nodes are referred as a manager.

 

As datacenter operations require deployment of at least one SNMP management system, we have decided to focus on SNMP and base our environment monitoring on SNMP. I would like to stress that there is absolutely no reason to use separate SNMP system just for environment monitoring. It can be integrated into your existing SNMP system following techniques, described below.

 

There are plenty of paid and free SNMP monitoring tools. Among them are Solarwinds, PRTG, ManageEngine OpManager, Nagios, Pandora, Cacti, Zenoss Core and many more. Any can do the job but for our project we selected Zabbix 4. Zabbix is a splendid enterprise-level software designed to monitor everything from performance and availability of servers, network equipment to web applications and databases. It is a server-agent system architecture where you get to install the agent on the server (client) to be monitored by the Zabbix server, but it supports SNMP agentless monitoring as well.

 

Zabbix servers can be installed on different Linux distributions. For our project we used Ubuntu 18.04 on a dedicated server, but there is absolutely not a problem to use a virtual machine as well. You can download Zabbix packages for your OS distribution from their web site https://www.zabbix.com

 

We don't go into details about Zabbix server installation and technical requirements for server hardware. You can find such relevant information on the Internet or in our other posts.

 

A Zabbix server can collect data from devices with SNMP agent versions 1, 2 or 3. Zabbix supports the standard SNMP MIB-2, as well as specific information on an Enterprise MIB. You can collect data by using the SNMP polling mode or receiving SNMP traps through the snmptrap daemon and zabbix_sender utility. Additionaly, you can get SNMP data with the use of dynamic indexes offering monitoring that is flexible and without performance degradation, since the resolution queries are cached.

 

As soon as data is collected, using different methods available in Zabbix, the process of evaluating the collected data begins. Data evaluation rules, or trigger expressions, in terms of Zabbix, provide logical definitions of a problem state for temperature data received from monitored sensors. When a certain threshold is reached, the trigger changes its state from OK to PROBLEM and also back when data is below threshold. In addition to that, Zabbix can be used to predict problems by analyzing the trend of incoming data and constructing a forecast of how things are likely to go, giving users the ability to act proactively.

 

Zabbix provides its users with very flexible, intelligent threshold definition options. While a threshold for a trigger may be as simple as "bigger than x", it is possible to use all logical expressions, such as division, multiplication, not equal, logical AND as well as OR. This can be of use when defining a threshold in case of extreme ambient temperatures, peak busy hours equipment load/power consumption or air conditioning maintenance periods.

 

Zabbix can use many different items obtained from different hosts to build a trigger expression and implement dependencies, when failure in one node is responsible for the failure of operation in many other parts of the environment. This allows to build a very complex, intelligent thresholds, which minimizes false positives and thus lets administrators concentrate on real issues. That can eliminate “alarm flooding” situations, where one fault causes occurrence of many alarms in different nodes, related with that single node. Using the complex trigger expression can diversify different alarms by severity level and not generate lower level alarms when a higher level alarm is present. For example you don’t need “High temperature alarm” with severity level “High”from each 6 sensors of every rack if there is room temperature alarm with severity level “Disaster” caused by air conditioning malfunctioning or power outage.

 

Zabbix also has a hysteresis function that allows to avoid flapping when incoming data is fluctuating around the threshold.

 

It is important to mention the excellent graphs generation capabilities of Zabbix. You can use automatic graphs, generated on runtime from each item. Controls allow to change the displayed time period or chose predefined, often used time periods as day, week, month, last week to date etc.

 

On the picture below you can see the temperature data of one sensor for the last 1 day.

 

Users can create custom graphs where data of several items can be compared or ad-hoc graphs for several items, where you can create a comparison graph.

 

Zabbix offers the creation of a different maps where the users can display values of different items over an optional background image for an user-friendly overview. An example of different racks in a computer room with top and bottom temperature sensor reading expression is shown on the picture below.

 

 

In addition, an user customizable dashboard and slide shows are available.

 

We will write more about the Zabbix configuration and creation of different graphs in another post.

 

Now we have good sensors to monitor the temperature in the datacenter. We have an excellent, open source platform for SNMP monitoring, which can collect, store, visualize temperature data and configure and generate alarms.

 

SNMP to 1-wire converter

 

Unfortunately, we are still missing the link between the 1-Wire interface and SNMP server. The best way to do that is by using Ethernet to 1-Wire server. Our team selected Ethernet to 1 Wire servers from Embedded Data Systems. More information about Embedded Data Systems products may be found at https://www.embeddeddatasystems.com

 

EDS have two devices HA7Net, which can handle up to 100 sensors and OW-SERVER-ENET-2, handling up to 22 sensors.

In order to ensure redundancy we recommend installing 3 devices of the same type. In case of technical failure, loss of power or loss of Ethernet connection of one unit you will still be able to monitor most of your sensors with the other two units.

 

With  three HA7Net devices you will be able to monitor up to 50 cabinets, while OW-SERVER-ENET-2 will let you monitor 11 cabinets/racks.

 

The HA7Net supports a variety of 1-Wire devices at a high level. For example, reading the current temperature from a DS18S20 doesn't require the user to learn the details of reading and parsing scratchpads. Instead, you simply ask the HA7Net for the current temperature and it handles all of the required low-level communications for you. HA7Net works well with our temperature sensor DS18B20, described above.

 

The HA7Net will allow you to use industry standard Ethernet products (switches, hubs, etc.) to build out the backbone of your 1-Wire based sensor / control system. This provides several important benefits: Firstly, 1-Wire networks can be easily distributed over great distances without having to deal with the challenges associated with the installation of physically large 1-Wire networks. By designing your sensor networks in a distributed star topology, the individual 1-Wire MicroLans can be kept to a smaller physical size, thereby improving reliability and decreasing installation costs. Secondly, since the HA7Net communicates via TCP/IP over standard Ethernet, you can take advantage of existing LANs in the datacenter for building out and maintaining the backbone of your sensor network. Another advantage is the number of readily available libraries and developer tools which are widely available for common platforms, and can be used to communicate with the HA7Net.

 

 

 

 

 

 

One of the very important decisions to make when building a datacenter is that the datacenter building is specially design and built to the purpose. Of course that is not always possible. Sometimes the building is rented and often it is not built on purpose to accommodate datacenter. In other cases the building was designed and build long time ago and not all current standards and good practices are accommodated. Sometimes implementing all special requirements is not possible or is not economically justified.

 

In this article we would like to share our vision of how to select the right location of a datacenter and how to design and construct the building. The considerations given below are valid also when, due to whatever reason, a decision to rent, instead of build is taken. Although we don’t recommend it, sometimes renting a premises is the only option.

 

Datacenter location

 

Let's start with selecting datacenter location. The careful selection of a datacenter site location is one of the most important decisions made at the early stage of Data center planning. Please bear in mind that often requirements are contradictive and fulfilling all of them is either impossible or may compromise performance. It is important to point that once selected, the location will stay forever. If all aspects and risks are not properly assessed that may have a long term effect on the datacenter operation. Often wrong locations may show no problems for decades until disaster strikes.  Let’s have a look on the following example:

 

At 5:37 am on 10th of December 2016 a cargo train with propane-butan and propilen tanks, travelling from the town of Burgas to the town of Russe in Bulgaria derailed when entering railway station Hitrino.  A leakage occurred from two of the wagons, which caused a fire. The fire and shock wave, generated during the ignition of the flammable gases, destroyed about 50 surrounding buildings and some people were buried and others suffered severe burns. An evacuation of all inhabitants of the village was done and 150 fire fighters managed to extinguish the fire by noon the same day.