The modern internet is powered and supported by data centers of various sizes.
Background
The modern internet is powered and supported by data centers of various sizes. The largest data centers, called Tier 4, are warehouse-sized and contain thousands of machines. The power requirements of these data centers are immense, and attempts to achieve optimal Power Usage Effectiveness (PUE) must take into account the absolute need for consistent and reliable power, as well as the huge potential cost of downtime due to outages. In recent years Power Management Systems (PMS) have brought the ability to access real-time temperature and power status information, and to make adjustments as necessary across the entire facility.
A multinational content provider was constructing a massive new data center in Taiwan, and needed to implement an advanced PMS in order to achieve maximum energy efficiency and reliability. The system would manage both primary and backup power sources, and span three large buildings across a large data center campus. For the networking infrastructure that would underpin this important system, the company turned to
EtherWAN.
The challenge
In the PMS application lower level, the Supervisory Control and Data Acquisition (SCADA) system provides real-time monitoring of a vast array of serial power meters and PLCs, and automatically takes action when a warning message is received. At the upper level, PSM servers and Open Platform Communication (OPC) servers use Modbus TCP and SNMP to monitor both the power meters and the status of all Ethernet switches. Reliable communication for this system is paramount, so a hardened solution with dependable network redundancy is required.
Application requirements
- Robust network redundancy with minimized failover time
- Support for SNMP protocol
- Link Fault Pass Through on fiber connections
Solution
The data center campus has three detached buildings – two larger and one smaller. The two larger buildings are equipped with 15 units each of
EX77000 hardened managed gigabit Ethernet switch, connected by Category 5e copper cable. Each switch is connected to a series of PLCs, sending a constant stream of temperature and device data back to the OPC servers, while also relaying instructions from the servers to the PLCs. The EX77000 is hardened against extreme temperatures, and has high immunity to electromagnetic interference, minimizing the danger of device failure due to environmental conditions. Two fiber optic cable links, supported by two EL2211 Series 10/100/1000BASE-TX to 1000BASE-SX/LX/BX media converters, connect the structures and form a large Alpha-ring. Because the media converters support Link Fault Pass Through (LFPT), any failure in a copper link will cause a fail state to be passed across, disabling the fiber link as well as the copper link on the opposite end. This prevents the connected switches from sending packets that would end up lost, and allows the Alpha-ring to instantly (in less than 15 milliseconds) enable the protocol-blocked port, repairing the connection and allowing the resumption of data throughput.
In the third, smaller building, a combination of nine EX77604 and EX73402 managed Ethernet switches are connected to PLCs and arranged in an Alpha-ring. Suitable for sustained operation in areas with extreme temperatures, they are also ISA12.12.01 Class I, Division 2 certified, making them qualified for use in explosive environments.
Conclusion
Power is a critical and capacity-limiting resource in large data centers. To maximize efficiency and power planning, facility-wide power monitoring and management systems are required. These systems not only improve disaster prevention and response capabilities, they also provide detailed power metrics, allowing an organization to improve data center operations and PUE. This kind of PMS requires maximized network availability in both the hardware and networking software aspects. EtherWAN’s proprietary redundancy protocols and technical expertise in large scale projects make the company well positioned to satisfy the requirements of data center facility networks.