Success-Case

Japanese Telco Leader KDDI Invents Immersion Cooling Small Data Center with GIGABYTE

by GIGABYTE
Japanese telco giant KDDI Corporation has invented a new class of data centers that are mobile and eco-friendly. These “container-type immersion cooling small data centers” employ “single-phase immersion cooling” to reduce power consumption by 43% and lower the PUE below 1.07. GIGABYTE Technology drew from its years of experience in the telco sector to provide the R282-Z93 and R182-Z91 Rack Servers for KDDI to use as the management and GPU computing nodes in the data center. KDDI benefits from the servers’ powerful 3rd Gen AMD EPYC™ CPUs, the scalable, high-density configuration of NVIDIA® GPUs in small form factors, and the servers’ optimized compatibility with the liquid-based data center cooling solution. GIGABYTE’s participation in KDDI’s project is in line with GIGABYTE’s long-term CSR and ESG efforts, which are focused on working with global industry leaders to “Upgrade Your Life” with high tech while building a greener, more sustainable environment for our future.
KDDI Corporation, which ranks alongside SoftBank Group Corp. and NTT Docomo Inc. as one of Japan’s top three telecommunications companies, started work in July of 2020 on a new, experimental design for data centers that may set the trend for edge computing and “net zero” green computing. These modular, containerized micro data centers, which KDDI refers to as “container-type immersion cooling small data centers” (コンテナ型液浸スモールデータセンター), are expected to drastically reduce both the carbon footprint and the installation lead time of data centers.

Learn More:
Visit GIGABYTE’s CSR Homepage to See How We are Giving Back to the World!
《Glossary: What is Data Center?
《Glossary: What is Edge Computing?

There are many benefits to such a design. For one thing, because these data centers are portable and convenient to install, they can quickly respond to a burst in demand in a specific location (such as an outdoor music festival—think Coachella) or an emergency situation (such as an earthquake). What’s more, since these data centers are environmentally friendly and sustainable, they can help us build the smart, interconnected world of tomorrow—without having an adverse effect on the environment.

This second point is worth going deeper into. As our lives become more digitized, the international community is building more and more data centers. They are an indelible part of our modern IT infrastructure, because they contribute to every high tech invention you can think of, from extended reality (XR) and generative AI to self-driving cars and space exploration. However, data centers eat up a lot of energy. By some estimates, data centers consume 2% of the world’s electricity and emit roughly as much carbon dioxide (CO2) as the entire aviation industry. How to build new data centers without harming the environment or dumping more carbon into the atmosphere—that is the million-dollar question (or billion-dollar, if you adjust for inflation) that IT experts everywhere are grappling with.

Learn More:
《Glossary: What is IT?
《Glossary: What is Metaverse?
ArchiFiction Achieves Naked-Eye 3D Virtual Reality with GIGABYTE Workstations
GIGABYTE’s ARM Server Boosts Development of Smart Traffic Solution by 200%
Lowell Observatory Looks for Habitable Exoplanets with GIGABYTE Servers

KDDI may have hit upon a solution: a revolutionary method of data center cooling known as “immersion cooling”. Immersion cooling is a liquid-based thermal management system that submerges the servers directly in a bath of dielectric, nonconductive coolant. Since this is more energy-efficient than air cooling, the data center’s PUE, which is its total energy consumption divided by its computing equipment’s energy consumption, can be lowered below 1.07, by KDDI’s estimates. That is a 43% reduction in electricity use when compared with air-cooled data centers, which may have an average PUE of around 1.7.《Glossary: What is PUE?

By adopting immersion cooling in its micro data centers, KDDI hopes to create a win-win situation that fulfills two important aspects of its corporate vision. There are two important reasons why KDDI is exploring possible new designs for data centers. The first is that such a design may contribute to the digital transformation (DX) of Japan. The second is to meet the carbon neutrality goals outlined in "KDDI GREEN PLAN 2030", which is the company’s roadmap to combating climate change, protecting biodiversity, and ushering in the circular economy.

Learn More:
Visit GIGABYTE’s Dedicated Solutions Page to Learn All about Immersion Cooling
How to Pick a Cooling Solution for Your Servers? A Tech Guide by GIGABYTE
KDDI’s “container-type immersion cooling small data center” is a micro data center housed within a 12ft shipping container. The servers within are cooled with the revolutionary “single-phase immersion cooling” technology, which can help the data center reduce power consumption by 43% and lower the PUE below 1.07.
In fact, there are two types of immersion cooling. In the “two-phase” solution, the coolant undergoes a continuous cycle of vaporization and condensation within the sealed tank, and the heat is extracted from the vapors through a condenser at the top of the tank. In the “single-phase” solution, the coolant does not vaporize; instead, it is pumped to a coolant distribution unit (CDU) to remove the heat. GIGABYTE Technology, a leading brand of data center and server solutions, has already built a two-phase immersion cooling solution for a prominent IC foundry giant that leads the world in advanced process technology.

Learn More:
《Press Release: Building a Two-phase Immersion Cooling Solution for IC Foundry Giant, GIGABYTE Is Looking to Set New Standards for Modern Data Center
As Good as Gold: Immersion Cooling Accelerates Oil Extraction

KDDI Corporation opted for single-phase immersion cooling. It invited GIGABYTE Technology to participate in its “container-type immersion cooling small data centers” project. GIGABYTE provided one R282-Z93 and two R182-Z91’s from its state-of-the-art line of Rack Servers to serve as the management and GPU computing nodes. Benefiting from GIGABYTE’s server products and expert knowledge, KDDI’s vision of “carbon neutral digital transformation” may be one step closer to reality.

Learn More:
《More information about GIGABYTE’s Rack Server
《Glossary: What is GPU?
《Glossary: What is Node?
Going Green on the Edge with GIGABYTE’s Industry-Leading Rack Servers
In Mitsubishi Heavy Industries’ Yokohama Hardtech Hub (YHH) in Yokohama, Japan, KDDI erected a 12ft shipping container measuring 3.66 meters in length, 2.44 meters in width, and 2.90 meters in height. Within, KDDI installed a micro-modular, rack-based immersion cooling system from GRC (Green Revolution Cooling) that offers 24 rack units (24U) of space for servers, and can cool up to 50kVA of heat. The coolant system is made by Mitsubishi Heavy Industries. The Japanese system integrator NEC Networks & System Integration Corp. (NESIC) also helped to set up KDDI’s innovative data center.

Learn More:
See It Now! GIGABYTE Provides Single-Phase Immersion Cooling with GRC
Single-phase Immersion Cooling with GIGABYTE
《Glossary: What is Rack Unit?

Here is how single-phase immersion cooling keeps the data center’s temperature in check: during operation, heat generated by the servers is absorbed by the coolant, which is then piped to the CDU installed in an outward-facing closet partitioned into one end of the container. The heat exchanger in the CDU transfers the thermal energy from the coolant to a chilled water loop, which is kept cool by an external radiator. This is more energy-efficient than air or even direct-to-chip (D2C) liquid cooling. By KDDI’s estimates, when the data center is operating at its maximum capacity, the servers will use 50kVA of power, while the data center’s total power consumption will only be around 53.5kVA—therefore lowering the PUE below 1.07.《Glossary: What is Liquid Cooling?
Unlike “two-phase immersion cooling” (which GIGABYTE also provides), “single-phase immersion cooling” cycles the coolant out to a CDU, which transfers the thermal energy to a chilled water loop. KDDI estimates that its prototype data center only consumes around 3.5kVA of energy for cooling, compared to 50kVA for computing.
In addition to being more energy-efficient, there are other advantages, as well. Because immersion cooling can support a higher density of servers, the entire system fits snugly within a 12ft container. Immersion cooling can help to protect the servers from the environment, whether it is high temperatures or salt and dusts in the air. Thanks to the modular design, a number of installation site restrictions can be eliminated; therefore, it is expected that the new data centers will be able to overcome the difficulties of space allocation and material delivery, making them more flexible, mobile, and space-saving.

With the help of GIGABYTE and other suppliers, KDDI may achieve its goal of deploying micro data centers that are not only mobile and eco-friendly, but can also expedite edge computing across Japan. The adoption of immersion cooling is a valuable learning experience for KDDI’s IT experts, and it may become a role model that can be emulated by other ICT companies in Japan—because it represents the best of many important tech trends, such as carbon-neutral computing, edge computing, and small-scale data centers.

“We are impressed that the GIGABYTE servers were able to adapt to our liquid-based cooling system so smoothly,” says Mr. Shintaro Kitayama, Assistant Manager and Server Infrastructure Engineer at KDDI. “GIGABYTE leveraged its past experience in the telco industry to overcome the challenges of immersion cooling, which is very beneficial to our project.”
 
GIGABYTE contributed to KDDI’s vision of a new class of container-type immersion cooling small data centers in two important ways:
1. Suitable server products: the R282-Z93 and R182-Z91 are uniquely suited for the management and GPU computing nodes. This is due to the scalable, highly dense configurations of powerful CPUs and GPUs within easily manageable small form factors.《Glossary: What is Scalability?
2. Expert knowledge: the R282-Z93 and R182-Z91 are designed for air cooling, but GIGABYTE leveraged its years of experience with data center cooling solutions to modify and optimize the hardware and software for single-phase immersion cooling.
Server Products: Powerful Processors in Scalable, Highly Dense Small Form Factors
The single R282-Z93 that GIGABYTE provided for KDDI contains dual AMD EPYC™ 7643 CPUs and triple NVIDIA® A100 Tensor Core GPUs in a 2U chassis, while the two R182-Z91’s contain AMD EPYC™ 7713 CPUs and NVIDIA® T4 Tensor Core GPUs in 1U chassis. Based on GIGABYTE’s years of experience providing enterprise server solutions for the telco sector, this combination of CPUs and GPUs (or more accurately, GPGPUs) in small form factors are eminently suitable for the management and GPU computing nodes in a micro data center.

Learn More:
《GIGABYTE Solutions: NVIDIA A100 Tensor Core GPU Offers Unprecedented Acceleration at Every Scale
《Glossary: What is GPGPU?
GIGABYTE provided the R282-Z93 and R182-Z91 Rack Servers to KDDI, for use as the management and GPU computing nodes in its container-type immersion cooling small data center. The 3rd Gen AMD EPYC™ CPUs and NVIDIA® A100 and T4 GPUs contained within offer KDDI the processing power it needs to offer a variety of services for customers.
The synergy between CPUs and GPGPUs, which is sometimes referred to as heterogeneous computing, is crucial for a data center that supports the public network. Not only are there a large number of users connecting to the data center at all times, the sheer amount of data in graphical form—photos, videos, even data that may be used by computer vision—is staggering. While the 3rd Gen AMD EPYC™ processors already contain an impressive number of cores and threads, it is still necessary to boost CPU performance with the NVIDIA® A100 and T4 GPUs. PCIe Gen 4.0 is used to enable faster data transmission between the processors.

Glossary:
What is Heterogeneous Computing?
What is Computer Vision?
What is Core?
What is Thread?
What is PCIe?

Mr. Kitayama explains, “We thought it necessary to supplement the CPUs with powerful GPUs to provide enough computing power on the data center and the edge site in a highly aggregated state, which will allow us to offer a rich variety of services for our customers. Immersion cooling opens the door to the most advanced GPUs, such as the A100, without encountering limitations in power consumption or cooling.”

The form factors of the R282-Z93 and R182-Z91, which can support a highly dense and scalable configuration of CPUs and GPUs, is another reason why the GIGABYTE servers are suitable for micro data centers. Immersion cooling has made it possible for the telco giant to experiment with high-density computing, which refers to the aggregation of a lot of computing resources within limited space. KDDI is able to reduce the data center’s physical footprint without giving up an iota of computing power, all the while retaining the option to scale out if necessary. The small size of the data center also makes it more convenient to load onto the back of a truck, where it can be carted around Japan to the locations that need it the most.《Glossary: What is Scale Out?
Expert Knowledge: Optimizing the Servers for Immersion Cooling
Before the R282-Z93 and R182-Z91’s were shipped to Yokohama, the GIGABYTE team in Taiwan dedicated weeks of time to modifying and optimizing the servers for immersion cooling. Air-based cooling components, such as the fans, heat sinks, and air shrouds were removed. The servers’ temperature sensors, which were calibrated for the lower ambient temperatures of an air-cooled data center, were also replaced. GIGABYTE was happy to draw upon its experience with data center cooling solutions to help KDDI achieve the vision of a containerized small data center that utilizes immersion cooling.

After the servers arrived in Japan, KDDI conducted a round of operation tests, performance tests, and heat resistance tests to evaluate the cooling effectiveness and structural integrity of the servers. This was done to see if the servers could maintain peak performance without running into issues. The configurable TDP (thermal design power) of the CPUs could go as high as 240W—this is why, in an air-cooled data center, the ambient temperature is usually limited to 30°C or 35°C. However, because immersion cooling is much more efficient at dissipating heat, a much higher ambient temperature may be acceptable. KDDI tested the whole set-up extensively to make sure the micro data center would be ready for field deployment.《Glossary: What is TDP?
Before shipping the servers to Japan, GIGABYTE optimized the R282-Z93 and R182-Z91 for immersion cooling. GIGABYTE’s wealth of experience in the field of data center cooling, coupled with its in-depth understanding of the telco industry, enables it to help KDDI achieve the vision of micro data centers that utilize immersion cooling.
In the course of testing, KDDI spotted other details to consider when adopting immersion cooling. For example, KDDI discovered that ink on labels may smudge when it comes in contact with the oil-based coolant, and this may form a pollutant that may be detrimental to the cooling system. This example proves that there are many things to watch out for when optimizing the servers for immersion cooling. Fortunately, GIGABYTE was glad to share its treasure trove of expert knowledge, as well as its extensive line of server products which are compatible with air, liquid, and immersion cooling, to help KDDI achieve its vision of the container-type immersion cooling small data center.

Learn More:
Waseda University Decodes the Storm with GIGABYTE’s Computing Cluster
Using GIGABYTE, NIPA Cloud Soars Among CSP Giants in Thailand
The Future of Data Centers: Mobile, Eco-Friendly, Immersion Cooling
KDDI Corporation has embarked on a bold venture that may revolutionize the way we design and build data centers. The outlook is optimistic. In April of 2022, KDDI tested the immersion cooling solution in the KDDI Oyama Technical Center (KDDI Oyama TC) in the Tochigi Prefecture, about 90 kilometers north of Tokyo. The system was tested against the “High Availability” standards of Tier 4 data centers, as outlined by ANSI/TIA-942-A. It was also tested to see if immersion cooling may contribute to overcoming the problems of space limitations and heat dissipation and evaluate the operability for construction and maintenance. The hope was that the container-type immersion cooling small data center may eventually see widespread deployment across Japan.

Learn More:
《Glossary: What is High Availability?
《Glossary: What is 5G?
Free Downloadable Tech Guide: How to Build Your Data Center with GIGABYTE?

“We are grateful for GIGABYTE’s active participation as a server manufacturer in our new cooling technology challenge, and for their provision of equipment and technical support,” says KDDI’s Mr. Masato Katou, Expert at Platform Technology Department, DX Solution Engineering Division, “As a manufacturer of IT equipment in the immersion field, KDDI hopes that GIGABYTE will lead the industry in the future with GIGABYTE’s advanced technological capabilities and know-how.”

GIGABYTE Technology is pleased to receive KDDI’s kind words of commendation. GIGABYTE also anticipates the widespread adoption of data center cooling solutions that will help industry leaders reduce carbon emissions, achieve carbon neutrality, and move the world toward a greener and more sustainable “net zero” future. This is in line with GIGABYTE Technology’s long-term CSR, ESG, and SDGs-related efforts, as well as its vision: “Upgrade Your Life”, which is focused on utilizing innovative high tech to deliver better business results while building a smarter, better world of tomorrow.

Learn More:
《More GIGABYTE Tech Guides: What is HPC, and How Can You Stand to Benefit?
Get the inside scoop on the latest tech trends, subscribe today!
Get Updates
Get the inside scoop on the latest tech trends, subscribe today!
Get Updates