Microsoft opens data compression algorithm and hardware for cloud – TechCrunch


The amount of data currently stored by major cloud providers is staggering. It is therefore not surprising that most of them store all this information as compressed data, in one form or another, as you did if you were in zip mode. , CD-ROM and low bandwidth connections. As a rule, these systems are well-kept secrets, but today, Microsoft open source, algorithm, hardware specification and Verilog source code explaining how it compresses the data in its Azure cloud. The company contributes all this to the Open Compute Project (OCP) project.

The Zipline project, as Microsoft calls this project, provides twice the compression ratio of the standard 64 KB Zlib-L4 model. To do this, the algorithm – and its hardware implementation – has been specifically designed for the kind of big datasets that Microsoft sees in its cloud. Because the system works at the system level, there is virtually no overhead, and Microsoft says it is actually able to handle higher throughput rates and lower latency than other algorithms currently allow. reach.

Microsoft points out that it also contributes to Verilog source code for the Registry Transfer Language (RTL), which is the low-level code that makes all this possible. "RTL's contribution to this level of detail as an open source for OCP is a benchmark in the industry," writes Kushagra Vaid, general manager of Azure's hardware infrastructure. "It sets a new precedent for frictionless collaboration in the OCP ecosystem for new technologies and paves the way for silicon hardware innovation."

Microsoft is currently using this system in its own Azure cloud, but is now partnering with other Open Compute project partners. These partners include Intel, AMD, Ampere, Arm, Marvell, SiFive, Broadcom, Fungible, Mellanox, NGD Systems, Pure Storage, Synopsys and Cadence.

"Over time, we expect Project Zipline compression technology to end up in several market segments and usage models such as network data processing, smart SSDs, archiving systems, cloud appliances, general purpose microprocessors, IoT and advanced devices, "Vaid writes.

Source link