Intel ISA-L

Erasure Code and Intel® Intelligent Storage Acceleration Library
好像是新的演算技術搭配 Intel 處理器就可以讓儲存裝置加速.

先來看看 Intel 怎麼介紹 Intel ISA-L !!

擷取影片裡面比較重要的內容來了解什麼是 Intel ISA-L !!
Data replication has become the popular choice to increase availability.But more cost and less capacity!! Wouldn’t it be great if you could increase data availability while reducing capacity?That’s where Erasure Code comes in.To Keep it simple, Erasure Code uses a sophisticated algorithm to deconstruct your data and spread the result across many nodes.The cool thing is that your data can be reconstructure even if many of those nodes go down at the same time.

為了增加儲存裝置的可靠度,通常會將資料數據複製成多份並儲存在不同的儲存結點上(如 RAID).但缺點是 IT 的花費會增加,而且相對可使用的容量也會跟著減少!! 如果有技術可以減少浪費儲存容量並增加資料的可靠度呢? Erasure Code 就可以做到.簡單的來說.Erasure Code 使用複雜的演算,重新解構您的資料數據並把結果分佈在多個節點上.最棒的是資料數據被重整過,所以就算多個儲存結點失效你的資料數據還是可以做存取.

But the cools thing about Erasure Code is that it requires only half the space of Triple Replication.While providing up to eleven nines (99.999999999%)of availability.eleven nines (99.999999999%) : Your data would be unavailable less than one second every 3,000 years!!

採用 Erasure Code 的儲存空間是 3倍資料複製技術所需的儲存空間的一半而已.而且可靠度高達 (99.999999999%),什麼是 11個9 (99.999999999%),以可靠度時間來算也就是 3,000年中只有一秒的時間會讓你的資料數據無法使用而已.

Simple: the performance penalty.Traditional processors used for storage just don’t have enough horsepower to perform the complex computations.We’ve designed new Intel architecture solutions for storage.We’ve written an optimized . Intel Intelligent storage acceleration library for those solutions.Enjoy roughly 10x more performance than traditional lookup table code.

那以前怎麼不用呢!!!
當然沒完美的世界,想要高效能是需要代價的.傳統的處理器沒有能力處理太過複雜的資料數據運算.但新設計的 Intel 處理器架構可以幫你解決儲存上的困境.我們重寫了一個優化版本的 Intel Intelligent storage acceleration library 來解決以上的問題.相較於傳統 lookup table code 的資料數據儲存, Intel 的 ISA-L 效能快上 10 倍.

看完簡介,現在可以開始做了,方式參考 ISA-L 的網站說明

https://01.org/sites/default/files/documentation/isa-l_open_src_2.8_0.pdf

硬體系統需求

• em64t: A system based on the Intel® Xeon® processor with Intel® 64 architecture.
• IA32: When available for 32-bit functions; A system based on the Intel® Xeon® processor or subsequent IA-32
  architecture based processor.

軟體需求:
Yasm Assembler: version at least v1.2.0.
Yasm is a complete rewrite of the NASM assembler. It supports multiple assembler syntaxes.

root@ben-virtual-machine:~# apt-get install yasm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  xinetd
Use 'apt-get autoremove' to remove it.
The following NEW packages will be installed:
  yasm
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 644 kB of archives.
After this operation, 1,692 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ raring/main yasm amd64 1.2.0-1 [644 kB]
Fetched 644 kB in 11s (53.9 kB/s)
Selecting previously unselected package yasm.
(Reading database ... 189054 files and directories currently installed.)
Unpacking yasm (from .../yasm_1.2.0-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up yasm (1.2.0-1) ...

開始編譯 ISA-L Erasure Code (請自行檢查 ISA-L 的最新版本 https://01.org/zh/storage-acceleration-library/downloads )

root@ben-virtual-machine:~# wget https://01.org/sites/default/files/downloads/storage-acceleration-library/isa-l_open_src_2.8.tar.gz
--2014-04-09 05:17:22--  https://01.org/sites/default/files/downloads/storage-acceleration-library/isa-l_open_src_2.8.tar.gz
Resolving 01.org (01.org)... 198.145.11.105, 2001:19d0:3:5::105
Connecting to 01.org (01.org)|198.145.11.105|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 326050 (318K) [application/x-gzip]
Saving to: ‘isa-l_open_src_2.8.tar.gz’

100%[======================================>] 326,050     72.2KB/s   in 4.4s

2014-04-09 05:17:29 (72.2 KB/s) - ‘isa-l_open_src_2.8.tar.gz’ saved [326050/326050]

root@ben-virtual-machine:~# tar zxvf

root@ben-virtual-machine:~# cd isa-l_open_src_2.8/

root@ben-virtual-machine:~/isa-l_open_src_2.8# make
mkdir -p bin
  ---> Building erasure_code/ec_highlevel_func.c
  ---> Building erasure_code/ec_base.c
  ---> Building erasure_code/gf_vect_mul_sse.asm
  ---> Building erasure_code/gf_vect_mul_avx.asm
  ---> Building erasure_code/gf_vect_dot_prod_sse.asm
  ---> Building erasure_code/gf_2vect_dot_prod_sse.asm
  ---> Building erasure_code/gf_3vect_dot_prod_sse.asm
  ---> Building erasure_code/gf_4vect_dot_prod_sse.asm
  ---> Building erasure_code/gf_5vect_dot_prod_sse.asm
  ---> Building erasure_code/gf_6vect_dot_prod_sse.asm
  ---> Building erasure_code/ec_multibinary.asm
  ---> Creating Lib bin/isa-l.a

接下來我就不懂要做什麼了!!!

好像是可以應用在 Hadoop 的 HDFS 上.