Moore s Law BC 1940 AD Antikythera Mechanism, Babbage Difference Engine. Analog Specialization

Transcription

1

2 Analog Specialization 2000 BC 1940 AD Antikythera Mechanism, Babbage Difference Engine Von Neumann Invention Instruction sets, virtual memory, caches Integration RISC, single-chip CPUs, integrated FPUs, caches Clock Frequency (+ ILP) Deep pipelines, speculation, large caches Moore s Law Multicore to 24 cores, on-chip networks Hardware Specialization ? Programmable logic, rapid ASICs, CGRAs

3

4 FPGAs ASICs Source: Bob Broderson, Berkeley Wireless group

5 Generality Efficiency CPUs 1000x CMPs Manycore GPGPUs ALU arrays ASICs Cloud: Two main challenges for specialization Want homogeneous (to the extent possible) server infrastructure Need five years of stability for ASICs (2 to design, 3 for use), software changes monthly Client: Area is precious, must be both general and efficient Uncanny valley between CPUs and ASICS (where accelerators go to die)

6 5.8+ billion worldwide queries each month 250+ million active users 400+ million active accounts 2.4+ million s per day 8.6+ trillion objects in Microsoft Azure storage 48+ million users in 41 markets 50+ million active users 1 in 4 enterprise customers 50+ billion minutes of connections handled each month 200+ Cloud Services 1+ billion customers 20+ million businesses 90+ markets worldwide

7 Huge infrastructure: Scale is the enabler Microsoft has datacenter capacity around the world and we re growing Quincy Cheyenne Chicago Boydton Des Moines San Antonio Dublin Finland Amsterdam Shanghai Hong Kong Singapore Japan Brazil Australia 1M+ servers Mega, Regional, Edge datacenters Dark fiber network

8 Azure scaling: Exponential growth Compute (VMs) Storage DC Network Capacity

9

10 Ubiquity Efficiency (ASICS)

11 Xeon CPU Search Acc. (FPGA) Search Acc. (ASIC) NIC Xeon CPU Search Acc. v2 (FPGA) Wasted Power, Holds back SW NIC Xeon CPU Math Accelerator Wasted Power, One more thing that can break NIC

12

13 1/2U rack-mounted 1 x 10Ge ports 1 x16 PCIe slot 12 Intel Westmere cores (2 sockets) 13

14

15 FPGA FPGA FPGA FPGA Web Search Pipeline

16 Math Acceleration Service Physics Engine Comp. Vision Service FPGA FPGA Web Search Pipeline FPGA FPGA

17 CS CS ToR ToR ToR ToR

18 Two 8-core Xeon 2.1 GHz CPUs 64 GB DRAM 4 HDDs, 2 SSDs 10 Gb Ethernet No cable attachments to server

19 Altera Stratix V D5 172,600 ALMs, 2,014 M20Ks, 1,590 DSPs PCIe Gen 3 x8 8GB DDR Powered by PCIe slot Torus Network 8GB DDR3 Stratix V PCIe Gen3 x8

20 Data Center Server (1U, ½ width)

21 4 GB DDR ECC SO-DIMM 4 GB DDR ECC SO-DIMM Shell DDR3 Core 0 DDR3 Core 1 Role Config Flash (RSU) JTAG Mb QSPI Config Flash Host CPU 8 x8 PCIe Core DMA Engine Application LEDs Temp Sensors I 2 C Inter-FPGA Router xcvr reconfig SEU North SLIII South SLIII East SLIII West SLIII

22 Front end Bing Pod IFM 0 L0 L0: Candidate finder Find all docs on this machine that contain the query terms Front end sends query that misses in the cache to the TLA, query is processed MLA 0 IFM 1 L1 L1: Fast filter Generate quick scores for each doc from index, choose top 4 TLA MLA 1 IFM 2 L2 (RaaS) FE FFE MLS L2: Expensive ranker Retrieve 4 docs from disk Compute numerical score for each. Milliseconds/doc MLA N IFM 47 IFM sends 4 scores to MLA MLA sends top 100/220 to TLA TLA sorts, generates captions, returns top 10 Microsoft Confidential

23 Selection as a Service (SaaS) Ranking as a Service (RaaS) Query SaaS IFM 1 IFM IFM 1 1 SaaS IFM 2 IFM IFM 2 2 SaaS IFM 3 IFM IFM 3 3 Selected Documents RaaS IFM 1 IFM IFM 11 RaaS IFM 2 IFM IFM 22 RaaS IFM 3 IFM IFM blue links SaaS IFM 48 IFM 44 IFM RaaS IFM 48 IFM 44 IFM Selection-as-a-Service (SaaS) - Find all docs that contain query terms, - Filter and select candidate documents for ranking Ranking-as-a-Service (RaaS) - Compute scores for how relevant each selected document is for the search query - Sort the scores and return the results

24 S1 S0 Send ranked scores for 4 documents back to MLA From L1: query + 4 document IDs Query compilation Read document from disk Docs Decompress and extract HV Hit vector per stream and static features FE: Feature Extraction Dynamic Features FFE: Free-Form Expressions Synthetic Features MLS: Machine learning scoring Position Term NumOccurrences_1_3 = 1 + DF90 NF91 > if SF1 1 / 1e ln + * max + DF SF13 + DF94 + DF93 DF92 DF88 DF89 DF91 2

25 Document {Query, Document} Query: FPGA Configuration NumberOfOccurrences_0 = 7 NumberOfOccurrences_1 = 4 NumberOfTuples_0_1 = 1 ~4K Dynamic Features ~2K Synthetic Features Score L2 Score

26 Document {Query, Document} NumberOfOccurrences_0 = 7 NumberOfOccurrences_1 = 4 NumberOfTuples_0_1 = 1 ~4K Dynamic Features FFE #1 =(2*NumberOfOccurrences_0 + NumberOfOccurrences_1) (2 * NumberOfTuples_0_1) ~2K Synthetic Features FFE #1 = 9 Score L2 Score

27 FE FFE MLS Stream Preprocessin g FSM Feature Transmissi on Network Control/Data Tokens Distribution latches FFE [1][3] FFE [1][2] FFE [1][1] FFE [1][0] FFE [0][3] FFE [0][2] FFE [0][1] FFE [0][0] MLT [3][11] MLT [3][10] MLT [3][9] MLT [3][8] MLT [3][7] MLT [3][6] MLT [3][5] MLT [3][4] MLT [3][3] MLT [3][2] MLT [3][1] MLT [3][0] MLT [2][11] MLT [2][10] MLT [2][9] MLT [2][8] MLT [2][7] MLT [2][6] MLT [2][5] MLT [2][4] MLT [2][3] MLT [2][2] MLT [2][1] MLT [2][0] MLT [1][11] MLT [1][10] MLT [1][9] MLT [1][8] MLT [1][7] MLT [1][6] MLT [1][5] MLT [1][4] MLT [1][3] MLT [1][2] MLT [1][1] MLT [1][0] MLT [0][11] MLT [0][10] MLT [0][9] MLT [0][8] MLT [0][7] MLT [0][6] MLT [0][5] MLT [0][4] MLT [0][3] MLT [0][2] MLT [0][1] MLT [0][0] >100 feature families ~90 State Machines Scheduling Logic Registers Constants FFE 1 Inst. FFE n Inst. Compression Thresholds Local ALU DSP DSP Basic Tile Basic Tile Complex ALU Ln,, div Basic Tile Basic Tile FFE: 64 cores / chip threads MLS: 48 MLT tiles/chip 240 ML processors 2880 ML units/chip

28 PCIe Compressed Document Stream Preprocessing FSM Control/Data Tokens Distribution latches Free Form Expression (FFE) Feature Gathering Network 196 feature families 54 state machines 2.6K dynamic features extracted in less than 4us (~600us in SW)

29 Outpu t Specialized processing engines Each core has a simple ALU (integer, logical, load/store, control flow operations) 4 HW threads, 16 registers per thread. 4kB shared memory Every six cores share a complex ALU Complex ALU performs ln, divide, exp and float to int conversions. Six cores + complex ALU = cluster 8+ clusters (192+ threads) per FPGA 551 synthetic features computed in less than 5us (~50us in SW) Cluster 0 Core 0 Core 1 Core 2 FST Complex Core 3 Core 4 Core 5

30 Document 8-Stage Pipeline FE: Feature Extraction FPGA 0 FPGA 1 Route to Head Route to Head RaaS Servers Server Server FFE: Free-Form Expressions FPGA 2 FPGA 3 FPGA 4 Document Scoring Request Return Score Server Server Server FPGA 5 Document Scoring Request Server Score Compute Score Compute Score FPGA 6 FPGA 7 Return Score Server Server

31 8-Stage Pipeline 8-Stage Pipeline FPGA 0 FPGA 5 FPGA 1 FPGA 6 FPGA 2 FPGA 2 FPGA 3 FPGA 0 FPGA 4 FPGA 1 FPGA 5 FPGA 2 FPGA 6 FPGA 3 FPGA 7 FPGA 4

32

33 1,632 Servers with FPGAs Running Bing Page Ranking Service (~30,000 lines of C++)

34

35

36

37