P4080DS U-boot & Linux

Transcription

1 June 22, 2010 P4080DS U-boot & Linux FTF-NET-F0683 Srikanth Srinivasan & Ted Peters NMG Systems and Applications Engineering Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

2 Session Introduction Board-bring up is a key activity for new silicon. Firmware and OS are the first software components run on the reference platforms. Initial bring-up involves debug of silicon, board and the software. This presentation details the important parts of firmware (u-boot) and OS (Linux) porting/bring-up for P4080DS reference platform. Freescale s Board Support Package (BSP) provides software that can be modified to run on the customer development systems. The talk will drive home the strategies, steps, concepts, and highlight relevant parts of the code needed most commonly during this effort. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 2

3 Learn more about the Freescale BSP for P4080/P4080DS Details of u-boot and Linux porting, and hypervisor concepts. Session Objectives Obtain and build u-boot and Linux from source Peruse source trees for making changes specific to your platforms Identify separation of tasks between SW components Identify key components of the reference platform as well as the software Learn multi-core boot sequence Hypervisor capabilities and use Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 3

4 Agenda Introduction 5 min U-boot 30 min Hypervisor 25 min Linux 40 min LWE 5 min Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 4

5 P4080 Block Diagram P Kbyte Backside L2 Cache Power Architecture e500-mc Core 1024-Kbyte Frontside L3 Cache 32-Kbyte 32-Kbyte 1024-Kbyte D-Cache I-Cache Frontside L3 Cache 64-bit DDR-2 / 3 Memory Controller 64-bit DDR-2 / 3 Memory Controller eopenpic PreBoot Loader Security Monitor Internal BootROM PAMU PAMU PAMU CoreNet Coherency Fabric PAMU PAMU Peripheral Access Mgmt Unit Power Mgmt SD/MMC SPI elbiu M2SB Security 4.0 Queue Mgr. Frame Manager Parse, Classify, Distribute Frame Manager Parse, Classify, Distribute SRIO Message Unit DMA Real Time Debug Watchpoint Cross Trigger DUART 2x I 2C 2x USB 2.0/ULPI Test Port/ SAP Pattern Match Engine 2.0 Buffer Mgr. 10GE Buffer 1GE 1GE 1GE 1GE 10GE Buffer 1GE 1GE 1GE 1GE PCIe PCIe SRIO PCIe SRIO Perf Monitor CoreNet Trace Aurora Clocks/Reset GPIO CCSR 18-Lane 5GHz SERDES Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 5

6 EC1 USB1 EC2 FM1[dTSEC2] 10/100/1G RGMII SerDes Lanes [0-17]: Slot 1: Bank1 A-B (PEX x2) Slot 2: Bank1 C-D (PEX x2) Slot 3: Bank1 E-H (4x SGMII x1, PEX x4, SRIO x4) : Bank1 I-J (Aurora x2 only) Slot 4: Bank 2 A-D (4x SGMII x1 or XAUI x4) Slot 5: Bank 3 A-D (4x SGMII x1 or XAUI x4) u-boot RCW[srds_prtcl] = 0x10 (RSX configuration) PCIe1 (5/2.5Gb) PCIe3 (5/2.5Gb) 4x SGMII (1) Bank 1 Bank 2 Bank 3 FM2 dtsec[1:4] Debug (5/2.5Gb) XAUI FM2 10GEC 4x SGMII FM1 dtsec[1:4] Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 6

7 Getting Started Master u-boot repository Master linux repository BSP release Follow instructions in the BSP documentation to extract software packages Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 7

8 Board Support Package (BSP) GNU Toolchain P4080 SDK Beta Components gcc-4.3.2, eglibc2.8, binutils 2.18, CodeSourcery release U-Boot Boot Loader Linux 2.6 kernel Linux device drivers with P4080 Datapath Acceleration Architecture (DPAA) support Embedded Hypervisor Software Light Weight Executive (LWE) LWE Libraries (LWE Libs) LWE Example Applications (LWE Apps) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 8

9 System requirements x86 linux with cross-tools Extracting Sources And Building Images LTIB $ mount o loop <BSP.iso> /mnt/p4080ds-ltib $ cd my-ltib-dir Local directory $ /mnt/p4080ds-ltib/install From the installed directory : $./ltib m prep p u-boot (Sources available in rpm/build/u-boot) $./ltib m scbuild p u-boot To configure packages : $./ltib --configure Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 9

10 git Version control done through git SCM of choice for many open-source projects including Linux Clone the tree locally git clone git://git.denx.de/u-boot.git Replicates the remote tree with change logs and tags When ready to contribute 1. Commit changes locally 2. Create a patch and mail it to the u-boot list 3. Peer review 4. Patch applied to master tree for everyone to use 5. Clean up patch for external world and repeat steps 2-4 A few keepers for the master tree Ensures safety of code base while still allowing group contribution and higher software quality Very convenient for large distributed projects Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 10

11 Summary Of Boot Flow From System-Wide Reset uboot Load hypervisor image and at least one guest OS image. Load device trees for guests. Hypervisor Hypervisor initializes and then starts one or more guests. It passes them a device tree from which they get information on resources allocated to them (and more). Guest OS One guest is designated to do global initialization of BMan, QMan, and FMan since they are used by many guests. Other guests await this initialization. Guests can bring up their network interfaces. A guest can be authorized to act as a partition manager. It can load and run more guests or reboot them. Example: Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 11

12 P4080DS u-boot Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 12

13 Introduction What is u-boot? Open source firmware for hardware platforms A boot loader widely used in embedded space What does it do? Brings the board to a sane state out of reset Load OS image onto board and start OS Origins DENX Software Engineering More at Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 13

14 U-boot Directories./u-boot board common cpu disk drivers include lib_* net doc dtt examples fs rtc tools post Relevant directories for modifications u-boot/. u-boot/board u-boot/lib_ppc u-boot/include u-boot/cpu u-boot/common Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 14

15 Boot Sequence Reset vector points to FLASH (0xfffffffc) Begins execution in FLASH Initializes DUARTs Checks for information on version numbers, clocks and displays on screen Initializes DDR Relocates itself to RAM Continues execution in RAM Releases other cores Initializes higher level devices ethernet, PCIe, SATA, USB, etc. Begins command interpreter Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 15

16 U-boot Memory Map for P4080DS /* * Memory map * * 0x0000_0000 0x7fff_ffff DDR 2G Cacheable * * 0x8000_0000 0x9fff_ffff PEX 1 Mem 512M non-cacheable * 0xa000_0000 0xbfff_ffff PEX 2/SRIO Mem 512M non-cacheable * 0xc000_0000 0xdfff_ffff PEX 3 Mem 512M non-cacheable * * 0xf800_0000 0xf802_ffff PCI IO range 256K non-cacheable * 0xf400_0000 0xf41f_ffff Bman 2M * 0xf420_0000 0xf43f_ffff Qman 2M * * Localbus non-cacheable * * 0xe000_0000 0xe80f_ffff Promjet/free 128M non-cacheable * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * * 0xffd0_0000 0xffd0_3fff L1 for stack 16K Cacheable TLB0 * 0xfe00_0000 0xfeff_ffff CCSR 16M non-cacheable */ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 16

17 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init/mp init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 17

18 Stack In Cache Drivers written in C code Need stack space to run Lock the cache Unlock before linux boot * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * * 0xffd0_0000 0xffd0_3fff L1 for stack 16K Cacheable TLB0 * 0xffe0_0000 0xffef_ffff CCSR 1M non-cacheable Files: cpu/mpc85xx/start.s Code Snippets: lis r6,fsl_booke_mas0(1, 14, 0)@h ori r6,r6,fsl_booke_mas0(1, 14, 0)@l lis r7,fsl_booke_mas1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@h ori r7,r7,fsl_booke_mas1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@l lis r8,fsl_booke_mas2(config_sys_init_ram_addr, 0)@h ori r8,r8,fsl_booke_mas2(config_sys_init_ram_addr, 0)@l lis r9,fsl_booke_mas3(config_sys_init_ram_addr, 0, (MAS3_SX MAS3_SW MAS3_SR))@h ori r9,r9,fsl_booke_mas3(config_sys_init_ram_addr, 0, (MAS3_SX MAS3_SW MAS3_SR))@l : : tlbwe Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 18

19 MMU, LAW Setup Configuring memory map Files: include/configs/p4080ds.h, corenet_ds.h board/freescale/corenet_ds/tlb.c, law.c cpu/mpc85xx/tlb.c drivers/misc/fsl_law.c include/asm-ppc/immap_85xx.h Code Snippets: /* *I*G* - Flash, localbus */ /* This will be changed to *I*G* after relocation to RAM. */ SET_TLB_ENTRY(1, CONFIG_SYS_FLASH_BASE, CONFIG_SYS_FLASH_BASE_PHYS, MAS3_SX MAS3_SR, MAS2_W MAS2_G, 0, 2, BOOKE_PAGESZ_256M, 1), SET_LAW(CONFIG_SYS_FLASH_BASE_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_LBC), Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 19

20 SYSCLK SYSCLK value needed to get started Platform frequency FPGA registers SCLK0, 1, 2 Input to ICS307 clock generator code Obtain the board and ddr clocks Files: include/configs/corenet_ds.h board/freescale/corenet_ds/corenet_ds.c Code Snippets: /* Calculate frequency being generated by ICS clock chip based upon * the control bytes being programmed into it. */ static unsigned long ics307_clk_freq(unsigned char cw0, unsigned char cw1, unsigned char cw2) { const unsigned long InputFrequency = CONFIG_ICS307_REFCLK_HZ; unsigned long VDW = ((cw1 << 1) & 0x1FE) + ((cw2 >> 7) & 1); : : : } unsigned long calculate_board_sys_clk(ulong dummy) { ulong val; u8 *pixis_base = (u8 *)PIXIS_BASE; val = ics307_clk_freq( in_8(pixis_base + PIXIS_VSYSCLK0), Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 20

21 2 DUARTs ( 4 UARTs) on P serial outputs on DS u-boot provides driver for NS16550 compatible DUARTs Needs platform frequency Serial port very useful for early debug Serial Files: include/configs/corenet_ds.h drivers/serial/ns16550.c, serial.c Code Snippets: #define CONFIG_CONS_INDEX 2 #undef CONFIG_SERIAL_SOFTWARE_FIFO #define CONFIG_SYS_NS16550 #define CONFIG_SYS_NS16550_SERIAL #define CONFIG_SYS_NS16550_REG_SIZE 1 #define CONFIG_SYS_NS16550_CLK (get_bus_freq(0)/2) #define CONFIG_SYS_NS16550_COM1 (CONFIG_SYS_CCSRBAR+0x11C500) #define CONFIG_SYS_NS16550_COM2 (CONFIG_SYS_CCSRBAR+0x11C600) #define CONFIG_SYS_NS16550_COM3 (CONFIG_SYS_CCSRBAR+0x11D500) #define CONFIG_SYS_NS16550_COM4 (CONFIG_SYS_CCSRBAR+0x11D600) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 21

22 DDR Setup Relocating code Supports multiple controllers and SPD-based (I2C eeprom) initialization Supports Interleaving Files: cpu/mpc8xxx/ddr include/configs/corenet_ds.h board/freescale/corenet_ds/ddr.c board specific parameters Memory Controller 1 Memory Controller 2 CS0 CS1 CS0 CS1 Code Snippets: #define CONFIG_NUM_DDR_CONTROLLERS 2 #define CONFIG_DIMM_SLOTS_PER_CTLR 1 #define CONFIG_CHIP_SELECTS_PER_CTRL (2 * CONFIG_DIMM_SLOTS_PER_CTLR) #define CONFIG_DDR_SPD #define CONFIG_FSL_DDR3 #define CONFIG_SYS_SPD_BUS_NUM 1 #define SPD_EEPROM_ADDRESS1 0x51 #define SPD_EEPROM_ADDRESS2 0x52 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 22

23 Misc CPU Init Set up local bus memory map BRn, ORn Enable timebase register Files: cpu/mpc85xx/cpu_init.c Code Snippets: void cpu_init_f (void) { volatile ccsr_lbc_t *memctl = (void *)(CONFIG_SYS_MPC85xx_LBC_ADDR); : : #ifdef CONFIG_FSL_CORENET corenet_tb_init(); #endif init_used_tlb_cams(); /* Invalidate the CPC before DDR gets enabled */ invalidate_cpc(); } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 23

24 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 24

25 Relocates u-boot code to top of memory For 2GB on P4080DS, in the last 1MB Set up of stack space, etc. Relocate To DDR Space Continues to run from DDR after relocation Files: cpu/mpc85xx/start.s lib_ppc/board.c Code Snippets: /* * void relocate_code (addr_sp, gd, addr_moni) * * This "function" does not return, instead it continues in RAM * after relocating the monitor code. * * r3 = dest * r4 = src * r5 = length in bytes * r6 = cachelinesize */.globl relocate_code relocate_code: mr r1,r3 /* Set new stack pointer */ mr r9,r4 /* Save copy of Init Data pointer */ mr r10,r5 /* Save copy of Destination Address */ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 25

26 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 26

27 CFI flash support Reads on POR Sets up structures much later in the driver NOR Flash Files: * Localbus non-cacheable * * 0xe000_0000 0xe80f_ffff Promjet/free 128M non-cacheable * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * Code Snippets: include/configs/corenet_ds.h drivers/mtd/cfi_flash.c This information is used by the driver to detect the flash using CFI query at both possible addresses and populate its info table #define CONFIG_FLASH_CFI_DRIVER #define CONFIG_SYS_FLASH_CFI #define CONFIG_SYS_FLASH_AMD_CHECK_DQ7 #define CONFIG_SYS_FLASH_BANKS_LIST {CONFIG_SYS_FLASH_BASE_PHYS + 0x , CONFIG_SYS_FLASH_BASE_PHYS} /* Protect RCW and FMAN microcode sectors for the NOR flash boot bank */ #define CONFIG_SYS_FLASH_AUTOPROTECT_LIST { {CONFIG_SYS_FLASH_BASE + 0x , 1}, \ {CONFIG_SYS_FMAN_FW_ADDR, 1} } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 27

28 PCIe Check for PCIe interfaces on the SOC Basic PCIe enumeration Useful when pcie devices are needed at boot time. e.g. RTC/NVRAM on nvidia 1575 /* * 0x8000_0000 0x9fff_ffff PEX 1 Mem 512M non-cacheable * 0xa000_0000 0xbfff_ffff PEX 2 Mem 512M non-cacheable * 0xc000_0000 0xdfff_ffff PEX 3 Mem 512M non-cacheable Files: include/configs/corenet_ds.h drivers/pci/fsl_pci_init.c board/freescale/corenet_ds/corenet_ds.c Code Snippets: void fsl_pci_init(struct pci_controller *hose) { : : } #define CONFIG_SYS_PCIE1_ADDR (CONFIG_SYS_CCSRBAR+0x200000) #define CONFIG_SYS_PCIE2_ADDR (CONFIG_SYS_CCSRBAR+0x201000) #define CONFIG_SYS_PCIE3_ADDR (CONFIG_SYS_CCSRBAR+0x202000) : : #define CONFIG_SYS_PCIE3_MEM_PHYS 0xc #define CONFIG_SYS_PCIE3_IO_PHYS 0xf Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 28

29 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 29

30 Power-on Reset Misc Init/MP Init Core0 comes out of reset at the reset vector 0xFFFF_FFFC and all other cores are in boot hold off mode Core0 runs u-boot After setting BSTRL/BSTAR to secondary core s start page, core0 kicks off each additional secondary core Each secondary core comes out of reset at secondary_start_page(boot page), initialize resources specific that that core (caches, MMU, etc.) and enters a spin loop Core0 loads Linux image and device tree and boots Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 30

31 Multicore Boot Architecture epapr describes specifics on how secondary CPUs are booted for a system with multiple CPUs Default boot architecture The boot program releases all CPUs from hardware reset One CPU is designated to be the client program s boot CPU All other CPUs are secondary and are placed into loop where the CPUs spin, waiting for a spin table field to change that directs them where to go Control is transferred to the client program on the boot CPU When the client program is ready for secondary cores to start, it releases them by writing the spin table field with the desired address The architecture allows for other custom-defined secondary CPU release mechanisms as well Reference: epapr (Power.org Standard for Embedded Power Architecture Platform Requirements (epapr). power.org, 2008.) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 31

32 SMP Boot Process: U-boot Boot Process In the u-boot source, the entry point is the reset vector 0x0_FFFF_FFFC containing a simple branch to _start_e500, which is at the base of default boot page at 0x0_FFFF_F000; both are found in cpu/mpc85xx/start.s Reset vector call _start_e500() _start_e500() will: _start_e500() Boot other cores Start command interpreter Prepare device trees 1. Enable L1 caches 2. Config interrupt vectors 3. Config MMU and LAWs 4. Config L1D RAM 5. Config local bus 6. Config DDR 7. Relocate to DDR 1. Setup BSTRL and BSTAR 2. Release all other cores from boot holdoff using BRR and wait for them to boot. cpu/mpc85xx/mp.c 1. Copy HW device tree to DDR 2. Copy HV config tree to DDR 3. Copy hv.uimage to DDR 4. Launch hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 32

33 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 33

34 FM Init Rx Data MAC (RX data) FM BMI (put MAC data in internal buffers) RISC (generate DMA to write data to BDs) DMA (write data) Enable FM BMI (allocate internal context buffers) RISC (generate DMA to read data from BDs) DMA (read data) BMI (read data) FMan Independent mode to get basic networking for tftp downloads Needs configuration data Configuration data resides at 0xEF FM BMI (free internal buffers) MAC (transmit TX data) END Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 34

35 FM Init Independent mode Files: include/configs/corenet_ds.h drivers/net/fm/* board/freescale/corenet_ds/corenet_ds.c Code Snippets: int board_eth_init(bd_t *bis) { : : p4080_setup_serdes(); /* FM 1 DTSEC */ if (is_pcie_configured(prtcl, SGMII_FM1)) { printf("fm1 SGMII\n"); #ifdef CONFIG_FM1_ETH0 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 0) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH1 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 1) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH2 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 2) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH3 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 3) fm1_idx++; #endif : : Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 35

36 Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 36

37 Other New Pieces BMAN, QMAN, FMAN LIODNs Device tree fixups for HV/linux cpu/mpc85xx/portals.c, liodn.c CoreNet Platform Cache Enabled by a flag in include/configs/corenet_ds.h (CONFIG_SYS_FSL_CPC) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 37

38 Building u-boot Using LTIB $./ltib m scbuild p u-boot From command line Set the compiler toolchain to compile u-boot Set PATH variable on the build machine Use ARCH=ppc and CROSS_COMPILE flags make CROSS_COMPILE=powerpc-linux-gnumake CROSS_COMPILE=powerpc-linux-gnumake CROSS_COMPILE=powerpc-linux-gnu- ARCH=ppc distclean ARCH=ppc P4080DS_config ARCH=ppc u-boot.bin Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 38

39 P4080DS NOR Flash Map Cheat Sheet (from Bank 0 perspective after U-Boot DDR relocation) 0xE xE8FFFFFF Bank 0 PBL Image Bank 7 SW7[1:4] = 0b0111 U-Boot 0xE xE xE9FFFFFF Bank 1 PBL Image Bank 6 SW7[1:4] =0b0110 0xEA xEAFFFFFF BANK 2 PBL Image Bank 5 SW7[1:4] = 0b0101 0xEB xEBFFFFFF Bank 3 PBL Image Bank 4 SW7[1:4] = 0b0100 0xEC xECFFFFFF Bank 4 PBL Image Bank 3 SW7[1:4] = 0b0011 0xED xEDFFFFFF Bank 5 PBL Image Bank 2 SW7[1:4] = 0b0010 0xEE xEEFFFFFF Bank 6 PBL Image Bank 1 SW7[1:4] = 0b0001 0xEF xEFFFFFFF Bank 7 PBL Image Bank 0 SW7[1:4] = -0b0000 0xEFF80000 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 39

40 Common (Helpful) Commands help Environment variables printenv setenv saveenv NOR Flash flinfo erase <start addr> <end addr> cp.b <from addr > <to addr in flash> <#bytes> Network setenv [ipaddr][gatewayip][netmask][serverip] <ip address> tftp <dest address in RAM> <file on server> Other imls - Lists all u-boot recognizable images in flash i2c - i2c commands to display, read, write i2c devices pci - display, read, write PCI/PCIe space Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 40

41 36-bit addressing U-boot Hooks To Launch Hypervisor U-boot uses 36-bit addressing but is limited to 4GB HV uses 36-bit addressing to support multiple partitions and resources Physical addresses of the IO are moved to the end of the 64GB address space DDR address space starts at 0; now has room to expand New commands Device trees for HW and configuration Pass HW device tree address to bootm command Location of configuration device tree is passed config-addr env variable => setenv bootcmd bootm e e => setenv bootargs config-addr=0xfe Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 41

42 P4080DS Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 42

43 SMP Symmetric Multiprocessing ASMP Asymmetric Multiprocessing (aka. AMP) CAMP Cooperative AMP Terminology Guest OS OS running in guest supervisor state under a hypervisor Hypervisor - Manages globally shared resources, more privileged than operating systems, enforces system security, virtualizes some resources Light Weight Executive (LWE) - Provides a communications mechanism between an application running in a partition and the hypervisor software Device Tree - A data structure used for representing a partition s physical and virtual devices Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 43

44 Partitioning With A Hypervisor Hypervisor Software Analogous to role of an operating system kernel in managing user processes More privileged than operating systems Enforces system security Manages globally shared resources Virtualizes some resources e.g. interrupt controller, UART Hypervisor partition App App partition App App partition App App Linux RTOS Legacy OS Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 44

45 e500mc Privilege Levels Guest State (GS) MSR bit Under Hypervisor Bare Metal partition partition User MSR[PR=1][GS=1] App App User MSR[PR=1][GS=0] App App Kernel/Supervisor MSR[PR=0][GS=1] OS Kernel/Supervisor MSR[PR=0][GS=0] OS Hypervisor MSR[PR=0][GS=0] Hypervisor CPU CPU Memory Memory I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 45

46 Operating System sees a virtual core plus hypervisor services Hypervisor Features Virtual CPU (like e500mc minus hypervisor features) Services via hypercall Emulation (privileged instructions) guest operating system hypercalls device tree Debug stub interface for debugging guest operating systems virtual CPU (e500vcpu) Hypervisor services PIC Partition Mgmt Doorbells GPIO IOMMU boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 46

47 Virtual CPU Like e500mc minus the hypervisor extensions Full emulation is used MMU tlb access instructions are emulated Guest receive and manage TLB miss interrupts directly Timer DEC, FIT, watchdog, timebase synchronization Cache control L1,L2 Intra-partition signaling (msgsnd/msgclr) Debug services (DBCRx, IACx, DACx, etc) Core performance monitor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 47

48 Virtual CPU Emulation Example Operating system manages it s own page tables. Hypervisor has no knowledge of OS page table format. Hypervisor traps and authenticates all TLB updates Applicati on lwz r3,(r4) tlb miss OS (tlb miss handler) tlbwe privilege trap Hypervisor Hypervisor validate tlb writes Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 48

49 Hypervisor Services hcalls Interrupt controller (MPIC) Byte-channels character I/O stream Inter-partition signaling doorbell Partition management Start/stop/image-loading In future: Create/destroy Partition management interrupts Power management change clock frequency, power states GPIO supports partitioning of GPIO pins IOMMU supports create/destroy mappings Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 49

50 Device Trees Guest #1 Guest #2 Guest Dev tree HW Dev tree hypervisor u-boot Guest Dev tree HV config tree dynamically created and loaded into guest memory Loaded into hypervisor memory by u- boot. Hardware dev tree /chosen node points to HV config tree Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 50

51 Boot Process Hypervisor initialization PAMU initialization Coherence domain setup set up LAWS, CSDIDs Error Configuration e.g. single bit ECC error thresholds Driver initialization DDR CPC CCM UART Release secondary CPUs Partition instantiation/creation The boot CPU for each partition takes care of partition creation Control transfer to guest OS Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 51

52 HV Commands Commands list and uses HV>? Commands: help (?) - Print command usage information version - Print the hypervisor version list-partitions (lp) - List partitions guest-device-tree (gdt) - Guest device tree operation master-device-tree (mdt) - Display master device tree paact - Dump PAMU's PAACT table entries start - Start a partition restart - Start a partition stop - Stop a partition pause - Stop a partition resume - Stop a partition HV> Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 52

53 P4080DS Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 53

54 Overview P4080 Linux Overview P4080 BSP Linux Bare Metal and Guest OS Linux Kernel Configuration Device Tree SMP Boot Process DPAA and the Linux Ethernet Driver DPAA Components Life of a Packet Ingress Egress Linux Driver Structure Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 54

55 P4080 Linux Bare metal Linux Linux runs directly on HW User state MSR[PR]=1 MSR[GS]=0 Supervisor state MSR[PR]=0 MSR[GS]=0 Guest OS Linux Hypervisor SW layer between Linux and HW Guest User state MSR[PR]=1 MSR[GS]=1 Guest Supervisor state MSR[PR]=0 MSR[GS]=1 Hypervisor state MSR[PR]=0 MSR[GS]=0 Guest OS requires porting to run on Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 55

56 Bare Metal SMP Linux App App SMP Linux Multicore System Hardware Shared Cache I/O Interrupt Controller CPU CPU CPU CPU Memory I/O I/O I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 56

57 Guest OS Linux partition partition partition App App App App App App SMP Linux RTOS Legacy OS Hypervisor Multicore System Hardware Shared Cache I/O Interrupt Controller CPU CPU CPU CPU Memory Memory Memory I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 57

58 Resources controlled by Hypervisor: Physical memory map Porting Guest OS (Linux) to Hypervisor Core: caches, MMU, SPRs (Guest OS sees a virtual CPU via emulation) Platform: CPC, MPIC, Partion management, Doorbell, Byte-channels, IOMMU, GUTs, DDR (Guest OS access via HCALLs) Direct resources (Direct I/O access by Guest OS) Device Tree aware (per epapr, libfdt library available) State of HW when Guest OS Runs (per epapr) Hypercall API (HCALLs for shared services, ie VMPIC) Traps (emulation - vcpu) Reference: epapr (Power.org Standard for Embedded Power Architecture Platform Requirements (epapr). power.org, 2008.) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 58

59 Building Linux LTIB BSP Configure default LTIB settings $./ltib Configure menu for P4080DS, then automatically build $./ltib --configure In configure menu, select: Select Configure kernel to alter the default kernel build configuration Select Leave the sources after building to leave the linux source in rpm/build/linux From GIT tree make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc menuconfig make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc uimage Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 59

60 P4080 Linux Kernel Configuration The easiest way to create a.config file is to simply copy the appropriate config file in./arch/powerpc/configs to.config. cp arch/powerpc/configs/p4080_ds_defconfig.config Typing make menuconfig results in a console-based, menu-driven utility that can be used to specify the desired configuration. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 60

61 Linux Directory Structure./linux-2.6.xx Doc. arch block crypto drivers fs include init kernel lib mm scripts security sound ipc net usr arch/powerpc/boot/dts/p4080ds.dts arch/powerpc/configs/p4080_ds_defconfig arch/powerpc/boot/ arch/powerpc/platforms/85xx/ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 61

62 P4080 Linux Device Tree Baremetal Linux Use the device tree compiler to build the appropriate dts file in the./arch/powerpc/boot/dts directory. dtc -I dts -O dtb -R 8 -b 0 -S 0x3000 p4080ds.dts > p4080ds.dtb OR make p4080ds.dtb (from linux-2.6/) Guest OS Linux The device tree for each partition is generated by Hypervisor from a system configuration file that maps the system partitioning and HW. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 62

63 Kernel Configuration Options - Hypervisor CONFIG_P4080_HV - Freescale P4080 Hypervisor platform CONFIG_FSL_HYPERVISOR - Freescale P4080 Hypervisor CONFIG_VMPIC - Freescale P4080 Hypervisor - Virtualized OpenMPIC driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 63

64 SMP Boot Process: Linux SMP Boot Process In the Linux kernel, the entry point is _start found in arch/powerpc/kernel/head_fsl_booke.s _start start _kernel() is in init/main.c, it will: _start_kernel()... setup_arch() call _start_kernel() mpc85xx_smp_init() { smp_ops = &smp_85xx_ops; } Kernel_init() smp_prepare_cpu() smp_ops->probe() in main.c smp_init() cpu_up() smp_cpus_done() smp_ops->kick_cpu() smp_ops->setup_cpu() Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 64

65 SMP Boot Process: Core1 Start Up (continued) Let s look at smp_mpc85xx_kick_cpu() found in: arch/powerpc/platforms/85xx/mpc85xx_smp.c smp_mpc85xx_kick_cpu() {. /* Get the BPTR */ bptr_vaddr = ioremap(get_immrbase() + MPC85xx_BPTR_OFFSET, 4); /* Set the BPTR to the secondary boot page */ oldbptr = in_be32(bptr_vaddr); bptr = (BPTR_EN ( pa((unsigned) secondary_start_page) >> 12)); out_be32(bptr_vaddr, bptr); /* Kick that CPU */ smp_85xx_release_core(nr); /* Wait a bit for the CPU to take the exception. */ while (( secondary_hold_acknowledge!= nr) && (++n < 1000)) mdelay(1); /* Restore the BPTR */ out_be32(bptr_vaddr, oldbptr); } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 65

66 SMP Boot Process: Core1 Kick-Off Process Core 0 kicks off core 1 by setting EEBPCR[CPU1_EN] Let s look at smp_85xx_release_core() in arch/powerpc/mpc85xx/mpc85xx_smp.c smp_85xx_release_core(int nr) { /* * Startup Core #nr. */ ecm_vaddr = ioremap(get_immrbase() + MPC85xx_ECM_OFFSET, MPC85xx_ECM_SIZE); pcr = in_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2)); pcr = EEBPCR_CPU1_EN; out_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2), pcr); } EEBPCR Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 66

67 SMP Boot Process: Core1 Kernel Code Flow Core1 starts up from secondary_start_page defined in arch/powerpc/kernel/head_fsl_booke.s secondary_start_page: 4K The code in this page must not exceed 1023 instruction. The1024th is a branch instruction. It will initialize I-cache and D-cache Use TLB1[1] to map 16M memory Jump to early_start early_start If it is core1, it will not call start_kernel(), but jump to secondary_start secondary_start jumps to start_secondary() start_secondary() defined in arch/powerpc/kernel/smp.c Call smp_ops->setup_cpu() Call cpu_idle() Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 67

68 Linux DPAA (Ethernet) Driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 68

69 Multicore Datapath Issues and Requirements Multicore SoCs, like the P4080, have a number of new requirements related to packet processing when compared to single core SoCs: Load spreading of arriving packets across pools of cores for parallel processing Packet ordering issues after processing Pipelined processing of packets using cores Share network I/O between cores Virtualizes hardware accelerators Inter-core communication Hardware Accelerator Core Core D$ Core I$ D$ Core I$ D$ I$ D$ I$ Network I/O Network Core D$ I$ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 69

70 More Multicore Datapath Requirements Addressing these requirements can lead to new requirements: Hardware managed queues lead to the need for hardware-supported active queue management Network interfaces must able to parse, classify, and distribute (PCD) frames. High-bandwidth network I/O as found on P4080 also drive datapath requirements: Queue congestion driven flow control Resource depletion driven flow control Hardware buffer management Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 70

71 QorIQ DPAA Components Infrastructure components Queue Manager (QMan) Buffer Manager (BMan) Network I/O Frame Manager (FMan) Hardware accelerators SEC cryptographic accelerator PME Pattern matching engine Cores CoreNet is not part of the DPAA but it provides the interconnect between the cores and the DPAA infrastructure as well as access to memory (DRAM) D$ D$ I$ I$ L2$ L2$ D$ D$ I$ I$ Frame Manager Frame Manager 1GE 1GE 10GE 1GE 1GE 10GE 1GE 1GE 1GE 1GE e500mc e500mc Core Core D$ D$ I$ I$ CoreNet Coherency Fabric Queue Manager Sec 4.0 PME 2 Buffer Mgr Multi-Lanes SERDES Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 71

72 6 FMan/QMan/BMan Ingress Packet Processing Software dequeues from QMan Software Portal and releases buffer to BMan References to Packet (Frame Descriptors) QMan Parse, classify/hash, select queue 5 4 Packet Data written to main memory subsystem Frontside Cache DDR SDRAM Packet Data Stored in H/W managed buffers FMan 2 Buffer Acquisition Request Packets in process Buffer Buffer Reference 3 BMan References to Buffers 10G 1G 1G 1G 1G 1 Packets Arriving Think ARP request Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 72

73 1 Software Acquires buffer from BMan FMan Egress Packet Processing QMan Think ARP response 8 Priority Work Queues 6 Optional packet response Class scheduler 2 Packet Data read from main memory subsystem Priority based packet scheduling 3 Frontside Cache DDR SDRAM Packets in process 5 FMan Releases Buffer to BMan Bman FMan Buffer 10G 1G 1G 1G 1G 4 Packets Transmitted Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 73

74 Aggregate Driver Structure DPA net_device Linux Driver (fsl-dpa) drivers/net/dpa BMan Linux Driver QMan Linux Driver FMan/Port Linux Driver drivers/hwalloc drivers/hwqueue drivers/net/dpa/netcomsw FMan/MAC Linux Driver drivers/net/dpa Low Level Driver FMan/PCD/Port (FMD) Low Level Driver MAC (FMD) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 74

75 Kernel Configuration Options Bare Metal CONFIG_FSL_BMAN - Freescale BMan support CONFIG_FSL_BMAN_PORTAL - BMan portal support - compiles support to detect and support BMan software CoreNet portals (as provided by the DT) CONFIG_FSL_BMAN_CONFIG - BMan device management - if this Linux image is running natively, you need this option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane") needs this option and the DT node CONFIG_FSL_QMAN - Freescale QMan support CONFIG_FSL_QMAN_PORTAL - QMan portal support - compiles support to detect and support QMan software CoreNet portals (as provided by the DT) CONFIG_FSL_QMAN_CONFIG - QMan device management - if this Linux image is running natively, you need this option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane") needs this option and the DT node CONFIG_FSL_QMAN_FQALLOCATOR - QMan FQ allocator - implemented using BMan BP 0 as a system wide simple FQ allocator/manager CONFIG_DPA - Freescale data-path (and FMan) Ethernet driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 75

76 Light Weight Executive Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 76

77 Light Weight Executive (LWE) LWE is a set of lower level platform libraries used for development of data plane networking applications LWE libraries enable application software to run in OS-less environment. Applications based on LWE run in a partitioned environment under the control of Freescale Embedded Hypervisor. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 77

78 LWE Application Programming Model Single application per partition Run to completion Thread model Global variables Per CPU variables Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 78

79 LWE Apps Provide an example of how to use LWE Libraries Out-of-box benchmark code LWE Applications Starting point for users to develop their own high performance data path code LWE Apps Examples Ethernet Bridging Hello World Simple Queueing (Pktwire) Queueing Torture Test Simple Crypto IPSecFwd FMan Test Simple PME Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 79

80 hv-4p 2-core Linux partition Single core Linux partition 4-core LWE partition Single core LWE partition Requires use of Hypervisor Partition Scenarios White Board Shared Memory CPU Linux Linux CPU CPU CPU CPU CPU CPU LWE CPU LWE Private Memory Ethernet Shared Memory Private Memory Private Memory Private Memory Data Path Shared Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 80

81 Session Summary U-boot, Hypervisor, Linux for P4080DS BSP Concepts Porting Building and deploying Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 81

82

83 What is DPAA and Why do I Need It? A multicore SoC must be able to: Load spread RX ed packets across multiple cores to facilitate parallel processing Ensure packets are TX ed in the same order that they were RX ed especially after parallel processing Support pipelining of cores Support inter-core communication Share network interfaces between cores On ingress when the different cores providing different services On egress Share hardware offload accelerators (e.g. SEC, PME) between multiple cores Address performance issues that arise from high bandwidth network interfaces (e.g. 10GE) prevalent on such an SoC Continue to support key functionality that exists in network interfaces today e.g. IEEE1588 QoS Loss-less flow control Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 84

84 Addressing Multi-Core Datapath Requirements Many of these requirements ( share, pipeline, inter-core communication ) result in the need for the ability to define communications paths between all the many blocks in the datapath Thus QMan Load spreading packets and sharing network interfaces require hardware to parse, classify, and distribute FMan s PCD functionality Effective load spreading requires that multiple consumers can share queued data Thus QMan s shared queues Support for high bandwidth network interfaces (10GE) FM policing QMan s WRED support BMan Helps offload overhead associated with managing buffers The intent of the DPAA is to address multicore requirements and provide optional software offload Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 85

85