June 22, 2010 P4080DS U-boot & Linux FTF-NET-F0683 Srikanth Srinivasan & Ted Peters NMG Systems and Applications Engineering Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink
Session Introduction Board-bring up is a key activity for new silicon. Firmware and OS are the first software components run on the reference platforms. Initial bring-up involves debug of silicon, board and the software. This presentation details the important parts of firmware (u-boot) and OS (Linux) porting/bring-up for P4080DS reference platform. Freescale s Board Support Package (BSP) provides software that can be modified to run on the customer development systems. The talk will drive home the strategies, steps, concepts, and highlight relevant parts of the code needed most commonly during this effort. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 2
Learn more about the Freescale BSP for P4080/P4080DS Details of u-boot and Linux porting, and hypervisor concepts. Session Objectives Obtain and build u-boot and Linux from source Peruse source trees for making changes specific to your platforms Identify separation of tasks between SW components Identify key components of the reference platform as well as the software Learn multi-core boot sequence Hypervisor capabilities and use Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 3
Agenda Introduction 5 min U-boot 30 min Hypervisor 25 min Linux 40 min LWE 5 min Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 4
P4080 Block Diagram P4080 128-Kbyte Backside L2 Cache Power Architecture e500-mc Core 1024-Kbyte Frontside L3 Cache 32-Kbyte 32-Kbyte 1024-Kbyte D-Cache I-Cache Frontside L3 Cache 64-bit DDR-2 / 3 Memory Controller 64-bit DDR-2 / 3 Memory Controller eopenpic PreBoot Loader Security Monitor Internal BootROM PAMU PAMU PAMU CoreNet Coherency Fabric PAMU PAMU Peripheral Access Mgmt Unit Power Mgmt SD/MMC SPI elbiu M2SB Security 4.0 Queue Mgr. Frame Manager Parse, Classify, Distribute Frame Manager Parse, Classify, Distribute SRIO Message Unit DMA Real Time Debug Watchpoint Cross Trigger DUART 2x I 2C 2x USB 2.0/ULPI Test Port/ SAP Pattern Match Engine 2.0 Buffer Mgr. 10GE Buffer 1GE 1GE 1GE 1GE 10GE Buffer 1GE 1GE 1GE 1GE PCIe PCIe SRIO PCIe SRIO Perf Monitor CoreNet Trace Aurora Clocks/Reset GPIO CCSR 18-Lane 5GHz SERDES Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 5
EC1 USB1 EC2 FM1[dTSEC2] 10/100/1G RGMII SerDes Lanes [0-17]: Slot 1: Bank1 A-B (PEX x2) Slot 2: Bank1 C-D (PEX x2) Slot 3: Bank1 E-H (4x SGMII x1, PEX x4, SRIO x4) --------: Bank1 I-J (Aurora x2 only) Slot 4: Bank 2 A-D (4x SGMII x1 or XAUI x4) Slot 5: Bank 3 A-D (4x SGMII x1 or XAUI x4) u-boot RCW[srds_prtcl] = 0x10 (RSX configuration) PCIe1 (5/2.5Gb) PCIe3 (5/2.5Gb) 4x SGMII (1) Bank 1 Bank 2 Bank 3 FM2 dtsec[1:4] Debug (5/2.5Gb) XAUI FM2 10GEC 4x SGMII FM1 dtsec[1:4] Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 6
Getting Started Master u-boot repository http://git.denx.de Master linux repository http://git.kernel.org BSP release http://www.freescale.com/powerbsp Follow instructions in the BSP documentation to extract software packages Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 7
Board Support Package (BSP) GNU Toolchain P4080 SDK Beta 2.1.0 Components gcc-4.3.2, eglibc2.8, binutils 2.18, CodeSourcery release 4.3.74-6 U-Boot Boot Loader Linux 2.6 kernel Linux device drivers with P4080 Datapath Acceleration Architecture (DPAA) support Embedded Hypervisor Software Light Weight Executive (LWE) LWE Libraries (LWE Libs) LWE Example Applications (LWE Apps) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 8
System requirements x86 linux with cross-tools Extracting Sources And Building Images LTIB $ mount o loop <BSP.iso> /mnt/p4080ds-ltib $ cd my-ltib-dir Local directory $ /mnt/p4080ds-ltib/install From the installed directory : $./ltib m prep p u-boot (Sources available in rpm/build/u-boot) $./ltib m scbuild p u-boot To configure packages : $./ltib --configure Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 9
git Version control done through git SCM of choice for many open-source projects including Linux Clone the tree locally git clone git://git.denx.de/u-boot.git Replicates the remote tree with change logs and tags When ready to contribute 1. Commit changes locally 2. Create a patch and mail it to the u-boot list 3. Peer review 4. Patch applied to master tree for everyone to use 5. Clean up patch for external world and repeat steps 2-4 A few keepers for the master tree Ensures safety of code base while still allowing group contribution and higher software quality Very convenient for large distributed projects Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 10
Summary Of Boot Flow From System-Wide Reset uboot Load hypervisor image and at least one guest OS image. Load device trees for guests. Hypervisor Hypervisor initializes and then starts one or more guests. It passes them a device tree from which they get information on resources allocated to them (and more). Guest OS One guest is designated to do global initialization of BMan, QMan, and FMan since they are used by many guests. Other guests await this initialization. Guests can bring up their network interfaces. A guest can be authorized to act as a partition manager. It can load and run more guests or reboot them. Example: Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 11
P4080DS u-boot Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 12
Introduction What is u-boot? Open source firmware for hardware platforms A boot loader widely used in embedded space What does it do? Brings the board to a sane state out of reset Load OS image onto board and start OS Origins DENX Software Engineering More at http://www.denx.de/wiki/u-boot/webhome Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 13
U-boot Directories./u-boot board common cpu disk drivers include lib_* net doc dtt examples fs rtc tools post Relevant directories for modifications u-boot/. u-boot/board u-boot/lib_ppc u-boot/include u-boot/cpu u-boot/common Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 14
Boot Sequence Reset vector points to FLASH (0xfffffffc) Begins execution in FLASH Initializes DUARTs Checks for information on version numbers, clocks and displays on screen Initializes DDR Relocates itself to RAM Continues execution in RAM Releases other cores Initializes higher level devices ethernet, PCIe, SATA, USB, etc. Begins command interpreter Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 15
U-boot Memory Map for P4080DS /* * Memory map * * 0x0000_0000 0x7fff_ffff DDR 2G Cacheable * * 0x8000_0000 0x9fff_ffff PEX 1 Mem 512M non-cacheable * 0xa000_0000 0xbfff_ffff PEX 2/SRIO Mem 512M non-cacheable * 0xc000_0000 0xdfff_ffff PEX 3 Mem 512M non-cacheable * * 0xf800_0000 0xf802_ffff PCI IO range 256K non-cacheable * 0xf400_0000 0xf41f_ffff Bman 2M * 0xf420_0000 0xf43f_ffff Qman 2M * * Localbus non-cacheable * * 0xe000_0000 0xe80f_ffff Promjet/free 128M non-cacheable * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * * 0xffd0_0000 0xffd0_3fff L1 for stack 16K Cacheable TLB0 * 0xfe00_0000 0xfeff_ffff CCSR 16M non-cacheable */ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 16
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init/mp init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 17
Stack In Cache Drivers written in C code Need stack space to run Lock the cache Unlock before linux boot * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * * 0xffd0_0000 0xffd0_3fff L1 for stack 16K Cacheable TLB0 * 0xffe0_0000 0xffef_ffff CCSR 1M non-cacheable Files: cpu/mpc85xx/start.s Code Snippets: lis r6,fsl_booke_mas0(1, 14, 0)@h ori r6,r6,fsl_booke_mas0(1, 14, 0)@l lis r7,fsl_booke_mas1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@h ori r7,r7,fsl_booke_mas1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@l lis r8,fsl_booke_mas2(config_sys_init_ram_addr, 0)@h ori r8,r8,fsl_booke_mas2(config_sys_init_ram_addr, 0)@l lis r9,fsl_booke_mas3(config_sys_init_ram_addr, 0, (MAS3_SX MAS3_SW MAS3_SR))@h ori r9,r9,fsl_booke_mas3(config_sys_init_ram_addr, 0, (MAS3_SX MAS3_SW MAS3_SR))@l : : tlbwe Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 18
MMU, LAW Setup Configuring memory map Files: include/configs/p4080ds.h, corenet_ds.h board/freescale/corenet_ds/tlb.c, law.c cpu/mpc85xx/tlb.c drivers/misc/fsl_law.c include/asm-ppc/immap_85xx.h Code Snippets: /* *I*G* - Flash, localbus */ /* This will be changed to *I*G* after relocation to RAM. */ SET_TLB_ENTRY(1, CONFIG_SYS_FLASH_BASE, CONFIG_SYS_FLASH_BASE_PHYS, MAS3_SX MAS3_SR, MAS2_W MAS2_G, 0, 2, BOOKE_PAGESZ_256M, 1), SET_LAW(CONFIG_SYS_FLASH_BASE_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_LBC), Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 19
SYSCLK SYSCLK value needed to get started Platform frequency FPGA registers SCLK0, 1, 2 Input to ICS307 clock generator code Obtain the board and ddr clocks Files: include/configs/corenet_ds.h board/freescale/corenet_ds/corenet_ds.c Code Snippets: /* Calculate frequency being generated by ICS307-02 clock chip based upon * the control bytes being programmed into it. */ static unsigned long ics307_clk_freq(unsigned char cw0, unsigned char cw1, unsigned char cw2) { const unsigned long InputFrequency = CONFIG_ICS307_REFCLK_HZ; unsigned long VDW = ((cw1 << 1) & 0x1FE) + ((cw2 >> 7) & 1); : : : } unsigned long calculate_board_sys_clk(ulong dummy) { ulong val; u8 *pixis_base = (u8 *)PIXIS_BASE; val = ics307_clk_freq( in_8(pixis_base + PIXIS_VSYSCLK0), Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 20
2 DUARTs ( 4 UARTs) on P4080 2 serial outputs on DS u-boot provides driver for NS16550 compatible DUARTs Needs platform frequency Serial port very useful for early debug Serial Files: include/configs/corenet_ds.h drivers/serial/ns16550.c, serial.c Code Snippets: #define CONFIG_CONS_INDEX 2 #undef CONFIG_SERIAL_SOFTWARE_FIFO #define CONFIG_SYS_NS16550 #define CONFIG_SYS_NS16550_SERIAL #define CONFIG_SYS_NS16550_REG_SIZE 1 #define CONFIG_SYS_NS16550_CLK (get_bus_freq(0)/2) #define CONFIG_SYS_NS16550_COM1 (CONFIG_SYS_CCSRBAR+0x11C500) #define CONFIG_SYS_NS16550_COM2 (CONFIG_SYS_CCSRBAR+0x11C600) #define CONFIG_SYS_NS16550_COM3 (CONFIG_SYS_CCSRBAR+0x11D500) #define CONFIG_SYS_NS16550_COM4 (CONFIG_SYS_CCSRBAR+0x11D600) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 21
DDR Setup Relocating code Supports multiple controllers and SPD-based (I2C eeprom) initialization Supports Interleaving Files: cpu/mpc8xxx/ddr include/configs/corenet_ds.h board/freescale/corenet_ds/ddr.c board specific parameters Memory Controller 1 Memory Controller 2 CS0 CS1 CS0 CS1 Code Snippets: #define CONFIG_NUM_DDR_CONTROLLERS 2 #define CONFIG_DIMM_SLOTS_PER_CTLR 1 #define CONFIG_CHIP_SELECTS_PER_CTRL (2 * CONFIG_DIMM_SLOTS_PER_CTLR) #define CONFIG_DDR_SPD #define CONFIG_FSL_DDR3 #define CONFIG_SYS_SPD_BUS_NUM 1 #define SPD_EEPROM_ADDRESS1 0x51 #define SPD_EEPROM_ADDRESS2 0x52 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 22
Misc CPU Init Set up local bus memory map BRn, ORn Enable timebase register Files: cpu/mpc85xx/cpu_init.c Code Snippets: void cpu_init_f (void) { volatile ccsr_lbc_t *memctl = (void *)(CONFIG_SYS_MPC85xx_LBC_ADDR); : : #ifdef CONFIG_FSL_CORENET corenet_tb_init(); #endif init_used_tlb_cams(); /* Invalidate the CPC before DDR gets enabled */ invalidate_cpc(); } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 23
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 24
Relocates u-boot code to top of memory For 2GB on P4080DS, in the last 1MB Set up of stack space, etc. Relocate To DDR Space Continues to run from DDR after relocation Files: cpu/mpc85xx/start.s lib_ppc/board.c Code Snippets: /* * void relocate_code (addr_sp, gd, addr_moni) * * This "function" does not return, instead it continues in RAM * after relocating the monitor code. * * r3 = dest * r4 = src * r5 = length in bytes * r6 = cachelinesize */.globl relocate_code relocate_code: mr r1,r3 /* Set new stack pointer */ mr r9,r4 /* Save copy of Init Data pointer */ mr r10,r5 /* Save copy of Destination Address */ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 25
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 26
CFI flash support Reads on POR Sets up structures much later in the driver NOR Flash Files: * Localbus non-cacheable * * 0xe000_0000 0xe80f_ffff Promjet/free 128M non-cacheable * 0xe800_0000 0xefff_ffff FLASH 128M non-cacheable * 0xffdf_0000 0xffdf_7fff PIXIS 32K non-cacheable TLB0 * Code Snippets: include/configs/corenet_ds.h drivers/mtd/cfi_flash.c This information is used by the driver to detect the flash using CFI query at both possible addresses and populate its info table #define CONFIG_FLASH_CFI_DRIVER #define CONFIG_SYS_FLASH_CFI #define CONFIG_SYS_FLASH_AMD_CHECK_DQ7 #define CONFIG_SYS_FLASH_BANKS_LIST {CONFIG_SYS_FLASH_BASE_PHYS + 0x8000000, CONFIG_SYS_FLASH_BASE_PHYS} /* Protect RCW and FMAN microcode sectors for the NOR flash boot bank */ #define CONFIG_SYS_FLASH_AUTOPROTECT_LIST { {CONFIG_SYS_FLASH_BASE + 0x8000000, 1}, \ {CONFIG_SYS_FMAN_FW_ADDR, 1} } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 27
PCIe Check for PCIe interfaces on the SOC Basic PCIe enumeration Useful when pcie devices are needed at boot time. e.g. RTC/NVRAM on nvidia 1575 /* * 0x8000_0000 0x9fff_ffff PEX 1 Mem 512M non-cacheable * 0xa000_0000 0xbfff_ffff PEX 2 Mem 512M non-cacheable * 0xc000_0000 0xdfff_ffff PEX 3 Mem 512M non-cacheable Files: include/configs/corenet_ds.h drivers/pci/fsl_pci_init.c board/freescale/corenet_ds/corenet_ds.c Code Snippets: void fsl_pci_init(struct pci_controller *hose) { : : } #define CONFIG_SYS_PCIE1_ADDR (CONFIG_SYS_CCSRBAR+0x200000) #define CONFIG_SYS_PCIE2_ADDR (CONFIG_SYS_CCSRBAR+0x201000) #define CONFIG_SYS_PCIE3_ADDR (CONFIG_SYS_CCSRBAR+0x202000) : : #define CONFIG_SYS_PCIE3_MEM_PHYS 0xc000000 #define CONFIG_SYS_PCIE3_IO_PHYS 0xf8020000 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 28
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 29
Power-on Reset Misc Init/MP Init Core0 comes out of reset at the reset vector 0xFFFF_FFFC and all other cores are in boot hold off mode Core0 runs u-boot After setting BSTRL/BSTAR to secondary core s start page, core0 kicks off each additional secondary core Each secondary core comes out of reset at secondary_start_page(boot page), initialize resources specific that that core (caches, MMU, etc.) and enters a spin loop Core0 loads Linux image and device tree and boots Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 30
Multicore Boot Architecture epapr describes specifics on how secondary CPUs are booted for a system with multiple CPUs Default boot architecture The boot program releases all CPUs from hardware reset One CPU is designated to be the client program s boot CPU All other CPUs are secondary and are placed into loop where the CPUs spin, waiting for a spin table field to change that directs them where to go Control is transferred to the client program on the boot CPU When the client program is ready for secondary cores to start, it releases them by writing the spin table field with the desired address The architecture allows for other custom-defined secondary CPU release mechanisms as well Reference: epapr (Power.org Standard for Embedded Power Architecture Platform Requirements (epapr). power.org, 2008.) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 31
SMP Boot Process: U-boot Boot Process In the u-boot source, the entry point is the reset vector 0x0_FFFF_FFFC containing a simple branch to _start_e500, which is at the base of default boot page at 0x0_FFFF_F000; both are found in cpu/mpc85xx/start.s Reset vector call _start_e500() _start_e500() will: _start_e500() Boot other cores Start command interpreter Prepare device trees 1. Enable L1 caches 2. Config interrupt vectors 3. Config MMU and LAWs 4. Config L1D RAM 5. Config local bus 6. Config DDR 7. Relocate to DDR 1. Setup BSTRL and BSTAR 2. Release all other cores from boot holdoff using BRR and wait for them to boot. cpu/mpc85xx/mp.c 1. Copy HW device tree to DDR 2. Copy HV config tree to DDR 3. Copy hv.uimage to DDR 4. Launch hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 32
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 33
FM Init Rx Data MAC (RX data) FM BMI (put MAC data in internal buffers) RISC (generate DMA to write data to BDs) DMA (write data) Enable FM BMI (allocate internal context buffers) RISC (generate DMA to read data from BDs) DMA (read data) BMI (read data) FMan Independent mode to get basic networking for tftp downloads Needs configuration data Configuration data resides at 0xEF000000 FM BMI (free internal buffers) MAC (transmit TX data) END Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 34
FM Init Independent mode Files: include/configs/corenet_ds.h drivers/net/fm/* board/freescale/corenet_ds/corenet_ds.c Code Snippets: int board_eth_init(bd_t *bis) { : : p4080_setup_serdes(); /* FM 1 DTSEC */ if (is_pcie_configured(prtcl, SGMII_FM1)) { printf("fm1 SGMII\n"); #ifdef CONFIG_FM1_ETH0 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 0) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH1 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 1) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH2 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 2) fm1_idx++; #endif #ifdef CONFIG_FM1_ETH3 SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 3) fm1_idx++; #endif : : Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 35
Initialization MMU, LAW setup Clocks Serial DDR Misc CPU init Relocate to DDR space Flash PCIe Misc init, MP init FM init Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 36
Other New Pieces BMAN, QMAN, FMAN LIODNs Device tree fixups for HV/linux cpu/mpc85xx/portals.c, liodn.c CoreNet Platform Cache Enabled by a flag in include/configs/corenet_ds.h (CONFIG_SYS_FSL_CPC) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 37
Building u-boot Using LTIB $./ltib m scbuild p u-boot From command line Set the compiler toolchain to compile u-boot Set PATH variable on the build machine Use ARCH=ppc and CROSS_COMPILE flags make CROSS_COMPILE=powerpc-linux-gnumake CROSS_COMPILE=powerpc-linux-gnumake CROSS_COMPILE=powerpc-linux-gnu- ARCH=ppc distclean ARCH=ppc P4080DS_config ARCH=ppc u-boot.bin Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 38
P4080DS NOR Flash Map Cheat Sheet (from Bank 0 perspective after U-Boot DDR relocation) 0xE8000000 0xE8FFFFFF Bank 0 PBL Image Bank 7 SW7[1:4] = 0b0111 U-Boot PBL @ 0xE0000000 0xE9000000 0xE9FFFFFF Bank 1 PBL Image Bank 6 SW7[1:4] =0b0110 0xEA000000 0xEAFFFFFF BANK 2 PBL Image Bank 5 SW7[1:4] = 0b0101 0xEB000000 0xEBFFFFFF Bank 3 PBL Image Bank 4 SW7[1:4] = 0b0100 0xEC000000 0xECFFFFFF Bank 4 PBL Image Bank 3 SW7[1:4] = 0b0011 0xED000000 0xEDFFFFFF Bank 5 PBL Image Bank 2 SW7[1:4] = 0b0010 0xEE000000 0xEEFFFFFF Bank 6 PBL Image Bank 1 SW7[1:4] = 0b0001 0xEF000000 0xEFFFFFFF Bank 7 PBL Image Bank 0 SW7[1:4] = -0b0000 U-boot @ 0xEFF80000 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 39
Common (Helpful) Commands help Environment variables printenv setenv saveenv NOR Flash flinfo erase <start addr> <end addr> cp.b <from addr > <to addr in flash> <#bytes> Network setenv [ipaddr][gatewayip][netmask][serverip] <ip address> tftp <dest address in RAM> <file on server> Other imls - Lists all u-boot recognizable images in flash i2c - i2c commands to display, read, write i2c devices pci - display, read, write PCI/PCIe space Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 40
36-bit addressing U-boot Hooks To Launch Hypervisor U-boot uses 36-bit addressing but is limited to 4GB HV uses 36-bit addressing to support multiple partitions and resources Physical addresses of the IO are moved to the end of the 64GB address space DDR address space starts at 0; now has room to expand New commands Device trees for HW and configuration Pass HW device tree address to bootm command Location of configuration device tree is passed config-addr env variable => setenv bootcmd bootm e8700000 - e8800000 => setenv bootargs config-addr=0xfe8900000 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 41
P4080DS Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 42
SMP Symmetric Multiprocessing ASMP Asymmetric Multiprocessing (aka. AMP) CAMP Cooperative AMP Terminology Guest OS OS running in guest supervisor state under a hypervisor Hypervisor - Manages globally shared resources, more privileged than operating systems, enforces system security, virtualizes some resources Light Weight Executive (LWE) - Provides a communications mechanism between an application running in a partition and the hypervisor software Device Tree - A data structure used for representing a partition s physical and virtual devices Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 43
Partitioning With A Hypervisor Hypervisor Software Analogous to role of an operating system kernel in managing user processes More privileged than operating systems Enforces system security Manages globally shared resources Virtualizes some resources e.g. interrupt controller, UART Hypervisor partition App App partition App App partition App App Linux RTOS Legacy OS Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 44
e500mc Privilege Levels Guest State (GS) MSR bit Under Hypervisor Bare Metal partition partition User MSR[PR=1][GS=1] App App User MSR[PR=1][GS=0] App App Kernel/Supervisor MSR[PR=0][GS=1] OS Kernel/Supervisor MSR[PR=0][GS=0] OS Hypervisor MSR[PR=0][GS=0] Hypervisor CPU CPU Memory Memory I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 45
Operating System sees a virtual core plus hypervisor services Hypervisor Features Virtual CPU (like e500mc minus hypervisor features) Services via hypercall Emulation (privileged instructions) guest operating system hypercalls device tree Debug stub interface for debugging guest operating systems virtual CPU (e500vcpu) Hypervisor services PIC Partition Mgmt Doorbells GPIO IOMMU boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 46
Virtual CPU Like e500mc minus the hypervisor extensions Full emulation is used MMU tlb access instructions are emulated Guest receive and manage TLB miss interrupts directly Timer DEC, FIT, watchdog, timebase synchronization Cache control L1,L2 Intra-partition signaling (msgsnd/msgclr) Debug services (DBCRx, IACx, DACx, etc) Core performance monitor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 47
Virtual CPU Emulation Example Operating system manages it s own page tables. Hypervisor has no knowledge of OS page table format. Hypervisor traps and authenticates all TLB updates Applicati on lwz r3,(r4) tlb miss OS (tlb miss handler) tlbwe privilege trap Hypervisor Hypervisor validate tlb writes Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 48
Hypervisor Services hcalls Interrupt controller (MPIC) Byte-channels character I/O stream Inter-partition signaling doorbell Partition management Start/stop/image-loading In future: Create/destroy Partition management interrupts Power management change clock frequency, power states GPIO supports partitioning of GPIO pins IOMMU supports create/destroy mappings Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 49
Device Trees Guest #1 Guest #2 Guest Dev tree HW Dev tree hypervisor u-boot Guest Dev tree HV config tree dynamically created and loaded into guest memory Loaded into hypervisor memory by u- boot. Hardware dev tree /chosen node points to HV config tree Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 50
Boot Process Hypervisor initialization PAMU initialization Coherence domain setup set up LAWS, CSDIDs Error Configuration e.g. single bit ECC error thresholds Driver initialization DDR CPC CCM UART Release secondary CPUs Partition instantiation/creation The boot CPU for each partition takes care of partition creation Control transfer to guest OS Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 51
HV Commands Commands list and uses HV>? Commands: help (?) - Print command usage information version - Print the hypervisor version list-partitions (lp) - List partitions guest-device-tree (gdt) - Guest device tree operation master-device-tree (mdt) - Display master device tree paact - Dump PAMU's PAACT table entries start - Start a partition restart - Start a partition stop - Stop a partition pause - Stop a partition resume - Stop a partition HV> Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 52
P4080DS Linux Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 53
Overview P4080 Linux Overview P4080 BSP Linux Bare Metal and Guest OS Linux Kernel Configuration Device Tree SMP Boot Process DPAA and the Linux Ethernet Driver DPAA Components Life of a Packet Ingress Egress Linux Driver Structure Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 54
P4080 Linux Bare metal Linux Linux runs directly on HW User state MSR[PR]=1 MSR[GS]=0 Supervisor state MSR[PR]=0 MSR[GS]=0 Guest OS Linux Hypervisor SW layer between Linux and HW Guest User state MSR[PR]=1 MSR[GS]=1 Guest Supervisor state MSR[PR]=0 MSR[GS]=1 Hypervisor state MSR[PR]=0 MSR[GS]=0 Guest OS requires porting to run on Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 55
Bare Metal SMP Linux App App SMP Linux Multicore System Hardware Shared Cache I/O Interrupt Controller CPU CPU CPU CPU Memory I/O I/O I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 56
Guest OS Linux partition partition partition App App App App App App SMP Linux RTOS Legacy OS Hypervisor Multicore System Hardware Shared Cache I/O Interrupt Controller CPU CPU CPU CPU Memory Memory Memory I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 57
Resources controlled by Hypervisor: Physical memory map Porting Guest OS (Linux) to Hypervisor Core: caches, MMU, SPRs (Guest OS sees a virtual CPU via emulation) Platform: CPC, MPIC, Partion management, Doorbell, Byte-channels, IOMMU, GUTs, DDR (Guest OS access via HCALLs) Direct resources (Direct I/O access by Guest OS) Device Tree aware (per epapr, libfdt library available) State of HW when Guest OS Runs (per epapr) Hypercall API (HCALLs for shared services, ie VMPIC) Traps (emulation - vcpu) Reference: epapr (Power.org Standard for Embedded Power Architecture Platform Requirements (epapr). power.org, 2008.) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 58
Building Linux LTIB BSP Configure default LTIB settings $./ltib Configure menu for P4080DS, then automatically build $./ltib --configure In configure menu, select: Select Configure kernel to alter the default kernel build configuration Select Leave the sources after building to leave the linux source in rpm/build/linux From GIT tree make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc menuconfig make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc uimage Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 59
P4080 Linux Kernel Configuration The easiest way to create a.config file is to simply copy the appropriate config file in./arch/powerpc/configs to.config. cp arch/powerpc/configs/p4080_ds_defconfig.config Typing make menuconfig results in a console-based, menu-driven utility that can be used to specify the desired configuration. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 60
Linux Directory Structure./linux-2.6.xx Doc. arch block crypto drivers fs include init kernel lib mm scripts security sound ipc net usr arch/powerpc/boot/dts/p4080ds.dts arch/powerpc/configs/p4080_ds_defconfig arch/powerpc/boot/ arch/powerpc/platforms/85xx/ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 61
P4080 Linux Device Tree Baremetal Linux Use the device tree compiler to build the appropriate dts file in the./arch/powerpc/boot/dts directory. dtc -I dts -O dtb -R 8 -b 0 -S 0x3000 p4080ds.dts > p4080ds.dtb OR make p4080ds.dtb (from linux-2.6/) Guest OS Linux The device tree for each partition is generated by Hypervisor from a system configuration file that maps the system partitioning and HW. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 62
Kernel Configuration Options - Hypervisor CONFIG_P4080_HV - Freescale P4080 Hypervisor platform CONFIG_FSL_HYPERVISOR - Freescale P4080 Hypervisor CONFIG_VMPIC - Freescale P4080 Hypervisor - Virtualized OpenMPIC driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 63
SMP Boot Process: Linux SMP Boot Process In the Linux kernel, the entry point is _start found in arch/powerpc/kernel/head_fsl_booke.s _start start _kernel() is in init/main.c, it will: _start_kernel()... setup_arch() call _start_kernel() mpc85xx_smp_init() { smp_ops = &smp_85xx_ops; } Kernel_init() smp_prepare_cpu() smp_ops->probe() in main.c smp_init() cpu_up() smp_cpus_done() smp_ops->kick_cpu() smp_ops->setup_cpu() Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 64
SMP Boot Process: Core1 Start Up (continued) Let s look at smp_mpc85xx_kick_cpu() found in: arch/powerpc/platforms/85xx/mpc85xx_smp.c smp_mpc85xx_kick_cpu() {. /* Get the BPTR */ bptr_vaddr = ioremap(get_immrbase() + MPC85xx_BPTR_OFFSET, 4); /* Set the BPTR to the secondary boot page */ oldbptr = in_be32(bptr_vaddr); bptr = (BPTR_EN ( pa((unsigned) secondary_start_page) >> 12)); out_be32(bptr_vaddr, bptr); /* Kick that CPU */ smp_85xx_release_core(nr); /* Wait a bit for the CPU to take the exception. */ while (( secondary_hold_acknowledge!= nr) && (++n < 1000)) mdelay(1); /* Restore the BPTR */ out_be32(bptr_vaddr, oldbptr); } Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 65
SMP Boot Process: Core1 Kick-Off Process Core 0 kicks off core 1 by setting EEBPCR[CPU1_EN] Let s look at smp_85xx_release_core() in arch/powerpc/mpc85xx/mpc85xx_smp.c smp_85xx_release_core(int nr) { /* * Startup Core #nr. */ ecm_vaddr = ioremap(get_immrbase() + MPC85xx_ECM_OFFSET, MPC85xx_ECM_SIZE); pcr = in_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2)); pcr = EEBPCR_CPU1_EN; out_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2), pcr); } EEBPCR Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 66
SMP Boot Process: Core1 Kernel Code Flow Core1 starts up from secondary_start_page defined in arch/powerpc/kernel/head_fsl_booke.s secondary_start_page: 4K The code in this page must not exceed 1023 instruction. The1024th is a branch instruction. It will initialize I-cache and D-cache Use TLB1[1] to map 16M memory Jump to early_start early_start If it is core1, it will not call start_kernel(), but jump to secondary_start secondary_start jumps to start_secondary() start_secondary() defined in arch/powerpc/kernel/smp.c Call smp_ops->setup_cpu() Call cpu_idle() Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 67
Linux DPAA (Ethernet) Driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 68
Multicore Datapath Issues and Requirements Multicore SoCs, like the P4080, have a number of new requirements related to packet processing when compared to single core SoCs: Load spreading of arriving packets across pools of cores for parallel processing Packet ordering issues after processing Pipelined processing of packets using cores Share network I/O between cores Virtualizes hardware accelerators Inter-core communication Hardware Accelerator Core Core D$ Core I$ D$ Core I$ D$ I$ D$ I$ Network I/O Network Core D$ I$ Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 69
More Multicore Datapath Requirements Addressing these requirements can lead to new requirements: Hardware managed queues lead to the need for hardware-supported active queue management Network interfaces must able to parse, classify, and distribute (PCD) frames. High-bandwidth network I/O as found on P4080 also drive datapath requirements: Queue congestion driven flow control Resource depletion driven flow control Hardware buffer management Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 70
QorIQ DPAA Components Infrastructure components Queue Manager (QMan) Buffer Manager (BMan) Network I/O Frame Manager (FMan) Hardware accelerators SEC cryptographic accelerator PME Pattern matching engine Cores CoreNet is not part of the DPAA but it provides the interconnect between the cores and the DPAA infrastructure as well as access to memory (DRAM) D$ D$ I$ I$ L2$ L2$ D$ D$ I$ I$ Frame Manager Frame Manager 1GE 1GE 10GE 1GE 1GE 10GE 1GE 1GE 1GE 1GE e500mc e500mc Core Core D$ D$ I$ I$ CoreNet Coherency Fabric Queue Manager Sec 4.0 PME 2 Buffer Mgr Multi-Lanes SERDES Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 71
6 FMan/QMan/BMan Ingress Packet Processing Software dequeues from QMan Software Portal and releases buffer to BMan References to Packet (Frame Descriptors) QMan Parse, classify/hash, select queue 5 4 Packet Data written to main memory subsystem Frontside Cache DDR SDRAM Packet Data Stored in H/W managed buffers FMan 2 Buffer Acquisition Request Packets in process Buffer Buffer Reference 3 BMan References to Buffers 10G 1G 1G 1G 1G 1 Packets Arriving Think ARP request Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 72
1 Software Acquires buffer from BMan FMan Egress Packet Processing QMan Think ARP response 8 Priority Work Queues 6 Optional packet response Class scheduler 2 Packet Data read from main memory subsystem Priority based packet scheduling 3 Frontside Cache DDR SDRAM Packets in process 5 FMan Releases Buffer to BMan Bman FMan Buffer 10G 1G 1G 1G 1G 4 Packets Transmitted Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 73
Aggregate Driver Structure DPA net_device Linux Driver (fsl-dpa) drivers/net/dpa BMan Linux Driver QMan Linux Driver FMan/Port Linux Driver drivers/hwalloc drivers/hwqueue drivers/net/dpa/netcomsw FMan/MAC Linux Driver drivers/net/dpa Low Level Driver FMan/PCD/Port (FMD) Low Level Driver MAC (FMD) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 74
Kernel Configuration Options Bare Metal CONFIG_FSL_BMAN - Freescale BMan support CONFIG_FSL_BMAN_PORTAL - BMan portal support - compiles support to detect and support BMan software CoreNet portals (as provided by the DT) CONFIG_FSL_BMAN_CONFIG - BMan device management - if this Linux image is running natively, you need this option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane") needs this option and the DT node CONFIG_FSL_QMAN - Freescale QMan support CONFIG_FSL_QMAN_PORTAL - QMan portal support - compiles support to detect and support QMan software CoreNet portals (as provided by the DT) CONFIG_FSL_QMAN_CONFIG - QMan device management - if this Linux image is running natively, you need this option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane") needs this option and the DT node CONFIG_FSL_QMAN_FQALLOCATOR - QMan FQ allocator - implemented using BMan BP 0 as a system wide simple FQ allocator/manager CONFIG_DPA - Freescale data-path (and FMan) Ethernet driver Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 75
Light Weight Executive Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 76
Light Weight Executive (LWE) LWE is a set of lower level platform libraries used for development of data plane networking applications LWE libraries enable application software to run in OS-less environment. Applications based on LWE run in a partitioned environment under the control of Freescale Embedded Hypervisor. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 77
LWE Application Programming Model Single application per partition Run to completion Thread model Global variables Per CPU variables Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 78
LWE Apps Provide an example of how to use LWE Libraries Out-of-box benchmark code LWE Applications Starting point for users to develop their own high performance data path code LWE Apps Examples Ethernet Bridging Hello World Simple Queueing (Pktwire) Queueing Torture Test Simple Crypto IPSecFwd FMan Test Simple PME Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 79
hv-4p 2-core Linux partition Single core Linux partition 4-core LWE partition Single core LWE partition Requires use of Hypervisor Partition Scenarios White Board Shared Memory CPU Linux Linux CPU CPU CPU CPU CPU CPU LWE CPU LWE Private Memory Ethernet Shared Memory Private Memory Private Memory Private Memory Data Path Shared Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 80
Session Summary U-boot, Hypervisor, Linux for P4080DS BSP Concepts Porting Building and deploying Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 81
What is DPAA and Why do I Need It? A multicore SoC must be able to: Load spread RX ed packets across multiple cores to facilitate parallel processing Ensure packets are TX ed in the same order that they were RX ed especially after parallel processing Support pipelining of cores Support inter-core communication Share network interfaces between cores On ingress when the different cores providing different services On egress Share hardware offload accelerators (e.g. SEC, PME) between multiple cores Address performance issues that arise from high bandwidth network interfaces (e.g. 10GE) prevalent on such an SoC Continue to support key functionality that exists in network interfaces today e.g. IEEE1588 QoS Loss-less flow control Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 84
Addressing Multi-Core Datapath Requirements Many of these requirements ( share, pipeline, inter-core communication ) result in the need for the ability to define communications paths between all the many blocks in the datapath Thus QMan Load spreading packets and sharing network interfaces require hardware to parse, classify, and distribute FMan s PCD functionality Effective load spreading requires that multiple consumers can share queued data Thus QMan s shared queues Support for high bandwidth network interfaces (10GE) FM policing QMan s WRED support BMan Helps offload overhead associated with managing buffers The intent of the DPAA is to address multicore requirements and provide optional software offload Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 85