3. Processing Core Intercommunication: Asymmetric Multiprocessing with FreeRTOS and Microblaze Soft-core Processor

Post date: Jun 24, 2014 9:20:11 PM

Introduction. Not to jump too far ahead, but this project deals with Xilinx's Zynq^TM Extensible Processing Platfom (EPP), specifically utilizing both of its hardcore processing cores, each of which running a real-time operating system (RTOS), and an additional soft-core processor. As opposed to the ZC702 Evaluation board utilized in previous projects, Avnet's Zynq Evaluation Development Board (ZedBoard) is this project's development board of choice.

More information on the specifics of the project's hardware and software will come later. For now, know the primary objectives of the project are having the three processing cores within Xilinx's Zynq^TM Extensible Processing Platform (EEP) intercommunicate, to explore a few of the different methods by which inter-processor communication can be implemented, and to measure how fast the communication can take place.

The layout of the post is as follows. The post will first start with a video and a description on how the user interacts with the project (see Demonstration below). Next, background information of the physical hardware is briefly explained (see Hardware below). The next two sections discuss the actual design of the entire project, starting with the Register-Transfer-Level (RTL) Project created in Xilinx's Vivado GUI (see Programmable Logic below) and then ending with the applications developed in Xilinx's Software Development Kit (SDK) (see Application Software below). Lastly, any final thoughts and observations I found significant are covered in the conclusion (see Conclusion below).

Demonstration. Below is a small demonstration video I uploaded to my website's YouTube account. In the video I try to quickly demonstrate the project and brush over some of details more explicitly described in this post.

Demonstration video

But, if the video is hard to understand, or see due to the video's poor quality, here is a brief explanation of what the project is programmed to do. First off, the user, which is myself in the video, interacts with the Zynq EPP through either the slide switches or the command-line interface (CLI). The slide switches are found on the development board (see Figure 1), whereas the CLI is accessed via the ZedBoard's UART (see Figure 2). The slide switches simply toggle the corresponding LED. With a set of commands, the CLI allows the user to post messages to the development board's organic light-emitting diode (OLED) display, invert the state of the LEDs, and display how long either of those actions take to complete.

Figure 1. The output shown through the ZedBoard's peripherals, specifically the OLED display and the LEDs

The numbers shown in Figures 1 and 2 correspond to each other. The console in Figure 2 shows what commands the user is issuing and the time taken to complete those commands in ticks. Ticks, of course, refer to the number of times the counter register of a timer has incremented. The set of pictures in Figure 1 shows the results of those commands through the ZedBoard's peripherals, the OLED and LEDs.

Figure 2. The projected is controlled via a CLI accessed by the ZedBoard's UART.

As far any visible actions go, the demonstration itself is pretty simple. Just a few lights, switches, and a small CLI for more involved user-interaction with the Zynq EPP's processors. Not necessarily a major objective of this project, but knowing how to either implement or use the peripherals will become more important in later, more advance projects.

For now, though, it is important just to know enough about the basics, for instance the hardware.

Hardware. As already mentioned, the ZedBoard is the development board that carries the Zynq EPP as its central processing unit (CPU), memory, and many other peripherals. The Zynq EPP comes in a few different versions, and the version on the ZedBoard is the Z-7020. The main memory includes 512 MB DDR3 DRAM and 256 MB of flash with a Quad-SPI interface. Some of the many peripherals include built-in USB-JTAG for programming and debugging, USB-to-UART bridge module, the LEDs and slide switches, and the 128x32 pixel OLED. The ZedBoard does have many other features unmentioned in this post, a few of which I plan to utilize in later projects, however I only mention the features I used for this project.

Figure 3. The ZedBoard

I have already covered the background information on the Zynq EPP in earlier posts. However, I will briefly go over some of the Zynq EPP's outstanding features here. The Zynq EPP falls under a somewhat new class of devices known as either System-on-Chip (SoC) Field Programmable Gate Arrays (FPGA) or All-Programmable SoCs. The latter term is what Xilinx refers to their class of SoC FPGAs as, namely the Zynq EPP. I will primarily stick with the former since, from my experience thus far, "SoC FPGA" seems relatively more common.

An SoC FPGA is a combination of a FPGA and a hard multi-core processor on the same die. That is not to say an SoC FPGA couldn't have a single, hard processing core --- which I have yet to see --- but one of the major ideas behind SoC FPGAs is having an integrated system with multiple processing cores, namely an SoC, and with the flexibility of re-configurable hardware, namely an FPGA.

Figure 4. This is an abstraction of the Zynq EPP. Please refer to the Zynq EPP's user guide for more reliable, specific information. L2C is the "Level 2 Cache", SCU is "Snoop Controller Unit", MIO is "Multiplexed Input/Output", AXI is "Advanced eXtensible Interface", GP is "General Purpose", EMIO is "Extended MIO", ACP is "Accelerator Coherency Port", and HP is "High Performance".

For most of the documentation written for the Zynq EPP, the FPGA is referred to as the programmable logic (PL) , whereas every other component, including the hard multi-core processor, is referred to as the processing system (PS). Such components include the PS's interconnect, interfaces, on-chip memory (OCM), and the hard multi-core processor itself. The Zynq EPP's documentation refers to the hard multi-core processor as the application processing unit (APU).

Now on to the specific on-chip hardware. The Z-7020, the type of Zynq EPP on the ZedBoard, is a combination of Xilinx's Artix 7 FPGA as the PL and a ARM© Dual-Core Cortex A9 processor as the APU.

Programmable Logic. Prior to writing the application software, the first set of steps for this project involve building the RTL project in Xilinx's latest development tool, the Vivado Design Suite. Specifically, the design of the programmable logic and the configuration of the Intellectual Property (IP) cores are done in the Design Suite's Vivado GUI. Of course, there are constraints defined in the user constraint file, but I will only mention their existence here.

Figure 5. The board design taken from Vivado GUI's IP Integrator

The hardware description language (HDL) of choice is Verilog, though most of the RTL project is created within the built-in IP Integrator, a tool where IP cores are graphically instantiated, configured, and interconnected as block designs. Other important features of the IP Integrator include the Address Editor, in which the address map or each processor are configured, and a validation check to quickly check for a critical faults in either the board design or the address map.

Figure 6. Address Editor taken from Vivado GUI's IP Integrator

Figure 5 shows how the board design can logically be divided based on the functionality of the IP cores and the external interfaces which define the port of the board design when instantiated in an HDL source file, namely the top module.

I will pretty much go over all the IP cores shown in the block design, here. In later posts, though, I may just divide the board designs into several hierarchies, which are groups of IP core blocks encapsulated as a single block that conceptually represent a subsystem within the IP Integrator. There are two IP cores I will gloss over now, the AXI Interconnect and the Processor System Reset. The AXI Interconnect is a hierarchy --- in other words, it is composed of smaller IP cores --- intended to bridge together one or more master AXI interfaces with one or more slave AXI interfaces. The AXI Interconnect blocks can also conveniently bridge together different AXI interfaces, such as AXI4 and AXI4Lite. The Processor System Reset is automatically generated by the IP Integrator and its function seemingly is to synchronize resetting all the IP cores and hardware modules with the PS (I have yet to read more into this IP).

Programmable Logic: Processing System. The most important IP, of course, is the ZYNQ7 Processing System core, through which the PS is configured. Configuration includes defining the available multiplexed input/output (MIO) interfaces, setting up the PS's ports, and configuring many other features not necessary to mention for the sake of brevity.

Figure 7. Processing System IP core block as seen in Vivado GUI's IP Integrator

The features most relevant to this project are the MIO Quad-SPI flash interface for storing the final boot file within flash memory, the extended MIO (EMIO) for accessing the the LEDs and slide switches, and configuring the Advanced eXtensible Interface (AXI) ports, specifically the general purpose (GP) and accelerator coherency port (ACP). Both AXI ports are methods by which the PS can communicate with the PL, although with differing levels of performance and approaches to how data is accessed. To simply put it, the AXI GP is relatively slower and the type of AXI GP ports instantiated allow the PS to write to and read from a connected IP core or hardware module (i.e. a module written in a HDL such as Verilog). On the other hand, the AXI ACP port allows an IP core or hardware module to access a portion of the PS's address map, including internal and external memory.

Each AXI GP port corresponds to each of the APU's two processing cores, whereas the AXI ACP grants the MicroBlaze access to memory through the PS.

I will talk in more detail on the PS's various ports for later projects. But, for those who have yet to read UG585, the performance advantages from the PS's AXI ACP port are because the AXI ACP port has direct access to cache memory through the APU's Snoop Control Unit (SCU). There are also the AXI High Performance (HP) ports that connect to the PS's memory interfaces; however, I will only mention their existence, here.

Programmable Logic: MicroBlaze. In short, the MicroBlaze is soft-core processor developed by Xilinx. Its purpose is to solely drive the OLED display through its AXI GP port and to sit quietly until one of the APU's cores sets a semaphore, signaling to the MicroBlaze when there is new information to post to the OLED. The MicroBlaze's data port is also conveniently configured as an AXI interface in order to access the PS's AXI ACP port and write to the OLED.

Figure 8. MicroBlaze IP core block as seen in Vivado GUI's IP Integrator

The other two blocks shown with the MicroBlaze in Figure 5 are a MicroBlaze Debug Module (MDM) IP for debugging/programming purposes and local memory that interfaces with the MicroBlaze's instruction and data local memory buses (LMB). For this project, the local memory is implemented as block RAM (BRAM), though the MicroBlaze can be configured such that external DDR3 RAM or the OCM is instead reserved for instructions and data.

Programmable Logic: Mailbox. The Mailbox IP core is built specifically for inter-processor communication and is implemented as a synchronous, bidirectional first-in-first-out (FIFO) queue in the PL, as opposed to memory. The APU's first processing core is programmed to wait for messages from the APU's second processing core. On receipt, the first processing core causes the LEDs to invert.

Figure 9. Mailbox IP core block as seen in Vivado GUI's IP Integrator

Programmable Logic: Peripherals. Last, but not least, are the peripherals! This group contains the interfaces and the IP core that enable the PS and the MicroBlaze to access the world outside of the block design and the Zynq EPP itself. Together, the AXI GPIO IP core and the GPIO_oled interface allow the MicroBlaze to drive the OLED's interface and thus enable the user to post messages. The GPIO_peripherals interface connects directly to the PS's EMIO port, which is labeled GPIO_0 on the ZYNQ7 Processing System block. The GPIO_peripherals consist of the nets that connect to the LEDs and slide switches. Finally, the DDR and FIXED_IO are both interfaces that connect to fixed pins on the Zynq EPP, including pins that interface with the external memory, MIO, and the PS's driving clock.

Figure 10. The set of interfaces and the IP core that together allow the processing cores to communicate external to the board design

Application Software. The majority of the high-level functionality of the project is most visible at the software level. Once the hardware-related files are exported from the Vivado GUI, the Software Development Kit is the development environment used to write software for the processing cores, generate the boot loader application, create the boot file, and to program the ZedBoard's flash. The operating systems (OS) of choice are Xilinx's Standalone (alternatively called bare-metal), which provides a small, though convenient software layer that provides basic drivers and configuration of the available processing cores, and Real Time Engineers Ltd's FreeRTOS, which is, of course, a RTOS that adds powerful tools, such as threads and timers, while also not interfering with the drivers provided with Standalone. The programming language is C (and small amounts of ARMv7).

Application Software: MicroBlaze. To review, the MicroBlaze's primary function is to post messages to the OLED. In the case of this project, the MicroBlaze checks for a semaphore and, if the semaphore is set to 1, the message is written to the OLED. Once the message has been written, the semaphore is set back to 0. Both the semaphore and the message are externally stored in DDR. Finally, MicroBlaze is only running the Standalone OS (though there exists a FreeRTOS port for the MicroBlaze).

Figure 11. (code snippet) High-level software written for the MicroBlaze

The code snippet shown in Figure 11 is a portion of the MicroBlaze's high-level source code, the part of the source code that most visibly demonstrates what the MicroBlaze is programmed to operate. Not shown, though, are important declarations, including the oled structure, pointers related to the oled structure (i.e. any variable whose name starts with oled), and the pointers related to the message and the semaphore.

The oled related variables, and related methods, are based entirely on a set of libraries developed by Adafruit. The original software (and this software) was written in C++ and thus needed to be re-written in C.

Application Software: APU's CPU0. The first processing core of the APU is programmed primarily to change the state of the LEDs, depending on the state of the slide switches and whether or not the state of the LEDs should be inverted. So, in addition to controlling the LEDs, the first processing core must also check the Mailbox IP core for a message indicating it is time to invert. Even though the first processing core is running FreeRTOS, only a single thread is needed to run the aforementioned tasks.

Figure 12. (code snippet) High-level software written for the APU's first processing core

Similar to the description of the MicroBlaze's software, only the software representative of the first processing core's highest-level of functionality is shown in Figure 12. The declarations and configuration-related function calls, such as initializing FreeRTOS and starting its dispatcher, are done in the main function, whereas the code shown in Figure 12 is called within a single thread.

Application Software: APU's CPU1. The final application is developed for the APU's second processing core, and its main role is to run the CLI, which is accessible through the ZedBoard's UART. In future projects, the CLI will support additional commands and also ways by which large amounts of data can easily be logged and organized for future use; but, for now, the CLI only supports the few commands necessary to carry out the project. Those commands include displaying a help message which enumerates the possible commands, posting a message to the OLED, and inverting the LEDs. Similar to the first processing core, the OS is FreeRTOS and the CLI is built to run as a single thread.

Figure 13. (code snippet) call-back function written for the CLI running on the APU's second processing core

Instead of showing the highest-level source as done with the MicroBlaze and the first processing core, the second processing core's functionality can more easily be explained with the CLI's callback function, as shown in Figure 13. The callback function is called whenever a carriage return is received via UART. The commandline structure and related methods are utilized in order to retrieve the parameters of any known commands. Other necessary parameters are also passed to the callback function, as well.

So as to minimize any confusion, know the Io pointers and related methods are used to read and/or write to something, where something, in the case of Figure 13, is the UART (i.e. the ioSerial pointer) and the shared memory from which the MicroBlaze reads (i.e. the ioShare pointer).

Conclusion. To recap, the user interacts with the ZedBoard via its slide switches, LEDs, CLI, and OLED. The user can change the state of the LEDs with each LED's respective slide switch or by inverting all the LEDs through the CLI. The user can also post messages to the OLED through the CLI. Internal to the Zynq EPP, the CPU of the ZedBoard, a soft-core and two hardcore processors are interconnected within the Zynq EPP's PL. The 3 processing cores are programmed such that the user can carry out the aforementioned tasks. And the primary objectives of the project can be summarized as inter-processor communication and thereof measuring the timing.

Again, most of the projects I have posted deal mostly with the internals of the Zynq EPP, rather than large embedded-system type projects wherein many different devices are connected and talking to each other. For now, since my Master's research demands it, I will focus primarily on the internals but improve the visual aspect of my projects with maybe graphs and a nicer CLI. Though, I do fully intend to do something big with the Zynq EPP, at some point.

Speaking of "at some point", at some point I will improve my recording situation with the videos. I am no film major, but I really do enjoy making the videos, even videos of my less visual projects. They all lack so much quality, though. Unfortunately, better videos probably will most likely be the last update to come.