Ring Buffers

Source code: https://github.com/DrMarty/STM32Utils/tree/master/UartIO

In my earlier post about redirecting printf() to a UART, I mentioned that the implementation was 'blocking'. This means that the calling code has to wait for all the data to be printed out the serial port before it can continue. If, for example, you are sending out a modest string of 20 characters at 115,200 baud, your code will be sitting idle for around 1.7ms, and at 100MHz clock speed, that translates to around 173,000 (single-cycle) instructions that could have done something productive in the meantime.
What you really want is to hand the character processing off to a background routine so your mainline code can execute as quickly as possible. One very effective way to do this is to implement a ring buffer.

Imagine an array of 16 characters and two indexes:

#define TX_BUFF_SZ 16

char TxBuffer[TX_BUFF_SZ];

int InsertIDX = 0;

int ExtractIDX = 0;

When we want to write data into the buffer, we use the InsertIDX to write into the next available slot, and then increment the index;

TxBuffer[InsertIDX++] = write_ch;

And when InsertIDX gets to the end of the buffer, we wrap it back to zero as follows:

If (InsertIDX >= TX_BUFF_SZ)

InsertIDX = 0;

This code works in the generic case, but if you initially set your buffer size to a power of 2, you can get a little fancy by logically AND'ing InsertIDX to mask off the high-order bits.

InsertIDX &= (TX_BUFF_SZ - 1);

Putting it all together, we can create a generic ring-buffer write function that also enables a transmitter empty interrupt as follows:

void rb_putchar(char ch)

{

   TxBuffer[InsertIDX++] = ch;

   InsertIDX &= (TX_BUFF_SZ - 1);

   TxEmptyInterruptEnable = 1;    //Enable the TxEmpty interrupt

}

So long as TxBuffer is at least as large as the largest amount of data we might want to send out in a single printf() call, the above code will allow us to buffer the data and return control back to the calling routine as quickly as possible - i.e. it is no longer 'blocking'.

Now let's deal with feeding each character one-by-one to the USART. This would be done inside an TxEmpty interrupt service routine (ISR). Observe that when ExtractIDX catches up to InsertIDX there is no more data in the ring buffer to send out so we need to disable further TxEmpty interrupts:

void ISR_TxEmpty(void)

{

   UartTX = TxBuffer[ExtractIDX++];

   ExtractIDX &=(TX_BUFF_SZ - 1);

   if (ExtractIDX == InsertIDX)

      TxEmptyInterruptEnable = 0; //Disable further TX empty interrupts

}

Caveats

To help keep the focus on the core concepts, I've kept the code above very simply, but in practice, you'll need to qualify your declarations of InsertIDX and ExtractIDX as volatile so that the compiler recognizes these variables can be updated outside of the normal function execution path.

stdio - to buffer or not to buffer?

Source code: https://github.com/DrMarty/STM32Utils/tree/master/UartIO

In my previous post, I showed how stdio could be redirected to send printf() output to an available USART. It was a basic implementation designed to get things moving, but you may encounter a little quirk when sending strings that don't end with a newline '\n' character.
By default, stdio buffers (holds onto) outgoing data until it sees a newline character; at which point it sends the entire line (to the serial port). The benefit of this is that it reduces the amount of low level interactions you need to have with the USART driver code, but the disadvantage is that you have to ensure each printf() call ends with a newline if you want the string to be immediately presented to the COM port.
I'll leave it up to you as to which implementation is best for your situation, but in case you decide to eliminate the input and output buffering, here's how to do it in C.
Somewhere in your initialization code, add the following lines:

setbuf(stdout, NULL);
setbuf(stderr, NULL);
setvbuf(stdin, NULL, _IONBF, 0);

The first line will remove buffering from stdout (which is where printf() goes by default)
The second line will remove buffering from stderr (if you are using that for directing error information)
The last line will remove buffering from stdin - which is the default for where scanf() will look for input.

The juicy details can be found here:
https://www.gnu.org/software/libc/manual/html_node/Controlling-Buffering.html

Implementing printf() on a UART

When developing on a new embedded (bare metal) platform, one of the first things I look for (after getting an LED to flash) is the ability to output text to a display. Usually the simplest path is to commandeer one of the UARTs and to start bashing characters out the TX pin and receiving them at the host end through a PuTTY terminal. So how to implement this efficiently?

The first question you'll need to resolve is how to get characters from your code into the UART. Hopefully the frawework you are using provides a simple setup process and wrapper to allow you to initialize the UART and send a single character. In the STM32CubeIDE environment, the UART properties can be setup fairly simply:

115200 is a pretty safe baud rate to start with. I've also shown the corresponding PuTTY settings too. Once you've proved that basic communications are working, you will probably be able to set the baud rate much higher; although take care when doing this. Even though the STM32 might be able to do 8Mbaud, FTDI-based USB Serial devices will top out at 3Mbaud: (https://www.ftdichip.com/Support/Knowledgebase/index.html?whatbaudratesareachieveabl.htm)

In any case, let's assume you've correctly setup the comms basics (don't forget to make sure the parity and stop bits match). Next job is to send something simple.

Once again, using the STM32CubeIDE environment as an example, ST offers a simple function call as part of the HAL (Hardware Abstraction Layer) interface:

HAL_StatusTypeDef HAL_UART_Transmit(UART_HandleTypeDef *huart, uint8_t *pData, uint16_t Size, uint32_t Timeout)

Assuming STM32CubeIDE has instantiated the UART handle as "huart1", you can make a simple call from your code like this:

char msg[] = "Hello World";
hres = HAL_UART_Transmit(huart1, msg, strlen(msg), 100);

Tying into printf()

Sending out "Hello World" might be initially very gratifying, but eventually you'll need to send out more advanced information such as variable values. One tempting option is to jump in and write your own MyPrintf() function, but that would involve a whole lot of re-inventing the wheel. A much better option is to tie into the existing functionality that already exists. Fortunately, the authors of most embedded frameworks understand that developers will want to redirect their output through various peripherals so they provide the infrastructure to readily support it.

If you dig deep enough, you'll find that after several layers of code, the standard printf() function distills down to one or two low-level functions. Take a look in the syscalls.c file for the _write() function:

__attribute__((weak)) int _write(int file, char *ptr, int len)
{
 int DataIdx;
 for (DataIdx = 0; DataIdx < len; DataIdx++)
 {
  __io_putchar(*ptr++);
 }
 return len;
}

Take note that this function is prefaced by __attribute__((weak)). This is a message to the compiler to look elsewhere in the code for a _write() function and use that one first, but if it can't find one then use this one. In effect, it allows you to implement your own _write() function within your codebase without needing to modify the library source files. A UART-based implementation would therefore be:

int _write(int file, char *ptr, int len)
{
 HAL_StatusTypeDef hres;
 hres = HAL_UART_Transmit(huart1, (uint8_t*)ptr, len, TX_BLOCKING_TIMEOUT);
 if (hres == HAL_OK)
  return len;
 else
  return -1;
}

With this function now in your code, you can access all of the wonderful formatting capabilities that come with printf() and your output will be magically directed to the serial port.

Concluding Remarks

The implementation above gets you up and running with basic character output capabilities but there's a hidden problem you may face further down the track. This implementation is 'blocking'; which means that it waits until all characters have been successfully piped out the serial port before it returns control back to the calling function. In timing critical applications or when sending large volumes of data, this could have an impact on the performance of your system, so in another blog entry, I'll talk about how to implement a ring-buffer and interrupts so your code doesn't have to wait around for the UART to finish its thing.

Source code: https://github.com/DrMarty/STM32Utils/tree/master/UartIO

fopen() - Text or Binary?

Its often convenient to control an embedded system remotely via some sort of communications protocol. In the good ole days, I'd devise some sort of homebrew communications protocol that was specific to my application's needs and I'd bash some data in and out of the serial port. Having grown up on 8-bit micros with limited RAM, every byte was valuable so there was a strong inclination to pack things tightly into binary streams, but the problem with that is how to test it... and before you know it I'm being drag into an endless rabbit hole of development on the PC side just so I could conveniently communicate with the embedded app.
Embedded processors have come a long way and right now, I'm getting up close and personal with some pretty nifty STM32 processor boards - this one in particular that I sourced from Banggood: https://sea.banggood.com/STM32F407VET6-Development-Board-Cortex-M4-STM32-Small-System-ARM-Learning-Core-Module-p-1460490.html

Compared with an 8051, these ARM devices are a beast; and the development boards are so cheap! It's time to cast off the constraints of decades past and update my comms routines... enter JSON.
JSON is sooo simple even I can wrap my head around it; it's text based which means testing it with my embedded app is simply a case of sending text files through a PuTTY terminal. Furthermore, because JSON is used so widely across the winternet, I'm pretty confident I'll be able to keep using it even if I upgrade my comms to Etherweb.
I dug around and quickly settled on a relatively compact JSON interpreter (called JSMN) from Serge Zaitsev (kudos @zsergo): https://zserge.com/jsmn/. It comes packaged as a single C header file which feels a little hacky but its nonetheless effective and I was able to get a basic command interpreter up and running on my target board pretty quickly.
At that point, I started thinking more about what sort of commands I might want to exchange and it became pretty clear that I was going to need some sort of generic command interpreter that was easy to abstract and extend.  That's going to be a work in progress but the relevance to today's discussion is how it drove me back to establishing a parallel development platform that would allow me to develop and test code on my PC rather than needing to run everything from my target (ARM) platform, and how I bumped into a nasty little PITA on how Windows processes text files.
I'm not going to post my original code because some dufuss will blindly cut and past it into their application and wonder why it doesn't work.. instead I'll post the working code:

 ============================================================================

 Name        : main.c

 Author      : Marty Hauff

 Version     :

 Copyright   : Use at own risk

 Description : Test scaffold for reading a file and processing...

 ============================================================================

 */

#include <stdio.h>
#include <stdlib.h>
#include <limits.h> //PATH_MAX
#include <unistd.h> //getcwd()
#include <sys/stat.h> //struct stat
#include <string.h> //strlen()

#define MAX_TOKENS 256

int main(void) {

FILE* fp;

char* Filename;

struct stat st;

char cwd[PATH_MAX];

char* json_string = {0};

printf ("\nJSON Test\n");

getcwd(cwd, sizeof(cwd));

if (cwd != NULL) {

printf ("Current working dir: %s", cwd);

} else {

perror("getcwd() error");

return EXIT_FAILURE;

}

Filename = "JSON Examples/JSON_2.json";

fp = fopen(Filename, "rb"); //MUST use binary mode otherwise \r\n sequences get messed up!!

if (fp == NULL)

{

printf ("\nFailed to open %s", Filename);

return EXIT_FAILURE;

}

printf ("\n\"%s\" opened successfully", Filename);

stat(Filename, &amp;st);

printf ("\nStat Size: %ld", st.st_size);

fseek(fp, 0, SEEK_END);

printf ("\nfseek Size: %ld", ftell(fp));

fseek(fp, 0, SEEK_SET);

json_string = malloc(st.st_size+1);

fread(json_string, st.st_size, 1, fp);

fclose(fp);

json_string[st.st_size] = '\0';

printf ("\nstrlen Size: %d", strlen(json_string));

printf ("\n%s",json_string);

jsmn_parser p;

jsmntok_t tkns[MAX_TOKENS];

int nodes = 0;

jsmn_init(&amp;p);

nodes = jsmn_parse(&amp;p, json_string, strlen(json_string), tkns, MAX_TOKENS);

printf ("\nFound %d nodes", nodes);

free(json_string);

return EXIT_SUCCESS;

}

The thing that got me stuck for a day was that pesky little 'b' character in the fopen command. Here's what the output looked like without it (I'm using some test JSON from: https://json.org/example.html):

JSON Test
Current working dir: C:\Users\user\Documents\Projects\WinTest
"JSON Examples/JSON_2.json" opened successfully
Stat Size: 253
fseek Size: 253
strlen Size: 253
{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}
âãäåæçèéêëì
Found -2 nodes

I couldn't work out what was causing the garbage at the end of the stream (second last line) and why jsmn was subsequently failing to parse the file. The clue is that there are the same number of garbage characters as there are lines in the JSON string. Take a look at a hex dump of the JSON file:

Take note of the 0D 0A sequence. Also take a look at these settings in Notepad++

Without the 'b' in the fopen() call, the file is opened as a text file and Windows strips out all the CR (0x0D '\r') characters when its reading the contents into a buffer. As a result, the string stored in memory is actually shorter than the number of characters read from disk.
Now look at the output when the 'b' is included in the fopen() call (i.e. as per the code listing above):
JSON Test
Current working dir: C:\Users\user\Documents\Projects\WinTest
"JSON Examples/JSON_2.json" opened successfully
Stat Size: 253
fseek Size: 253
strlen Size: 253
{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}
Found 26 nodes

The CR character remains in the text stream so the console output (on Eclipse) double spaces each line, but take note that we no longer get the garbage at the end of the stream; AND jsmn has managed to parse the file correctly.

Take note of the moral of the story - consider reading a file in binary mode even if you know the contents are text.

Now to work out how to build a generic JSON command interpreter.

Return from hiatus

After almost 10 years since my last blog, I've decided it's about time I started writing up a few things again. It's been an eventful 10 years... not the least of which was living in China for almost 8 of them. The great firewall of China blocks this blogger site so it was increasingly difficult to make entries and, for the most part, I was so busy doing life that finding time to update this blog just became one of those things that fell off the priority list.
In any case, kids are now living their own independent lives back in Australia and a year ago wifey and I moved to Singapore and I've got a lot more control over my time.  And with this Covid-19 shenanigans I've been recently put out of (working for the man) work and so I've got plenty of time to pursue some projects of my own. Over the past couple of years, I've also bought some reasonably good test equpment (DSO, PSup, Sig Gen & Desktop Multimeter) so I'm pretty well equipped to do development at home. One of these days I'll do a writeup of my lab setup.
For now, this is a quick hello to the world and I hope to be keeping this site much better fed.