Continuous Integration of Embedded Software in the Cloud – Docker, Simplicity Studio & CircleCI

Uncannier Software November 29, 2018

In this article, I’ll show you how to create a Docker image for Continuous Integration of embedded software projects. I’ll use Silicon Labs Simplicity Studio and Circle CI as an example. However the basic principle could be applied to many embedded software tools and many CI tools.

Embedded Complication

In recent years there’s been a big move towards continuous integration in the cloud with the rise of tools like Travis CI, CircleCI, Bitbucket Pipelines and more. CI/CD in the cloud is challenging for many embedded software projects because of our arcane tools and ancient licensing models. Putting together a cloud build environment is non-trivial. And it’s not like CircleCI are going to offer a standard Keil uVision Docker image any time soon.

Simplicity Studio

Silicon Labs’ Simplicity Studio is a YACE. Yet Another Customized Eclipse. Like Atollic TrueSTUDIO, like NXP MCUXpresso and many more.

Simplicity Studio has some characteristics that make it difficult to wrangle it into a CI build environment. As at version 4.2.0.201811151901-172 of Simplicity Studio ARM IDE:

The base installation, which can be done headlessly, does not include any SDKs or build tools. For example, the Bluetooth SDK and the GNU ARM Toolchain.
To install the SDKs and build tools, it’s necessary to first login to your Silicon Labs account. It’s currently only possible to login via the Simplicity Studio GUI. Headless login is not possible.
Once logged in, it’s presently only possible to install the desired SDKs and build tools using the GUI. Headless installation is not possible. However, having contacted Silicon Labs support, I can tell you that headless installation is coming soon.
Simplicity Studio very much intends for your projects to be managed make IDE projects. Unlike stock Eclipse CDT, there’s no out-of-the-box support for Makefile projects. Code generation is strongly encouraged through the AppBuilder Bluetooth Configurator. “Install New Software” is removed from the Help menu – they really don’t want you to install non-Silabs plugins.

The bottom line is that it’s presently impossible to fully automate creation of a build environment through something like a Dockerfile. You are forced to at least partially handcraft the environment. This situation will be sadly true of many embedded IDEs, and certainly true of the likes of Keil uVision.

Mercifully, Simplicity Studio at least does not suffer any licensing complication. No node locked license, no floating license, no worries.

Docker Image

The summary is this: hand-install the IDE and tools, then build a Docker image by copying the installation tree.

Here are my repositories:

My development host is an Ubuntu 18.04 64-bit VM. If you insist on maximizing your suffering by working under Windows, you’ll have to adapt the steps accordingly.

Of course, I assume you have Docker installed, and you know how to use it.

OK, now here’s the detailed recipe of how to create the Docker image.

Handcraft Your IDE Installation

In my case, I did a fresh install of Simplicity Studio to /opt. I then logged in and installed my desired SDKs and tools:

Bluetooth SDK 2.10.1.0 (with Gecko Platform 2.4.1.201811021046-152)
GNU ARM Toolchain v7.2.2017.q4
and a few other bits

Wrapper Script

Before a Dockerfile can be used to build the Docker image, it’s first necessary to copy the Simplicity Studio to the Docker context. To automate that, I’ve wrapped the Docker build command in a shell script (docker-build.sh). The shell script does the copy, but also prunes out some stuff that isn’t needed for the CI environment, and retains only one version of the SDK. This trims the resultant Docker image size.

With this automation, it’s easy to generate an updated Docker image when Silicon Labs release updated SDKs.

https://github.com/gregbreen/uncannier-thunderboard-docker/blob/master/docker-build.sh

#!/bin/bash
 
# Check that the Gecko SDK Suite version has been passed as an argument
if [ $# -ne 1 ]; then
	echo "Usage: $0 [Gecko SDK Suite Version Number. e.g. 2.4]"
	exit 1
fi
 
# Confirm that the Gecko SDK Suite version actually exists
if [ ! -d "/opt/SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1" ]; then
	echo "Gecko SDK Suite $1 not found in your Simplicity Studio installation"
	exit 1
fi
 
# Simplicity Studio does not support headless install of SDKs, so we
# copy our installation into the Docker context so we can build a
# Docker image with the necessary build tools
echo Copying Simplicity Studio installation to the Docker context ...
mkdir -p SimplicityStudio_v4
cp -R /opt/SimplicityStudio_v4 .
 
# Get rid of all SDKs except the one we currently need to build the Thunderboard projects
mv SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1 v$1
rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/*
mv v$1 SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1
 
# Clean out other big things we don't need for CI of Thunderboard projects
rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/app
rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/hardware
rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/platform/hwconf_data
rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/util/third_party/emwin
 
echo Running Docker build ...
docker build --tag uncannier-thunderboard:gecko-sdk-suite-v$1 .
 
echo Cleaning up ...
rm -r SimplicityStudio_v4
 
echo Push as "docker push gregbreen/uncannier-thunderboard:gecko-sdk-suite-v$1"

Dockerfile

After setting up the context, the wrapper script triggers the build of the Docker image. The only real trick with the Dockerfile is that it should use the same OS base image as your host. This ensures that the copied Simplicity Studio installation will be able to execute in the Docker container.

https://github.com/gregbreen/uncannier-thunderboard-docker/blob/master/Dockerfile

FROM amd64/ubuntu:18.04
 
LABEL maintainer "Greg Breen <greg@uncannier.com>"
LABEL description "Simplicity Studio in a container for CI of Thunderboard projects"
 
# To the best of my knowledge, Silicon Labs do not support headless install of 
# Simplicity Studio. We can do the base install headlessly, but we can't install
# the SDKs (Gecko and Bluetooth for example) without running the GUI. Thus we
# build this Docker image by copying a Simplicity Studio installation that already
# has the SDKs installed. Therefore this Dockerfile should be executed in a Linux distro
# that is the same as the Linux distro used as the base image here. Ubuntu 18.04 amd64 
# in my case.
RUN mkdir -p /opt/SimplicityStudio_v4
COPY SimplicityStudio_v4 /opt/SimplicityStudio_v4
 
# Need to install Simplicity Studio's prerequisites - we use its setup script to do
# this. As a consequence, we need to setup udev rules even though this image is only 
# used for building.
RUN dpkg --add-architecture i386
RUN apt-get update
RUN mkdir -p /etc/udev/rules.d
RUN sed -i "s/sudo apt-get install/apt-get install -y/" "/opt/SimplicityStudio_v4/setup.sh"
RUN /opt/SimplicityStudio_v4/setup.sh
 
# Need 64-bit edition of Qt5 for Simplicity Commander to run
RUN apt-get install libqt5gui5
 
# Need to install the following package to avoid warnings during builds
RUN apt-get install -y libcanberra-gtk-module
 
# Put Simplicity Commander in the path - our build calls Commander
ENV PATH="/opt/SimplicityStudio_v4/developer/adapter_packs/commander:$PATH"

With thanks to Daniel Meer for the sed trick: https://github.com/meerdan/docker-simplicity-studio

Docker Push

Even with the trimming, the resultant image is a tad hefty. A Docker push will get it to Docker Hub, or your private repository, but you’ll need some patience.

If you consult Dr Google, you’ll find lots of self-styled Docker gurus saying that image sizes should never be in the GB. These people haven’t suffered with embedded software tools.

CircleCI

At last, we get to the payoff.

I won’t go to detail on how to setup your projects in CircleCI. Let’s just pick the eyes out.

YAML File

Here’s the config.yml as added to the Uncannier Thunderboard Sense 2 project.

I elected to not use Silicon Labs’ Python scripts for headless builds: https://www.silabs.com/documents/public/application-notes/an1121-headless-builds.pdf. I think they offer no material advantage over using the vanilla Eclipse command-line approach.

version: 2
jobs:
  build:
    docker:
      - image: gregbreen/uncannier-thunderboard:gecko-sdk-suite-v2.4
    steps:
      - checkout
      - run:
          name: Build the Uncannier Thunderboard Sense 2 bootloader
          command: /opt/SimplicityStudio_v4/studio -nosplash -application org.eclipse.cdt.managedbuilder.core.headlessbuild --launcher.suppressErrors -import uts-bootloader -cleanBuild uts-bootloader/Release
      - run:
          name: Build the Uncannier Thunderboard Sense 2 application
          command: /opt/SimplicityStudio_v4/studio -nosplash -application org.eclipse.cdt.managedbuilder.core.headlessbuild --launcher.suppressErrors -import uts-application -cleanBuild uts-application/Release
      - run:
          name: Package the build artifacts
          command: |
            cd uts-application/Release
            mv uts-image.hex uts-image-$CIRCLE_BUILD_NUM.hex
            mv uts-application.gbl uts-application-$CIRCLE_BUILD_NUM.gbl
            mv uts-apploader.gbl uts-apploader-$CIRCLE_BUILD_NUM.gbl
            tar -cvzf uncannier-thunderboard-sense2.tar.gz uts-image-$CIRCLE_BUILD_NUM.hex uts-application-$CIRCLE_BUILD_NUM.gbl uts-apploader-$CIRCLE_BUILD_NUM.gbl
      - store_artifacts:
          path: ./uts-application/Release/uncannier-thunderboard-sense2.tar.gz
          destination: uncannier-thunderboard-sense2.tar.gz

Build History & Logs

Here are the Uncannier Thunderboard projects on CircleCI:

Click through to see the build logs.

Conclusion

I have many CI battle scars, and most of them are with Atlassian Bamboo and various AWS EC2 build agents or locally-hosted VM build agents. Docker and cloud CI, in my view, has a lower barrier to entry. If you can get the lasso onto your embedded development tools, there’s no real excuse to not have CI on your embedded software project.

Uncannier Pull Requests

References

Tagged ci, devops, docker, embedded, thunderboard

14 thoughts on “Continuous Integration of Embedded Software in the Cloud – Docker, Simplicity Studio & CircleCI”

Daniel says:

January 11, 2019 at 3:07 pm

First off, ace series, I’m always amazed how little CI there is in the embedded world. Thank you for putting these articles together!
I’ve got ESP32 builds using docker in CircleCI, but have got stuck with the Silabs EFM8BB series.
It uses the KEIL C51 compiler, which you don’t need a license for below a certain code size, so builds in CircleCI work perfectly there.
However, over a certain code size, you need to register a license which is linked to specific hardware Ids.
Have you managed, or had thoughts about how you’d get hardware linked compiler licenses working on Cloud CI tools like CircleCI?

Reply
1. Uncannier Software says:
  
  January 16, 2019 at 5:02 pm
  
  Thanks Daniel.
  
  I have done CI for Keil uVision (MDK-ARM). That is with Atlassian Bamboo, and with an elastic build agent in AWS EC2. The trick is to use a floating license, instead of a node-locked license. So you need a FlexLM server, which I also have in AWS EC2. The uVision in your Docker image will need to be configured with the IP address and TCP port of your FlexLM server. Then you’re in business.
  
  The downside is you have to buy a floating license.
  
  Reply
  1. Wally4u says:
    
    March 23, 2019 at 5:07 am
    
    To my surprise ARM does not seem to have any intention of using CI for their tools.
    
    Personally I would love to see a blog post on how you managed to get Keil uVision (MDK-ARM) running with Bamboo on AWS EC2. I actually tried this a year or so ago and it was a nightmare even on a local windows server with a local agent.
    Basically I got so far that running a local script to execute the project but it was messy (local privileges etc).
    
    In an ideal world I would love to see a Keil uVsion installation inside a docker Windows Core image.
    Spin up a instance, run the docker image, build & cleanup.
    
    Any idea if that would be even possible?
    ThankS
    
    Reply
    1. Uncannier Software says:
      
      March 23, 2019 at 9:37 am
      
      Groeten Wally!
      
      You are probably right I should blog about it. In the meantime, here are the steps I did, in a nutshell:
      
      Launch Atlassian’s stock Windows build agent: https://confluence.atlassian.com/bamboo/stock-images-289277463.html
      
      Install Keil uVision.
      
      Configure uVision to obtain a floating license from your FlexLM server.
      
      Create a new AMI from this running instance.
      
      Configure your Bamboo server to launch this new AMI automatically.
      
      Then you can configure your build task like this:
      
      for %%x in (Debug,Release) do (
      
      echo Building target %%x …
      
      uv4 -j0 -cb WallysExcellentProject.uvprojx -t”%%x” -o “buildlog.txt”
      
      if ERRORLEVEL 1 (
      type buildlog.txt
      echo Build error level: %ERRORLEVEL%
      exit /B 1
      ) else (
      type buildlog.txt
      )
      )
      
      That will build all of the configurations in the project, fail the build on any warnings, and put all of the uVision build output into the Bamboo logs.
      
      To my surprise ARM does not seem to have any intention of using CI for their tools.
      
      I think embedded software tools are generally not modern. Many are Windows-only, and none of the cool kids use Windows. CI, and virtualization in general, is just more painful with Windows.
      
      In an ideal world I would love to see a Keil uVsion installation inside a docker Windows Core image.
      Spin up a instance, run the docker image, build & cleanup.
      
      I am 99% sure it’s possible. I personally have never made a Windows-based Docker image though.
      
      More generally, I’d ask why Keil? In my experience, Keil produces only marginally more optimal code than GCC ARM. Not enough that it’s worth the money and complication IMO. If you leave Keil behind, the whole world of Linux opens up.
      
      Reply
      1. Wally4u says:
        
        March 23, 2019 at 10:04 pm
        
        Awesome (bedankt!)
        
        I’ll give it a try upcoming weeks to see if I can get it working.
        
        Why Keil? It because the entire dev team works on Windows/Keil (Long History) and we do a lot of bare metal programming (M0 to M4) on custom hardware poking bits etc.
        ARMCC 6 is based on LLVM which is a lot better than ARMCC 5 these days perhaps we will switch someday.
        
        First step is CI and TDD to improve turnaround and quality 🙂
Andrew Keener says:

December 20, 2019 at 2:44 am

Hey Greg,

Having an issue with CircleCI and this image building. My build fails after simplicity starts up headless and here is the output:

Unable to init server: Could not connect: Connection refused
Studio: Cannot open display:
Studio:
GTK+ Version Check
Create.
Opening ‘ooler_firmware_bgm111_v13’.
Refreshing ‘/ooler_firmware_bgm111_v13’.
Unable to find full path for “gcc”
Unable to find full path for “g++”
Unable to find full path for “gcc”
Unable to find full path for “g++”
Studio:
An error has occurred. See the log file
/root/SimplicityStudio/v4_workspace/.metadata/.log.

Exited with code exit status 13

Its almost as if its not seeing the GNU toolchain. But I’ve checked the image and it is definitely their in the folder.

Thanks for any help you can give me.

Reply
1. Uncannier Software says:
  
  December 20, 2019 at 8:25 pm
  
  Hi Andrew,
  
  Good to hear from you.
  
  Forgetting the Docker container and forgetting CircleCI for a moment, can you build from the command line on your development machine? I think that should be established first, if you haven’t already.
  
  However I think there’s a clue in the exit status 13. See here: https://stackoverflow.com/questions/27019786/eclipse-java-was-started-but-returned-error-code-13
  
  I don’t really see how that mismatch could happen with Simplicity Studio since it comes with its own JRE embedded in the installation. But perhaps something to look into.
  
  Reply
  1. Andrew Keener says:
    
    December 20, 2019 at 10:15 pm
    
    Thanks for the quick reply Greg,
    
    So everything is building in the image where I have Simplicity Studio installed. I used your script and dockerfile almost verbatim to build the image I then push to CircleCI. I should say that I’m doing this all on windows but I use my working docker image (same Ubuntu 18.04) to do the docker build of the final image by piping docker into the image with:{ docker run -ti -v /var/run/docker.sock:/var/run/docker.sock docker} and then running docker build.
    
    Further, after building my image and running it locally. I experience the same problem. So something is not getting set up correctly in the docker build. I was going to post them but as I stated above, pretty much verbatim from this guide.
    
    I’ll try running the build and see what version of JVM is being installed.
    
    Thanks!
    
    Reply
    1. Andrew Keener says:
      
      December 23, 2019 at 11:38 pm
      
      By editing the build script and pulling out the “cleanup” portions I was able to get the build running on CircleCI. This is in all likelihood to do with the weird dependencies in our project. The original project was authored by a contractor and is a bit of a mess.
      
      Either way. Thanks for your help Greg! The tutorials are awesome!
      
      Thanks,
      
      Reply
      1. Uncannier Software says:
        
        December 24, 2019 at 5:22 am
        
        I’m glad to hear it’s working. By “cleanup portions” do you mean these 4 lines of my docker-build.sh file?
        
        rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/app
        rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/hardware
        rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/platform/hwconf_data
        rm -r SimplicityStudio_v4/developer/sdks/gecko_sdk_suite/v$1/util/third_party/emwin
        
        That would certainly make some sense to me.
      2. Andrew Keener says:
        
        January 3, 2020 at 3:37 am
        
        Yep. Removed those lines. I don’t think the app folder or conf data mattered but my project was looking for something in hardware and emwin I believe.
Rafael Ribeiro says:

May 28, 2020 at 6:37 pm

I’m having a bit of trouble Installing the SDK’s within Simplicity Studio, are there some gotchas that someone has to watch out when creating the VM?

Reply
Rafael Ribeiro says:

May 30, 2020 at 1:13 am

Hi. I was able to follow this tutorial to setup my own CI environment, however I also have this error:
java.lang.RuntimeException: Application “org.eclipse.cdt.managedbuilder.core.headlessbuild” could not be found in the registry.

I’m fairly certain the problem is recreating the Silabs environment on Linux. I’m an infrastructure guy but i’m helping out the Firmware guys setting up a CI for this.

I installed :
– Bluetooth SDK 2.11.5
– GNU ARM Toolchain q4 2017
– Silabs Commander

My Dockerfile is really similar to yours and I’m also setting up the VM automatically with Vagrant and Ansible (except for the manual parts where you have to login and install the above packages). I’ll be happy to share it with you if you want.

My guess is that I’m missing a component for a working environment, but from the Package Manager can’t quite figure out which one.
Also to help matters, the firmware guys develop on Windows, so they didn’t help me a whole lot..

Do you have any opinion on what It could be?

Reply
1. Uncannier Software says:
  
  June 7, 2020 at 11:08 am
  
  Hi Rafael.
  
  Sorry for the slow reply. Yes you are seemingly missing a package from your Simplicity Studio installation. So I presume it’s not in in the installation on your VM, or your Dockerfile is failing to put it into the container image.
  
  I suggest you try to perform a (headless) build inside your VM. And confirm your Simplicity Studio installation there is good. Then you can isolate the problem as being in the VM or in the container.
  
  I’d love to see the Vagrant and Ansible scripts. I setup my VM by hand because it’s my dev VM, but for sure I agree automation is better for creating the build environment.
  
  Reply

Uncannier Software