Open Source Firmware - Escape the proprietary cave

1 Preface

II have been doing firmware development for more than 12 years for a large German industrial group. I have started my professional firmware development career with x86 based systems. Back then, the old BIOS solution was currently phasing out, but I was able to get a good insight into the techniques and development flow coming from this approach. Then it was replaced by a new, standardized solution called UEFI. After a few years of development with UEFI on our devices and all the pain it brought from my point of view, I moved on and finally settled with coreboot, an open source project to serve the same demand, booting our devices into the OS.

This article will take a closer look at proprietary firmware solutions and the development workflow I was working with. Further on, the modern practices in the open source firmware development will be presented and benefits as well as drawbacks will be discussed.

All the aspects and arguments in this article are reflecting my experience with the different solutions and development flows in an active product development. For sure there can be different perceptions out there about the content which is mentioned in this article. Of course all of them might be applicable depending on the different situation they are coming from but the aim of this article is not to take each and every pathway into account.

2 What is meant by firmware in this article

The term ‘firmware‘ describes a piece of software that is tightly coupled to the hardware. It is stored on non-volatile memory devices directly located on the given circuit board. The firmware installation is usually done as part of the device manufacturing process by the OEM (Original Equipment Manufacturer) or ODM (Original Design Manufacturer) and the device will not start up properly if the firmware is missing or corrupt.

This article describes in particular firmware for x86 based systems. A very common term for this is the BIOS (Basic Input/Output System). In the last two decades the legacy BIOS was more and more suppressed by a different firmware stack called UEFI (Universal Extensible Firmware Interface). Both of the aforementioned solutions are closed source, proprietary implementations which usually come with a license fee. In addition, there are open source alternatives available to serve the same demand, one of these is called coreboot, which is a GPL project freely available to everyone.

3 The typical development flow with proprietary firmware

Proprietary firmware for x86 systems is usually provided by so called Independent BIOS Vendors (IBVs). These companies are specialized in cross-platform (client and server side) enablement including source code development and maintenance services, custom OEM tooling and bug fixes during the entire product life-cycle. Let’s evaluate how a new hardware product (e.g. PC mainboard of a consumer device) is brought to life at the hardware vendor(OEM or ODM). An OEM is developing a design but does not sell it by himself. Instead, a company which is then really selling a device to end customers reaches out to an ODM which will take the design from an OEM and make a real product out of it on behalf of the selling company.

It starts with the hardware design phase which takes care of the schematic and board layout. At this level the hardware feature set is defined by the silicon selection and the interconnects on the circuit board. Typically, the hardware design and board manufacturing is managed by the ODM with guidance of an OEM. Once this phase is completed and the hardware is manufactured, the next step is to populate it with a firmware. Usually, the hardware vendors do not have the needed knowledge or resources to develop the firmware for their product on their own from scratch. Instead, they reach out to IBVs which provide standardized firmware stacks. As the developed hardware is never 100% identical to all the other hardware designs out there, a certain level of adaptation is required on the firmware side. There are a few options how this adaptation can be achieved:

• Option 1: Ask the IBV to do this as a service
• Option 2: Ask a third party company to do the needed adaptation. This company will likely work closely with an IBV.
• Option 3: Acquire the firmware stack from an IBV and do the adaptations by dedicated firmware engineers at the hardware vendor

In any case the IBV’s firmware stack is used as the basis for this adaptation. The first option requires the hardware vendor to share development details like schematic with the IBV. This can be very demanding and not every vendor is willing to do this for different reasons. Option 2 is even problematic as a third company is involved and needs to know hardware and firmware details, which is often hard to deal with. From my experience the often chosen option is therefore option 3, which means the needed adjustments will be performed at the hardware vendor side. To show the full development flow and all its dependencies, let’s take the last option here as an example (this is how I was working with UEFI).

So the hardware vendor purchases the firmware stack from the IBV of its choice. The delivery is, in the case of the nowadays standard UEFI firmware, a source code package with multiple ten thousands of files. It contains the real source code as well as the needed build system to compile the source code to a single image. The firmware engineers of the hardware vendor need to handle this giant amount of code in this scenario. This task is in practice very cumbersome due to the nature of proprietary firmware. There are some reasons and explanations I would like to point out as follows:

First, the IBVs have a huge development team contributing to the firmware stack. The aim is to support all the features of a given platform in a single firmware stack as the real use case of this stack is defined by the hardware capabilities. The code base grows over time. Some parts have been ported over from previous versions of the firmware stack and might not have been cleaned up properly to match a possible new structure. Other parts may have been provided by a 3rd party. A large amount was newly developed just to cover the needs of the new platform. Often there are complex override mechanisms for a big portion of the code which results in having the same files being present multiple times in the code tree, at different locations with in-transparent dependency rules. All this increases the number of source code files in the firmware stack and its complexity.

Secondly, the complex nature of modern silicon designs cannot be handled by the IBVs themselves. The silicon vendor is the only reliable source of knowledge for the latest silicon designs. Additionally, the silicon vendors often have their own reference firmware implementation which they need for early silicon commissioning. To ease the silicon platform integration at the IBVs, the silicon vendors provide their reference code (or parts of it) to the IBVs which is now included into the firmware stack at the IBV. Since this reference code is written by the silicon vendor and not by the IBV, it often does not match the structure and architecture the IBV uses in their firmware stack. And often there is no common code style available across these parts. Therefore, abstraction layers and a lot of glue code are used inside the firmware stack to enable the reference code usage. In some cases the silicon vendor is not able or willing to provide source code for a part of the platform (e.g. because this part was acquired from a supplier by the silicon vendor and the license does not cover code redistribution). In these cases the delivery form is a binary image (blob) with a defined API to call into.So in the end the firmware engineer at the hardware vendor has to deal with a huge code base which was developed by two or more big parties, at least the IBV and the silicon vendor. It even may contain blobs with little possibilities for modification. Figure 1 shows the code composition the hardware vendor needs to handle in this scenario.

Now this code base needs to be tailored carefully to match the use case the new hardware implements. The closer the hardware implementation is to the reference design of a silicon vendor, the less adaptations are needed in the firmware stack. If one has a more specialized hardware for industrial or IoT use cases, the amount of required adaptations rises quickly and the risk grows to configure or adapt the provided firmware stack in a bad or inappropriate way.

3.1 Problems with proprietary firmware development

If we look further in the drawn scenario, the hardware vendor’s firmware engineer is now responsible for the adaptation of the provided firmware stack. The engineer needs a deep knowledge of the firmware source code in order to implement the required changes in the right way, without introducing hidden bugs or even security issues. This knowledge, once built up at the hardware vendor, cannot always be transferred to a new platform smoothly as in the meantime the IBV continues developing his firmware stack. So at the time a new CPU platform is used on a new design a few years later, the then acquired code base can be quite different compared to the last one from the same IBV. The engineer at the hardware vendor needs to go through the steep learning curve again, investing a lot of time and making new mistakes. If a different IBV is used, the whole case is just worse.

Now imagine there is an issue found by the engineer in the provided source code, either a general one or a special one dedicated to his use case. In such scenarios the usual process would be to reach out to the IBV and ask for help. This support is often covered by dedicated service contracts which the hardware vendor needs to pay for. In some cases the IBV might be able to provide a fix for the issue quickly. In other cases the issue might be more complex and the needed support hours to reproduce, find and fix the issues at the IBV’s side might count up fastly. In even more complex scenarios the root cause for the issue might be in the portion of the code which was delivered by the silicon vendor or, even worse, by its suppliers. Now, the IBV has to reach out to the silicon vendor and ask for help, for which the IBV needs to be able to reproduce the issue on his side and provide detailed information to the silicon vendor. Often it is the customer’s duty to provide a way to reproduce the issue on a reference design. Otherwise, neither the IBV nor its supplier will start an investigation for the root cause. This again increases efforts at the hardware vendor. One can easily see how quickly this model will become overloaded with a lot of parties involved in the solution with service hours growing quite fast. Of course the hardware vendor has to pay for all these service hours. Beside this, the time to get a fix in such a scenario can reach months. This in turn slows down the firmware adaptation process on the hardware vendor side a lot.

The other aspect of issues in such a development process is the code quality. The most important thing from the IBVs point of view is to deliver a functional code base in time. The code quality is not in the focus as a customer usually pays for functionality. This strategy leads to new problems as over time the code base will become really hard to maintain. Bugs will be hard to find or track which again will increase the requested service hours and slow down development dramatically.

Even when the hardware vendor was able to fix a bug in the purchased code base of an IBV by himself, there is no guarantee that this finding will ever find its way back to the IBV. And even if it will be reported, the implementation in the code base at the IBV is not necessarily done. The next customer of this IBV might get the old version of the code tree delivered and can step into the same pitfall again, investing now additional time to find and fix the same issue at his end. There is no established workflow between the IBV and its customers when it comes to reporting issues to the IBV which again results in bad code quality over time.

Sometimes there are contracts between the IBV and the hardware vendor that forces the hardware vendor to pass the ownership of the code changes the engineer at the hardware vendor develops at his end to the IBV. This makes it highly difficult for the hardware vendor to add additional, differentiating features to his product as they need to be passed back to the IBV and the IBV, being the owner of them per contract, can freely sell these features to the next customer (which easily can be a competitor of the original hardware vendor).

3.2 What will you get with proprietary firmware

As described, the development process with proprietary firmware can be tedious and cumbersome. But there is a second view which reveals benefits of this firmware approach.

First, there is a contract between the IBVs and their customer. Usually such a contract includes support from the IBV which the customer can count on if any cases or issues are coming up. Of course this service needs to be paid for but on the other hand it provides the customer a convenient way to get knowledge from the IBV and fix arising issues faster.

The second benefit is the fact that proprietary firmware is still mainstream. If a hardware vendor uses a proprietary firmware on its product, this product will be able to boot all the operating systems out there out of the box. The OS vendors treat the proprietary firmware case as de facto standard and provide all the needed implementation for such a scenario. All the tests are done with proprietary firmware in mind.

Once shipping of the product with proprietary firmware has started, the maintenance of the firmware becomes hard. The hardware vendor usually provides firmware updates in the first few months up to a year or two of the product life cycle. These updates are provided to extend functionality (add support for newer versions of silicon or memory modules), fix minor bugs that have been found at the hardware vendor side or provide security updates. Improvements in the firmware stack driven by the IBVs frequently do not flow back into this product since at this time the support contracts between the IBVs and their customers may have expired. So the end customer, who owns this device now, often has to live with issues in the firmware limiting the user experience. As there is no way that an owner of the device can have access to the firmware stack, an incident in the firmware cannot be fixed by the owner or other enthusiasts. Therefore, often fixes for a firmware bug are applied on the operating system level (the Linux kernel has a lot of such ‘fixes’) which in turn shifts the maintenance burden to the wrong level and makes the maintenance of the operating system complex.

4 Open source firmware

The open source software approach is available now for decades and has proven its power and benefits in various software projects all around the world. Starting with small applications driven by enthusiasts reaching to big enterprise solutions and operating systems all of us have had contact with open source products. No wonder that this model was adopted even for x86 firmware in the late 1990s already. In the meantime the number of projects around open source firmware has increased. Projects like coreboot, Tinaocore or Slim Bootloader are well known and actively developed. And the usage of such open source firmware grows continuously.

4.1 Open source firmware development flow

Let’s have a look at how a development process in the open source firmware domain looks like. Unlike the proprietary firmware there is no dedicated company that owns the open source firmware. Contrary, an open source project is driven by an open source community which consists of different members. There are pure enthusiasts spending their free time and passion to support an open source project development. There are members working for companies which use the open source firmware in their products or even offer services like an IVB for the open source domain. There are even members from different silicon vendors if the silicon vendor is interested in support for his latest platforms in a given firmware project. All of the mentioned members are working collaboratively in the open source firmware project and composing the community. Usually, the development inside the community is transparent and everybody has access to the changes and the whole code tree, even people who are not part of the community. And everybody is welcome to participate at any time. The well mixed community usually ensures high code quality as a broad and publicly visible review process can be established and maintained.

Of course this development model needs rules in order to keep the project in a good shape. These rules cover processes which describe how the development is done (where the repository is stored, how patches are pushed upstream, what the merge policy looks like, ...), include coding guidelines and describe the test infrastructure. They are accessible for everybody so that the overall project policy is known to all the members and possible users of an open source firmware.

Let’s now take the depicted example from chapter 3 (the proprietary workflow) where a device manufacturer chooses an open source firmware for its new product this time. Let us further assume that the hardware vendor is an active community member (the other case will be discussed later). The adaptation needed in the firmware for the new product now needs to be driven by the hardware vendor. So the engineer creates patches for the open source firmware and pushes them upstream. Now the major difference, compared to the proprietary flow, is visible: Before the patches can land in the code tree, a public review happens. Numerous community members can have a look at the proposed changes and send suggestions for improvement. This step is crucial because now the quality of the proposed change is increased. It is not just one single engineer who implements the feature or change on its own, it is a big community taking care of the change and its quality. Experienced members can judge if the proposed change matches the architecture and policy of the open source project and reply with change requests if needed. At the first glance this may look like an additional burden compared to the proprietary development flow. But with a closer look one can see easily how this is massively increasing the code base quality and therefore reducing the maintenance efforts inside the open source firmware project. And good code quality leads to a good firmware quality, decreasing the number of bugs and providing stability at execution time.

Beside the review there are often other helpers in place to help with the development. Code style checkers help to achieve a common coding style across the whole project, automated build tests ensure that a single commit is not harming the code base or leads to compile errors. Only after all the installed code checkers and build tests have been passed and a positive feedback was gained in the public review, the patch is considered to be good enough for merging. In some cases just being able to compile the code without errors and warnings is not enough to guarantee functionality of the code at runtime. Therefore it is crucial to reach out to platform owners (often community members who have access to various platforms, but everybody is welcome here) and ask them for a real test on dedicated hardware to ensure seamless functionality of a code change. For a quick fix of the flaw in the codebase a fast response is substantial in cases where a change introduces runtime issues.

From the developers point of view the open source approach has the benefit that the development of the code base can be observed continuously and the developer can stay familiar with the code. And even more, an active community member has always the possibility to shape the project by participating in the daily development. This will pay off quickly if more than one device of a vendor is using the open source firmware. Features implemented once can easily be used on other devices without the need to re-implement them. This is especially a benefit compared to the proprietary firmware as it varies a lot from platform to platform and features implemented earlier by the hardware vendor often cannot be just re-used later as they are but need adaptation or even re-implementation. In addition, a merged feature will most likely be maintained in the open source code base by the community (this depends on the community, though). So if the code will be refactored and a given feature is affected by this refactoring, the one who drives the refactoring will make sure that the feature stays functional.

Of course the hardware vendor could have chosen not to be an active community member and just fork the project on his side. As long as the license obligations are fulfilled, this model can be used. But one needs to be aware that all the mentioned benefits of community driven development will vanish in this model.

4.2 Code composition with open source firmware

The goal of an open source firmware project is always to keep the complete code base open source. Unfortunately, especially the firmware layer has a high dependency on the hardware and big silicon vendors are not always that open source friendly as they could be. There are many different reasons for that: There might be license issues which bind a vendor to stay closed or just the fear to open up some internals of a given IP, or even just the good old “We never did it the open way before!” phrase. Therefore, critical parts of the hardware initialization (memory and silicon initialization) are often covered by binary code parts provided by the silicon vendor. For modern x86 systems this blob is called the Firmware Support Package (FSP) and open source firmware needs to use this blob in order to be able to support a modern platform. Of course we now have this hard dependency on the silicon vendor and its willingness to provide the FSP. And to be honest, no recent platform would be able to boot without such a blob. This is the reason why there are activities out there to improve this situation (see blog posts here and there). The following figure shows the typical code composition in an open source firmware project for modern x86 systems.

Issues arising in the blobs need to be addressed at the silicon vendor. It often is easier to handle such issues with direct contact to the silicon vendor without the IBV in between. On the other hand this requires a good relationship with the silicon vendor so that there is a way at all to reach out to him and get support in the form of fixes in the blob. This of course needs additional time compared to a fully open source approach but for now we do not have a better way to support modern platforms in the open source firmware domain.

4.3 What will you get with open source firmware

Generally, there is no limitation that open source firmware has compared to proprietary. Whatever is technically doable can be done the same way with open source firmware. The major difference right now might be the different policy open source firmware follows: Do just as little as needed to get your platform to boot the OS! This results in the fact that all the fancy (but not really required) features of proprietary firmware are just not implemented in open source firmware. Things like sophisticated setup UIs or network stacks are usually missing in nowadays open implementations. Not because it is not doable but because it contradicts the open source firmware philosophy. On the other hand open source firmware provides anything that is needed to boot into modern OSes, be it Linux or MS Windows.

The other thing that you get with open source firmware is the openness of the code base. You can easily perform audits on the code which will provide certainty in regards to the used firmware layer. This is not easily possible with proprietary firmware as the NDAs that are usually in place in such a case make it hard to impossible to share the code with an entity that can do the code audit. The next benefit of open source firmware is the higher code quality as it, like described earlier, usually has a mandatory code review in place before changes land in the code base.

One other benefit that becomes more and more important is the development speed. The open source development process is a modern approach designed for collaborative development across the world. Everybody has the ability of proposing changes to cover the needs. A proposed change can be quickly introduced and uploaded for review. Though the mandatory review of a change, which is essential to merge it, can take a while (depending on the change complexity), it will ensure high code quality of the patch. With this approach changes can be implemented on someone's own request quickly without the need of asking the IBVs and waiting for their implementation. And development speed will be the key in the future as the product cycles will become shorter over time.

Speaking of the coreboot project as an example here, there are other benefits in place:

• coreboot is per architecture designed to be simple and fast. Its complexity is magnitudes lower compared to the proprietary solution UEFI.
• The code base was designed to be quite similar to the one of the Linux kernel. This provides the benefit that a lot developers can be fast familiar with its code base which lowers the entry burden for the developers.
• The community takes care of bug fixing, there is no maintenance cost required at the product price level.
• Fixes are done at tree level. If a fix is in a common code path, all OEMs and SoCs can benefit from it equally.

4.4 What are the issues with open source firmware

The nature of the open source firmware is of course its openness. This means that implementations are freely available to others, including competitors (this depends on the chosen license). This might reveal information of a newly developed product (the used CPU generation, number of interfaces a board has, used interface types, etc.). Therefore, the hardware vendor, to stay within the depicted example, needs to act carefully. Depending on the license there might be ways to overcome this issue by not publishing the adaptations for a board. Or the vendor could choose to publish its modifications later where the new development is already revealed at a fair or on a website.

The other issue with open source firmware at the moment might be the limited feature set it provides (for a good reason, though). If an end user switches for example from an UEFI implementation to let’s say coreboot, then the most obvious difference will be the missing UI the user might be more familiar with. So all the fancy settings are not possible in the environment with coreboot. This limitation of course can be overcome with development in the open source firmware as like already mentioned there is no technical reason for not having it. And this brings us to the next limitation of open source firmware: The resources big companies spend for open source development are just limited at the moment. Often the bleeding edge development still happens in the proprietary space because the paradigm of the past decades kind of dictates it. But to emphasize at this point, this will probably not remain the case in the future. We are already seeing a shift away from proprietary to open source firmware. If this trend continues, open source firmware will become ever stronger.

Another issue of an open source project can be the way how new features are introduced. Contrary to a proprietary solution, where just one IBV is responsible for the code base and hence owns the features, there are multiple different players in the open source community. While one contributor would like to get a new feature in, he still needs to get acceptance in the project for this new feature. It can easily happen that someone invests time and resources to develop a new feature and then, when it gets presented to the community in the form of a patch train, it receives negative feedback or even a rejection. There might be  requests from some community members to modify the way this feature was implemented heavily which will result in additional efforts at the introducer’s side. Or there is even somebody in the community who, for whatever reason, does not want this feature to be merged at all. Therefore, it is much better to stay in a close collaboration with the community and push out patches in small chunks early to get the other’s feedback and prevent huge throwbacks. In the end it might result in a long discussion before a new feature is accepted and provided in the code tree.

In addition, an open source firmware project heavily depends on the willingness of a SoC vendor to get involved in the project by either contributing to the project or at least openly providing enough information in order to enable the community to write code based on this information. If this type of information is simply not available or is kept under wraps by NDAs, the developing support for new SoCs in an open source project is hardly feasible because then you have to resort to techniques such as clean room design (see [1]) or trial-and-error, which are very time consuming and offer no guarantee for reliable results. It is therefore important that more companies share the open source vision and leave the proprietary approach in favor of the open source approach. Otherwise, open source will not work out in the long term.

5 Why is the firmware the wrong place for product differentiation

In discussions with various hardware and firmware vendors I often get the impression that the firmware is one of the big playgrounds for product differentiation. The IBVs enhance the firmware layer more and more by adding fancy features to it. If one has a look at a modern UEFI driven system and its setup screen, one will see a fully integrated GUI with HD graphics, animations and support for a pointing device. Even full network stacks with DHCP support and file system drivers as well as e-mail clients are implemented. Indeed, the proprietary firmware is a full OS nowadays. But what is the use case here? I recently was setting up a brand new off the shelf PC system and I visited the UEFI setup twice: The first time to update the firmware to the latest available version (the shipped mainboard includes a really old firmware though I just got it) and the second time to disable all the fancy features I do not want to be enabled in the firmware layer (really, I do not want my firmware to drive a network stack). After this, the rest happens in the OS. The only thing I need from the firmware layer now is: please transfer the control to the OS fast! Because all these fancy features in the firmware requires code to execute and this in turn will need time, time that the user has to wait before the system can be used for the task it was acquired for.

To me, the real differentiation of a system is defined by its hardware capabilities and then by the operating system and the applications that are used on this system. The firmware is just this piece of software that enables the operating system to take care of the hardware properly.

6 Recap

In the firmware space the proprietary approach is still dominant. It seems like it gives the decision makers more confidentiality while it pretends to reduce the risk of a product development. But the world has changed dramatically in the firmware space over the past 15 years. Given that firmware becomes more and more complex due to newer CPU generations, a modern software development flow is required to rule the challenges that the firmware is facing. And since development cycles are becoming shorter, there is no time to lose by re-implementing features or searching for issues caused by a questionable code base quality. The goal shall be to re-use as much as possible and spend the limited development resources to add support for new platforms and features. We need to change the paradigm and welcome the open source firmware approach as the new first class citizen as the benefits this approach provides just outweighs the old style firmware development flow. If there are more development forces flowing into the open source firmware, we will soon overcome the remaining limitations and provide a stable and reliable firmware experience that does what it was meant to do: boot the system into OS!

[1]: https://en.wikipedia.org/wiki/Clean_room_design