This document describes an strategy to manage platforms based on the Yocto Project. Basically, we’ll describe how to use the repo tool and manifest files to manage multiple git projects as if they were a single one and manage platform specific settings in an integrated way.

Platform manifest files

Platform manifest files are XML files that describe the git repositories your projects use. They are used by the repo tool to manage the git repositories required by your projects. Roughly speaking, platform manifest files are a way to turn multiple git repositories into a single project that can be managed by a tool on top of git.

Platform manifest files are usually stored under their own git repository, which can be cloned by repo (using git behind the scenes). Here’s an example (from http://code.ossystems.com.br/#/admin/projects/ossystems-yocto-platform):

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

<remote fetch="http://code.ossystems.com.br" name="code" review="code.ossystems.com.br"/>
<remote fetch="git://git.yoctoproject.org" name="yocto"/>
<remote fetch="git://github.com/Freescale" name="fslc"/>
<remote fetch="git://git.openembedded.org" name="oe"/>

<default revision="invalidRevision" sync-j="4"/>

<project remote="yocto" revision="master" name="poky" path="sources/poky"/>
<project remote="oe" revision="master" name="meta-openembedded" path="sources/meta-openembedded"/>

<project remote="code"  revision="master" name="ossystems-yocto-base-scripts" path="sources/base">
  <copyfile dest="setup-environment" src="setup-environment"/>
</project>

<!-- O.S. Systems Yocto BSP -->
<project remote="code" revision="master" name="meta-ossystems-base" path="sources/meta-ossystems-base"/>
<project remote="code" revision="master" name="meta-ossystems" path="sources/meta-ossystems"/>
<project remote="code" revision="master" name="meta-fsl-arm" path="sources/meta-fsl-arm"/>
<project remote="code" revision="master" name="meta-fsl-arm-extra" path="sources/meta-fsl-arm-extra"/>
<project remote="code" revision="master" name="meta-fsl-demos" path="sources/meta-fsl-demos"/>

<!-- conf -->
<project remote="code" revision="master" name="ossystems-yocto-platform" path="sources/conf/repo"/>
<project remote="code" revision="master" name="ossystems-yocto-config" path="sources/conf/tools"/>

</manifest>

Let’s take a closer look at the things specified in that XML document.

The

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
…
</manifest>

block defines a manifest. In that block we’ll define projects and remote locations for projects' source code repositories.

The <remote> tag specifies a location for repositories. Let’s see an example:

<remote fetch="git://git.yoctoproject.org" name="yocto"/>

This line specifies a remote named yocto, whose URL is git://git.yoctoproject.org. name will be used by the <project> tag to specify projects' complete URLs.

Note that remotes can have a review attribute, which can be used the integrate your projects to, for example, the Gerrit code review tool. In those cases, all your changes to the source code repositories under repo control will be submitted to the code review tool before hitting the actual source repository.

As an example of the <project> tag, let’s take a closer look at the following line:

<project remote="yocto" revision="master" name="poky" path="sources/poky"/>

That line specifies a project named poky which uses the yocto remote. It means that repo will use that project to fetch a project named poky from the URL specified for the yocto remote. That would give us git://git.yoctoproject.com/poky. repo will fetch the sources from that location using git and save them under the sources/poky directory. Note that <project> tags have a revision attribute, which can be very useful for specifying the exact commit you want your code to be at.

Also note an interesting trick we use to add the manifest repository itself to the list of projects:

<project remote="code" revision="master" name="ossystems-yocto-platform" path="sources/conf/repo"/>

With that line, the repository which holds the manifest file will also be fetched and saved under sources/conf/repo. So, if you ever need to change the manifest file, you can do it from the copy you have using repo.

Using repo

repo is the command line tool that uses the platform manifest files to aggregate all projects in a single tree. Let’s see how to use it using the ossystems-yocto-platform project as a practical example.

Assuming you have repo installed (if you don’t, see http://source.android.com/source/downloading.html#installing-repo), run the following commands:

$ mkdir ossystems-yocto-platform
$ cd ossystems-yocto-platform
$ repo init -u http://code.ossystems.com.br/ossystems-yocto-platform

Those commands will initialize the repo metadata under the directory you created (ossystems-yocto-platform). The projects' source code repositories are still not there. To fetch/update them, run

$ repo sync

That command tells repo to fetch the code repositories for all the <project> entries in the platform manifest file. The projects' source code repositories will be fetched to the sources directory.

Under the sources directory, you’ll find clones of all the projects you specified in the manifest file. That means you can use git as you would do if they where individual projects. repo provides some interesting features to ease the management of multiple projects. For example, if you need to search for some string in all projects, you can use repo grep <pattern> instead of running git grep <pattern> on each git repository. The same for repo diff, instead of manually iterating git diff over all the repositories.

To change repositories, you can use git commit as you would do in a regular git workflow. To submit changes, you can use repo upload. repo will submit the changes to the right locations, as they are specified in the manifest file.

Managing platform versions

Since the platform manifest file lives in its own git repository, and projects specified in the manifest file have a revision attribute, we have all we need to version the whole platform. To do that, all we need is filling all projects' revision attribute with the hashes of the commits corresponding to the version we want to tag. After setting up the revisions for projects, we commit the manifest file and set a tag on the repository. By doing that, we have tagged the whole platform — it is now possible to check out the full state of a version at any time.

setup-environment hooks

O.S. Systems provides an improved setup script which allows users to configure hooks that can be run in a per layer basis. That script can be found in the ossystems-yocto-config project.

setup-environment will look for Python scripts under <layer>/conf/setup-environment.d, load and run them. Hooks are Python functions that receive no argument. Hooks are run at two stages: before running Poky’s oe-init-build-env (which actually writes conf/local.conf and conf/bblayers.conf) and after Poky’s oe-init-build-env (so that you can overwrite things in conf/local.conf and conf/bblayers.conf).

Here’s an example of hook script:

def __my_hook_before_init():
   import os
   import sys

   try:
       os.environ['MACHINE']
   except:
       sys.stderr.write("ERROR: You must set 'MACHINE' before setting up the environment.\n")
       sys.exit(1)


def __my_hook_after_init():
   import os

   append_var('INHERIT', 'buildhistory')
   set_var('PACKAGE_CLASSES', 'package_ipk')

   PLATFORM_ROOT_DIR = os.environ['PLATFORM_ROOT_DIR']

   append_layers([ os.path.join(PLATFORM_ROOT_DIR, 'sources', p) for p in
                   ['meta-browser',
                    'meta-chicken',
                    'meta-java'] ] )

run_before_init(__before_init_my_hook)
run_after_init(__after_init_my_hook)

The first definition (my_hook_before_init) specifies what is to be run before Poky’s oe-init-build-env. We register that hook with run_before_init(before_init_my_hook). That hook simply checks if the MACHINE environment variable has been set in the environment.

The second definition (__my_hook_after_init) specifies what is to be run after Poky’s oe-init-build-env. At this point, both conf/local.conf and conf/bblayers.conf have been created, so we can use the `setup-environment’s Python API to edit those files. At the example you can see some functions of the `setup-environment’s Python API. Here’s a more detailed documentation for them:

append_var(<var>, <val>)

Append <val> to <var> in conf/local.conf.

set_var(<var>, <val> [, op=<operator>])

Set <var> to <val>, using the = operator if op is omitted. op can be any of the assignment operators supported by BitBake.

append_layers(<layers>)

Append <layers> to the BBLAYERS variable in the conf/bblayers.conf file. <layers> is a Python list of strings. Each string must be the path to a layer directory. append_layers can be useful when your layer depends on another layer(s). In that case you can use append_layers to automatically add the layers you layers depend on to BBLAYERS in conf/bblayers.conf (note that specifying what layers have to be fetched must be done in the platform manifest file).

End-user-license agreement API

Hook scripts also have the ability to prompt users for EULA (End-user license agreement) acceptance right at the moment the setup-environment script is executed, so all EULAs can be accepted (or not!) before the build process starts.

There is no standard for specifying EULA acceptance and EULA text location, so layers may use different variables to indicate EULAs acceptance/rejection and may place the text anywhere in the layer directory layout.

The interface for specifying EULAs acceptance provided by O.S. Systems' setup-environment script is quite flexible in that regard. Below you can see an example for the meta-fsl-arm layer:

eulas.accept['meta-fsl-arm/EULA'] = 'ACCEPT_FSL_EULA = "1"'

eulas.accept is a Python dictionary whose key is path to the EULA file (including the layer directory), and its value is the EULA acceptance expression to be added to local.conf in case the user accepts the EULA.