This document addresses the problem of managing product releases, assuming the structure O.S. Systems has developed to make the management of complex projects possible.
Product releases must be:
we must be able to track down changes made in specific product releases and see what changed between releases.
we must be able to check out any release and build the whole product from that point.
This document also introduces a set of tools developed by O.S. Systems, which are available to its customers to aid them in the release of their products.
Overview of a product’s structure
O.S. Systems has developed a structure to organize software products into a set of git repositories. This structure has proved to be flexible enough to satisfy the requirements of all of its customers so far. This section describes the structure assumed for products.
Usually, a product is composed by a set of projects. Projects are stored in VCS (Version Control System) repositories, usually git.
Products have an index of projects it depends on. This index is
called the platform manifest or just manifest. It is a special
git repository managed by the
repo tool. See the
platforms based on the Yocto Project document for more information
on that repository.
The platform manifest file points to all repositories which are part
of the product. Each reference to project repositories has the
information about the point at the repository the project should be
cloned (see the
revision attribute for the
The figure below illustrates this scheme:
+--------------------+ | Platform manifest | +--------------------+ +-----------+ | | +----------------------+ | Revisions | | <project name=a |------->| Project a's git |--->| * abc | | revision=abc ...> | | repository | | * def | | | +----------------------+ | * ... | | | +-----------+ | | | | +-----------+ | | +----------------------+ | Revisions | | <project name=b |------->| Project b's git | | * klm | | revision=nop ...> | | repository |--->| * nop | | | +----------------------+ | * ... | | | +-----------+ | | | | +-----------+ | | +----------------------+ | Revisions | | <project name=c |------->| Project c's git |--->| * rst | | revision=rst ...> | | repository | | * uvw | | | +----------------------+ | * ... | +--------------------+ +-----------+
Being in its own git repository, the platform manifest can also be tagged, have branches etc.
So, at a given point in the platform manifest’s repository history, we have a reference to all projects' state.
Yocto Project’s layers
Projects referenced from the platform manifest file are likely to be Yocto Project’s layers. Layers are git repositories which contain metadata for Yocto Project. Among the metadata are the so called recipes. Recipes in the Yocto Project’s jargon are files with instructions about how to build software.
Recipes build software from their source code. Recipes contain a reference to the software source code location (an URI). If the URI points to a VCS repository, the recipe also specifies a reference to a revision in the repository (e.g., a git commit hash).
It’s not uncommon to have in your company’s layers recipes to describe how to build software developed by your own company. For example, your product is a kiosk system and one of the software shipped with it is user interface (KUI), which is developed by your company. KUI is likely to be in its own git repository and constantly being updated.
So, while in the development phase, you’ll probably want to use the tip of your projects' repositories (e.g., tip of master branch). On the other hand, for releases, you want revisions at a point the projects have been properly tested.
This approach can be tricky to handle manually. Updating recipes' reference to software’s VCS every time they change is not ideal, since it would demand a lot of manual work and would pollute the layers' history.
O.S. Systems provides some tools to automate that. They will be described in the next sections.
The problem we are trying to address with this document is the release of product versions. What should Product version 1 mean?
As we’ve seen on the Overview of a project’s structure section, a product can be composed of a set of git repositories. We have a chain of repositories like this:
Platform manifest (git) | |`> Layer 1 (git) | | | |`> recipe 1 | | | | | `> software 1 (git) | ... | | | `> recipe n | | | `> software n (git) ... | `> Layer n (git) | |`> recipe 1 | | | `> software 1 (git) ... | `> recipe n | `> software n (git)
Answering the question stated at the first paragraph: a given product version is a tag in the platform manifest repository. The platform manifest file points to the exact git object (tag, commit hash) of each project (layer) it references. Layers at the commit object referenced by the platform file contain recipes that reference the exact git object of the software whose build steps they describe.
To be able to have that, we need to:
tag each software repository referenced by layers
update their corresponding recipes to point to the just created tags
tag the layer repositories
update the platform manifest file with the layer tags just created
tag the platform manifest file
The next section describes a set of tools developed by O.S. Systems to help its customers to accomplish all the aforementioned tasks.
O.S. Systems tools
ossystems-tools is a set of tools developed by O.S. Systems to aid
the release of new product versions. These tools assume the product
structure described by this document and significantly simplify the
ossystems-tag-projects is a tool to tag the whole set of projects used by a product. It assumes the existence of a platform manifest repository which references all the projects required by a product. Note that this platform manifest file is not the same as the one to describe the product. This one is just a convenience format to group all the git repositories for the product, independently of Yocto Project-related repositories. It would put together, for example, all the software components showed in the The problem section’s diagram.
ossystems-tag-projects assumes the projects platform clone is updated. So, before running it, you probably want to explicitly run:
$ repo sync
in the projects platform directory.
ossystems-tag-projects will check for each project, if they have
changes after the latest tag. If they have been changed,
ossystems-tag-projects will ask you if it should bump the tag to
create a new version.
Here’s an example of
ossystems-tag-projects running on one of
O.S. Systems' internal projects:
$ ossystems-tag-projects --all Loading configuration file: /home/mario/os/ossystems-projects-platform/conf/tools/tools.conf Starting branch pending-1416420719.0 in the srcrevs repo The following changes will be applied: ++ liboslicense: bump version from 13.0.4 to 13.0.5 ++ liboshwcollect: bump version from 13.0.6 to 13.0.7 Proceed? (yes/no) no
In the example above,
ossystems-tag-projects detected the projects
liboshwcollect have changes after the latest tag,
so it asks you whether you want to bump them. If you answer
nothing will be changed.
If you answer
ossystems-tag-projects will tag and push tags
for the projects it detected that changed. Next it will ask you
whether you want to generate SRCREV files in the
Generate the SRCREV file in /home/mario/os/ossystems-projects-platform/conf/srcrevs? (yes/no)
If you answer
yes, it’ll show you all the projects' tags and their
commit hashes that will be written to the
srcrevs.inc file is used by a Yocto Project’s class
ossystems-srcrev-handler.bbclass) in the meta-ossystems-base layer
to set the project revisions to the stable versions (at release time)
or to the repository tip (at development time).
You can always use ossystems-tag-projects with the
argument to check what it would do without actually changing things.
Instead of using
--all, you can also specify a list of individual
projects separated by space on the command
ossystems-tag-projects will only check the provided projects in
You may have noticed in the examples that
a configuration file if it finds one in
With this configuration file you can specify:
a version number to start from: the tag to be used for projects which have not been tagged (their repositories have no tag); or if you want to start a new version series from a base version.
a version mask to specify which version part to increment. ossystems-tag-projects need a criterion to increment versions. Usually, versions follow a scheme like
ossystems-tag-projectsdetects that a project has changes after the latest tag, it needs to increment the version somehow. What version part? That’s where the version mask comes into play: it’s a string like
x.x.@(the default mask), where
@indicates the version part to be incremented.
Here’s an example of configuration file:
$ cat conf/tools/tools.conf ;; -*- scheme -*- (use ossystems-config-params) (version-mask "x.x.@") (version-to-start-from "13.0.0")
Those options can also be specified on the command line (see the
ossystems-tag-projects' help message):
$ ossystems-tag-projects --help Usage: ossystems-tag-projects [ <options> ] [ <proj1> <proj2> ... <projn> ] This program should be used from the "platform" directory. <proj1> <proj2> ... <projn> (optional) are project directories in the sources directory. If none is provided, nothing is done (unless --all is provided). <options>: --version-to-start-from=<version> This option is to be used when we want to start a new version series. All projects that have modifications are tagged with a tag starting from the provided version. --version-mask=<version mask> Use the provided <version mask> to increment versions. If not provided, x.x.@ is used. --all Tag all projects in the sources directory. --dry-run Don't actually write tags.
ossystems-release is the tool that actually tags a new product
version. It tags the product’s platform manifest file. This tool is
supposed to be used after you have tagged all the projects that are
part of the product (see the documentation for the
ossystems-tools should be run from the Yocto Projects root directory
(the directory where you source
Here’s an example usage session:
$ repo sync
Like with the projects platform manifest, it’s always recommended to update the product platform manifest copy.
$ ossystems-release Loading configuration file: /home/ossbot/src/ossystems-products-platform/sources/conf/tools/tools.conf Skipping hook before-create-tag Create tag 13.0.15? (yes/no/abort) yes
yes here will make
ossystems-release fill up the
platform manifest file with the revision of all projects referenced
from it, and tag the platform manifest repository with the release
tag. With this procedure, we establish a release state, to witch in
the future we can return, in case it is necessary.
The release tag determined based on the current tag and the version
mask. The version mask can be specified on the command line or in the
conf/tools/tools.conf configuration file (just like for
On the example output, you can see the Skipping hook
ossystems-release may run some hook
scripts at some points. Hook scripts are looked for into the directory
pointed by the
release-hooks-dir configuration option in the
configuration file (the default value is
dir>/ossystems-release-scripts) or via the
command line option. Scripts should be named after the hook name.
ossystems-release calls each hook for each project, giving them the
project tag, the project name and the project branch as arguments on
the command line.
One of the hooks is the
before-create-tag hook. That hook allows
you to run an arbitrary script right before the new tag is created.
Other hooks available are:
Errors in hook scripts (except
after-push-tag) will cause
ossystems-release to abort. At the time of this writing,
ossystems-release has no means to roll back what has been previously
ossystems-release will ask if you want to see the changes in
the platform manifest:
Show changes in the platform directory? (yes/no/abort) yes
It’s always recommended to look at the differences before actually
checking the changes in. If you agree with the changes, you can
yes to the next question, then
ossystems-release will commit
changes and ask if you want to run the
Commit changes in the platform directory? (yes/no/abort) yes Run hook after-create-tag? (yes/no/abort) yes
The new tag must be checked in before the build starts because your product probably needs the new tag at build time (for example, to automatically get the tag from the repository at build time to build up an About screen for your project).
after-create-tag hook is the script which actually starts the
Run hook after-create-tag? (yes/no/abort) yes Running hook after-create-tag for project poky (tag: 13.0.15, branch: danny) ...
after-create-tag hook finishes successfully (i.e., the build
process has been successfully finished),
ossystems-release will ask
you if you want to revert back to development mode.
Revert back to development mode? (yes/no/abort) yes
Let’s describe this step with more details. When
showed you the differences in the platform manifest file and asked you
whether you wanted to commit the changes, it checked in a commit which
set the revisions for each project it references. So, at that point,
your platform manifest file was now referencing projects' git objects
for the current version. Once your build is finished and your new
version is released, you’ll want to set the project revisions in the
platform manifest file to their original revisions (usually the branch
name for each project).
Push changes to platform? (yes/no/abort) yes
Now that everything is done, the only step left is actually pushing the changes and tag to the remote repository.
ossystems-diff-releases shows the difference between two releases.
The differences can be shown in the git log format or in the git diff
format (you can use the
--git-command command line option to specify
which format you want — the default is git log format).
If you don’t specify projects,
ossystems-diff-releases will show the
differences for all projects.
Here’s the help message for
$ ossystems-diff-releases --help Usage: ossystems-diff-releases [ <options> ] tag1 tag2|HEAD [ project1 project2 ... ] <options>: --verbose --git-command=<command> Git command to use. Available options are: log and diff. If not provided, log is used. --git-command-args=<args> Arguments to be passed to git command.