Build_system_refactoring

Build_system_refactoring#

Overview#

The build system in FreeIPA 4.4 has following problems:

  • It is ineffective. This costs developer time every time we build IPA. Every developer typically builds IPA several times a day. Also, it slows down CI because it needs to build the packages as well.

  • The build system does not directly support rapid development approaches: Every patch-copy-test cycle requires either manual work (which is error prone) or full build (which is very slow). It slows down rapid prototyping.

    • Repeated build with minimal changes (like changing one file) takes long time, which prevents us from doing rapid devel cycle.

    • Theoretically the process should be very fast as proper Makefile would detect what can be skipped.

    • In reality the build takes 88 % of the time from the first build!

  • Is not reliable. Often, “git clean -xdf” is needed to get the build to pass.

  • It is hard to maintain and debug.

The proposal is to re-write the build system using current best practices and standard tools. Re-write from scratch is deemed to be more effecient than fixing bugs one by one because parts of current system are unnecessairly complex or can be replaced with standard solutions.

User stories#

Every user natually wants wants reliable build.

  • Developer building a development version of FreeIPA:

    • Expects very fast build

    • Wants very simple and quick installation of the new version

    • Needs easy way to extend/modify the build system as components change

  • Tester building latest version for test execution

    • Wants easy-to-follow procedure to build distribution packages with latest version of FreeIPA

  • Release engineer releasing new upstream version

    • Wants to use standard tools for creating tarball which is then consumed by packager

  • Packager preparing a package for Linux Distribution

    • Wants to use standard tools for packaging, without need for hacks

Design#

The proposed solution. This may include but is not limited to:

  • High Level schema (Example 1, Example 2)

  • Information or update workflow

  • Access control (may include new permissions)

  • Compatibility with other (older) version of FreeIPA. Think if the feature requires a minimum Domain level.

For other hints what to consider see general considerations page.

Implementation#

  • Use autotools suite to orchestrate the build on the top-level

  • For Python parts: use setuptools instead of distutils

  • For C parts: use autotools

  • For Internationalization support: use standard gettextize framework + custom enhancements to support Zanata workflow

  • Web UI build was not touched nad is usin Dojo builder as it was before.

  • Dependencies:

    • New build-time depedency gettext-devel is needed for gettextize framework.

Feature Management#

CLI

Command

Desired effect

./configure

test system configuration and prepare build according to configuration options

./makerpms.sh

build RPMs from clean Git tree: autoreconf -i && ./configure && make rpms

make all

build complete FreeIPA

make install

install FreeIPA files into paths so FreeIPA can be executed right away

make uninstall

uninstall FreeIPA files so the system is clean again and new installation can be done (e.g. installation from different branch)

make rpms

create SRPM and RPMs

make clean

erase the files built by make all

make

See Automake: Standard Targets

Configuration#

configure script will have several configuration options (replacing RPM-specific SPEC file magic).

Configure option

Default value

Desired effect

PYTHON (environment variable)

python in $PATH

path to Python interpreter (use this to select Python 2/3)

–disable-i18n-tests

not present (tests enabled)

do not execute ipatests/i18n.py (depends on python-polib)

–disable-server

server build is enabled

do not build server components

–enable-pylint

don’t run pylint

run pylint on Python packages using $PYTHON for make lint

–with-jslint

jsl in $PATH

run JS lint

–with-ipaplatform

autodetect

select Fedora/RHEL IPA ipaplatform to build for

–without-ipatests

include ipatests

don’t include ipatests

–with-vendor-suffix

(empty string)

vendor suffix; used in VENDOR_VERSION string stored in i papython/version.py; e.g. “-1.fc24”; this should be used from SPEC file, not necessary for upstream-only builds

Platforms which are missing some of the tools will be able to use –without-feature and –disable-feature options to disable part of the build or check.

Versioning#

Current versioning scheme is partly responsible for the slow build. Developer build with IPA_VERSION_IS_GIT_SNAPSHOT=1 changes version values in version.m4 during each build. As a result, the whole autotools machinery needs to be re-executed on each build.

Me and jcholast decided to keep this behavior for option IPA_VERSION_IS_GIT_SNAPSHOT=1. If you want fast build, disable it. The main reason is that getting rid of this problem would require significant effort which would include code changes outside of build system.

For reference, here are pieces of the old build system which concern versioning:

Here is plan what we should do with these variables:

Variable

What to do with it

IPA_VERSION_MAJOR

move to VERSION.m4

IPA_VERSION_MINOR

move to VERSION.m4

IPA_VERSION_RELEASE

move to VERSION.m4

IPA_VERSION_ALPHA_RELEASE

merge to IPA_VERSION_PRE_RELEASE in VERSION.m4

IPA_VERSION_BETA_RELEASE

merge to IPA_VERSION_PRE_RELEASE in VERSION.m4

IPA_VERSION_RC_RELEASE

merge to IPA_VERSION_PRE_RELEASE in VERSION.m4

IPA_VERSION_PRE_RELEASE

new variable; string is appended to .. to form version number like “1.0.0rc1”

IPA_VERSION_IS_GIT_SNAPSHOT

move to VERSION.m4

IPA_DATA_VERSION

move to VERSION.m4

IPA_API_VERSION_MAJOR

move to VERSION.m4

IPA_API_VERSION_MINOR

move to VERSION.m4

IPA_VENDOR_VERSION_SUFFIX (currently defined in SPEC)

move to configure –with-vendor-suffix

IPA_RPM_RELEASE (currently reads RELEASE file)

remove, replaced by –with-vendor-suffix

When IPA_VERSION_IS_GIT_SNAPSHOT is enabled, the configure script will touch VERSION.m4 file. On subsequent builds, this will trigger automatic re-execution of configure script (assuming AM_MAINTAINER_MODE is disabled).

How to Use#

All users can use multiple CPUs by running make with parameters “-j” or alternativelly “-l”. It is handy to specify these parameters in variable MAKEFLAGS like this:

$ export MAKEFLAGS="-j16"

so it applies to all make jobs by default.

Developer#

  • First round - build & install RPMs once to get all the depedencies and scriptlets ran:

$ rm Makefile  # if Makefile exists, remove it
$ ./makerpms.sh  # this runs configure with paths appropriate for subsequent installation
$ dnf install dist/rpms/*.rpm

  • Subsequent rapid development:

$ make install

make install will quickly rebuild files as needed and install new files onto development system, so the new build can be tested immediatelly.

  • Installing files to a remote machine:

The install target supports variable DESTDIR which specifies where to copy the files. This can be easily used together with SSHfs which mounts complete root filesystem from a VM to developer’s machine:

$ mkdir /tmp/vm
$ sshfs -o transform_symlinks root@``\ ``:/ /tmp/vm
$ make install DESTDIR=/tmp/vm

This snippet will synchronize all files from developer’s machine onto a VM. Just keep in mind that it will not bump version in RPM database and things depending on this might break.

To remove all files from the latest build, you can use target uninstall:

$ make uninstall DESTDIR=/tmp/vm

Uninstallation ensures that there are no leftovers from the current version so new version can be safely installed. (Again, keep in mind that this will not touch RPM database.)

As an optimization for lower-bandwidth/high-latency links you can use rsync instead of sshfs. Is is just additional step after make install:

$ mkdir /tmp/vm
$ make install DESTDIR=/tmp/vm
$ rsync -rlK /tmp/vm/ root@``\ ``:/

Tester#

$ autoreconf -i
$ ./configure
$ make rpms

Or alternatively:

$ ./makerpms.sh

will produce RPMs suitable for further FreeIPA testing.

Release engineer#

$ autoreconf -i
$ ./configure
$ make dist

will produce version.tar.gz suitable for further packaging

Packager#

$ autoreconf -i
$ ./configure
$ make install DESTDIR=

will install FreeIPA into correct paths in build root so it is very easy to take all installed files and just package them.

Packager - client only build#

$ autoreconf -i
$ ./configure --disable-server --without-ipatests
$ make install DESTDIR=

will install FreeIPA into correct paths in build root so it is very easy to take all installed files and just package them.

Note: This use case does not fully work yet. See progress in #6417

Translation maintainer#

  • Generate a new .pot file for Zanata:

$ make ipa.pot-update

  • Strip untranslated strings from .po files downloaded from Zanata:

$ make strip-po

  • Test all strings and translation system:

$ make polint

Test Plan#

Test scenarios that will be transformed to test cases for FreeIPA Continuous Integration during implementation or review phase. This can be also link to source in cgit with the test, if appropriate.