summaryrefslogtreecommitdiff
path: root/Build-system.md
blob: 02823d6d8247ab8adf7a0db9de0f2a5ba8a83c2e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
The build system is the heart of the project. On this page you can see the initial plan.

## Files in git repository
For packages that need some customizations, the build system will use a repository that consist from the following files:
- _[package_name]_/_[architecture_name]_.sh: shell script, which will run in the same directory where the original PKGBUILD found. It can do the following things: apply sed lines, apply patches, or even replace the original PKGBUILD file with a custom one.
- blacklist: text file, list of packages that will be ignored by the build system (e.g. cannot build for this architecture).
- build-manually: text file, list of packages that will be added to the build list on update, but need manual interaction before build. Packages that depend on this package will be also hold back.
- update-once: text file, list of packages that will be updated next time even if nothing changed. This list will be cleared out on next update. This list overrides the 'build-manually' list.

## Build list
The build list is a text file, and will contain the list of packages that need to be built in the proper build order. One package per line, a line contains the package name, the git revision number and the repository name.
Along with the build list, there will be a state directory, where the build process could create files for each package, and mark their current status. The files are named as _package name_-git revision_. Possible statuses:
- locked: this means that a build process is currently running for this package.
- broken: this means that a build process fails to build the package, requires fixing.
- blocked: this means that a package is updated, but it's on the 'build-manually' list.

## Processes
The build system will consist from the following shell scripts:
- get-package-updates: check for packages that need to be built, and build a list in the proper build order
- get-assignment: retrieve a build assignment from the master build server
- build-packages: build packages one by one, then upload the binary package to the repository server
- arch32build: build packages in a clean chroot (a fork of [archbuild](https://git.archlinux.org/devtools.git/tree/archbuild.in))
- return-assignment: add binary packages to the repository, and remove them from the build list (could have been a fork of [db-update](https://git.archlinux.org/dbscripts.git/tree/db-update), but isn't)

### get-package-updates
1. Update git repositories ([official packages](https://git.archlinux.org/svntogit/packages.git/), [community packages](https://git.archlinux.org/svntogit/community.git/) and the repository of package customizations).
2. Read previous git revision numbers from files.
3. Create a lock file for build list.
4. Check modified packages from the last update, and put them to the build list. If a package is updated, but already on the rebuild list, then just update the git revision number. If a package is deleted, remove from the rebuild list, and add it to the deletion list. If a new package is added, then ensure that it's not on the deletion list.
6. Put the list in the proper build order.
7. Write the current git revision numbers to files.
8. Remove the lock file.

### get-assignment
1. Create a lock file for build list.
2. Read the build list and try to advertise the packages one by one for building.
3. Check if the current package or any of its dependencies are locked, broken or blocked (if a state file is available). If it's not, proceed with step 5.
4. Hand out the package which break the most dependency loops.
5. Create a state file for the current package and set it as locked.
6. Assign the build order to the caller of the script.
7. Remove the lock file.

### build-packages
1. Get a build assignment by calling `get-assignment` on the build master
2. Copy (or download) the source package and the customizations locally (this is currently done via checking out the respective revisions in git).
3. ~~If it's an architecture-independent package (marked as 'any'), then copy the built package from the official repository, and skip the following three steps.~~ (this doesn't gain much and only calls for trouble, e.g. different keys for signing, not applied modifications, outdated packages on the x86_64 mirrors, ...)
4. Apply customizations by ~~running the script if exist in a chroot~~ appending to `PKGBUILD` and replacing all other files by existing files in the package customizations repository.
5. Run arch32build.
6. If the package fails to build, then try again once with `-c`, once with source downloaded from `sources.archlinux.org` and once more with `-c`. If it still fails, then mark it as broken (report back to the build master by calling `return-assignment ... ERROR`).
7. After a successful build, sign the package with the build slave's PGP key, and pack and upload the binary package to the staging directory by calling `return-assignment` on the build master ~~together with a file which contains the git revision number~~.
9. ~~Run db-update.~~

### arch32build
Build the package in a clean chroot environment with a custom makefile.conf.

### return-assignment
1. Create a lock file for build list.
2. If error is reported, mark package as broken and proceed with step 6.
3. Check if the git revision of the binary package is the same as on the build list. If it's different, then drop the package. Otherwise add binary packages to the proper staging repository.
4. Check package for sanity.
5. Remove the package from the build list, and remove the corresponding package state file.
6. Remove the lock file.