Git log: commit b1ef4797a4d1afe0debee9114dabea7fc83070a7
Merge: 94dfe830 49d51a5c
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Aug 26 17:44:01 2023 +0000
Merge branch 'disk-storage-limits' into 'master'
Implement a per-package disk storage limit
See merge request ci-team/debci!257
commit 49d51a5c7a70cd236f7f24efb714a362ffb4cb3d
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Aug 12 12:33:44 2023 -0300
Limit amount of storage each package can use
By Default, each package is allowed to use up to 200MB of disk space
with logs and artifacts.
Gbp-Dch: full
commit c144136e054517bec4f14253e236321c1dd4e4c4
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Aug 12 12:07:37 2023 -0300
Debci::Job: add method to calculate per-job disk usage
commit 9902db9180589451d813b99889ba7d045469f574
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Aug 12 12:05:26 2023 -0300
config: define a disk storage limit per package
Initially the default value is 200 MB.
commit c35086e6f151ee1d4dffb18ee3b72357bcf9614c
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Aug 12 11:45:04 2023 -0300
spec: add missing `require 'spec_helper'` to specs
Gbp-Dch: ignore
commit 8e5dd89b97e601065c6e2545c3f5b2f64537ef6a
Author: Antonio Terceiro <terceiro@debian.org>
Date: Tue Aug 8 12:14:51 2023 -0300
Debci::Job: keep track of purged files separate from the expiration date
This solves several problems:
- it allows the data retention period to be increased without making the
system act as if files previously deleted -- from past jobs that would
not be expired based on the new retention period -- were available.
- it will allow for jobs to be purged before they pass the data
retention period (e.g. because the package they belong to is using too
much disk storage).
- it avoids hitting the disk when generating job listings, which
improves performance a bit.
commit 434152f3625a9edbeb41159f1dd54aaf6af6091e
Author: Antonio Terceiro <terceiro@debian.org>
Date: Mon Aug 7 07:19:42 2023 -0300
debci-publisher: resist corrupted result archives
commit 94dfe8300c6405934b62c3b5eb3b6beb40698478
Author: Helmut Grohne <helmut@subdivi.de>
Date: Fri Jul 28 16:32:27 2023 -0300
Allow passing arguments to autopkgtest backends
A configuration may specify new variables
debci_autopkgtest_args_$backend that are to be passed to the respective
backend when running an autopkgtest. For instance, this allows changing
the --ram-size for a qemu backend.
Closes: #1037245
commit f33c8b5ede2767eabaa7a0763f11a95d276f693b
Merge: 9d8c2f08 7d6f4486
Author: Antonio Terceiro <terceiro@debian.org>
Date: Fri Jul 28 19:08:20 2023 +0000
Merge branch 'munin' into 'master'
import and install munin plugins from debian-ci-config
See merge request ci-team/debci!256
commit 7d6f4486c4c332f78dbf2cb26306a9937c71d349
Author: Helmut Grohne <helmut@subdivi.de>
Date: Fri Jul 7 14:08:22 2023 +0200
import and install munin plugins from debian-ci-config
This copies the munin plugins from debian-ci-config as of 24458ea90331
("Merge branch 'munin-no-template' into 'master'"), which was the moment
where none of them were templates anymore.
The plugins are installed into the standard location
/usr/share/munin/plugins. Since debian-ci-config installs plugins
directly into /etc/munin/plugins, this will not affect the main
installation.
The plugin-conf.d files are directly installed to the standard location
/etc/munin/plugin-conf.d. These have the same content as in
debian-ci-config for now.
Once this is deployed, debian-ci-config should stop installing the
plugin-conf.d files and enable the plugins by placing symbolic links to
the files in /usr/share/munin/plugins. Thus debian-ci-config can drop
the actual plugins finishing this transition.
commit 9d8c2f083cc5f9559d67d14c75c44b94369aea78
Merge: 05249dc5 f7d395d7
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sun Jun 25 14:22:38 2023 +0000
Merge branch 'invalid-payload' into 'master'
Improve handling of incomplete test results
See merge request ci-team/debci!254
commit f7d395d77e1c9e9dbc1a99c42960cc9e8c54a170
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 16:10:49 2023 -0300
Make codespell ignore certain words
Initially, ignore requestor ==> requester corrections, as requestor is
also valid English.
Gbp-Dch: ignore
commit 8139107f7102724505ebec3c0d6b8fadacc812d5
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 15:56:49 2023 -0300
.gitlab-ci.yml: disable autopkgtest-stable for now
debci now needs autopkgtest > 5.29, but bookworm backports is not open
yet.
Gbp-Dch: ignore
commit d2cb42a5f043ff406c10dcf1dccfbe2ccd0c9bbf
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 15:52:21 2023 -0300
Debci::Job.receive: improve handling of incomplete test results
When test results transmission from workers to the collector fails, the
payload may fail to uncompress, and the results diretory will not
contain all files we usually expect. Be more forgiving about missing
files, and work with whatever we can.
In particular, this fixes a crash when log.gz is missing.
commit fdd9536490f4598e72d76263cf0622e93dd30693
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 15:50:38 2023 -0300
job_spec: create all files in the test results directory at once
Gbp-Dch: ignore
commit 232a8f481ccb8e9cbdb72a60721f66c54003e64f
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 15:40:20 2023 -0300
spec_helper: give a proper name to temporary directories
Gbp-Dch: ignore
commit 3fa77e451c6fff6eccc39e195fcb6f2952e8c46d
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Jun 24 14:42:34 2023 -0300
collector_spec: fix typo
Gbb-Dch: ignore
commit 05249dc5cbb338b7d798cf4b61b676270a0d198a
Merge: cd323528 41f9bedf
Author: Antonio Terceiro <terceiro@debian.org>
Date: Tue Jun 20 15:56:02 2023 +0000
Merge branch 'drop-devscript-dep' into 'master'
Drop dependency on devscripts
See merge request ci-team/debci!253
commit 41f9bedfe61c014fcc84499f54dacb7431082730
Author: Antonio Terceiro <terceiro@debian.org>
Date: Tue Jun 20 10:50:44 2023 -0300
debian/tests/control: drop uneeded Ruby dependencies from non-Ruby test suite
Gbp-Dch: ignore
commit dee251ff242caa8f4d96c07ddd55d814b68af2c2
Author: Antonio Terceiro <terceiro@debian.org>
Date: Tue Jun 20 07:47:23 2023 -0300
Drop dependency on devscripts
`file` was being pulled as part of devscripts, and used in the test
suite. Rewrite that test to use a ruby-magic instead.
commit 3f4504d8a2a860bbdd0d8203130ef217ccfc9da8
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 21:45:29 2023 -0300
debci-publisher: fix obtaining data for logging
commit 266c5c0f5841b790af5a523a7cf6204d0d683f4e
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 15:30:13 2023 -0300
Makefile: ignore more autogenerated files
Gbp-Dch: ignore
commit cd32352816b8474653c25c3f3f45a643d88af15a
Merge: 09cfc029 e15f1a4a
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 18:25:05 2023 +0000
Merge branch 'add-dbgsym' into 'master'
backends/lxc/create-testbed: do not enable --dbgsym for Debian derivatives
See merge request ci-team/debci!252
commit e15f1a4a294abd90e02af9fa2a525d2adf1002ac
Author: Helmut Grohne <helmut@subdivi.de>
Date: Wed May 3 17:16:47 2023 +0200
backends/lxc/create-testbed: do not enable --dbgsym for Debian derivatives
The --dbgsym option adds a debug archive that always uses deb.debian.org
regardless of the --mirror option. In most cases where a custom
--keyring is used, this does not make sense, so skip --dbgsym in those
cases.
commit 09cfc02901887324032fcb4c9e3ab20af50046c4
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 09:20:19 2023 -0300
Bump dependency on autopkgtest to >= 5.29~
Commits 96a6135 and 88d1a1b depend on new functionality only present
after autopkgtest 5.28.
commit 0ab189e592ea6be54d5154b407e8e0c034975257
Merge: 0373d3be 47970b91
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 09:05:42 2023 -0300
Merge branch 'merge-requests/250'
commit 47970b9198bf67df07ce4ea4f1a4350f03334ffb
Author: Helmut Grohne <helmut@subdivi.de>
Date: Tue May 9 16:56:57 2023 +0200
debci-setup: conditionalize installation of auto-apt-proxy
For one thing, e.g. lxc can configure a hard coded proxy. Having both a
hard coded proxy and auto-apt-proxy interacts badly, so only have one.
Also skip installation of auto-apt-proxy when it is unavailable in the
target suite.
commit 0373d3be81a2a6da0483b6d5a88f8ef00640bbe9
Merge: ace13ed7 a2192dec
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 11:59:22 2023 +0000
Merge branch 'reduce-temporary-apt-sources' into 'master'
backends/lxc/create-testbed: use fewer different sources.lists
See merge request ci-team/debci!247
commit ace13ed7c1510e1d204b4b4b3f6d28bf541bafb2
Merge: 7ad37341 96a61358
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Jun 14 08:55:49 2023 -0300
Merge branch 'merge-requests/248'
commit 96a6135874983b9b93b445f5e38f6e6a29239f0a
Author: Helmut Grohne <helmut@subdivi.de>
Date: Wed Jun 7 11:39:31 2023 +0200
backends/lxc/create-testbed: pass keyring via new --keyring to autopkgtest-build-lxc
commit 88d1a1b9949792994abf35ef1cfef5b90b45ae90
Author: Helmut Grohne <helmut@subdivi.de>
Date: Fri May 5 14:19:28 2023 +0200
backends/qemu/create-testbed: forward --keyring
commit 85504d693da01cc30c87cd8d544885a185e4340b
Author: Helmut Grohne <helmut@subdivi.de>
Date: Wed May 3 17:06:59 2023 +0200
debci setup: add --keyring option
debci specifically handles the derivatives Kali and Ubuntu by using
separate lxc-templates that know about their keys. Continuing this way,
we'd have to add every possible derivative to debci and lxc-templates.
Alternatively, we can treat very simple derivatives as being Debian and
pass their differences along. The most fundamental difference will be
the keyring using, so by providing this as an option, we can use debci
with simple derivatives while continuing to use the debian lxc template
and without further modification to lxc-templates or debci.
commit a2192dec0e6770b0bbc0a102508630583486fb0b
Author: Helmut Grohne <helmut@subdivi.de>
Date: Wed Jun 7 10:47:31 2023 +0200
backends/qemu/create-testbed: use fewer different sources.lists
This is a similar simplification as the one done to lxc: Rather than
have autopkgtest generate a sources.list and then overwrite that in a
customization step, generate it ahead and tell autopkgtest-build-qemu to
use the one we want.
commit 7ad37341c31de451ea9cb2a16e9f1746b02a7908
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sun May 28 11:47:00 2023 -0300
Drop schroot backend
It has been several years that ci.debian.net no longer uses schroot at
all, and there is no point in keeping this code around.
- /var/lib/debci/chroots is no longer created on debci.postinst
- The entire /var/lib/debci is already removed by debci.postrm when
purging
commit 64338d435422a760261e48537cb97767ddfa84f1
Merge: 27b42265 12e0796e
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sun May 28 14:31:49 2023 +0000
Merge branch 'undefined-proxy-variable' into 'master'
backends/qemu: fix evaluation of undefined variable
See merge request ci-team/debci!249
commit 12e0796e0195b04cb8cb67a79ee0f72cc4e3d64f
Author: Helmut Grohne <helmut@subdivi.de>
Date: Fri May 5 09:47:25 2023 +0200
backends/qemu: fix evaluation of undefined variable
When no proxy is configured, evaluating RES does not set PROXY, so it
may legitimately be unset. Avoid terminating the customization step in
that case.
commit 153e2d3e7432981fda0b18f28e2bd47ae5f88ec4
Author: Helmut Grohne <helmut@subdivi.de>
Date: Wed May 3 13:42:54 2023 +0200
backends/lxc/create-testbed: use fewer different sources.lists
When creating a lxc testbed, we change the apt sources.list multiple
times. Initially, the lxc-debian template creates one via lxc-create.
This has been called from autopkgtest-build-lxc, which has its own ideas
about sources.list and replaces it. That happens to be called from
backends/lxc/create-testbed, which again has its own ideas and replaces
the sources.list again. If any of these sources.lists does not work, the
setup fails.
In order to simplify this, we remove the one from autopkgtest-build-lxc
from the equation. Rather than having backends/lxc/create-testbed
replace the sources.list, we tell autopkgtest-build-lxc to just use the
one we'd be overwriting directly.
commit 27b4226549f1db6b247f5cd491ea3aba3137cb34
Merge: 4cb44568 26cbbbe5
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Apr 19 18:08:53 2023 +0000
Merge branch 'rm-obsolete-code' into 'master'
Drop obsolete scripts and mentions to them
See merge request ci-team/debci!246
commit 26cbbbe5b927e6ec7f458a62e4f95296d4a51f01
Author: Antonio Terceiro <terceiro@debian.org>
Date: Wed Apr 19 14:38:39 2023 -0300
Drop obsolete scripts and mentions to them
Since debci-batch has been removed in favor of external scheduling of
tests, some infrastructure left behind is not really useful anymore.
Right now, drop debci-list-dependencies, debci-list-packages,
debci-setup-chdist and references to them (which are unused anyway).
Gbp-Dch: full
commit 4cb44568812ec1abff83123883017f34ca6d784d
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Apr 15 12:54:40 2023 -0300
spec_helper: bump required test coverage to 92%
commit ec91c2b882a73e7c5879f1cbfb41dc429b55fd85
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Apr 15 12:41:21 2023 -0300
self_service: allow multiple API keys and add to manage them
commit 29a0e2b345801d541789d24d4e969b2ee98fc0db
Author: Antonio Terceiro <terceiro@debian.org>
Date: Mon Mar 13 16:30:10 2023 -0300
debci-generate-apt-sources: fix checks for non-free-firmware for suite codenames
It was only working correctly for the suites in use in ci.debian.net,
e.g. unstable, testing, stable etc. When given a suite codename (e.g.
bullseye, stretch) it would produce the correct response.
commit 691fe69c8954fd14e7c453a0c9655ba6eb6357c8
Merge: de2cabce 047a0fb6
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Mar 11 20:14:18 2023 +0000
Merge branch 'consistent-logging' into 'master'
Make logging for all daemons consistent
See merge request ci-team/debci!244
commit 047a0fb67fbdd2651f5f1d1c6909995af161dc97
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Mar 11 16:44:10 2023 -0300
debci-worker, debci-publisher: log errors even in quiet mode
commit 105032056e6ff8fe752114dba6ada5c02f827387
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Mar 11 16:36:44 2023 -0300
Make logging for all daemons consistent
All daemons will now report using the same format:
$package $suite/$architecture/$backend ($run_id) $status [$duration]
This should make it easier to correlate logs from daemons that run on
different hosts.
commit f9d297f82c3ce5f81475c155aa143700003c8350
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sat Mar 11 16:33:37 2023 -0300
Debci::Job: store and report backend
Report default backend if the database field is empty, to be backwards
compatible with jobs created before this was introduced.
commit de2cabce65d9afb552ea7f1ddc33546cf30c6981
Author: Antonio Terceiro <terceiro@debian.org>
Date: Sun Mar 5 20:57:37 2023 -0300
Split publishing of results into a separate daemon
We have been having reliability issues on s390x and ppc64el, and some
evidence that points to the publishing of results. By separating running
jobs from publishing their results, this should ease debugging
connectivity issues, while keeping the results around so that they can
be published later when the issue is solved.
The protocol between the worker and the publisher is very simple: the
worker will copy the results tarball in the designated results dir, and
after that will create a matching stamp file. The publisher waits for
events on stamp files, and tries to publish the corresponding results
file.
The publisher will try publishing at most 3 times, with a delay of 5
seconds between each attempt. On failure, the publisher will try again
later (e.g. when new results arrive).
Another property of this new design is that regardless of how many
workers a host is running, there will be a single process actually
pushing the results back to the master.
