Git log: commit 6118a758fda9d589cdbcff126f6c3c8004e430b4
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 30 02:07:11 2023 +0100
make-firmware-image: remove non-free from FIRMWARE_COMPONENTS
See previous commit for rationale.
commit 3d879d5e467be7df90c0726005a06a670000f909
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 30 02:01:11 2023 +0100
CONF.sh: remove non-free from NONFREE_COMPONENTS
Almost all packages initially identified as interesting for us[1] have
been moved to non-free-firmware. An updated list of packages has been
published[2], which can be summed up this way:
- amd64-microcode and intel-microcode would require specific code
anyway (#1029804);
- zd1211-firmware shows inconsistencies (archive vs. repository).
1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
2. https://lists.debian.org/debian-boot/2023/01/msg00235.html
Therefore, stop looking for non-free firmware packages in non-free,
and concentrate on non-free-firmware only.
commit d94e9eb5b727a1b610d81577faffc7c371631e0c
Author: Samuel Thibault <sthibault@debian.org>
Date: Sun Jan 29 22:03:55 2023 +0100
Fix missing %install% replacements
E.g. the ontimeout line of spkgtk.cfg includes both the kernel and initrd
paths on the same line, so need several %install% replacements.
commit 492b08530504bfc8d3da5737bcfe024a3777b805
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 07:38:40 2023 +0100
make_disc_trees.pl: move local vs. base_components safeguard
We have two safeguards related to the local component now, one if it's
present, one if it's absent: group them via if-then-else.
commit 2e61e76d86845633fdf74b1e1d67de0ff340385c
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 07:38:40 2023 +0100
generate_firmware_task: prettify output
Use a format string to align both columns (Package and Filename fields).
commit d415c60a1250c75bc10ad981c28a6581a3c94f3b
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 07:38:40 2023 +0100
generate_firmware_patterns: prettify output
Use format strings to align all entries (%4d entries).
Use standard double-space indentation found almost everywhere else.
commit 59978789f08bb694e4038b0b32261f138b9e017c
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 07:38:40 2023 +0100
apt-selection: order contrib before non-free*
Components are ordered “main contrib non-free” almost everywhere, let's
use the standard order here too. There should be no functional changes.
Note: This controls the sources.list file used during the build, but
doesn't control “deb cdrom:” lines in sources.list during or after
installation, which are the result of running apt-cdrom (which sorts
components alphabetically, see its pkgCdrom::ReduceSourcelist function).
commit ccaa1a9c693f3f18604405caa51d6935dddc0e8a
Merge: 8bba3df8 7ce8d0c2
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:27:02 2023 +0100
Merge branch 'clarify-fix-improve-support-for-local-packages'
WARNING: This is an attempt at making it easier for users to enable
support for local packages. This branch is really believed to make
the whole situation easier and clearer on the long run, but it might
introduce some regressions, or at least behavorial changes depending
on existing configurations.
In particular, users might have a hierarchy under LOCALDEBS that's
different from the one getting documented in eb6d140f48, possibly with
a top-level local/ (see also bfebeded70 and 7ce8d0c28f).
It's advised to move files around to match the documented layout, and
to report any regressions that would be spotted: many tests where run,
trying to torture this patch series as much as possible, imagining
borderline cases, but in the end, that's not a feature that's getting
fully battle-tested since it's obviously not used for official builds.
commit 7ce8d0c28f8c79ee7c1587305a6ea7d16eb52f6c
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:24:03 2023 +0100
start_new_disc: remove /dists/local commodity symlink
When local packages are enabled (LOCAL=1), various things happen in the
target /dists directory. One of them is the creation of a /dists/local
symlink, pointing to (/dists/)<CODENAME>. This leads to a very confusing
situation, where “local” is initially about a component, but effectively
ends up looking like a distribution.
It is theoretically possible that cleaning this up might require some
follow-up fixes in other tools, but even that would be better than
keeping bear traps around.
commit ae8490ba7099e216c0a01e55c889db0234e4bc5f
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:24:03 2023 +0100
make_disc_trees.pl: rewrite base_components if needed
It's possible to set LOCAL=1 to enable support for local packages, which
automatically results in “local” being listed in /.disk/base_components,
which is used to build the --components parameter for debootstrap.
Users might have various sets of *.deb and/or *.udebs, be it over time,
or for various codenames, and it's possible for /dists/<CODENAME>/local
to be entirely missing in the generated image.
In this case, /.disk/base_components must not contain “local”, as that
would break debootstrap; this commit detects and fixes this possible
inconsistency.
commit c24949141b4c00b17ec2a1a73360ea6459b442fb
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:24:03 2023 +0100
Packages-gen: write compressed Packages indices again
This is a regression introduced while trying to catch up with apt's
expectations, in 2183089968c04c4765d2119d176e021431b26353: iterating over
Packages files is done by running the “zcat” utility on Packages.gz and
Packages.xz; replacing Packages.gz with an uncompressed Packages broke
iterating over Packages files referencing udebs.
commit bfebeded7083c1bfcbb1e9b48e795beb7a6cb1a3
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:24:03 2023 +0100
make_disc_trees.pl: merge “local” udebs into “main” explicitly
This might have worked by accident depending on the exact layout under the
LOCALDEBS directory (see previous commit for the big picture).
Let's focus on where input files for d-i should be stored:
<LOCALDEBS>/dists/<CODENAME>/local/debian-installer/binary-amd64/*.udeb
Those *.udeb end up being stored under the “local” component[1] (making it
clear they're not from usual components on a Debian mirror), but they are
referenced in Packages for the “main” component[2], possibly replacing a
different version found in the archive.
/dists/<CODENAME>/local/debian-installer/binary-<ARCH>/*.udeb [1]
/dists/<CODENAME>/main/debian-installer/binary-<ARCH>/Packages [2]
This makes it possible for d-i to find those *.udeb automatically, as it
uses a single Packages file.
commit eb6d140f48db94cbbcc640f4ab195bec0ad48749
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 05:24:03 2023 +0100
CONF.sh: clarify expected layout below LOCALDEBS
Getting support for local packages (*.deb and/or *.udeb) to work
consistently has been a gigantic time sink, never again!
Document the expected layout as precisely as possible, so that
unsuspecting users have a better chance of getting things right!
commit 8bba3df8659b0f6253b3632be208a7defa6250be
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:37:06 2023 +0100
make_disc_trees.pl: fix block calculation for --rollback
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
commit 5e39a81adbb69878e7aa12bae5f9d1d4469e50e8
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:37:06 2023 +0100
make_disc_trees.pl: fix block calculation for multi-arch
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
commit 9fcb75094853d921e5b164ed8631664c7353bb93
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:22:43 2023 +0100
README: clarify link between FORCE_FIRMWARE and NONFREE_COMPONENTS
commit 88b10ce0377d943b49f1dc20268fcb84e0ff2791
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:22:43 2023 +0100
README: clarify both NONFREE and NONFREE_COMPONENTS must be exported
commit e6b19960aabe21314c6e3df071834fc4402e9664
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:22:43 2023 +0100
CONF.sh: a link between FORCE_FIRMWARE and NONFREE_COMPONENTS
commit ee1efda6dea4be6e553592d1fb3a82f9e61e955a
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 27 00:22:43 2023 +0100
CONF.sh: unset NONFREE_COMPONENTS initially
The idea is to start from a clean slate every time, do the same as for
other variables.
commit 2674f2606045beadd09c8d16e6abac0be0c506c6
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:47:45 2023 +0100
CONF.sh: unset CONTRIB for consistency
It looks strange to have NONFREE unset and CONTRIB=1 in the default config,
since packages in contrib are usually there because they depend on non-free
stuff.
commit 95df8ab5246710bcab58777364d85590f5596d92
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
make-firmware-image: implement firmware/dep11 generation for bookworm and above
Mimic what make_disc_trees.pl does, making it possible for hw-detect to
spot not only the /firmware/dep11 directory in netinst images, but also
on installation images with firmware added manually.
Rename pseudo-option to clarify it returns filenames and indices.
Keep generation disabled for bullseye (as an extra safeguard, even if we
have separate, per-release branches anyway): That's helpful information
but bullseye is a stable release, and changing even unofficial material
might not be the best idea. Be cautious by default, it's easy enough to
enable if desired.
commit 45db7e15820385a9c38c80a721570c387b3b4ac2
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
generate_firmware_task: build lists of Packages files and iterate on them
The next step for make-firmware-image is to use generate_firmware_patterns
for each firmware package that was detected. That command requires DEP-11
metadata, so needs to know which Packages file each firmware package comes
from.
Instead of building long $pkgfiles and $bp_pkgfiles strings, and passing
them to catz, build lists instead and iterate over them. This means a few
more catz calls, but that also means being able to derive the DEP-11
directory from Packages paths.
There should be no functional changes.
commit a2c722b7354f2247f412148eb1e0480dd5ff6d93
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
make-firmware-image: use generate_firmware_task to find firmware
Since that tool might be used without a caller that sources CONF.sh,
be explicit about the components in which firmware packages must be
searched:
- use "main contrib non-free" for bullseye;
- use "main non-free non-free-firmware" for bookworm and above.
non-free can go away once we have all we need in non-free-firmware.
Export some environment variables for generate_firmware_task (also
renaming SUITE to CODENAME).
commit bd13392570b8930e5cc926f0bb64297e9f934dec
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
generate_firmware_task: implement support for --list-filenames
Treat the output file argument specifically when it's “--list-filenames”:
instead of actually generating a task file with a list of packages (and a
few comments), output the Filename field for each firmware package
matching our criteria.
Move some status messages to STDERR accordingly.
This makes the “find suitable firmware packages” reusable, e.g. by
make-firmware-image.
commit 90739b8cde293b13cf8bd09e6bc3d1a23219801f
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
generate_firmware_task: move code around
This is a preparatory step for the following commit, there should be no
functional changes.
commit 14d266445921875328c4a0fa57ba74cadd461504
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 23:22:40 2023 +0100
generate_firmware_task: pick and choose components
We have a few fully-free firmware packages in main, and a few downloaders
(e.g. b43-installer and b43legacy-installer) in contrib; the latter might
not be really interesting without an alternative connectivity during the
installation process.
Regarding non-free vs. non-free-firmware, we started moving packages[1] from
the former to the latter so that we could build official installation images
based on main and non-free-firmware only.
1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
At the moment, NON_FREE_COMPONENTS still lists both, but the plan is to move
to non-free-firmware only once all the packages we want are available there.
The relevant variables regarding components include:
- CONTRIB
- NONFREE
- EXTRANONFREE
- FORCE_FIRMWARE
This commit implements:
- using main all the time;
- using contrib if CONTRIB=1;
- using NONFREE_COMPONENTS if NONFREE=1 or EXTRANONFREE=1 or FORCE_FIRMWARE=1.
At the moment it seems the only existing caller is the top-level Makefile,
only when FORCE_FIRMWARE is set. But it seems to make sense to match what
tools/apt-selection does, to avoid any surprises.
commit 4a31c04b815aec1b5ab40001c68f4c1db5ac821c
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 26 21:14:20 2023 +0100
make_disc_trees.pl: remove workaround for missing bookworm/non-free-firmware/dep11
This directory appeared in the archive earlier today.
commit ee40a3bf239440026362bf94a66ffae52b1c15e0
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 25 01:38:34 2023 +0100
make_disc_trees.pl: add workaround for missing bookworm/non-free-firmware/dep11
There's no dep11 directory for bookworm's non-free-firmware component
for the time being, and we've just decided to be more cautious about
input files, which breaks the bookworm build.
Until the issue is fixed on the appstream.debian.org side, and until dak
can indeed import that directory into the Debian archive, use sid's
non-free-firmware dep11 directory instead of bookworm's.
commit 214e2dd88c74fbf649de6c06f19e4dc4580843e7
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 25 01:38:34 2023 +0100
make_disc_trees.pl: use write_file() for .component files as well
commit 957ee269b4f129fc2039143b70fcc3b5eb150b82
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 25 01:38:34 2023 +0100
make_disc_trees.pl: create /firmware/dep11/README.txt and count it
Move the README.txt creation from generate_firmware_patterns to its
caller, so that it's done just once, and so that added blocks are
counted properly.
commit b56af4dbcc05695d7d3ea272bff1dc1cf15ec106
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 25 01:38:33 2023 +0100
generate_firmware_patterns: rewrite to match caller's expectations
Initially it was interesting to pass “--package ALL” to create *.patterns
files for all packages found in the specified Components-* file(s), but
make_disc_trees.pl uses a call per package, so drop support for looping.
Remember whether modalias information was found for the requested package,
and only apply the firmware-sof-signed workaround when that's indeed the
requested package and when modalias information wasn't found (in case that
gets added to DEP-11 metadata at some point in the future). When using the
workaround, include a reminder in the logs.
commit 9d95f65bd9350e90c7e70745cd03d510be98df32
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 25 01:38:33 2023 +0100
generate_firmware_patterns: stop creating the output directory
The output directory's presence is the caller's responsibility; while
it might have been useful during development, actual builds require
make_disc_trees.pl to keep track of blocks getting added, so let it
take care of fulfilling prerequisites.
commit 2183089968c04c4765d2119d176e021431b26353
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 23 20:25:21 2023 +0100
easy-build.sh: create top-level Release file for local packages
Without this, apt would try and download way too many indices, resulting
in 128 “Ign:” lines. With this, we're down to 4 “Ign:” lines (Release.gpg
and InRelease, twice).
Funnily enough, that changes apt's behaviour, which suddenly requires
uncompressed Packages to be present.
commit cc53129b15506c6d8f7a3efa6899751ca2531e06
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 23 19:30:42 2023 +0100
apt-selection: mark local repositories as trusted
Without this, apt won't like local repositories.
commit f2a4a832a2b19560adc895c6582d8b6412eb62d9
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 23 19:30:39 2023 +0100
Replace all gzip --best calls with pigz -9nm ones.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
commit 14064e08a10da7131ba2d50ad3cf277df696a7c3
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 23 19:30:35 2023 +0100
Replace all gzip -9 calls with pigz -9nm ones.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
commit de66d9f692b04fc83a7857712c23d080e2dfc129
Author: Cyril Brulebois <kibi@debian.org>
Date: Mon Jan 23 19:30:32 2023 +0100
Add pigz to Depends for parallel gzip compression.
commit f7925da6cd9f02fe5bb590205e26028c28d8ee23
Author: Cyril Brulebois <kibi@debian.org>
Date: Sat Jan 21 18:54:38 2023 +0100
make_disc_trees.pl: store .component alongside .patterns for firmware packages
This makes it possible for hw-detect's install-firmware hook to enable the
right components if it installs firmware packages based on modalias
information. The components found in Contents-firmware indices are only
useful for firmware packages detected by the check-missing-firmware.sh
script.
commit 4308e4666cc3340645e55e681d26d49812b0db8f
Author: Cyril Brulebois <kibi@debian.org>
Date: Sat Jan 21 16:40:58 2023 +0100
make_disc_trees.pl: add components to Contents-firmware indices
All three columns could be aligned, but I'm not sure how many characters
should be allocated to the package filename. Alternatively, we could
scratch alignment altogether, and let interested users use column -t on
the file when debugging.
commit bbda95c283dcdc4b58376a45d665c00bb72fd706
Author: Cyril Brulebois <kibi@debian.org>
Date: Sat Jan 21 16:40:56 2023 +0100
make_disc_trees.pl: fix misnamed $dist variable
$dist would suggest distribution, an alternative name for suite or
codename (!), let's make it clear we mean component here.
commit 8c497447f9ca58c8487bd52000df2a2eef2a1b93
Author: Cyril Brulebois <kibi@debian.org>
Date: Sat Jan 21 16:33:35 2023 +0100
generate_firmware_patterns: make sure dep11 metadata files are present
Otherwise we might be failing to generate firmware patterns for some
firmware packages. Being cautious about this lets us submit patches
against dak: https://salsa.debian.org/ftp-team/dak/-/merge_requests/267
commit a2378ed25db6a2c2aeba7d67137d9adad734015c
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 20 16:23:17 2023 +0100
make_disc_trees: create /firmware before adding symlinks there
Otherwise it would only be created after failing to create the first symlink.
See previous commit.
commit b1cb4c43c235616879eb0782d3967aa1a55fc0ba
Author: Cyril Brulebois <kibi@debian.org>
Date: Fri Jan 20 16:22:42 2023 +0100
make_disc_trees: error out if firmware symlink creation fails
Otherwise failures would go unnoticed, logs would pretend firmware-b43-installer
(the first firmware package getting processed) is indeed installed there while
it's not. See next commit.
commit 47ae1a293b6f4e20215c92efbc2eba145f4f91d4
Author: Cyril Brulebois <kibi@debian.org>
Date: Thu Jan 19 18:16:19 2023 +0100
Add uuid-runtime to Depends.
That's needed for uuidgen, added in 6d1fdf2446.
commit 5aa89ab269861e6707ba1e05401b986d4c649074
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 22:43:23 2023 +0100
Add leading / to paths in Contents-firmware indices.
With this change, hw-detect's check-missing-firmware.sh can use grep -f
with a file listing absolute paths to required firmware files.
commit 81baccabbf35018dc8f864838d459217c800c00e
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 22:07:19 2023 +0100
Refresh references in comments.
commit 51bf02a44c3d6ef9cd93a74b94777441110c6d73
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 21:59:31 2023 +0100
Avoid duplicates.
commit 80d19dcb42d2ccca4b1e63dded505fb88ffa5650
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 21:43:28 2023 +0100
Refresh @sof_aliases for unstable.
Bump to the current linux-image ABI in unstable, and look at
snd-soc-*-sof modules in addition to the snd-soc-sof ones.
commit fb2eac45ebf7aee5f62e0a7af505036e8c0d243b
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 21:35:29 2023 +0100
Refresh @sof_aliases for bullseye.
Update comments:
- Bump linux-image package to the latest found in bullseye.
- Include snd-soc-sof.*\.ko in the lookup.
- Delete the TODO: the list was built using snd-soc-sof*.ko already,
since a80f013b57230991eb99316789d3d941d6d7e84f.
commit 139041f33229a364b461ed74b8c595166257bac5
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 21:25:48 2023 +0100
Reindent without mixing tabs and spaces.
No functional changes.
commit 723fdc9ee839c1de6c3db20c2ff8ef6635ff8a9f
Author: Cyril Brulebois <kibi@debian.org>
Date: Wed Jan 18 21:17:44 2023 +0100
Delete TODO.
It was addressed in 897bf4754dbbd74904e1140ec3a6e6b26a599e18.
