Git log: commit 560724267c442eb4a6fa3d67a581a1aaf7c6c3c1
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 14:31:10 2020 -0500
update changelog
commit 71842b0e9f8150fc48283409c80552283683c42d
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 14:29:55 2020 -0500
* Backport new patches that fix ABI mistake in 0.2.0
commit b08ef1a5fb40594cf23ced6c85e311375b85f20e
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 11:20:52 2020 -0500
remove extra build log files
commit 911772125542e2e81d74fd0965a812f76b838d99
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 10:58:24 2020 -0500
update changelog
commit 8ca70d472f965c6961dee7f3b38f350d081adf3b
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 10:56:45 2020 -0500
Update packages for soname bump that broke ABI
commit bde852efbf3aca2f91d60c67a894ddbb2417e97d
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 10:56:06 2020 -0500
drop all patches, now upstream
commit 33226fef0872ab7805a301ccccc5288b82f40321
Merge: e273494 b4f6076
Author: Mario Limonciello <mario.limonciello@dell.com>
Date: Tue Aug 18 10:47:02 2020 -0500
Merge tag '0.2.0' into debian
Release libxmlb 0.2.0
commit b4f60764ec3a03ae666d81d8b61bc8dbbccf2b85
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Aug 18 15:36:23 2020 +0100
Release libxmlb 0.2.0
commit 770d0e31d19a99aaffe6a39ae62fa17de9221fec
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Jul 28 10:25:34 2020 +0100
Add the missing TEXT:INTE cases which caused the afl issues
Although comparing text to an integer is not very widely used, it is valid
XPath and that is what afl reminds us.
commit 9860e0b5d5142779307c42dafe17d98c4936d212
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Jul 27 13:49:25 2020 +0100
Add GError to xb_stack_pop() and xb_stack_push()
Many places were calling these and returning FALSE without setting the GError,
leading to critical warnings and crashes. The error code was also different
depending on the caller.
Set the optional GError in the low-level function for afl sanity.
Additionally, do not use g_assert_not_reached() when comparisons are not
handled as it's better to return with failure than just crash the caller.
commit 60aff1fbcbbe76071fc8f43d7797282f397bbfc2
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Jun 23 12:57:11 2020 +0100
Actually make XB_QUERY_FLAG_FORCE_NODE_CACHE work
The XbSiloQueryHelperFlags are different to the XbQueryFlags.
commit 313143ee66ccafd729fc8d7fcbf481fc56f5571c
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Jun 23 12:26:04 2020 +0100
trivial: Fix header includes when used outside libxmlb
commit 324d616ad8b92ef9e7fc8e7ccac27c30bf0d26b3
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Jun 23 11:17:06 2020 +0100
trivial: Disable the ABI checker on the development branch
commit 629acfa24ef72568e013bb07073056d199d25c82
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Jun 22 20:35:21 2020 +0100
Change the default value for the node cache
I've changed my mind slightly here; if libxmlb breaks API it's only going to be
packaged in the development versions of distros, and thus fwupd will also be up
to date.
commit 7bba2b941d898ce69fff57aeb456de032c50f8c8
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Jun 22 20:28:34 2020 +0100
Allow specifying the node cache behaviour for the query
This allows us to change the default for enable_node_cache in the future. Even
fwupd only needs the node cache in one place, and enabling it for the entire
XbSilo seems like huge overkill for a handful of <release> nodes.
commit 0013dff7f687279cd13512e2621ebea109c9cb2e
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Jun 22 22:24:40 2020 +0100
Export xb_node_get_silo()
We need the silo for a standalone XbNode when building prepared queries using
xb_query_new() and xb_query_new_full().
Split it out to a new header to avoid an #include loop.
commit e19c0ad4549cf271d76c6c29ebe27172377ae5cb
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Jun 22 17:23:11 2020 +0100
Do not use the node cache when building indexes
commit 52b1ba1aad777df0c2e8440bd0a2df1edd341d3d
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 10 19:00:03 2020 +0100
xb-silo: Don’t unconditionally create GTimer objects
Most of the time, profiling is disabled, and these are unnecessary. Each
call to `g_timer_new()` calls the `clock_gettime()` syscall, which is
not particularly cheap.
This reduces `clock_gettime()` calls by around 11000 while starting up
gnome-software.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 3a0d23057cf4a69a8d7f0e6657e08bb2d3bcc23d
Author: Philip Withnall <withnall@endlessm.com>
Date: Thu Jun 4 12:10:51 2020 +0100
xb-node: Document that xb_node_{get,set}_data() needs node caching
See the previous commit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 50273f52c7f7c3d1ca5207e0546732463cac8904
Author: Philip Withnall <withnall@endlessm.com>
Date: Thu Jun 4 12:08:01 2020 +0100
xb-silo: Conditionally remove XbNode cache
This cache has existed since libxmlb was originally written, and
contained a strong reference to every `XbNode` ever constructed by the
`XbSilo`, presumably to avoid the cost of constructing new `XbNode`
objects repeatedly.
Some profiling of it shows that, for the two `XbSilo`s in use by
gnome-software:
• Silo 1: 4512 entries, hit rate 2352/6864 (34%)
• Silo 2: 2613 entries, hit rate 1601/4214 (37%)
(After applying the previous commits in this MR, which reduce the size
and usage of the cache by about 1/3 on its previous levels.)
Most of those hits are fairly localised in time, and once a particular
piece of XML has been processed (for example, by gnome-software’s
appstream plugin), its nodes are not touched again, but continue to sit
in the cache forever.
Dropping the cache reduces locking overhead for accesses to the
`XbSilo`, and drops 1.8MB of heap memory usage from gnome-software.
(This was split between 990KB for the `XbNode` instances and 830KB for
their entries in the cache hash table.) It does not affect the speed of
jobs in gnome-software’s appstream plugin on startup, which remain the
same (to the nearest 100ms).
Unfortunately, fwupd relies on the cache memoising every `XbNode` so
that `xb_node_{get,set}_data()` can be used to side-load data into the
XML tree so it can be passed around between otherwise-disconnected parts
of the fwupd code. There’s no easy way to rearchitecture fwupd to avoid
this dual use of the `XbNode` tree as a representation of the XML and as
an implementation of the higher level objects.
To preserve compatibility, a new `XbSilo:enable-node-cache` property has
been created, enabled by default. Most clients will want to disable it
shortly after constructing an `XbSilo`.
`XB_BUILDER_COMPILE_FLAG_NO_NODE_CACHE` does this when compiling an
`XbBuilder`.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 78e9ec613c026126dac0537a17e18a7544438de9
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 3 14:49:59 2020 +0100
xb-node-query: Use XbSiloNodes as intermediate results
`xb-node-query.c` has various helper functions which return results
derived from `XbNode`s, such as an attribute or the node’s text content.
Previously these functions queried for an `XbNode` and then derived
their results from it, then threw the `XbNode` away. Since they’re used
quite commonly in the appstream parsing in gnome-software, this resulted
in a lot of `XbNode`s being added to the silo’s node cache and never
being hit in the cache again.
Avoid that by operating on `XbSiloNode`s instead, so they never have to
be wrapped by an `XbNode`.
This introduces no functional changes, but should speed up these queries.
It reduces the size of the gnome-software silo caches from:
• 5745 → 4512
• 3315 → 2613
on my machine. It reduces the cache hit rate slightly from 40% (both) to
34% and 37%, respectively. The time taken to do operations on the
appstream plugin when starting gnome-software remains the same (to the
nearest 100ms, the resolution given by gnome-software logging).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit d71bcb6ad4f36dfe3aa3d6688956ceb9236a0b78
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 3 14:48:14 2020 +0100
xb-silo-export: Change function to operate on XbSiloNode
This will help in upcoming commits to avoid using `XbNode`s where their
creation would be temporary and expensive.
This is private API, so it’s not an API break.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit abd2f3dd538a142bba84e28e2ae4b91fc1fe0fff
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 3 14:41:15 2020 +0100
xb-silo-query: Add variant of xb_silo_query_with_root() avoiding XbNodes
This is a new internal variant of `xb_silo_query_with_root()` which
returns results as `XbSiloNode`s instead of as `XbNode`s. This will be
used internally in following commits to optimise some of the functions
in `xb-node-query.c`.
This commit introduces no functional changes by itself.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit b879704492156109bbbe4a5f3916f0e8789a04b1
Author: Philip Withnall <withnall@endlessm.com>
Date: Thu Jun 4 12:31:03 2020 +0100
xb-builder-node: Don’t crash if priv->children is NULL
Since commit d12765d0, `priv->children` is loaded lazily, and can be
`NULL`. Commit b7f2509c2b3 (#50) introduced a crash in that case.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit b7f2509c2b322d9e4e4d808e746b6060cc32220b
Author: Richard Hughes <richard@hughsie.com>
Date: Tue Apr 28 22:04:24 2020 +0100
Do not show a critical warning for invalid XML
If we ignore a node due to XB_BUILDER_COMPILE_FLAG_SINGLE_LANG then also ignore
the child nodes. This ensures we do not warn when presented with something like:
<description xml:lang=de>
<ul><li xml:lang=de></li><li></li></ul>
</description>
<description>
<ul><li></li></ul>
</description>
Fixes https://github.com/hughsie/libxmlb/issues/50
commit f4694c150bc2e94472e6bef7a3356adda910b52c
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 3 09:58:53 2020 +0100
xb-builder-source: Add XB_BUILDER_SOURCE_FLAG_WATCH_DIRECTORY flag
This allows callers to specify that a source comes from a directory full
of other sources, and hence it would be better to monitor the directory
for changes rather than every file within it.
This reduces the number of file monitors used by gnome-software by an
order of magnitude as only the appstream and appdata directories need to
be monitored, rather than O(N) apps within them.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 32850eb987e93b1fb09180903be632e352c5bee6
Author: Philip Withnall <withnall@endlessm.com>
Date: Tue Jun 2 15:47:03 2020 +0100
xb-silo: Look up monitors by GFile rather than by path
This saves about 20KB of path allocations, in particular even on the
‘fast’ path where the file monitor already exists.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 93c2cd1755754cc578080a03a0d6d1d68d8f492a
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Jun 3 10:42:06 2020 +0100
xb-builder-node: Avoid using weak pointers
Weak pointers were previously used to clear the `XbBuilderNode.parent`
pointer from a node to its parent when the parent was destroyed. Each
parent has strong references to its children, however, so has all the
information available to clear their parent pointers manually.
This saves a load of locking within GObject, and 6.8MB of heap memory
used to store the weak pointers and an (otherwise empty) `GDatalist` per
GObject, when generating the libxmlb caches on gnome-software startup.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 97903e4d02fbf04d1718712cc328ad406f065abb
Author: Philip Withnall <withnall@endlessm.com>
Date: Tue Jun 2 15:09:35 2020 +0100
xb: Drop redundant parent_instance members from GObject private structs
The `parent_instance` member is only needed when creating a derived
`GObject` without a separate private data struct (for example, when
using `G_DECLARE_FINAL_TYPE`).
When using `G_DEFINE_TYPE_WITH_PRIVATE` the `MyTypePrivate` struct only
needs to contain the private members, no boilerplate.
This saves around 500KB of memory in the `XbNode`s created by the silo
when opening gnome-software.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit d12765d0cd9b965934add708c547d74d8267b0d7
Author: Philip Withnall <withnall@endlessm.com>
Date: Fri May 29 13:16:25 2020 +0100
xb-builder-node: Only allocate children array when it’s needed
Before this commit, 10.2MB of `GPtrArray`s were being allocated to store
`children` in `XbBuilderNode`. With this commit, that’s down to 1.1MB — so
9.1MB of new `GPtrArray`s were empty and never used.
This reduces peak memory consumption in gnome-software when it’s
rebuilding xmlb caches.
From some profiling, of 500080 `XbBuilderNode`s created when rebuilding
the xmlb caches on Fedora 32 (triggered using `find
~/.cache/gnome-software -name '*.xmlb' -delete`):
* 435513 (87%) have 0 children
* 22308 (4%) have 1 child
* 13315 (2%) have 2 children
* 8143 (1%) have 3 children
* 20798 (4%) have 4–1000 children
* 3 (<1%) have 1001–2237 children
* 0 have ≥2238 children
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 6250d04050e079501ca2de2dca140f26d2e6467d
Author: Philip Withnall <withnall@endlessm.com>
Date: Fri May 29 12:24:29 2020 +0100
xb-builder-node: Only allocate attrs array when it’s needed
Before this commit, 10.2MB of `GPtrArray`s were being allocated to store
`attrs` in `XbBuilderNode`. With this commit, that’s down to 1.9MB — so
8.3MB of new `GPtrArray`s were empty and never used.
This reduces peak memory consumption in gnome-software when it’s
rebuilding xmlb caches.
From some profiling, of 500080 `XbBuilderNode`s created when rebuilding
the xmlb caches on Fedora 32 (triggered using `find
~/.cache/gnome-software -name '*.xmlb' -delete`):
* 397949 (79%) have 0 attributes
* 62924 (12%) have 1 attribute
* 15523 (3%) have 2 attributes
* 23579 (4%) have 3 attributes
* 105 (<1%) have 4 attributes
* 0 have ≥5 attributes
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 5e0accc0a3b5f3c2fbbfecd3b7479df7e1d72014
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Apr 27 16:17:44 2020 +0100
trivial: Remove two more deprecated symbols
commit 52f64aa8bbf785244a7f3a2b3881e265c01eff82
Author: Philip Withnall <withnall@endlessm.com>
Date: Thu Apr 2 17:27:29 2020 +0100
Do not allocate opcodes individually
On a simple test of loading gnome-software and switching between a couple of
categories, this reduces calls to `g_slice_new0()` from around 21 million to
around 5 million.
It reduces the time to show (refine) the ‘Games’ category from around 9s to
around 5s the first time it’s shown (in a very unrigorous test).
This makes `XbOpcode` stack-only, and removes all API which requires it to be
heap-allocated. For the most part, this only requires mechanical changes in the
rest of the codebase to operate on stack-allocated opcodes, removing one layer
of indirection in some places.
The largest other change needed was to rewrite the optimiser to avoid using
`GArray`, as tracking the ownership of `XbOpcode`s through several different
`GArray`s was getting too complex. The new implementation of the optimiser
works as before, except it only uses one helper stack rather than a helper
stack and two temporary arrays. This brings its own inherent performance
benefits, as around 5 million XPath queries are done when gnome-software starts
up, so making the optimiser run quickly is important.
This ABI breaks API and ABI, and thus also bumps soname.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 22629295bd1580f07676fb6ea7e1d41975b93ea9
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Apr 27 15:42:35 2020 +0100
trivial: Disable the Windows CI build for now
commit 6a0b6b0d4004e969686b0a971ba38264322b056c
Author: Richard Hughes <richard@hughsie.com>
Date: Mon Apr 27 12:45:39 2020 +0100
trivial: Post branch version bump
commit 5831d9fd2d19052f75ef43ea6e444dbce1002191
Author: Philip Withnall <withnall@endlessm.com>
Date: Fri Apr 24 17:55:37 2020 +0100
xb-builder: Avoid recursion when setting flags if possible
If a flag has already been set, avoid recursing into all the descendent
nodes of an `XbBuilderNode` again, which results in an exponential
number of function calls. In my test of starting gnome-software up from
cold, before this patch there were 89 million calls to
`xb_builder_node_add_flag()`.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit b33e2600e60e53d59ce92f715f06e3a4b76d2905
Author: Philip Withnall <withnall@endlessm.com>
Date: Thu Apr 23 14:01:06 2020 +0100
xb-machine: Fix a typo in a debug message
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 484fd2983075ee94b4557f2b06d1366f9ac713f4
Author: Philip Withnall <withnall@endlessm.com>
Date: Wed Apr 22 14:17:32 2020 +0100
docs: Fix a typo in README.md
Signed-off-by: Philip Withnall <withnall@endlessm.com>
commit 5b82697fa2b3cb4d0b07461fc332743e69c08361
Author: Robert Ancell <robert.ancell@canonical.com>
Date: Thu Apr 2 10:41:52 2020 +1300
Report silo versions when versions mismatch
commit 45c559ca1ecf2071ba1c4ea87362f0b0b4972489
Author: Richard Hughes <richard@hughsie.com>
Date: Wed Mar 4 15:55:13 2020 +0000
trivial: post release version bump
