Git log: commit c4955a41c1dd7a43b5af706af532c9ef4485b183
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 22 14:53:23 2022 +0200
Only create the Message Queues when the section requires them. (#110)
commit 88a668bf23d81c1434a501cc382198fa74c7cd9e
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 21 16:05:59 2022 +0200
Fix docs for pgcopydb dump schema --target etc. (#109)
commit 93f80b8af2c5fc3d495b9f1bd2b288cf3636bbb4
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 21 15:54:40 2022 +0200
Improve pgcopydb follow docs, and some more. (#108)
Add documentation for the sentinel table, and add an --json option to the
command pgcopydb stream sentinel get.
Also document that pg_dumpall --roles-only needs superuser privileges.
commit 513bdca78728fe9a972744f61d24fd72c13461b3
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 21 14:01:08 2022 +0200
Add more pg restore object descriptions, from shared logs. (#107)
commit 0595dda18aa5f7c44e38f60284221baca861018b
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Tue Sep 20 19:21:24 2022 +0200
Fix/index jobs (#106)
* Add OS details about fork failures (%m).
* Review some DEBUG messages to make them NOTICE instead.
* Fix preparing copy data specs for constraints.
* Review the process model for pgcopydb.
In order to handle database schemas with a very large number of objects,
change the pgcopydb process model to use a stable number of processes during
all operations.
To be able to do that, we need a communication system between processes so
that we can start the CREATE INDEX commands for a given table only when that
table data are known to have been COPYd entirely already. For that we
introduce a support layer for System V message queues (see msgget(2) etc).
In passing, also improve the schema querying done by pgcopydb: at startup
catalog queries are used to fetch the list of all tables to handle, and all
indexex to handle. Previous to this work, pgcopydb would still run a
separate query for each table to list its indexes. Instead, pgcopydb now
links the indexes to their table in its in-memory data structures for the
source schema, in a way that we don't have to run more queries later.
Finally, prior to this work VACUUM ANALYZE jobs would be created as soon as
a table COPY would complete, creating as many sub-processes as we have
tables total. Now a static set of VACUUM ANALYZE sub-processes are created
and a message queue is used to start working on a table when it is ready.
* Update docs for the new concurrency handling of pgcopydb.
* Fix list indentation.
* Fix race condition when indexes are all built.
It is necessary that all indexes have been built already to trigger the
creation of the constraints, and also that only one process is assigned with
this task.
commit f7a27c82f7a23f1a5a9f6e2915d810724c1c83e8
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 14 18:06:12 2022 +0200
Improve docs: intro, design, concurrency. (#102)
The concurrency section needed an update to match with the current code, and
to cover the pgcopydb clone --follow process tree.
The intro section needed a re-think. Most readers care about how to use
pgcopydb and what it does, not so much (yet) about how it's implemented.
commit 28099be16ae6dcf7222bba4ad82608118e3ac83c
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Sep 12 15:57:02 2022 +0200
Improve clone --follow sub-process error handling. (#100)
Specifically, if the clone sub-process fails there is no point for the
follow process to continue looping until we're ready to apply the changes,
that won't happen. It's best to let know the user that something went wrong,
even if that means we stop pre-fetching.
commit e0bc5879a802cb906d21a60ca0fa48153d0e1ec4
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Sep 12 15:17:41 2022 +0200
Fix reltuples processing: it might be NULL. (#99)
commit 56217d87768190d0e2b13183f195e1225804b1e7
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 8 16:01:41 2022 +0200
A round of documentation updates. (#98)
* A round of documentation updates.
* Improve section about pgcopydb stream commands being for unit testing.
Some of those commands are not for testing only, not anymore. Document that,
and link to detailed examples that have been added to the main pgcopydb
clone --follow documentation.
commit 5b591c1d3a2eeb8c4f494e74cf08007866b9022f
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 7 15:19:33 2022 +0200
Implement --debug and --trace, adding to --verbose. (#97)
Some users are confused by the Unix way of using -vvv to mean “very, very
verbose” and accumulate verbosity that way. Introduce --debug as a synonym
to -vv and --trace as a synonym to -vvv.
The only drawback to also implement --debug and --trace when we could use
--verbose several times is to use yet another short option letter. It seems
to be worth it given the user feedback though.
Also introduce a new log level NOTICE between INFO (default verbosity) and
DEBUG (extra verbosity) so that we have a new verbose setting that makes
sense. The log_notice() macro isn't used much as yet, following patches and
PRs will introduce more usage for it.
commit 79d1c3c62fcb36f393d3a2a9e946840203a3e2cb
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Sep 7 15:14:35 2022 +0200
Add some other missing pg_restore list descriptions. (#96)
commit 055404ace2ee30b6114ec5b85426ac635cac8571
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Tue Sep 6 17:03:40 2022 +0200
Implement pgcopydb list progress [ --json ] (#95)
* Implement pgcopydb list progress [ --json ]
* Refrain from committing test-only code.
* Refactoring: introduce src/bin/pgcopydb/progress.c.
* Add process summary information to the progress JSON output.
This includes the pid of the process currently working on the table or
index, its start time (as epoch number and also as a formatted string), and
the command that's currently running when it's a table.
commit 41165f48721ea0c2129a3798ecee42238aa6faed
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Sep 5 14:08:47 2022 +0200
Store the schema elements to a schema.json file. (#94)
* Store the schema elements to a schema.json file.
This allows an external process to list tables, indexes, and sequences that
pgcopydb is processing internally.
* Fix memory too-early free, add metadata to schema.json.
* Document pgcopydb list schema.
* Add a log mention about the JSON file for the schema.
commit 1d5bda92d60da44f9245d815092348b13413d717
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Fri Sep 2 13:34:56 2022 +0200
Fix version number when building Docker images. (#93)
* Fix version number when building Docker images.
* Attempt to run a command in the checkout to set VERSION in environment.
* Fix for git checkout action that leaves the repo in detached head.
* Attempt to fix the re-attaching for PRs.
* Typo fix.
* Another try at using GITHUB_HEAD_REF.
* Try using git switch -c then.
* Improve `make echo-version` dependencies, avoid make include.
* Clean-up Makefile output in `make echo-version`.
* Try using actions/checkout@v3 with a ref.
* Try fetch:0 for actions/checkout@v3.
* Try fetch-depth: 0 in actions/checkout@v3 in PRs too.
* Update run-tests workflow too.
* Remove more unwanted output.
* Build a 'version' file and COPY it in the Docker context.
* Use the version file in the docker build workflow too.
* Add some verbosity to `make version`.
* Leave actions/checkout@v3 'ref' to default on non-PR builds.
Closes #77
commit 96b4b7da3d445571a1da6354fcc737fa7e32fe75
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 13:57:55 2022 +0200
Yes another attempt at fixing cosign in GitHub workflow. (#92)
commit 9a5d8aa6063dd82561ef981dbf19391aa1e6af65
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 13:46:35 2022 +0200
Another attempt at fixing the cosign bits of GitHub workflow. (#91)
commit 4bf05d3ac5255ed3090a23d60236eace82208c10
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 13:16:38 2022 +0200
Another attempt to fix cosign: ignore the digest (it's wrong). (#90)
commit 6dd7ba8d7e76de7484343e30750b6f3a7fd8a0d8
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 13:01:41 2022 +0200
Review cosign installation and use in github workflow. (#89)
commit 41ee72ea6bf592df4617bd37230e9246fbdb4149
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 12:49:56 2022 +0200
Update cosign installer from 1.4.1 to 2.5.1 in GitHub Workflow. (#88)
commit 216c230268c7f287bbb29fd1b2c142bd3e18d47b
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Sep 1 12:34:39 2022 +0200
Improve docs and help strings. (#87)
- Add coverage for XDG_DATA_HOME in the docs
- Add --split-tables-larger-than in the help string for pgcopydb clone
commit 593e660b1aefc29af4116bd5b6ec80fac0f77c1f
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Aug 31 18:09:11 2022 +0200
Fix environment variable names for table and index jobs. (#86)
A copy-paste mistake had the names wrong. Replace PGCOPYDB_TARGET_TABLE_JOBS
with the expected PGCOPYDB_TABLE_JOBS, and same with INDEX instead of TABLE.
commit 99b153567282b60e97d54831bcbdd4cb7b835b0e
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Aug 31 17:56:40 2022 +0200
Implement same-table concurrency. (#85)
* Implement same-table concurrency.
The idea is to split (or partition) larger tables into many COPY processes.
The threshold --split-tables-larger-than 100GB is also used to compute how
many processes to start for any single given table (table size / threshold).
When using same-table concurrency then we can't TRUNCATE in the same
transaction as where the COPY happens, because we now have several
concurrent connections doing the COPY. It means we can't use the COPY FREEZE
option anymore when splitting the COPY operation in concurrent processes.
The truncate operation still happens, it is implemented in a critical
section by the first concurrent COPY process that gets there.
The COPY source partitioning is computed with a SQL query that uses the SRF
generate_series, and then a COPY (SELECT ...) form is used to restrict the
source COPY to the computed range. Ranges are non-overlapping by construct
so we use (SELECT * FROM table WHERE partKey BETWEEN min AND max).
At the moment the partKey must be a single integer column (int2, int4, or
int8) with a unique constraint (either UNIQUE or PRIMARY KEY). Tables larger
than the threshold --split-tables-larger-than but without a candidate
partKey are still handled by a single process for the COPY operation.
* Fix DEFAULT_SPLIT_TABLES_LARGER_THAN value.
Using the ^ operator to mean "power" is a common pitfall of the C language,
enough so to still fall into it after all those years. That said, a good
default for the single-table concurrency is to refrain from any, so zero.
* Implement pretty-printed bytes string parsing.
This allows users to specify --split-tables-larger-than 500GB for example.
* Fix the summary display, count indexes only once.
* Remove debug information.
* Document the same-table concurrency feature.
* Improve docs.
commit 2903ac5b0d971c2307def5e4b0c406a58973bf78
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Fri Aug 12 12:58:32 2022 +0200
When pgcopydb clone --follow is used, setup before forking subprocesses. (#83)
This allows to stop processing in case we failed to do the Change Data
Capture setup, which involves creating a replication slot with the logical
decoding plugin wal2json. When that plugin is not available on the source
database, it's best to exit early with a clear error message.
Without this change pgcopydb clone --follow would implement the full base
copy fo the database and only then exit with an error because the follow
subprocess failed early.
commit ab7bc8dc787c576f7d9cfe069d9c8dfe6561d516
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Aug 10 16:39:43 2022 +0200
Improve documentation of the --follow option. (#81)
commit 5ff06e469fee62fbbeb735ea7c4ad369961bf5d5
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Aug 10 16:34:41 2022 +0200
Fix unit testing to follow latest changes in the pagila database. (#82)
The schema and data of the pagila database have been updated recently to
version 3 where they're not using 2020 anymore but rather use 2022. Adjust
to the change.
commit 023569303222b2963d0ef5978634f4c2a8daa771
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Jul 21 18:03:10 2022 +0200
Reset sequences at the end of pgcopydb clone --follow. (#76)
commit 91cc5d91710ef9b629c41bd0de3a60fe1868a9f4
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Jul 20 11:53:22 2022 +0200
Fix GIT-VERSION-GEN default version to 0.8. (#75)
This script works very well with development versions but I couldn't yet
figure out how to properly manage released (tagged) version numbers when
building in a docker container. Oops.
