Git log: commit 34fecb66761ee958b179a86ac5c1d7eff3ccc8f2
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Wed Mar 27 19:30:12 2024 +0530
Problem: Post merge docker push fails (#724)
There are changes in Sigstore's Trusted root used for signing, details
are explained here => https://blog.sigstore.dev/tuf-root-update/
Solution: Switch to cosign-release v2.2.3
Fixes https://github.com/dimitri/pgcopydb/issues/723
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 6ebbc3bc8a2dd62fcf029bc88bcf7168b8f47220
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Wed Mar 27 17:42:32 2024 +0530
Problem: interrupted clone resume fails with index relation already exists error (#699)
The clone operation employs table copy workers to copy data. Once completed, it delegates index creation to index workers. If interrupted during index creation, resuming is impossible since the underlying table has already been copied.
Solution: On resume, let the table-data skip data copy if it is done but
still signal the index workers to proceed with creating indexes. Index
worker shall ignore if the index is already exists, else creates one.
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 01872e58104c6f308c039d4b71df30799b16bc6f
Author: Georgy Shelkovy <rekgrpth@gmail.com>
Date: Tue Mar 26 16:18:32 2024 +0500
Fix building with older libpq (pg 11 and 12) (#722)
Postgres 11 and 12 only exposed PG_MAJORVERSION in libpq, we can't rely on PG_MAJORVERSION_NUM.
commit 14fed359c24b4179bc9bf95b7428c735792b753e
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Mon Mar 25 22:30:11 2024 +0530
Problem: CDC replication/apply is slow (#704)
* Problem: CDC replication is slow
Our existing implementation is basic and it will execute a statement and waits for its result to come before issuing next statement. We already improved DML statements targetting same table with in a transaction by coalscing them into multi-value insert statement, but it will be beneficial only for logical decoding transaction which has multiple DML statements with in a txn.
However, we have encountered few cases where the transaction had only one DML statement. This kind of txn will be slower because statements are executed sequentially from the client's perspective.
Here is an example of how a single statement transaction will be executed now,
1. Execute ["BEGIN"]()
2. Execute [bunch of "SET" statements]() which are related to replication session setup
3. Prepare [DML statement]()
4. Execute [prepared statement with values]()
5. Execute procedure to update [replication origin progress]()
6. Execute [COMMIT]()
7. Map [target current insert lsn to commit lsn]() for feedback reporting (i.e. sentinel replay_lsn)
**Solution**: Use [pipeline
API](https://www.postgresql.org/docs/current/libpq-pipeline-mode.html) from libpq client library
The proposed implementation would enter into pipeline mode as soon as a new libpq connection is created for the ld_apply/ld_replay process. All the statements would be executed on a pipeline by default except few for statements which needs response immediately(e.g. step 7). A dedicated connection would be used to serve (step 7), because it needs synchronous response.
The following functions are being called on the target PGSQL handle from ld_apply & ld_replay.
* pgsql_begin
* pgsql_set_gucs
* pgsql_execute
* pgsql_replication_origin_xact_setup
* pgsql_prepare
* pgsql_execute_prepared
* pgsql_current_wal_insert_lsn
* pgsql_current_wal_flush_lsn
Among all of the above function, only pgsql_current_wal_insert_lsn and pgsql_current_wal_flush_lsn returns values and other functions are write only.
The idea is to have 2 PGSQL connection handles, 1 for all write activity which can go through pipeline and another one could be used for reading.
Pipeline connection has to be synced/drained at some point to avoid accumulating results on the server & client which would end up eating lots of heap memory. The current implementation syncs based on the time interval(i.e. for every 1s). There are other methods like statement/txn count based sync, which may or may not be efficient.
The following command can be used to generate loads to understand the performance improvement made by this commit,
```
CREATE TABLE metrics (
time TIMESTAMP NOT NULL,
name TEXT,
id NUMERIC,
value FLOAT
);
```
```
-- insert_metric.sql
\set id random(1, 1000000)
\set value random(0,100)
INSERT INTO metrics (time, name, value) VALUES (NOW(), 'metric_' || :id, :value);
```
```
pgbench -n -c 40 -j 1 -t 10000 -f insert_metric.sql $SOURCE
```
- `-c` number of database connection to utlize (i.e. server side concurrency)
- `-j` number of threads to create on the client machine (i.e. client side concurrency)
synchronous_commit=off)
```
pgbench -n -c 1 -j 1 -t 10000 -f insert_metric.sql $TARGET
```
```
tps = 1177 (without initial connection time)
```
```
tps = 175
```
```
tps = 1652
```
**This commit improves the single statement txn throughput by 10x**
Ideally, we should aim to get performance number close to direct ingestion(i.e. 1200 txn/s). We are 40% performing better than the baseline in this iteration. However, we can aim more as in the real system there will be more than 1 connection will be utilized. We can't really race against multiple connection doing steady ingestion around 1000 txn/s per connection, but lets optimize the single connection throughput to the max!
This change will be a foundation to the future improvements which steers towards that.
1. Optimize step 2 - Instead of executing bunch of SET statements for every txn, run once in the beginning for the session
2. Optimize step 9 - Probably we can simply use pg_replication_origin_progress as replay_lsn?
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 4da852d7af86fd92182f4172229128516ad17f36
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Fri Mar 22 16:25:43 2024 +0100
Force sequences reset at the end of the follow process. (#716)
* Force sequences reset at the end of the follow process.
* Re-fetch current sequences values at source before resetting them.
commit b04ef4ddaf9425dd9c581362bc17ea40ce2b58fb
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Thu Mar 21 18:27:23 2024 +0530
Fix "database is locked" due to concurrent writes between follow and clone. (#712)
Also, for test_decoding plugin we would require the catalog information so it's better to let it initialise first and then we start the prefetch.
commit 0ce70cbcdd8f9ce65f5504e9d76d7eae3711c77d
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Thu Mar 21 17:11:16 2024 +0530
Problem: Resume fails after interrupted replication connection (#707)
When the replication connection is interrupted due to network or server issues, although there is a retry logic in place, it fails to work effectively. This is because the interruption may leave the last transaction in a partial state, leading to a failure when the ld_transform module attempts to transform the message.
Solution: Add a ROLLBACK message when the last transaction is partially written and the stream is in retry mode.
Use the following query on psql to test the network interruption behaviour while doing DML on source.
```.sql
SELECT pg_cancel_backend(pid) FROM pg_stat_activity where application_name like '%pgcopydb%' and query ilike '%START_REPL%';
\watch 5
```
The following error message will be loged on console when it is interrupted,
> 11:40:54.061 94512 ERROR unexpected termination of replication stream:
11:40:54.061 94512 ERROR ERROR: canceling statement due to user request
11:40:54.061 94512 ERROR CONTEXT: slot "pgcopydb", output plugin "test_decoding", in the change callback, associated LSN 7B/704E6A18
11:40:54.061 94512 WARN Streaming got interrupted at 7B/704E6A18, reconnecting in 1s
Without this fix, ld_transform would fail with the following console error logs,
>2024-02-28 18:44:44.785 37 ERROR ld_transform.c:1099 Failed to parse BEGIN: transaction already in progress
2024-02-28 18:44:44.785 37 ERROR ld_transform.c:924 Failed to parse JSON message: {"action":"B","xid":"3560764890","lsn":"DD68/8ADC210","timestamp":"2024-02-28 17:59:31.756879+0000","message":{"action":"B","xid":3560764890}}
2024-02-28 18:44:44.789 37 ERROR ld_transform.c:686 Stream transform worker encountered 1 errors, see above for details
2024-02-28 18:44:44.789 37 INFO follow.c:758 Transform process has terminated
2024-02-28 18:44:44.855 19 ERROR follow.c:993 Subprocess transform with pid 37 has exited with error code 12
2024-02-28 18:44:45.016 19 ERROR follow.c:471 Failed to transform 2 messages from the queue, see above for details
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>
commit f226f07dcb31abdf20117ffde06b586d67db0b62
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Thu Mar 21 17:00:17 2024 +0530
Fix list progress when there are partitions. (#713)
commit 3d5301889097e90704f06cfa68265ff99ba020e4
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Thu Mar 21 16:54:35 2024 +0530
Fix list progress due to missing delete in catalog `process` table. (#714)
This fixes the case where copydb_table_data_worker receives a
`QMSG_TYPE_STOP` and we return within the switch statement without calling
catalog_delete_process outside the swtich body.
commit e37d14be014b25b06113cf58b295d27c73a5bbe6
Author: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Date: Tue Mar 19 19:44:18 2024 +0300
Fix issues with CFLAGS (#709)
There were several issues with the CFLAGS in the pgcopydb Makefile. This patch fixes them.
1. Removal of conditional logic for USE_SECURITY_FLAGS
Historically, we needed to support some linux distros such as RHEL6 that
did not support the subset of security flags that are required for
Microsoft Compliance. However, we are now at a point where we can remove
this conditional logic and always use those security flags. This patch
removes the conditional logic and always uses the security flags.
2. Fix typo in JENKINS_CFLAGS
The JENKINS_CFLAGS variable was misspelled as CFAGS. This patch fixes
the typo.
3. Remove unnecessary override of CFLAGS
The override of CFLAGS was unnecessary and was causing the security
flags to be overridden. This patch removes the override. If we do not
have -Wformat option, -Wformat-security which is a required compiler
flag for compliance is ignored.
Relevant error message:
cc1: error: -Wformat-security ignored without -Wformat [-Werror=format-security]
commit 386485f7c5c49c00772f54884f60aa40bffc2e1c
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Mon Mar 18 23:18:26 2024 +0530
Fix SQL syntax error in group_concat. (#708)
commit 227db6fdb119bc43fa1d70d34031ec6239f0d3f9
Author: James Guthrie <JamesGuthrie@users.noreply.github.com>
Date: Tue Mar 12 23:07:05 2024 +0100
Respect FK constraints for extension config tables (#702)
Extension configuration tables can have foreign key dependencies between
them. If the configuration tables are restored in the incorrect order,
it can cause foreign key constraint violations.
This change ensures that the configuration tables are ordered according
to the dependency order in `pg_constraint`.
commit ff6b47389fd68447ad9e2039a418b97fce9e3186
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Mon Mar 11 18:38:33 2024 +0530
Problem: Resume after interrupted clone always fails (#698)
We always call `summary_start_timing` regardless of whether the section has been already done or not., which causing insertion(insert or replace) of default value to `done_time_epoch` because `summary_start_timing` won't be called for already done section.
Solution: Check whether the section is already completed using `runState` and exit early without calling `summary_start_timing` for already done section.
Related to https://github.com/dimitri/pgcopydb/issues/692
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 67df98b20ce62f8186c1a209cad20e6ca73a70d5
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Tue Mar 5 03:53:44 2024 +0530
Problem: Still old variant of strdup has been used (#691)
pgcopydb recently switched to Boehm-Demers-Weiser Garbage Collector[1] and replaced
malloc family with GC_malloc and friends. However, strdup is not replaced
to GC_strdup.
Solution: Use macros to replace strdup/strndup with it's GC variant.
[1] https://github.com/dimitri/pgcopydb/pull/647
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 7707207caf2c8d050b95de5f4da5b323fc9b7cd6
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Mar 4 15:04:03 2024 +0100
Review table attributes listing and usage of attnum. (#690)
Given ALTER TABLE ... DROP COLUMN ... we could well have a table without an
attribute with attnum == 1. Our code used to rely on that to happen in order
to allocate our SourceTableAttribute array in-memory, before filling it in.
Instead, use a window-function in the SQL query that we run against our
internal SQLite catalogs, using count(*) over(order by attnum) to map
whatever the Postgres attnum are into a proper gapless series that starts at
one.
commit cb55ea14ac669da43e9943f1a015ef985f9412f1
Author: VaibhaveS <56480355+VaibhaveS@users.noreply.github.com>
Date: Mon Mar 4 18:44:21 2024 +0530
Add skip tablespaces in restore options. (#683)
commit fdf3d906fecde76495277075543e1bac73047096
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Mar 4 13:56:10 2024 +0100
Cleanup schema_drop_pgcopydb_table_size. (#689)
This function does not exists anymore, but the header was still defined and
the call site was not removed yet.
commit 40af8d75252a377037bc5eb5fd2a735578a8d18b
Author: Gokhan Gulbiz <ggulbiz@gmail.com>
Date: Fri Mar 1 15:54:18 2024 +0300
Add ARM based OSX support to Makefile (#685)
commit 242fa27363a338b5eea76d57095a6320ca811823
Author: Gokhan Gulbiz <ggulbiz@gmail.com>
Date: Fri Mar 1 15:37:41 2024 +0300
Move table size queries into pgcopydb catalogs (#684)
Instead of storing the table size in the source database, store it in our internal catalogs in the SQLite database. This allows better compatibility with read-only source databases.
At the moment this removes the possibility for the user to provide their own table size data in order to avoid the `pg_table_size()` query which is known to be a long running query in some platforms. A later improvement could be done to enable external table size input again.
commit f9f382d0e022911d94bdf29ae384c3f9c623c2d9
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Feb 29 17:49:09 2024 +0100
Fix duplicate hazards when listing extension namespaces dependencies. (#687)
commit 374f11cb9a8477a3ec88e7993d84a0d570f6ea74
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Feb 29 17:36:20 2024 +0100
Fix documentation typo for hyperlink. (#686)
commit 0487b8895e92830418590cd6260414ce6f8d0a7c
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Mon Feb 26 20:13:28 2024 +0530
Problem: xid is missing in test_decoding commit for a continued txn (#682)
Missing xid triggers the following error during apply/replay.
"BUG: parseTxnMetadataFile is called with transaction xid: 0"
Solution: Use xid from commit message as a transaction xid.
Unlike wal2json, test_decoding don't have xid in the DML logical
messages. So we use the xid from the COMMIT message to update the
transaction xid.
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit dd84827a02622f6c33b2c4a96210a7326a32a2e1
Author: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Date: Mon Feb 26 20:06:57 2024 +0530
Problem: Snapshot id printed on stdout doesn't appear on console (#681)
Solution: fflush stdout after printing the snapshot id. By default stdout is
buffered and flushed only when the buffer is full.
Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
commit 298d9a4b0e19e2b1e9d8a32320094286a6899465
Author: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Date: Wed Feb 21 18:15:10 2024 +0300
Fix ordering of CFLAGS for sqlite (#680)
commit a7889dfa93385c2fcd82de4d9b3c96e48810524a
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Tue Feb 6 12:00:21 2024 +0100
Compute COPY attribute list using SQL. (#661)
Before embedding SQLite we could not depend on aggregates FILTER clause to
be available, and as a result we had to aggregate the attribute list in our
own C code. Turns out we had strange bugs around that, that hopefully the
very simple SQL query will not have.
commit 4b1863024980c31a3e95aad3400fe4047547f7c9
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Feb 5 15:38:54 2024 +0100
Fix `make deb` target, improve build-from-sources docs. (#670)
commit 1bd4f1e4d3aa3157956c624bdbd83851437776af
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Feb 5 15:20:38 2024 +0100
Improve documentation (resume, snapshots, tutorials). (#669)
commit 32fb16ea52ad01c049005fa988203f84ba158d43
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Fri Feb 2 15:55:07 2024 +0530
Escape column names while transforming for wal2json plugin. (#663)
Follow up of #595
commit bda0cb9ecb0d8cd7a61a10407733ee5444fbb0c6
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Jan 31 15:39:26 2024 +0100
Vendor SQLite amalgamation distribution. (#660)
This allows to ensure we're always using a known version of the SQLite lib
and allows pgcopydb code to use some "recent" features on all OS targets.
Such features include the RETURNING clause for DMLs, the FILTER clause for
aggregate function calls, and time formatting function or JSON functions.
See https://sqlite.org/amalgamation.html for details about that distribution
mode for SQLite.
commit 2ce0ae2233ebe3864914da9f743afcc96b9e62a7
Author: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Date: Wed Jan 31 13:26:00 2024 +0300
Overhaul of EditorConfig rules (#652)
Before this change we had 153 files on the main branch that did broke at
least one of our editorconfig rules. After this change, all of our files
honor the rules that are partially updated.
There are 2 places that are exempt from our rules:
- external libraries in src/bin/lib/
- debian directory in debian/
- test file from outside source in tests/extensions/
You can see the .editorconfig file in each of these directories for more
details.
commit 6b6db5d50f0514c4b64ffcc5af1ad79109cfe87d
Author: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Date: Tue Jan 30 15:00:13 2024 +0300
Introduce CONTRIBUTING.md (#653)
I used a copy of Citus' CONTRIBUTING.md as a starting point.
I removed sections that are not relevant to pgcopydb, and added some
more sections about documentation and running tests.
I also added references to the CONTRIBUTING.md file in the README.md and
in documentation page about installing pgcopydb. While I was at it, I
also removed the outdated installation instructions from the README.md
file, and replaced them with a reference to the documentation page.
Co-authored-by: Dimitri Fontaine <dim@tapoueh.org>
commit a3cc2eb8874debd032b5a4eb27a259258ac7af36
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Tue Jan 30 12:42:31 2024 +0100
Fix confusion around reporting operations in progress. (#658)
* Fix confusion around reporting operations in progress.
The SQLite refactor meant rewriting all the SourceTableArray loops to use a
query result iterator instead. Well there is a place for a SourceTableArray
loop still, and that's the pgcopydb list progress thing.
* Fix the table-in-copy SQLite query.
The query was meant to report actual duration and transmitted bytes for each
table in-progress, but the join and sum() operations are creating problems
in the SQLite query and the rest of the progress.c code was not prepared to
benefit from the extra information anyway.
The SQLite problem is that the sum(s.duration), sum(s.bytes) causes a single
row output when the dataset is empty, and the other columns are all set to
NULL. That might be because of the lack of a GROUP BY clause.
Simplify that aspect by removing the extra information for now.
commit 2b4c98d1b65eece04635d47396e9f16bbf410a57
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Fri Jan 26 14:43:42 2024 +0100
Fix pgcopydb list progress. (#654)
Since the move of the summary and process information to the SQLite
catalogs, we don't have a progress JSON file on-disk with parts of the setup
anymore.
The code needed to be updated to behave correctly without having
--table-jobs and --index-jobs information.
commit 29a42685f678f9ffa87eadb28a3f996f3e3a6ed1
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Thu Jan 25 18:13:09 2024 +0100
Use Boehm-Demers-Weiser Garbage Collector. (#647)
* Use Boehm-Demers-Weiser Garbage Collector.
From their own documentation:
> The Boehm-Demers-Weiser conservative garbage collector can be used as a
> garbage collecting replacement for C malloc or C++ new. It allows you to
> allocate memory basically as you normally would, without explicitly
> deallocating memory that is no longer useful. The collector automatically
> recycles memory when it determines that it can no longer be otherwise
> accessed.
In pgcopydb C code base it's been hard to maintain proper malloc/free
concerns while also implementing proper error-handling, which is a common
problem when writing C code. In order to ease the maintenance of the code
and reduce production hazards, the best way to proceed looks like automating
the memory management altogether by using a Garbage Collector.
* Revert removal of regree() calls.
* Get rid of LinesBuffer.ownsBuffer.
Now that libgc is handling the memory for us, we don't need to keep track of
the ownership of the buffer anymore.
commit 69c1340f051d4e07a1f075767d0f45525a174805
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Thu Jan 25 16:38:25 2024 +0530
[sqlite] Set PRAGMA only once for catalogs. (#651)
I observed frequent `[SQLite] PRAGMA journal_mode = WAL` when it should be
run only once as it's a sqlite database property, not something that has be
set for every connection. This might fix
https://github.com/dimitri/pgcopydb/actions/runs/7614380466/job/20736769276?pr=647
```
2024-01-22 16:45:01.923 44 ERROR catalog.c:7129 [SQLite]: PRAGMA journal_mode = WAL failed: database is locked
```
commit 88a34e8ff46ca86b86e684af409a6b8e2a6c421c
Author: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
Date: Wed Jan 24 16:54:39 2024 +0300
Improve documentation templates (#618)
- Adds CI to check that the documentation templates are up-to-date.
- Add a new `help` command to the documentation templates. This command
prints the help message for all the sub-commands of `pgcopydb` in a
single file. This is useful for users who want to see all the
sub-commands in one place.
- Remove dependency on installing the binary before updating the
documentation templates. This allows users to update the documentation
templates without installing the binary. One can set the `PGCOPYDB`
environment variable to point to the binary to use for updating the
documentation templates.
- Fix a casing issue in command descriptions. The first letter of the
first word in the description of each command should be capitalized.
commit 6caad8e4a7975adc683b3ac8af45ba12fac902b1
Author: VaibhaveS <56480355+VaibhaveS@users.noreply.github.com>
Date: Tue Jan 23 18:32:02 2024 +0530
Fix database name identifier in ALTER DATABASE properties using PQescapeIdentifier. (#645)
commit 773b3c4bad5d2d369a68a05cd1f95fbe83a0c6e0
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Jan 22 15:33:48 2024 +0100
Fix cache invalidation of zero --split-tables-larger-than setup. (#648)
When registering the catalog setup, which includes the same-table
concurrency threshold, only consider that value when the table partitions
have been populated in the cache already. Otherwise the table partitions are
going to be computed again anyway.
commit 81fb66cfbaa3fee0332f6d14af83e0570eda325b
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Mon Jan 22 17:39:10 2024 +0530
Improve sqlite catalog setup performance. (#646)
* Improve SQLite write performance with journal_mode as WAL.
* Batch initial catalog setup write in a single transaction.
commit 7633458c1af05886e5c5a99c03b9668d9de1595e
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Jan 17 16:45:06 2024 +0100
Fix eager extension/collation skipping. (#642)
The SQLite patch series introduced a bug where we would skip extensions even
when the option --skip-extension is not used.
Also adjust the "pgcopydb copy extensions" to the recent changes where we
use a sub-process under a supervisor to simplify snapshot handling.
commit 49725e5d70fb96e45192a102dd8d8a4ebc9b511b
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Wed Jan 17 12:30:54 2024 +0100
Review zero-byte allocation hasards. (#641)
Make sure we don't try and allocate a zero-size memory area, which can lead
to "corrupted size vs. prev_size". Review all the calloc() call sites and
protect them better when they need to be.
commit b207fd41cc36c91d9b2113f4265d0b967b84dca7
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Tue Jan 16 17:17:46 2024 +0100
Review the splitLines() API to avoid allocating zero bytes. (#638)
Make it systematic to first count the lines and then allocate the right
amount of memory for the array of pointers to the start of each line, and
make sure we don't try and allocate a zero-size array, which can lead to
"corrupted size vs. prev_size" errors at run-time.
commit 4013f969efc2f6cf0e7ec82f6f725b6362169992
Author: Shubham Dhama <shubhamdhamaofficial@gmail.com>
Date: Tue Jan 16 18:55:07 2024 +0530
Skip building debugParameters when not necessary. (#637)
commit 63b148adf025907a36b787552048d991ace320e4
Author: Dimitri Fontaine <dim@tapoueh.org>
Date: Mon Jan 15 14:56:11 2024 +0100
Blind attempt to fix a segfault in copy extensions code. (#636)
commit fe250034d224d2cc5736092c8108a77b4dfd8d40
Author: Marc Kline <marc.kline@turner.com>
Date: Mon Jan 15 07:02:52 2024 -0500
use int64 for lastValue (#633)