Git log: commit f5dfde5caff5601aa60e5f73767e01d64e093af1
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Thu May 28 09:40:20 2020 +0000
Add new error injection constant ERRINJ_AUTO_UPGRADE
In upgrade testing we need an ability to control running of upgrade.lua
script execution. When constant ERRINJ_AUTO_UPGRADE is set to true
tarantool will skip upgrade.lua script execution.
Part of #4801
commit e215737a90e11a52e83402d15b8c2b2287877323
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun May 31 07:23:04 2020 +0300
Correct error message at pwd.lua
During investigation of the issues found that error message had
not user friendly format:
local pwgr_errstr = "get%s failed [errno %d]: %s"
produced:
[001] LuajitError: builtin/pwd.lua:101: getgetgrall failed [errno 2]: No such file or directory
while it had to be:
[001] LuajitError: builtin/pwd.lua:101: getgrall failed [errno 2]: No such file or directory
Fixed the error message.
commit b92549ad24f356a5a9a5db82964cdb148d17455d
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun May 31 07:29:53 2020 +0300
test: remove skip condition of app-tap/pwd test
During creating patch:
e3d9d8c97edd39438cbc98fccac996e61ce4984c ("build: add CentOS 8 into CI / CD")
was found the issue:
[001] app-tap/pwd.test.lua [ fail ]
[001] Test failed! Output from reject file app-tap/pwd.reject:
[001] TAP version 13
[001] 1..6
[001] ok - checking user by id
[001] ok - checking user by name
[001] ok - checking group by id
[001] ok - checking group by name
[001]
[001] Last 15 lines of Tarantool Log file [Instance "app_server"][/mnt/test/var/001_app-tap/pwd.test.lua.tarantool.log]:
[001] tarantool: /lib64/libcurl.so.4: no version information available (required by tarantool)
[001] builtin/pwd.lua:169: getpwall failed [errno 2]: No such file or directory
Currently it was tried to reproduce the issue, but all the attempts
didn't got the needed fail. After discussion of the issue decided to
close the current one as not the issue for the test and continue the
fixes for getpwall/getgrall functions within issue #5034.
Closes #4592
commit c85f5dbfb31a92477d4bd2579a80e4bf0cbe28b9
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Fri May 22 01:05:27 2020 +0300
lua/log: do not allow to set json for boot logger
For some reason we've missed that say_set_log_format
doesn't support json format not only for syslog but
for boottime logging as well.
Part-of #689
Reviewed-by: Oleg Babin <olegrok@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 94ef2cf48be1106fe210ac82ba3c635f0a746cd4
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Fri May 22 00:57:20 2020 +0300
lua/log: put string constants to map
This allows us to reuse them instead of copying
opencoded constants. They are highly bound to
ones compiled into the C code.
Part-of #689
Reviewed-by: Oleg Babin <olegrok@tarantool.org>
Reviewed-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit b9e68ab5a69fcf692a54654dd3123d259c50b479
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed May 20 20:08:51 2020 +0300
lua/log: declare say_logger_init and say_logger_initialized
We gonna use it to provide a way to initialize
logger subsystem early.
Part-of #689
Reviewed-by: Oleg Babin <olegrok@tarantool.org>
Reviewed-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 3be17311b4d5417aec7f49028623cfb3d7dcdb9e
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Thu May 28 13:52:18 2020 +0300
core/say: use say_logger_initialized in say_logger_free
To eliminate opencoded comparision.
Reviewed-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Reviewed-by: Oleg Babin <olegrok@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 2df4e1539e5f258280fd558b8ae1383fd0be2433
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed May 20 14:16:32 2020 +0300
core/say: do not reconfig early set up logger
We gonna support logger configuration on its own
without requirement to call `box.cfg{}`. Thus lets
say_logger_init() to skip processing if we already
did.
Note it is preparatory for next patches. Currently
the init called once.
Part-of #689
Reviewed-by: Oleg Babin <olegrok@tarantool.org>
Reviewed-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit a70675e0e869e3573ad557e1420cbc592586169d
Author: Leonid Vasiliev <lvasiliev@tarantool.org>
Date: Tue May 19 22:20:06 2020 +0300
socket: fix error while closing socket.tcp_server
The tcp server starts in a separate fiber.
When server's socket is closed from another fiber,
an exception will be thrown in server's loop from
check_socket function.
A "socket is close" check has been added at server's
fiber and now server's fiber terminates correctly.
Fixes: #4087
commit c84a8533db85282f1d52a0c3c8ca3f1e31b70b47
Author: Leonid Vasiliev <lvasiliev@tarantool.org>
Date: Thu May 28 10:48:35 2020 +0300
error: add format string usage to compose a CustomError message
For the CustomError the ability to create a message
using a format string was added.
Closes #4903
@TarantoolBot document
Title: Add format string usage to compose a CustomError message
When an error created using the separate members mode
(box.error(code, errtext[, errtext ...])) in the case of
ClientError error creation, a pre-defined format is used
(corresponding to the error code) (nothing has changed), in
the case of CustomError error creation, a format string can
be used to compose a message.
ClientError(nothing has changed):
```Lua
box.error(code, reason args)
```
Example for ER_CREATE_SPACE:
```Lua
box.error(9, "my_space", "reason")
```
Result:
```Lua
error: 'Failed to create space ''my_space'': reason'
```
CustomError:
```Lua
box.error(type, reason format string, reason args)
```
Example:
```Lua
box.error("MyCustomType", "The error reason: %s", "some error reason")
```
Result:
```Lua
error: 'The error reason: some error reason'
```
commit 366b2de7cb244fcafeb6d3bc57ed444169b2c433
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Mon May 18 19:12:29 2020 +0300
Allow to set directory for copying DSO before load
Make it possible to set temporary directory where
module will be copied before load.
@TarantoolBot document
Title: Justify module (re-)loading semantics
It is now possible to set directory where temporary
copies of modules to be loaded will be created.
It is done by setting $(TMPDIR) variable. It
will be "/tmp" if variable was not set.
Follow up #4945
commit 4dd421fbd2fcbb2c27a6f6f00d90e1ce63341b97
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Mon May 18 18:56:31 2020 +0300
Copy DSO module before load instead of symlink-ing
Tarantool module reload mechanism allows to reload a module even
if there're functions running. It is implemented by refcounting
each invocation of module's routines.
If reload is called, then refcounter is checked:
- If it is 0, then no routines are in flight and module is
reloaded by simple pair of dlclose()/dlopen().
- If it is non-zero, then there're routines in flight. To allow
to load multiple versions of modules it is loaded not from the
DSO specified. Symlink to tempdir is created and dlopen() is
invoked against it (w/RTLD_LOCAL flag to avoid conflicts).
This trick was implemented in order to fool a dynamic linker: one
cannot invoke dlopen() against same file, so let's pretend
there're to independent DSOs.
The problem is that dynamic linker is smart enough. It tracks not
filenames, but i-nodes. Moreover it is smart enough to do stat -L
against DSO to follow symlinks! [1][2] So, any attempts to create
a symlinks to fool dynamic linker fail and instead of doing actual
load it just increments internal refcounter in map w/
corresponding inode, which in turn leads to not-reloading.
This wasn't caught by test since old module was always unlinked
before new one is copied in place.
The patch always copies DSO instead of creating a symlink. Also
it fixes the test so in SEGFAULTs without the change.
Closes #4945
[1] - https://sourceware.org/git/?p=glibc.git;a=blob;f=elf/dl-load\
.c;h=a5318f9c8d1d42745a254479cf6bb1cd2acd516f;hb=58557c229319a3b8d\
2eefdb62e7df95089eabe37#l898
[2] - https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/pos\
ix/dl-fileid.h;hb=58557c229319a3b8d2eefdb62e7df95089eabe37#l33
commit 943b507a3a22c300d3fd0dd668a9408ebcfcfd32
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri May 29 14:51:06 2020 +0300
travis-ci: disable OSX testing
OSX testing completely implemented on local MAC hosts with OSX 14
and 15 versions. So no more need to test and support issues with
testing OSX with 14 version on travis-ci.
Part of #4893
commit e8c72d4fe66ea94e357af2e527cb5cc4727f09da
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 25 23:46:01 2020 +0200
test: stop linking msgpuck lib with test modules
Test modules are shared libraries. They link with Tarantool at
runtime and resolve missing symbols then. However in case they
already contain some of the same symbols used by tarantool, that
leads to a conflict in ASAN build. Nothing serious, there are
strict rules how a linker should behave in this case, but there
is still a warning. This is how it looked (the text is shortened):
==25624==ERROR: AddressSanitizer: odr-violation (0x000001123b60):
[1] size=1024 'mp_type_hint' /builds/nZUxDh2c/0/tarantool/tarantool/src/lib/msgpuck/hints.c:39:20
[2] size=1024 'mp_type_hint' /builds/nZUxDh2c/0/tarantool/tarantool/src/lib/msgpuck/hints.c:39:20
These globals were registered at these points:
[1]:
#0 0x478b8e in __asan_register_globals (/builds/nZUxDh2c/0/tarantool/tarantool/src/tarantool+0x478b8e)
#1 0x7ff7a9bc9d0b in asan.module_ctor (/tmp/tntz2FLhA/function1.so+0x6d0b)
[2]:
#0 0x478b8e in __asan_register_globals (/builds/nZUxDh2c/0/tarantool/tarantool/src/tarantool+0x478b8e)
#1 0xab990b in asan.module_ctor (/builds/nZUxDh2c/0/tarantool/tarantool/src/tarantool+0xab990b)
The symbol mp_type_hint is defined both in tarantool executable,
and in function1.so dynamic library. This is because both build
with msgpuck static library.
However the modules don't need to be built with msgpuck. They can
link with it at runtime from tarantool executable.
This patch removes msgpuck from test dynamic modules, and exports
all msgpuck non-inlined symbols explicitly, so they couldn't be
removed by the linker in future.
Follow-up #2971
Follow-up #5001
Closes #5023
commit f262b64fdd98163c515d421a4008851b1598fca7
Author: Oleg Piskunov <o.piskunov@tarantool.org>
Date: Fri May 15 20:37:41 2020 +0300
gitlab-ci: parallelize perf testing
- Adding 2 servers (sh1, sh2) for performance testing.
- Rebalance performance testing between servers:
- sh1 (tag: docker_sh1_perf) used for benches:
nosqlbench_hash, nosqlbench_tree
- sh2 (tag: docker_sh2_perf) used for benches:
ycsb_hash, ycsb_tree, cbench
- sh3 (tag: docker_sh3_perf) used for benches:
sysbench, tpcc
- sh9 (tag: docker_perf_ssd) used for benches:
linkbench_ssd
- Changing gitlab-ci tag for performance docker images
from 'perf' to 'deploy'.
Closes #4868
commit 47dbbf5f90b69b4f84127b6819205c0d23704679
Author: Oleg Piskunov <o.piskunov@tarantool.org>
Date: Thu May 21 14:05:50 2020 +0300
gitlab-ci: cleanup temporary perf images
Each temporary docker perf image is about 5 GB
and we don’t have space to keep them localy on servers.
- add cleanup after perf docker
image preparation (perf_bootstrap job).
- cleanup perf docker images on execution hosts
after perf testing (cleanup stage).
Closes #5003
commit 513e13ebf3a14d4dea9772cf4cf5409192af1370
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Thu May 21 10:34:17 2020 +0300
test: set unix sockets for iproto at core = app
Set hard-coded unix sockets for iproto connections at core = app.
Added its option in *-tap/suites.ini files and fixed test for it.
Fix helped to handle the problem with 'Address already in use' error.
Check the previous commit that set the use of sockets:
60f84cbfca24e3a91cea067c923e006b44ee589f ('test: use unix sockets for iproto connections')
Test sql-tap/gh-4077-iproto-execute-no-bind.test.lua removed from
fragiled list.
Closes #4459
commit d1647590ec4263592d6408cd4c16c2c2d55a7847
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu May 28 01:19:36 2020 +0200
fk: fix wrong sizeof() in fk_constraint_def_sizeof()
The function returns a number of bytes needed to store
an fk_constraint_def object with its name and links.
However it used sizeof(struct fk_constraint) instead
of sizeof(struct fk_constraint_def) to calculate
base object size. This worked only because
fk_constraint is bigger than fk_constraint_def.
commit df4c69ec15bfa86fcb7826a2359356845ab0c64a
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Tue May 19 15:54:19 2020 +0300
core/say: drop redundant declarations
Part-of #689
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 6e567facd740484f54996f976d8372cf3632afc9
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed May 20 15:38:45 2020 +0300
core/say: drop vsay declaration
The implementation has been removed in
commit ef2c171d3e8f1d675c1cab548838a03ae67a2a66
Part-of #689
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 74ac4263e15068c7e6e256224e1153e04e8f730f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat May 23 18:42:09 2020 +0200
build: link bit library to vclock library
Vclock depends on bit library, but wasn't linked to it.
commit 7305fe5a38cf659b601b7e96b1a75000a22ff3e4
Author: Nikita Pettik <korablev@tarantool.org>
Date: Tue Apr 28 03:06:14 2020 +0300
vinyl: unthrottle scheduler on manual checkpoint
Before this patch box.snapshot() bails out immediately if it sees
that the scheduler is throttled due to errors. For instance:
box.error.injection.set('ERRINJ_VY_RUN_WRITE', true)
snapshot() -- fails due to ERRINJ_VY_RUN_WRITE
box.error.injection.set('ERRINJ_VY_RUN_WRITE', false)
snapshot() -- still fails despite the fact that injected error is unset
As a result, one has to wait up to a minute to make a snapshot. The
reason why throttling was introduced was to avoid flooding the log
in case of repeating disk errors.
What is more, to deal with schedule throttling in tests, we had to
introduce a new error injection (ERRINJ_VY_SCHED_TIMEOUT). It reduces
time duration during which the scheduler remains throttled, which is
ugly and race prone. So, let's unthrottle scheduler when checkpoint
process is launched via manual box.snapshot() invocation.
Closes #3519
commit 00c582f0db032bf7a980ac917047de3186a6f86a
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon Apr 27 21:33:51 2020 +0300
engine: add is_scheduled arg to engine->begin_checkpoint
In some cases it may turn out to be useful to know whether checkpoint
process was launched manually (explicitly calling box.snapshot()) or
automatically via checkpoint daemon. In particular, to unthrottle vinyl
scheduler when it comes for manual checkpoints. So let's extend engine's
vtab method begin_checkpoint() with corresponding argument.
Needed for #3519
commit f6cb92898a38a7b86d93a449dbcfe8dcd0e8d169
Author: Nikita Pettik <korablev@tarantool.org>
Date: Thu May 21 03:43:40 2020 +0300
vinyl: fix assert in vy_tx_write()
Assert in vy_tx_write() validates that upsert applied on the top of
cached statement is always replace. In fact, it is not always so. If
vy_apply_upsert() fails due to the fact that PK is modified, it returns
old statement (i.e. statement which upsert is applied on). In this
regard, if tuple cache contains insert and invalid upsert is executed,
vy_apply_upsert() will return insert. As a result, assert will fire.
Let's fix it and take into account that vy_apply_upsert() is able to
return inserts as well.
Closes #5005
commit 2cf4450f245778ff2b959c5ccd3009837320f905
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu May 21 16:46:18 2020 +0200
vinyl: add missing diag_set in space creation
vinyl_engine_create_space() didn't set an error in the
diagnostics area when region_alloc() failed. This
could lead to a crash, although was almost impossible
to happen.
commit 36927e540549fbdfd156ac3518616dbf4642711f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 20 00:09:25 2020 +0200
build: turn off LTO for exports.c
There were lots of errors of kind:
/builds/M4RrgQZ3/0/tarantool/tarantool/src/exports.h:395:1: error: variable ‘uuid_nil’ redeclared as function
EXPORT(uuid_nil)
^
/builds/M4RrgQZ3/0/tarantool/tarantool/src/lib/uuid/tt_uuid.c:39:22: note: previously declared here
const struct tt_uuid uuid_nil;
when LTO was enabled. That happened because exports.c file, to
take symbol addresses, declared lots of functions and variables
from all the code base as
extern void <symbol>(void);
This is crazy, but it worked for normal builds. Because symbol is
symbol. The compilers couldn't find conflicting declarations,
because they never met in one compilation unit.
However the lie was revealed by linker with LTO enabled. It could
see, that actual symbol definitions didn't match their exports in
exports.c. It could live with mismatching function-function or
variable-variable cases, but couldn't withstand function-variable
mismatches. When a symbol was declared as a variable in one place
and as a function in another.
This was the case for variables:
- uuid_nil
- tarantool_lua_ibuf
- log_pid
- log_format
- crc32_calc
- _say
- log_level
The errors were false positive, because the symbols were never
used for anything except taking their addresses. To calm the
linker down exports.c now does not participate in LTO.
Closes #5001
commit 5ca12a403aff06f17840daf966f98da71ba42aa8
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Wed May 20 16:06:35 2020 +0300
test: recover FreeBSD skip conditions
Due to inaccuracy FreeBSD skip conditions were overwritten by OpenBSD
skip conditions.
Closes #5004
commit 55a29d1a2697e15a7e99f2158c33ab6c90863ec9
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu May 14 00:19:43 2020 +0200
uuid: fix unaligned memory access
Functions tt_uuid_is_nil() and tt_uuid_is_equal() used struct
tt_uuid as an array of two uint64. But it is wrong, because struct
tt_uuid has 4 byte alignment, while uint64 requires 8. That led to
unaligned memory access in these 2 functions.
The patch makes these functions use memcmp() - it appers to be
faster (disassembly shows much smaller code and without jumps),
and does not depend on alignment.
Part of #4609
Reviewed-by: Aleksandr Lyapunov <alyapunov@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
commit 43a3620221d9e37f5bb92e4414bdb61b25c3e1cb
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu May 14 00:19:19 2020 +0200
bit: fix unaligned memory access and UB bit shift
Lib/bit library in some functions dereferenced a byte array as a
pointer at uint64 or uint32. It led to unaligned memory access,
when the array didn't have the needed alignment. That made clang
'alignment' sanitizer crazy.
Also there were places, where bitwise shift was done for the type
size. For example, a uint32 number was allowed to make 32 bit
shift in both sides. This is undefined behaviour.
The patch fixes these UBs. Bitwise shift of too long size is
banned. Unaligned access is fixed in some places by changing
uint64 to uint8, when it is possible without perf loss. And in the
others by using explicitly specified unaligned load. So it works
the same, but the sanitizer is aware of that.
Part of #4609
Reviewed-by: Aleksandr Lyapunov <alyapunov@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
commit 8b5657aa6b387028e014e52c79a7a2c2151bb06d
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 12 00:28:46 2020 +0200
msgpuck: activate MP_EXT custom serializers
Msgpuck functions mp_snprint() and mp_fprint() now support
customizable MP_EXT serializer. This patch makes them able to
correctly serialize all tarantool's MP_EXT extensions: MP_DECIMAL,
MP_UUID, MP_ERROR.
Closes #4719
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Serge Petrenko <sergepetrenko@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 1104179c4ef8209d451c2cb6d7327db71606c853
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun May 10 20:26:45 2020 +0200
error: provide MP_ERROR extension serializer
Msgpuck functions mp_snprint() and mp_fprint() now support
customizable MP_EXT serializer. This patch adds such for MP_ERROR.
All extension serializers will be activated in a separate commit.
Part of #4719
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Serge Petrenko <sergepetrenko@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 261a5c672ce9dc910082e5dd412df18bde5136d8
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun May 10 20:26:28 2020 +0200
uuid: provide MP_UUID extension serializer
Msgpuck functions mp_snprint() and mp_fprint() now support
customizable MP_EXT serializer. This patch adds such for
MP_UUID. It is not activated yet, but will be later, when last
extension - MP_ERROR - also gets its serializer.
Part of #4719
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Serge Petrenko <sergepetrenko@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 5e32b0806dd5c11258cf546a9ffa8a62be037f8e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun May 10 20:26:08 2020 +0200
decimal: provide MP_DECIMAL extension serializer
Msgpuck functions mp_snprint() and mp_fprint() now support
customizable MP_EXT serializer. This patch adds such for
MP_DECIMAL. It is not activated yet, but will be later, when other
extensions also get their serializers.
Part of #4719
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Serge Petrenko <sergepetrenko@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 0fde38d301df718fd171459bbeb3aabb6487f96d
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun May 10 20:30:44 2020 +0200
msgpuck: bump version to enable extension printer
In the new version an API is added to be able to customize MP_EXT
serializer for mp_snprint() and mp_fprint() functions.
Part of #4719
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Serge Petrenko <sergepetrenko@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 66024d5c928a757e95b7fa63f30733c938185c38
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu May 21 15:50:19 2020 +0300
test: update test-run
It is hotfix for the previous test-run update, where 'core = app' test
suites starts to support 'use_unix_sockets_iproto' option. We found a
case when unix socket names may clash and attempt to listen on the
socket will report 'Address already in use' error. It is fixed in this
test-run update.
See more in [1].
[1]: https://github.com/tarantool/test-run/pull/211
commit b7b9a59f8f210744a3907ea517dc9a11a0d821f5
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu May 21 00:43:16 2020 +0300
test: update test-run
Support 'use_unix_sockets_iproto' suite.ini option in 'core = app' test
suites. It'll allow to workaround TCP port choosing race (see more in
[1]) by using Unix sockets instead of TCP ones: we did the same for
'core = tarantool' test suites in [2].
[1]: https://github.com/tarantool/test-run/issues/141
[2]: 60f84cbfca24e3a91cea067c923e006b44ee589f ('test: use unix sockets
for iproto connections')
Part of #4459.
commit 4a98a9bb0b7d587be3d69738ed1d5ce77c1bf3b7
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon May 18 18:57:58 2020 +0300
vinyl: fix comment to vy_read_iterator_cmp_stmt()
vy_read_iterator_cmp_stmt() uses tuple_compare_with_key() under the hood
which follows next convention:
(a) it returns 0 if key_fields of tuple equal to given key;
(b) < 0 if key_fields less than given key;
(c) > 0 otherwise.
However, comment to vy_read_iterator_cmp_stmt() says that:
(a) it returns -1 if first statement precedes second one;
(b) 0 if they at the same position;
(c) +1 if first statement supersedes second one.
so, since there's no "normalization" of return code from
tuple_compare_with_key() it would be correct to say that
vy_read_iterator_cmp_stmt() returns:
(a) it returns arbitrary integer value < 0 if first statement precedes
second one;
(b) 0 if they at the same position;
(c) arbitrary integer value > 0 if first statement supersedes second one.
commit bb7c3d1679924513b4cb3d8fb92acc639454b1c3
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Wed May 20 15:32:52 2020 +0300
small: bump new version
Add MAP_STACK to mmap() flags.
commit d48406d58c54231b7a53c8f8ed4ba9c1d5275a59
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun May 10 09:28:23 2020 +0300
test: add more tests to packaging testing
Changed number of tests to packaging testing jobs from
suites: "unit/ app/ app-tap/ box/ box-tap/ engine/ vinyl/",
to all suites except 'replication/' suite which has special
issue for it's enabling:
https://github.com/tarantool/tarantool/issues/4798
Also added testing to the all available packing jobs except
'CentOS 6'.
Removed test-run option that run testing inline to be
able to run it in parallel. Also changed 'test-run' to
'make test-force' command.
Closes #4599
commit f5880666227cb86b3119a3557afadb1178a1b809
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun May 10 09:28:15 2020 +0300
test: return tests to packaging testing
Found that issues #1227 and #1322 were closed, returned
the tests blocked by it into the testing.
Part of #4599
commit b304a78df0f8d900136c47dc3945061115382805
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun May 10 09:22:20 2020 +0300
build: fix rpm packages build on Fedora 32
Tarantool packages build scenario at rpm/tarantool.spec
installs Python packages for Tarantool testing. In some
latest OS like Fedora 32 old Python packages based on
Python 2 version were deprecated and changed to Python 3.
Currently Tarantool testing based on Python 2 which
packages stored at local backport repository and it
should be used from there calling it as python2-* packages.
Issue:
...
No matching package to install: 'python-yaml >= 3.0.9'
...
Not all dependencies satisfied
Error: Some packages could not be found.
Needed for #4599
Closes #4965
commit 64d1336a3663854d4fcb9c5373dc8e4e7bff81e1
Author: Alexander.V Tikhonov <avtikhon@tarantool.org>
Date: Wed May 6 13:06:59 2020 +0300
Enable script for saving RPM packages in S3 for modules
Found that modules may have only sources packages w/o binaries
packages. Script updated to be able to work with only binaries
either sources RPM packages. The same fix was already done for
DEB packages at commit 4527a4da9640a63601d0a1d0b2bee7f8cb4c0c85.
Close #4839
commit 430c0e80e19069eeb0b39cded00b9569708f9ca2
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue May 12 12:23:22 2020 +0300
test: mark tests as fragile, 2nd group
Fragiled flaky tests from parallel runs to avoid
of flaky fails in regular testing:
app/fiber.test.lua ; gh-4987
app/fiber_channel.test.lua ; gh-4961
app/socket.test.lua ; gh-4978
box/alter_limits.test.lua ; gh-4926
box/misc.test.lua ; gh-4982
box/role.test.lua ; gh-4998
box/rtree_rect.test.lua ; gh-4994
box/sequence.test.lua ; gh-4996
box/tuple.test.lua ; gh-4988
engine/ddl.test.lua ; gh-4353
replication/box_set_replication_stress ; gh-4992
replication/recover_missing_xlog.test.lua ; gh-4989
replication/replica_rejoin.test.lua ; gh-4985
replication/wal_rw_stress.test.lua ; gh-4977
replication-py/conflict.test.py ; gh-4980
vinyl/errinj_ddl.test.lua ; gh-4993
vinyl/misc.test.lua ; gh-4979
vinyl/snapshot.test.lua ; gh-4984
vinyl/write_iterator.test.lua ; gh-4572
xlog/panic_on_broken_lsn.test.lua ; gh-4991
Part of #4953
commit 33cc961c7f7f77ea0a06456d899cc1dc4744cb73
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Sat Apr 25 22:57:09 2020 +0300
Mention OpenBSD as supported platorm
Part of #4967
commit 54d682e2ec4704d4fc0f10021d3989c3b297de74
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Wed Apr 22 11:17:03 2020 +0300
Add README.OpenBSD
Part of #4967
commit c244c0b254b68082e6e351c7b7e62117a14a4a62
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Thu May 7 11:24:27 2020 +0300
test: mute broken tests on OpenBSD
Part of #4967
commit 643084843ec54037f498c71c2bc4dd323e320dd0
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Tue May 12 14:27:05 2020 +0300
say: fix compilation on OpenBSD
- define macros LOG_MAKEPRI() on OpenBSD as it is absent
- replace sigtimedwait() by sigwait() as latter is unsupported on OpenBSD
Part of #4967
commit fd653200d09643910439b16475db804484847328
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Thu May 7 01:27:45 2020 +0300
Include libgen.h when building on OpenBSD
Part of #4967
commit 43ffbba60a552474b3a4185cf7f6970cc555194b
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Tue Apr 21 18:18:42 2020 +0300
sql: use mremap() on OpenBSD
Part of #4967
commit 15601e09a327277a316b781d7a11f1055b2dcc49
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Thu Apr 23 23:02:27 2020 +0300
Fix building of tt_pthread_attr_getstack() on OpenBSD
Part of #4967
commit 291c392a16a33c574cfb0ed660abfd2f69940d42
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Tue Apr 21 18:13:08 2020 +0300
build: skip linking with -ldl on OpenBSD
OpenBSD includes DL library in a base system
Part of #4967
commit 378e48aae8c2c6389ab4055e761212f3bf2da4db
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Tue Apr 21 18:16:34 2020 +0300
build: test pthread_stackseg_np() on OpenBSD
Part of #4967
commit 0b516c107743dbf09679043fb7ba507771f1819d
Author: Sergey Bronnikov <sergeyb@tarantool.org>
Date: Tue Apr 21 18:14:55 2020 +0300
build: introduce openbsd build target
Part of #4967
commit 0afba959e215ebd877af6774f0867e83eac90893
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon May 18 02:36:29 2020 +0300
test: popen: fix popen test 'hang' under test-run
killpg() on Mac OS may don't deliver a signal to a process: it seems
that there is a race when a process is just forked. It means that
popen_handle:close() may leave a process alive, when `opts.setsid` and
`opts.group_signal` are set.
There is simple reproducer, which does not leave alive `sleep 120`
processes on Linux, but does it on Mac OS (within three-four runs in
rows):
| #include <signal.h>
| #include <unistd.h>
| #include <fcntl.h>
|
| int
| main()
| {
| char *child_argv[] = {
| "/bin/sh",
| "-c",
| "sleep 120",
| NULL,
| };
| pid_t pid;
| int fd[2];
| pipe(fd);
| fcntl(fd[0], F_SETFD, FD_CLOEXEC);
| fcntl(fd[1], F_SETFD, FD_CLOEXEC);
|
| if ((pid = fork()) == 0) {
| /* Child. */
| close(fd[0]);
| setpgrp();
| for (int i = 0; i < 10; ++i) {
| /* Proceed with killpg. */
| if (i == 5)
| close(fd[1]);
| if (fork() == 0) {
| /* Child. */
| execve("/bin/sh", child_argv, NULL);
| }
| }
| } else {
| /* Parent. */
| close(fd[1]);
| char c;
| read(fd[0], &c, 1);
| killpg(pid, SIGKILL);
| }
| return 0;
| }
Compile it (`cc test.c -o test`) and run several times:
$ for i in $(seq 1 1000); do \
echo $i; \
./test \
&& ps -o pid,pgid,command -ax | grep [s]leep \
&& break; \
done
This is the reason why `sleep 120` process may be alive even when the
whole test passes.
test-run captures stdout and stderr of a 'core = app' test and waits EOF
on them. If a child process inherit one of them or both, the fd is still
open for writing and so EOF situation will not appear until `sleep 120`
will exit.
This commit doesn't try to overcome the root of the problem, but close
stdout and stderr for the child process that may not be killed / exited
in time.
Aside of this, updated found Mac OS peculiars in API comments of C and
Lua popen modules.
Fixes #4938
@TarantoolBot document
Title: popen: add note re group signaling on Mac OS
Copyed from the popen_handle:signal() updated description:
> Note: Mac OS may don't deliver a signal to a process in a group when
> opts.setsid and opts.group_signal are set. It seems there is a race
> here: when a process is just forked it may be not signaled.
Copyed from the popen_handle:close() updated description:
> Details about signaling:
>
> <...>
> - There are peculiars in group signaling on Mac OS,
> @see popen_handle:signal() for details.
Follows up https://github.com/tarantool/doc/issues/1248
commit 01211e8e458365462fac721919d9773a79919e31
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon May 18 01:48:17 2020 +0300
popen: fix access to freed memory after :close()
popen_delete() always frees a handle memory even when it reports a
failure to send SIGKILL, see [1]. We should reflect this contract in
popen_handle:close() and mark the handle as closed despite
popen_delete() return value.
There are cases, when killpg() fails with EPERM on Mac OS, so
popen_delete() reports a failure. See [1] for more information.
[1]: 01657bfbb9b34997f20d27405226a9affdeeb520 ('popen: always free
resources in popen_delete()')
Fixes #4995
commit a38c861e841a6f39beac313125fdcb854af8244a
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 4 23:38:58 2020 +0200
session: remove box.session.push() 'sync'
Closes #4689
@TarantoolBot document
Title: box.session.push() 'sync' is deprecated
box.session.push() had two parameters - data to push and 'sync'.
The sync was a request ID with which the out of bound data should
be pushed into a socket.
This was introduced as a workaround for #3450, and is useless
since its resolution.
A user anyway can't push to different sessions, where that
parameter could be useful. And pushing into requests of the same
session, on the contrary, is something not really needed anywhere,
not portable to non-binary session types (console, background),
and is just dangerous since it is easy to add a bug here.
The patch removes the parameter. Now there will be thrown a
'Usage' error at attempt to use 'sync' parameter. In version 2.4
it is deprecated, prints warnings into logs, but still works. In
2.5 it is removed completely.
commit 61d2cb64fee735045e492609cbbd2c000b1380c3
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue May 19 12:36:07 2020 +0300
small: revert previous bump
Revert 03790ac commit in small library part.
commit 03790ac5510648d1d9648bb2281857a7992d0593
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Apr 11 20:18:44 2020 +0200
cmake: remove dynamic-list linker option
dynamic-list (exported_symbols_list on Mac) was used to forbid
export of all symbols of the tarantool executable except a given
list. Motivation of that was to avoid hacking the linker with
false usage of symbols needed to be exported. As a consequence,
symbols not listed in these options became invisible.
Before these options, when a symbol was defined, but not used in
the final executable, the linker could throw it away, even though
many symbols were used by Lua FFI, or should be visible for user's
dynamic modules. Where the linker, obviously, can't see if they
are needed.
To make the linker believe the symbols are actually needed there
was a hack with getting pointers at these functions and doing
something with them.
For example, assume we have 'test()' function in 'box' static
library:
int
test(void);
It is not used anywhere in the final executable. So to trick the
linker there is a function 'export_syms()' declared, which takes a
pointer at 'test()' and seemingly does something with it (or
actually does - it does not matter):
void
export_syms()
{
void *syms[] = {test};
if (time(NULL) == 0) {
syms[0]();
syms[1]();
...
}
}
Some users want to use not documented but visible symbols, so the
patch removes the dynamic-list option, and returns the linker
hack back. But with 0 dependencies in the export file.
Closes #2971
commit 5517189531d7402cae88c5129d3ab7556c6a05a6
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Apr 8 23:39:43 2020 +0200
cmake: remove double usage of some source files
src/cpu_feature.c, src/box/error.cc and src/box/vclock.c were used
twice. As a result it could lead to duplicate symbols.
commit 19f6e42b9cabb5545ac540321b589df29db9a828
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Fri May 8 14:17:11 2020 +0300
test: extend console lua test
To make sure ULL constants are not broken.
Part-of #4682
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit ae7e21036b5b3709b5c34d04ddff2b7ac08fabfb
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Apr 29 11:53:39 2020 +0300
box/console: switch to new lua serializer
I do not drop the serpent module for a while
since I need to make sure that everything work
as expected as we've not break backward compatibility
significantly (though we didn't claim the lua mode
output is stable enough).
Fixes #4682
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 35058d2262e0aff5d2f4a02daad979e09d6a3a24
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Apr 29 11:53:39 2020 +0300
box/console: implement lua serializer
When we print output in console (especially in Lua mode)
it is impossible to find out the internal type an entry
represents, which in turn leads to inability to encode
"ULL" entries properly. Moreover we always introduce new
types (for example decimals, uuids) and the serpent module
we currently use for encodings has no clue about them.
Thus lets implement own lua serializer similarly to the
yaml encoder. This allows us to properly detect every
field and encode it accordingly.
Part-of #4682
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit ec34fcdcadfb58d56cf2ad0e686c812dd6b6219b
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Apr 29 11:18:16 2020 +0300
box/console: rename luaL_yaml_default to serializer_yaml
As we gonna implement lua output serializer lets
rename luaL_yaml_default to serializer_yaml which
will be more general name, for other serializers
we will use same serializer_ prefix.
Part-of #4682
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit acc83768e61dddd8a48f08b69eb94e04621b0c19
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Jan 15 13:46:45 2020 +0300
box/console: rename format to format_yaml
This will allow us to implement own formatter for
Lua output mode, so to be precise which exactly formatter
is caller lets rename general "format" to "format_yaml".
Part-of #4682
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 7225ba7171908981c4c180d64ee49d789043e3ee
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Tue May 5 13:34:35 2020 +0300
box/console: use tabs instead of spaces in consolelib
For some reason we're using spaces here to adjust code.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit a2e34240fbb46b50f47e1d9280b376da754103d9
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Jan 15 13:24:36 2020 +0300
box/console: console_session_vtab -- use designated initialization
For better maintainance.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 15b340123eb39ac815017b9e64ebc78abfdb1ff6
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Fri May 15 14:12:31 2020 +0300
box/memtx: use uint32_t for number of indices
The member is defined as unsigned 32 bit value,
so we have to use an exactly the same type here.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 7f20272ea863e0e10f5f4d8b0ad446b46f3202fd
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri May 1 19:36:44 2020 +0200
test: make app-tap/init_script produce less diff
When a new option is added, app-tap/init_script
outputs big diff. Because all options are printed with
ordinal indexes. Addition of a new option changes
indexes of all options after the new one.
The patch removes indexes from the output making diff
smaller, when a new option is added.
commit 413b563ea9f40aabc0bdbc39741fb2fe768ae11b
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Thu May 14 13:12:05 2020 +0300
gitlab-ci: disable perf tesing in scheduled runs
Release branches should be regularly run using gitlab-ci pipeline
schedules:
https://gitlab.com/tarantool/tarantool/pipeline_schedules
It will help to detect flaky issues. But there is no need to rerun
too long running performance testing, to block it in schedules the
option 'schedules' in 'except:' field was set.
Part of #4974
commit 95364f02968cf7705a84ad68d03cbda00c5bee50
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Apr 10 09:46:29 2020 +0300
gitlab-ci: set OSX to full testing
Set all test suites at OSX testing.
Close #4818
commit caf739137382f86d08564f6f6daec5d525d28329
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Apr 13 17:55:03 2020 +0300
replication: remove unnecessary errors on replicating from an anonymous instance
Since the anonymous replica implementation, it was forbidden to
replicate (join/subscribe/register) from anonymous instances.
Actually, only joining and register should be banned, since an anonymous
replica isn't able to register its peer in _cluster anyway.
Let's allow other anonymous replicas, but not the normal ones, to subscribe to
an anonymous replica.
Also remove unnecessary ER_UNSUPPORTED errors from box_process_join()
and box_process_register() for anonymous replicas. These cases are
covered with ER_READONLY checks later on, since anonymous replicas must
be read-only.
Note, this patch doesn't allow normal instances to subscribe to
anonymous ones. Even though it is technically possible, it may bring
more problems than profits. Let's allow it later on if there's an explicit
demand.
Closes #4696
commit 4e5d60d819c68caea1923311cda6fd9cda2ec3b5
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Thu Apr 23 14:17:15 2020 +0300
replication: add box.info.replication_anon
Closes #4900
@TarantoolBot document
Title: add new field to box.info: replication_anon
It is now possible to list all the anonymous replicas following the
instance with a call to `box.info.replication_anon()`
The output is similar to the one produced by `box.info.replication` with
an exception that anonymous replicas are indexed by their uuid strings
rather then server ids, since server ids have no meaning for anonymous
replicas.
Note, that when you issue a plain `box.info.replication_anon`, the only
info returned is the number of anonymous replicas following the current
instance. In order to see the full stats, you have to call
`box.info.replication_anon()`. This is done to not overload the `box.info`
output with excess info, since there may be lots of anonymous replicas.
Example:
```
tarantool> box.info.replication_anon
---
- count: 2
...
tarantool> box.info.replication_anon()
---
- 3a6a2cfb-7e47-42f6-8309-7a25c37feea1:
id: 0
uuid: 3a6a2cfb-7e47-42f6-8309-7a25c37feea1
lsn: 0
downstream:
status: follow
idle: 0.76203499999974
vclock: {1: 1}
f58e4cb0-e0a8-42a1-b439-591dd36c8e5e:
id: 0
uuid: f58e4cb0-e0a8-42a1-b439-591dd36c8e5e
lsn: 0
downstream:
status: follow
idle: 0.0041349999992235
vclock: {1: 1}
...
```
Note, that anonymous replicas hide their lsn from the others, so
anonymous replica lsn will always be reported as zero, even if anonymous
replicas perform some local space operations.
To know the anonymous replica's lsn, you have to issue `box.info.lsn` on
it.
commit 32f59756af8d9e025e43f130b48adad116f713e4
Author: Nikita Pettik <korablev@tarantool.org>
Date: Thu Apr 30 18:10:37 2020 +0300
vinyl: drop wasted runs in case range recovery fails
If recovery process fails during range restoration, range itself is
deleted and recovery is assumed to be finished as failed (in case of
casual i.e. not forced recovery). During recovery of particular range,
runs to be restored are refed twice: once when they are created at
vy_run_new() and once when they are attached to slice. This fact is
taken into consideration and after all ranges are recovered: all runs of
lsm tree are unrefed so that slices own run resources (as a result, when
slice is to be deleted its runs unrefed and deleted as well). However, if
range recovery fails, range is dropped alongside with already recovered
slices. This leads to unrefing runs - this is not accounted. To sum up
recovery process below is a brief schema:
foreach range in lsm.ranges {
vy_lsm_recover_range(range) {
foreach slice in range.slices {
// inside recover_slice() each run is refed twice
if vy_lsm_recover_slice() != 0 {
// here all already restored slices are deleted and
// corresponding runs are unrefed, so now they have 1 ref.
range_delete()
}
}
}
}
foreach run in lsm.runs {
assert(run->refs > 1)
vy_run_unref(run)
}
In this case, unrefing such runs one more time would lead to their
destruction. On the other hand, iteration over runs may turn out to
be unsafe, so we should use rlist_foreach_entry_safe(). Moreover, we
should explicitly clean-up these runs calling vy_lsm_remove_run().
Reviewed-by: Vladislav Shpilevoy <vshpilevoi@mail.ru>
Closes #4805
commit 9d4ac029a7b0ebfb679a4ae4582057970621c9ca
Author: Nikita Pettik <korablev@tarantool.org>
Date: Wed Apr 29 17:45:31 2020 +0300
errinj: introduce delayed injection
With new macro ERROR_INJECT_COUNTDOWN it is possible to delay error
injection by iparam value: injection will be set only after iparam
times the path is executed. For instance:
void
foo(int i)
{
/* 2 is delay counter. */
ERROR_INJECT_COUNTDOWN(ERRINJ_FOO, {
printf("Error injection on %d cycle!\n", i);
});
}
void
boo(void)
{
for (int i = 0; i < 10; ++i)
foo(i);
}
box.error.injection.set('ERRINJ_FOO', 2)
The result is "Error injection on 2 cycle!". This type of error
injection can turn out to be useful to set injection in the middle of
query processing. Imagine following scenario:
void
foo(void)
{
int *fds[10];
for (int i = 0; i < 10; ++i) {
fds[i] = malloc(sizeof(int));
if (fds[i] == NULL)
goto cleanup;
}
cleanup:
free(fds[0]);
}
"cleanup" section obviously contains error and leads to memory leak.
But using means of casual error injection without delay such situation
can't be detected: OOM can be set only for first cycle iteration and in
this particular case no leaks take place.
Reviewed-by: Vladislav Shpilevoy <vshpilevoi@mail.ru>
commit cb06201765911142e0a6a9afefa6941d55faec9d
Author: Nikita Pettik <korablev@tarantool.org>
Date: Fri Apr 24 21:10:08 2020 +0300
vinyl: add test on failed write iterator during compaction
vy_task_write_run() is executed in auxiliary thread (dump or
compaction). Write iterator is created and used inside this function.
Meanwhile, creating/destroying tuples in these threads does not change
reference counter of corresponding tuple formats (see vy_tuple_delete()
and vy_stmt_alloc()). Without cleaning up write iterator right in
write_iterator_start() after fail, this procedure takes place in
vy_task_compaction_abort() or vy_task_dump_abort(). These *_abort()
functions in turn are executed in the main thread. Taking this into
consideration, tuple might be allocated in aux. thread and deleted in
the main thread. As a result, format reference counter might decrease,
whereas it shouldn't change (otherwise tuple format will be destroyed
before all tuples of this format are gone).
Fortunately, clean-up of write iterator in another thread was found only
on 1.10 branch, master branch already contains fix but lacks test
(2f17c92). So let's introduce test with following scenario:
1. run compaction process;
2. add one or more slice sources in vy_write_iterator_start():
corresponding slice_stream structures obtain newly created tuples
in vy_slice_stream_next();
3. the next call of vy_write_iterator_add_src() fails due to OOM,
invalid run file or whatever;
4. if write_iterator_start() didn't provide clean-up of sources, it
would take place in vy_task_dump_abort() which would be executed in
the main thread;
5. now format reference counter would be less than it was before
compaction.
Closes #4864
commit 521a6fbdcf07e7fe02e8647c997bd6a57fc261d1
Author: Nikita Pettik <korablev@tarantool.org>
Date: Wed Apr 8 23:28:46 2020 +0300
vinyl: clean-up unprocessed read views in *_build_read_views()
vy_write_iterator->read_views[i].history objects are allocated on
region (see vy_write_iterator_push_rv()) during building history of the
given key. However, in case of fail of vy_write_iterator_build_history()
region is truncated but pointers to vy_write_history objects are not
nullified. As a result, they may be accessed (for instance while
finalizing write_iterator object in vy_write_iterator_stop) which in
turn may lead to crash, segfaul or disk formatting. The same may happen
if vy_read_view_merge() fails during processing of read view array.
Let's clean-up those objects in case of error takes place.
Part of #4864
commit d3a7dd17d494770b57bbac4e462f6423c82ea22b
Author: HustonMmmavr <huston.mavr@gmail.com>
Date: Thu May 7 21:28:08 2020 +0300
static build: dockerfile entrypoint set to exec form
According to dockerfile reference, there are two forms of specifying
entrypoint: exec and shell. Exec form is preferrable and allows use
this image in scripts.
Close #4960
commit 76157ef6cd903e0299b2f63878ed06a24e25e54c
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Jan 21 17:24:29 2020 +0300
gitlab-ci: add Catalina OSX 10.15
Added Catalina OSX 10.15 to gitlab-ci testing and removed OSX 10.13,
due to decided to have only 2 last major releases, for now it is
10.14 and 10.15 OSX versions. Also changed the commit job for branches
from 10.14 to 10.15 OSX version.
Additional cleanup for 'box_return_mp' and 'box_session_push',
added API_EXPORT which defines nothrow, compiler warns or errors
depending on the build options.
Part of #4885
Close #4873
commit faf7e4827a1e6b04f6cc13587ec93592f4c4e34e
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Thu Apr 30 12:44:45 2020 +0300
test: mark tests as fragile in a test's configs
Fragiled flaky tests from parallel runs to avoid
of flaky fails in regular testing:
box-py/snapshot.test.py ; gh-4514
replication/misc.test.lua ; gh-4940
replication/skip_conflict_row.test.lua ; gh-4958
replication-py/init_storage.test.py ; gh-4949
vinyl/stat.test.lua ; gh-4951
xlog/checkpoint_daemon.test.lua ; gh-4952
Part of #4953
commit eeb501ec22b5e6e23a8a2132d566afb45a4b2009
Author: Oleg Piskunov <o.piskunov@tarantool.org>
Date: Wed May 6 14:18:15 2020 +0300
gitlab-ci: keep perf results as gitlab-ci artifacts
Gitlab-ci pipeline modified in order to keep
performance results into gitlab-ci artifacts.
Closes #4920
commit a4f4adeb47cc5c8aea4cbea2774c8ea497ee647d
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Tue Dec 24 19:16:40 2019 +0300
wal: simplify rollback
Here is a summary on how and when rollback works in WAL.
Disk write failure can cause rollback. In that case the failed and
all next transactions, sent to WAL, should be rolled back.
Together. Following transactions should be rolled back too,
because they could make their statements based on what they saw in
the failed transaction. Also rollback of the failed transaction
without rollback of the next ones can actually rewrite what they
committed.
So when rollback is started, *all* pending transactions should be
rolled back. However if they would keep coming, the rollback would
be infinite. This means to complete a rollback it is necessary to
stop sending new transactions to WAL, then rollback all already
sent. In the end allow new transactions again.
Step-by-step:
1) stop accepting all new transactions in WAL thread, where
rollback is started. All new transactions don't even try to go to
disk. They added to rollback queue immediately after arriving to
WAL thread.
2) tell TX thread to stop sending new transactions to WAL. So as
the rollback queue would stop growing.
3) rollback all transactions in reverse order.
4) allow transactions again in WAL thread and TX thread.
The algorithm is long, but simple and understandable. However
implementation wasn't so easy. It was done using a 4-hop cbus
route. 2 hops of which were supposed to clear cbus channel from
all other cbus messages. Next two hops implemented steps 3 and 4.
Rollback state of the WAL was signaled by checking internals of a
preallocated cbus message.
The patch makes it simpler and more straightforward. Rollback
state is now signaled by a simple flag, and there is no a hack
about clearing cbus channel, no touching attributes of a cbus
message. The moment when all transactions are stopped and the last
one has returned from WAL is visible explicitly, because the last
sent to WAL journal entry is saved.
Also there is a single route for commit and rollback cbus
messages now, called tx_complete_batch(). This change will come
in hand in scope of synchronous replication, when WAL write won't
be enough for commit. And therefore 'commit' as a concept should
be washed away from WAL's code gradually. Migrate to solely txn
module.
commit c7341a3d4fd06c0efde3e9cf8048f71fba52aaba
Author: Roman Khabibov <roman.habibov@tarantool.org>
Date: Fri Apr 10 14:31:17 2020 +0300
console: check on_shutdown() before exit
Add check that on_shutdown() triggers were called before exit,
because in case of EOF or Ctrl+D (no signals) they were ignored.
Closes #4703
commit 4dcba1b54dea8242acd8adfc9f6df6abf9eb172c
Author: Nikita Pettik <korablev@tarantool.org>
Date: Wed Apr 8 22:12:17 2020 +0300
vinyl: init all vars before cleanup in vy_lsm_split_range()
If vy_key_from_msgpack() fails in vy_lsm_split_range(), clean-up
procedure is called. However, at this moment struct vy_range *parts[2]
is not initialized ergo contains garbage and access to this structure
may result in crash, segfault or disk formatting. Let's move
initialization of mentioned variables before call of
vy_lsm_split_range().
Part of #4864
commit 9f28197856ef5a2a83ddb7d02bf71788bedcb0cd
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri May 1 08:05:37 2020 +0300
gitlab-ci: add Ubuntu Focal to S3 list
Found that in commit 'travis-ci/gitlab-ci: add Ubuntu Focal 20.04'
forgot to add Ubuntu Focal to the list of the available Ubuntu
distributions in the script for saving built packages at S3.
Follow up #4863
commit 765f338ee87bf3ebf99727c6cccd7b173de9b4be
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Apr 28 17:37:24 2020 +0300
travis-ci/gitlab-ci: add Ubuntu Focal 20.04
Closes #4863
commit b136a61e51ac871f32658b611d4ce7b7f7d5363c
Author: Sergey Ostanevich <sergos@tarantool.org>
Date: Mon Apr 13 18:59:37 2020 +0300
Code cleanup: sync declarations and definitions
API_EXPORT defines nothrow, so compiler warns or errors depending on the
build options.
Closes #4885
commit f81dae2d87c50107ddeefabd4f8dee12e79df173
Author: Aleander V. Tikhonov <avtikhon@tarantool.org>
Date: Mon Apr 27 02:02:45 2020 +0300
test: fix flaky replication/skip_conflict_row test
Fixed flaky upstream checks at replication/skip_conflict_row test,
also check on lsn set in test-run wait condition routine.
Errors fixed:
[024] @@ -66,11 +66,11 @@
[024] ...
[024] box.info.replication[1].upstream.message
[024] ---
[024] -- null
[024] +- timed out
[024] ...
[024] box.info.replication[1].upstream.status
[024] ---
[024] -- follow
[024] +- disconnected
[024] ...
[024] box.space.test:select()
[024] ---
[024]
[004] @@ -125,11 +125,11 @@
[004] ...
[004] box.info.replication[1].upstream.message
[004] ---
[004] -- Duplicate key exists in unique index 'primary' in space 'test'
[004] -...
[004] -box.info.replication[1].upstream.status
[004] ----
[004] -- stopped
[004] +- null
[004] +...
[004] +box.info.replication[1].upstream.status
[004] +---
[004] +- follow
[004] ...
[004] test_run:cmd("switch default")
[004] ---
[004]
[038] @@ -174,7 +174,7 @@
[038] ...
[038] box.info.replication[1].upstream.status
[038] ---
[038] -- follow
[038] +- disconnected
[038] ...
[038] -- write some conflicting records on slave
[038] for i = 1, 10 do box.space.test:insert({i, 'r'}) end
Line 201 (often):
[039] @@ -201,7 +201,7 @@
[039] -- lsn should be incremented
[039] v1 == box.info.vclock[1] - 10
[039] ---
[039] -- true
[039] +- false
[039] ...
[039] -- and state is follow
[039] box.info.replication[1].upstream.status
[039]
[030] @@ -201,12 +201,12 @@
[030] -- lsn should be incremented
[030] v1 == box.info.vclock[1] - 10
[030] ---
[030] -- true
[030] +- false
[030] ...
[030] -- and state is follow
[030] box.info.replication[1].upstream.status
[030] ---
[030] -- follow
[030] +- disconnected
[030] ...
[030] -- restart server and check replication continues from nop-ed vclock
[030] test_run:cmd("switch default")
Line 230 (OSX):
[022] --- replication/skip_conflict_row.result Thu Apr 16 21:54:28 2020
[022] +++ replication/skip_conflict_row.reject Mon Apr 27 00:52:56 2020
[022] @@ -230,7 +230,7 @@
[022] ...
[022] box.info.replication[1].upstream.status
[022] ---
[022] -- follow
[022] +- disconnected
[022] ...
[022] box.space.test:select({11}, {iterator = "GE"})
[022] ---
[022]
Close #4457
commit ba206b484ae691e5816d81a28d38b2a385596479
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Apr 28 13:54:15 2020 +0300
travis-ci: don't deploy 2.5+ pkgs to packagecloud
Now we have S3 based infrastructure for RPM / Deb packages and GitLab CI
pipelines, which deploys packages to it.
We don't plan to add 2.5+ repositories on packagecloud.io, so instead of
usual change of target bucket from 2_N to 2_(N+1), the deploy stage is
removed.
Since all distro specific jobs are duplicated in GitLab CI pipelines and
those Travis-CI jobs are needed just for deployment, it worth to remove
them too.
Follows up #3380.
Part of #4947.
commit b56484d670426dfab336c321c0fa2819e8de5df0
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Apr 15 01:11:03 2020 +0200
schema: fix internal symbols dangling in _G
A couple of functions were mistakenly declared as 'function'
instead of 'local function' in schema.lua. That led to their
presence in the global namespace.
Closes #4812
commit fcce05a44f152720798706bf678a1160ff0d3e1c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Apr 15 01:10:47 2020 +0200
schema: fix index promotion to functional index
When index:alter() was called on a non-functional index with
specified 'func', it led to accessing a not declared variable in
schema.lua.
commit 4d82478f4d8978cc7d9d623ea2c3223b926c4a34
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Apr 23 00:28:09 2020 +0200
box: replace port_tuple with port_c everywhere
Port_tuple is exactly the same as port_c, but is not able to store
raw MessagePack. In theory it sounds like port_tuple should be a
bit simpler and therefore faster, but in fact it is not.
Microbenchmarks didn't reveal any difference. So port_tuple is no
longer needed, all its functionality is covered by port_c.
Follow up #4641
commit dd36c6108869be1cf45ce03092872904fae2381b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Mar 8 16:47:29 2020 +0100
box: introduce box_return_mp() public C function
Closes #4641
@TarantoolBot document
Title: box_return_mp() public C function
Stored C functions could return a result only via
`box_return_tuple()` function. That made users create a tuple
every time they wanted to return something from a C function.
Now public C API offers another way to return - `box_return_mp()`.
It allows to return arbitrary MessagePack, not wrapped into a
tuple object. This is simpler to use for small results like a
number, boolean, or a short string. Besides, `box_return_mp()` is
much faster than `box_return_tuple()`, especially for small
MessagePack.
Note, that it is faster only if an alternative is to create a
tuple by yourself. If an already existing tuple was obtained from
an iterator, and you want to return it, then of course it is
faster to return via `box_return_tuple()`, than via extraction of
tuple data, and calling `box_return_mp()`.
Here is the function declaration from module.h:
```C
/**
* Return MessagePack from a stored C procedure. The MessagePack
* is copied, so it is safe to free/reuse the passed arguments
* after the call.
* MessagePack is not validated, for the sake of speed. It is
* expected to be a single encoded object. An attempt to encode
* and return multiple objects without wrapping them into an
* MP_ARRAY or MP_MAP is undefined behaviour.
*
* \param ctx An opaque structure passed to the stored C procedure
* by Tarantool.
* \param mp Begin of MessagePack.
* \param mp_end End of MessagePack.
* \retval -1 Error.
* \retval 0 Success.
*/
API_EXPORT int
box_return_mp(box_function_ctx_t *ctx, const char *mp, const char *mp_end);
```
commit 4c3c9bda02e0ffb6092d5d07c7dcbffe39f60699
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Mar 7 18:27:47 2020 +0100
box: introduce port_c
Port_c is a new descendant of struct port. It is used now for
public C functions to store their result. Currently they can
return only a tuple, but it will change soon, they will be able to
return arbitrary MessagePack.
Port_tuple is not removed, because still is used for box_select(),
for functional indexes, and in SQL as a base for port_sql.
Although that may be changed later. Functional indexes really need
only a single MessagePack object from their function. While
box_select() working via port_tuple or port_c didn't show any
significant difference during micro benchmarks.
Part of #4641
commit 0edb4d97c65287c05d5bcf99497142548de30a54
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Apr 22 18:44:01 2020 +0300
applier: follow vclock to the last tx row
Since the introduction of transaction boundaries in replication
protocol, appliers follow replicaset.applier.vclock to the lsn of the
first row in an arrived batch. This is enough and doesn't lead to errors
when replicating from other instances, respecting transaction boundaries
(instances with version 2.1.2 and up).
However, if there's a 1.10 instance in 2.1.2+ cluster, it sends every
single tx row as a separate transaction, breaking the comparison with
replicaset.applier.vclock and making the applier apply part of the
changes, it has already applied when processing a full transaction
coming from another 2.x instance. Such behaviour leads to
ER_TUPLE_FOUND errors in the scenario described above.
In order to guard from such cases, follow replicaset.applier.vclock to
the lsn of the last row in tx.
Closes #4924
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 72ce442c933fb4c97a05789cf070d4fb480b84bc
Author: Roman Khabibov <roman.habibov@tarantool.org>
Date: Mon Apr 13 05:03:54 2020 +0300
sql: fix sorting rules for values of SCALAR type
Function implementing comparison during VDBE sorting routine
(sqlVdbeCompareMsgpack) did not account values of boolean type in some
cases. Let's fix it so that booleans always precede numbers if they are
sorted in ascending order.
Closes #4697
commit d6d69c9fb16c704c5e89335770308ed0a21b6dd7
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Apr 22 22:35:47 2020 +0300
cbus: fix inconsistency in endpoint creation
The notification of wait variable shall be done under
a bound mutex locked. Otherwise the results are not
guaranteed (see pthread manuals).
Thus when we create a new endpoint via cbus_endpoint_create
and there is an other thread which sleeps inside cpipe_create
we should notify the sleeper under cbus.mutex.
Fixes #4806
Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit d7fa6d34ab4e0956fe8a80966ba628e0e3f81067
Author: Leonid Vasiliev <lvasiliev@tarantool.org>
Date: Tue Apr 21 19:16:17 2020 +0300
build: fix compilation on Alpine 3.5
The cbus hang test uses glibc pthread mutex implementation details.
The reason why mutex implementation details is used:
"For the bug reproducing the canceled thread must be canceled
during processing cpipe_flush_cb. We need to synchronize
the main thread and the canceled worker thread for that.
So, thread synchronization has been realized by means of
endpoint's mutex internal field(__data.__lock)."
Therefore, it should not compile in case of using another library.
commit 098324556aadf9f7e6b41758fbb62cc91f32d284
Author: Olga Arkhangelskaia <arkholga@tarantool.org>
Date: Mon Mar 23 10:42:14 2020 +0000
say: fix syslog format
While refactoring of say module in commit
5db765a76ab7270b350960a2dd6d89dcd3bdc032 (say: fix non-informative error
messages for log cfg) format of syslog was broken.
Closes #4785
commit ad13b6d572e99b1ab5f2eb57b1ec76724e62275c
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Mon Apr 20 16:34:21 2020 +0300
Dummy commit
