Git log: commit 94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Oct 16 21:04:25 2020 +0300
test: gh test hangs after gh-4957-too-many-upserts
Found that if the previos test like gh-4957-too-many-upserts.test.lua
leaves some garbage on the current Tarantool server, then the next
running gh.test.lua test hangs. To resolve it decided to use workaround
and to set delay before the test start with fiber.sleep(1) to be able
to get out of one loop iteration.
Closes #5141
commit 0c40bd0a7b801cd1a1758160157625694d3feb1f
Author: Timur Safin <tsafin@tarantool.org>
Date: Fri Oct 16 19:47:52 2020 +0300
module api: fix asan failures in test
Adding forgotten resource cleanup to the
`test_tuple_validate_formatted` test, to make ASAN
happy again.
Closes #5432
Relates to #5384
commit c9447a7a7af04ca582899d0dc1a100d59a8e9932
Author: Igor Munkin <imun@tarantool.org>
Date: Fri Oct 16 14:08:27 2020 +0300
test: adjust the path to LuaJIT test directory
To ease the further maintenance of the tests located in LuaJIT repo,
particular hardcoded paths are replaced with the root path to test
directory within LuaJIT third party submodule.
Closes #5425
Reviewed-by: Timur Safin <tsafin@tarantool.org>
Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
commit 69e830e13d8fb339325772c913ec58d07c1a9439
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Fri Oct 16 16:15:43 2020 +0300
luajit: bump new version
* test: introduce the root CMakeLists.txt
commit cf6a9c64dea6a4385f3fd4719e44d0d9dc9ffee9
Author: Ilya Konyukhov <runsfor@gmail.com>
Date: Fri Sep 18 19:46:26 2020 +0300
feedback: add repository URL into report
This patch looks into tarantool repo files, extracts
download url and adds it into feedback report.
Currently only centos, fedora (yum) and ubuntu (apt) are
supported.
Closes #5281
commit 1e667e42b4d731f5a83f390ae5994e35bd11ef53
Author: Ilya Konyukhov <runsfor@gmail.com>
Date: Fri Sep 18 11:45:40 2020 +0300
feedback: make test not fail on subsequent runs
When running test without using test-run, i.e:
```
tarantool test/box-tap/feedback_daemon.test.lua
```
test would fail on second run, since database was not removed.
commit 259a6d1df002c8e0c73722e6a96fa2a49e203e02
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Sep 30 17:16:59 2020 +0300
gitlab-ci: add OSX 13 to testing
Restored OSX 13 testing to pass homebrew commit criteria.
Added MACOSX_DEPLOYMENT_TARGET=10.13 environment variable.
Closes #5374
commit 7a7d2e9ec9221a87068ac3809ad484fa866431b9
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Thu Oct 15 18:47:35 2020 +0300
.gitignore: add coverity, patches and special ignore
- coverity stands for coverity scanner results
- .pc for quilt
- .git-ignore directory for various things which
are not supposed to be in repo, just for convenience
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit aa78a941430a196e24d146734cd1f64bd9735e4f
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Thu Oct 15 17:25:55 2020 +0300
test/uint: fiber
When optimization level increased we've found that
memset calls might be optimized, moreover there
were no check if calls themselves would override
the stack.
Lets rework and use number of calls instead. Strictly
speaking the test is still a bit fragile and precise
testing would rather require asm level code.
Repored-by: "Alexander V. Tikhonov" <avtikhon@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit f85e886e945646cb90af127f843da3e4789599e6
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Oct 16 00:40:07 2020 +0200
raft: fix crash when leader resigned from its role
Nodes with disabled Raft keep listening for Raft events and
persist them. To be able to quickly enroll into the process if
they are configured to be candidates.
The same for the voter nodes - they can't be a leader, but watch
and persist all what is happening.
However when a leader resigned from its role, the voter and
disabled nodes tried to start a new election round, even though
they were not supposed to. That led to a crash, and is fixed in
this patch.
Closes #5426
commit eb989bc7c68fa3be0b8dcfc8566a497f4013708a
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Thu Oct 15 17:00:49 2020 +0300
test: merge possible failed test outputs for election_qsync_stress
Instead of having all possible 2^10 diffs in case select{} fails,
log the select{} output in case the space contains less than 10
elements.
Requested by @avtikhon for easier flaky test handling.
Related to #5395
commit 6f5886751b17404e0c338b8dadd1513650080675
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Oct 14 10:06:36 2020 +0300
gitlab-ci: add out-of-source build
Added out of source build make targets and added test job to gitlab-ci.
Closes #4874
commit 2b07601922138fee02554ba69c989bee567e812e
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed May 13 09:37:12 2020 +0300
build: enable cmake in curl build
Initially tried to change autoconf tools in Curl build to cmake and
found the following build issue on:
CentOS 6
CentOS 7
Debian 8
Ubuntu 14.04
Issue found:
CMake Error at CMakeLists.txt:41 (cmake_minimum_required):
CMake 3.0 or higher is required. You are running version 2.8.12.2
To fix the issue check is removed of the version from curl sources
in Tarantool's third party '<root Tarantool sources>/third_party/curl':
12af024bc85606b14ffc415413a7e86e6bbee7eb ('Enable curl build with old cmake')
After this fix completely changed autoconf to cmake in curl build.
Autoconf part was completely removed and code cleaned up for cmake.
For curl cmake build all autoconf options were ported to cmake
configuration call, check the accurate list of the change in [1].
Also the following issues resolved:
1. Found that CURL cmake configuration file:
third_party/curl/lib/CMakeLists.txt
has installation part for built libcurl.a library:
install(TARGETS ${LIB_NAME}
EXPORT ${TARGETS_EXPORT_NAME}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
where it changes CMAKE_INSTALL_LIBDIR to appropriate name with
suffix of the architecture, like:
lib
lib64
x86_64
Found that find_library routine from the file:
cmake/FindLibCURL.cmake
returns only 'lib' value and it breaks the building of the depends
binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to
cmake call:
-DCMAKE_INSTALL_LIBDIR=lib
2. Found issue with building on CentOS 6:
Linking C executable curl
build/ares/dest/lib/libcares.a(ares__timeval.c.o): In function `ares__tvnow':
ares__timeval.c:(.text+0x15): undefined reference to `clock_gettime'
collect2: error: ld returned 1 exit status
It was fixed with added "-lrt" flag to CMAKE_C_FLAGS and
CMAKE_CXX_FLAGS build flags, when cmake version is lower
than 3.0 and RT library had needed function.
3. Found issues with building Tarantool statically on Debian 9 and
its package build on CentOS 8.
Building statically got the issues with openssl linking, like [2]:
static-build/openssl-prefix/lib/libcrypto.a(threads_pthread.o): In function `CRYPTO_THREAD_lock_new':
threads_pthread.c:(.text+0x45): undefined reference to `pthread_rwlock_init'
It happened because in openssl radically changed how threads-related
things were handled before 1.1.0 version. It required the application
to provide us with the locking functionality in form of callbacks.
Since 1.1.0, these matters are entirely internal, so libcrypto
requires the presence of the platform specific thread implementation
of our choosing, which is pthread on everything.
After '-pthread' added to C compile flags package build on CentOS 8
failed with the issue [3]:
/build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:101:4: error: #error "SIZEOF_CURL_OFF_T not defined"
# error "SIZEOF_CURL_OFF_T not defined"
^~~~~
/build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c: In function 'curlx_uztoso':
/build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:192:40: error: 'CURL_MASK_SCOFFT' undeclared (first use in this function); did you mean 'CURL_MASK_SINT'?
return (curl_off_t)(uznum & (size_t) CURL_MASK_SCOFFT);
^~~~~~~~~~~~~~~~
CURL_MASK_SINT
To avoid of the issue decided to use '-pthread' flag only for openssl
when it had not this flag in openssl compilation.
4. Found issue with static build using CentOS 7, where SSL cmake rule
failed. Building the image got the issue:
[ 1%] Performing configure step for 'bundled-libcurl-project'
CMake Warning at CMakeLists.txt:50 (message):
the curl cmake build system is poorly maintained. Be aware
-- curl version=[7.66.0-DEV]
-- Found c-ares: /tarantool/build/ares/dest/lib/libcares.a
Found *nroff option: -- -man
CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:278 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:347 (find_package)
Root cause of the issue that Dockerfile uses globaly
installed openSSL with:
cmake ... -DOPENSSL_ROOT_DIR=/usr/local ...
Its cmake build file:
/usr/share/cmake/Modules/FindOpenSSL.cmake
fails on parsing the SSL version:
it has:
REGEX "^#define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")
but it should to use:
REGEX "^#[\t ]*define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")
Anyway we want to use the same OpenSSL library for libcurl, as is used
for Tarantool itself. So the path to it set for cURL build:
list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake")
5. In CMake build CMAKE_USE_LIBSSH2 flag is enabled by default, while
in autoconf --with-libssh2 was disabled by default. We need to switch
CMAKE_USE_LIBSSH2 flag off with:
list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_LIBSSH2=OFF")
to avoid of linking issues, like:
ld: libssh2.c:(.text+0x4d8): undefined reference to `libssh2_*...
this issue exists in curl issues [4].
Furthermore the following defaults are also disabled to keep the
configuration consistent with autoconf one:
list(APPEND LIBCURL_CMAKE_FLAGS "-DPICKY_COMPILER=OFF")
list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_CURL_EXE=OFF")
Closes #4968
Closes #5019
Closes #5396
[1] - https://github.com/tarantool/tarantool/issues/4968#issue-617183031
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/779176133#L6021
[3] - https://gitlab.com/tarantool/tarantool/-/jobs/778309145#L3060
[4] - https://github.com/curl/curl/issues/1146
commit cca763ad3f8fb9f29eaa73c82ebb53a3b64c2740
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Oct 14 11:15:38 2020 +0300
build: generate bootstrap.h in CMAKE_BINARY_DIR
Prior to these changes bootstrap.h was generated right in the source
directory even for out of source build. Firstly such approach doesn't
respect the idea of building outside the source files. Furthermore
this leads to build failures when the source directory is located on
read-only file system.
As a result of the patch bootstrap.h is generated within the build
tree and include directories are adjusted the corresponding way.
Also changed destination build directory from source to binary path.
Part of #4968
commit ead9c748692f1cc99a7dd6c7b6aa1c2bddeda85e
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Oct 14 10:26:55 2020 +0300
curl: enable curl build with old cmake
Blocked check if the current cmake of the older than
3.2...3.16 version to be able to build curl at the old OS:
- CentOS 6
- CentOS 7
- Debian 8
- Ubuntu 14.04
Used in 'third_party/curl' repository branch:
curl-7_71_1-tarantool
Needed for #4968
commit 3dc6a76c8955f720e7b487e6ad1d8b6aa1f6d2c0
Author: Timur Safin <tsafin@tarantool.org>
Date: Sun Oct 11 16:08:02 2020 +0300
module api: luaT_toibuf instead of luaL_checkibuf
* made `luaL_checkibuf` public and then renamed to
`luaT_toibuf` to follow naming convention
(it's not raising error, simply returning ibuf
structure).
Closes #5384
commit f56d31ddab21d8b4d520610186de58896a9135ac
Author: Timur Safin <tsafin@tarantool.org>
Date: Mon Sep 14 13:22:18 2020 +0300
module api: export box_key_def_dup
Exporting `box_key_def_dup` as accessor to the internal `key_def_dup`
Part of #5384
commit e7307b42b8c7e4f4c91556905c609788e53b30bb
Author: Timur Safin <tsafin@tarantool.org>
Date: Mon Sep 14 12:15:34 2020 +0300
module api: export box_tuple_validate
For external merger we need means to validate tuple data,
thus exporting `box_tuple_validate` which is wrapper around
`tuple_validate` without revealing access to tuple
internals.
Part of #5384
commit feb1fe53537ef49e54feffeabb5d8de7fb8d52a9
Author: Timur Safin <tsafin@tarantool.org>
Date: Tue Oct 13 02:41:37 2020 +0300
module api: box_ibuf_* wrappers
Introduced the bare minimum of ibuf accessors, which make
external merger possible:
- box_ibuf_reserve
- box_ibuf_read_range
- box_ibuf_write_range
Part of #5384
commit a312e27ebb369bf56f94cf878ddb0391c12d5b14
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Oct 12 20:34:20 2020 +0300
module api: add box_key_def_validate_full_key()
box_key_def_validate_key() verifies a probably partial key, while the
new function is to apply the extra restriction on the key part count: it
should be the same as in a given key definition.
Fixes #5273
commit 4a12985f1a2ad8779e7bb25565f5fc18fc8b55ae
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed Sep 16 22:45:16 2020 +0300
module api: add box_key_def_validate_key()
The function allows to verify a key against a key definition. It accepts
a partial key.
Part of #5273
commit 97ef85982b001977bc11687a8a718dca2f64763e
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed Sep 9 00:09:19 2020 +0300
module api: expose box_key_def_extract_key()
Unlike box_tuple_extract_key() it accepts a key_def structure, not
space_id, index_id pair.
Another difference from box_tuple_extract_key() is that this function
allows to pass so called multikey index. See commit 2.2.0-259-gf1d9f2575
('box: introduce multikey indexes in memtx') for details.
Note: The <multikey_idx> parameter is ignored on the backported version
of the patch on 1.10.
Part of #5273
commit 1327db7fed811fe24031025b4780d89ac9a81a4b
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Sep 8 22:48:49 2020 +0300
module api: expose box_key_def_merge()
Part of #5273
commit 4ce916e55b39202ce63b91de9476a3be17afce5f
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Sep 8 13:23:25 2020 +0300
module api: expose box_key_def_validate_tuple()
Part of #5273
commit 62a83e2a613d535ba4e21bf5b4680cbd450b8853
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Sep 21 18:42:59 2020 +0300
module api: add box_key_def_dump_parts()
The function dumps an opaque <box_key_def_t> structure into a non-opaque
array of <box_key_part_def_t> structures in order to allow an external
module to obtain information about the key definition.
Part of #5273
commit 741b11744d11db6cdf0151d4184b626cab4a06ab
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Sep 21 18:35:50 2020 +0300
module api: add box_key_def_new_v2()
Unlike box_key_def_new() it allows to set nullability, collation and
JSON path.
Note: JSON paths are not supported in the backported version of the
patch for 1.10.
Provided public non-opaque key part definition structure to create a key
definition. The next commit will also use this structure to dump a key
definition.
There are several technical points around the box_key_part_def_t
structure. They are mainly about providing stable ABI.
- Two uint32_t fields are placed first for better aligning of next
fields (pointers, which are usually 64 bit wide).
- A padding is used to guarantee that the structure size will remain the
same across tarantool versions. It allows to allocate an array of such
structures.
- The padding array is not a field of the structure itself, but added as
a union variant (see the code). It allows to get rid of manual
calculation of cumulative fields size, which is hard to do in a
platform independent way.
- A minimal size of the structure is guaranteed by the union with
padding, but a static assert is required to guarantee that the size
will not overrun the predefined value.
- PACKED is added as an extra remedy to make the structure layout
predictable (on given target architecture).
- A bit flag is used for is_nullable. bool is considered as too
expensive (it requires 8 bits). bitfields (int:1 and so on) do no
guarantee certain data layout (it is compiler implementation detail),
while a module is compiled outside of tarantool build and may use
different toolchain. A bit flag is the only choice.
- A collation is identified using a string. Different IDs may be used on
different tarantool instances for collations. The only 'real'
identifier is a collation name, so using it as identifier in the API
should be more convenient and less error-prone.
- A field type is also identified using a string instead of a number. We
have <enum field_type> in the module API, but it cannot be used here,
because IDs are changed across tarantool versions. Aside of this, size
of a enum is compiler defined. Anyway, we can expose field types as
numbers and implement number-to-name and name-to-number mapping
functions, but IMHO it would just add extra complexity.
The dependency on the fiber compilation unit is added for key_def: it is
purely to obtain the box region in the module API functions. The key_def
compilation unit does not use anything fiber related in fact.
Part of #5273
commit 8e9558475082bdf8383f239f19bfb9dcc4518510
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Sep 4 23:45:28 2020 +0300
module api: add API_EXPORT to key_def functions
It is the rule of thumb to use API_EXPORT with module API functions.
To be honest, I don't see strict reason to use this macro except for
unification with other module API functions.
It adds 'extern', which is default for functions.
It adds __attribute__((visibility("default"))), but symbols to be
exported are listed in extra/exports or src/export.h (depending of
tarantool version, see [1]).
It adds __attribute__((nothrow)), which maybe allows a compiler to
produce more optimized code and also catch an obvious problem and emit a
warning. I don't know.
So the reason for me is unification.
Part of #5273
[1]: 2.5.0-42-g03790ac55 ('cmake: remove dynamic-list linker option')
[2]: 1.6.8-71-g55605c5c9 ('Add __attribute__((nothrow)) to API_EXPORT macro')
commit ddceb1830b78bd543f3f46a1691899ca34a1e038
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Sep 8 16:56:48 2020 +0300
module api/lua: add API_EXPORT to tuple functions
The reason is unification of declarations. It is the rule of thumb to
use API_EXPORT with module API functions.
Part of #5273
commit 224988f6ef9157c0d9555887d56b63b27431958e
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Sep 7 23:05:36 2020 +0300
module api/lua: expose luaT_tuple_new()
It is convenient wrapper around box_tuple_new() to create a tuple from a
Lua table (or from another tuple).
Part of #5273
commit ab95ddaacaac30339ff3e5609adb08efdc6c9b62
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Oct 6 19:56:34 2020 +0300
module api/lua: add luaT_tuple_encode()
It is the same as luaT_tuple_new(), but returns raw MsgPack data (not
<box_tuple_t>) allocated on the box region.
The reason to expose this function is to provide ability to use
box_tuple_compare_with_key() function from an external module for a key
passed as a Lua table. The compare function has the following signature:
| API_EXPORT int
| box_tuple_compare_with_key(box_tuple_t *tuple_a, const char *key_b,
| box_key_def_t *key_def);
The second parameter is a key encoded as an MsgPack array, not a tuple
structure. So luaT_tuple_new() is not applicable here (it is not
worthful to create a tuple structure if we need just MsgPack data).
Some complexity was introduced to support encoding on the Lua shared
buffer and the box region both. The motivation is the following:
- luaT_tuple_encode() is exposed with encoding to the box region,
because it is more usual to the module API. In particular a user of
the API able to control when the tuple data should be released.
- Encoding to the Lua shared buffer is kept internally, because there is
no strong reason to change it to the box region for box.tuple.new().
Part of #5273
commit ec9a7fa7ebbb8fd6e15b9516875c3fd1a1f6dfee
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Oct 6 08:46:19 2020 +0300
lua: don't raise a Lua error from luaT_tuple_new()
This change fixes incorrect behaviour at tuple serialization error in
several places: <space_object>:frommap(), <key_def_object>:compare(),
<merge_source>:select(). See more in #5382.
Disallow creating a tuple from objects on the Lua stack (idx == 0) in
luaT_tuple_new() for simplicity. There are no such usages in tarantool.
The function is not exposed yet to the module API. This is only
necessary in box.tuple.new(), which anyway raises Lua errors by its
contract.
The better way to implement it would be rewritting of serialization from
Lua to msgpack without raising Lua errors, but it is labirous work. Some
day, I hope, I'll return here.
Part of #5273
Fixes #5382
commit 59149d6a9572ef7bf28e533c90c3c84fafc12afd
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Oct 5 22:09:27 2020 +0300
lua: factor out tuple encoding from luaT_tuple_new
It simplifies further changes around encoding a tuple: next commits will
get rid of throwing serialization errors and will expose a function to
encode a table (or tuple) from the Lua stack on the box region to the
module API.
Part of #5273
Part of #5382
commit 6a63b7c0c20b0bd2093b121b0ee7e29c2376f36f
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu Sep 3 13:10:38 2020 +0300
module api/lua: add luaL_iscdata() function
It is useful to provide a module specific error when cdata expected, but
a value of another type is passed.
Alternative would be using of lua_type() to check against LUA_TCDATA,
but this constant is not exposed for modules. See more in the
luaL_iscdata() API comment.
Part of #5273
commit c61179092c08ab1c3e693d4e17b479e119d1b07b
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Aug 4 01:51:28 2020 +0300
module api: expose box region
It is the better alternative to linking the small library directly to a
module. Why not just use the small library in a module?
Functions from an executable are preferred over ones that are shipped in
a dynamic library (on Linux, Mac OS differs), while a module may use the
small library directly. It may cause a problem when some functions from
the library are inlined, but some are not, and those different versions
of the library offer structures with different layouts. Small library
symbols may be exported by the tarantool executable after the change of
default symbols visibility (see [1]). See more details and examples in
[2].
So it is better to expose so called box region and get rid of all those
compatibility problems.
[1]: 2.5.0-42-g03790ac55 ('cmake: remove dynamic-list linker option')
[2]: https://lists.tarantool.org/pipermail/tarantool-discussions/2020-September/000095.html
Part of #5273
commit a25a3e0d2556b221b587793d383b23be246664a8
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Sat Sep 19 02:43:43 2020 +0300
module api: get rid of typedef redefinitions
Technically C99 forbids it. Clang complains about typedef redefinitions
when C99 is used (it is default prior to clang 3.6):
| error: redefinition of typedef 'box_tuple_t' is a C11 feature
| [-Werror,-Wtypedef-redefinition]
| error: redefinition of typedef 'box_key_def_t' is a C11 feature
| [-Werror,-Wtypedef-redefinition]
The generated module.h file should define a type using typedef once.
This patch moves extra definitions out of public parts of header files.
Reordered api headers to place usages of such types after definitions.
Set C99 for the module API test in order to catch problems of this kind
in a future. Fixed 'unused value' warnings, which appears after the
change (it is strange that -Wall was not passed here before).
Part of #5273
Fixes #5313
commit 0f575e01a20a2da76d73e72e77f6ebd0080ef1b1
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu Oct 15 15:51:22 2020 +0300
gitlab-ci: fix tag removal for a branch push job
We should remove a tag after fetching of a remote repository.
It is hotfix of commit 0f564f3497e6a8afd42da2175104118ba5213593
('gitlab-ci: remove tag from pushed branch commit').
Follows up #3745
Co-authored-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
commit c798e18c0cec0e6810cd667addc4cf03a1a23484
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Oct 15 13:08:34 2020 +0300
luajit: bump a new version
* jit: fix cdatanum addressing for GC64 mode on x86
commit 69fb3ccfa30ed82527d515627e868135df01d6d9
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Oct 15 12:20:34 2020 +0300
luajit: bump new version
* test: force enable assert checks in release build
commit 218ac1382db53d06d9edb1c6654d5f60cfdae15f
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Oct 14 19:08:10 2020 +0300
build: fix Werror warning in src/box/txn.c
Found on GCC 4.8.5 on CentOS 7 issue:
build/usr/src/debug/tarantool-2.6.0.144/src/box/txn.c:944:30: error: ‘limbo_entry’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
if (txn_limbo_wait_complete(&txn_limbo, limbo_entry) < 0)
Set limbo_entry variable to NULL on initialization.
Needed for #4941
commit 88ddc838194f3c2d52de26c3b9d3a6b5df447d1d
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Oct 15 11:39:31 2020 +0300
luajit: bump new version
* jit: fix cdatanum addressing for GC64 mode on x86
commit 0f564f3497e6a8afd42da2175104118ba5213593
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Oct 14 23:49:56 2020 +0300
gitlab-ci: remove tag from pushed branch commit
Drop a tag that points to a current commit (if any) on a job triggered
by pushing to a branch (as against of pushing a tag). Otherwise we may
get two jobs for the same x.y.z-0-gxxxxxxxxx build: one is run by
pushing a branch and another by pushing a tag. The idea is to hide the
new tag from the branch job as if a tag would be pushed strictly after
all branch jobs for the same commit.
Closes #3745
Co-authored-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 375f2d17924f350f49cfa1abb34fdb3289ea1188
Author: Nikita Pettik <korablev@tarantool.org>
Date: Fri Oct 2 14:26:16 2020 +0300
vinyl: remove squash procedures from source code
After previous commit, there's no need in these functions. However,
someday we may want to reconsider it squash optimization and make it
work without breaking upsert associativity rule. So to not lost initial
squash implementation, let's put its removal in a separate patch.
Follow-up #5107
commit 5a61c4717e63de24359a02c9f3e603444ab5b637
Author: Nikita Pettik <korablev@tarantool.org>
Date: Tue Jul 21 16:34:06 2020 +0300
vinyl: rework upsert operation
Previous upsert implementation had a few drawbacks which led to number
of various bugs and issues.
Issue #5092 (redundant update operations execution)
In a nutshell, application of upsert(s) (on top of another upsert)
consists of two actions (see vy_apply_upsert()): execute and squash.
Consider example:
insert({1, 1}) -- terminal statement, stored on disk
upsert({1}, {{'-', 2, 20}}) -- old ups1
upsert({1}, {{'+', 2, 10}}) -- new ups2
'Execute' step takes update operations from the new upsert and combines them
with key of the old upsert. {1} + {'+', 2, 10} can't be evaluated since
key consists of only one field. Note that in case upsert doesn't fold
into insert the upsert's tuple and the tuple stored in index can be
different. In our particular case, tuple stored on disk has two fields
({1, 1}), so first upsert's update operation can be applied to it:
{1, 1} + {'+', 2, 10} --> {1, 11}. If upsert's operation can't be executed
using key of old upsert, we simply continue processing squash step.
In turn 'squash' is a combination of update operations: arithmetic
operations are combined so we don't have to store actions over the same
field; the rest operations - are merged into single array. As a result,
we get one upsert with squashed operations: upsert({1}, {{'+', 2, -10}}).
Then vy_apply_upsert() is called again to apply new upsert on the top of
terminal statement - insert{1, 1}. Since now tuple has second field,
update operations can be executed and corresponding result is {1, -9}.
It is the final result of upsert application procedure.
Now imagine that we have following upserts:
upsert({1, 1}, {{'-', 2, 20}}) -- old ups1
upsert({1}, {{'+', 2, 10}}) -- new ups2
In this case execution successfully finishes and modifies old upsert's
tuple: {1, 1} + {'+', 2, 10} --> {1, 11}
However, we still have to squash/accumulate update operations since they
may be applied on tuple stored on disk later. After all, we have
following upsert: upsert({2, 11}, {{'+', 2, -10}}). Then it is applied
on the top of insert({1, 1}) and we get the same result as in the first
case - {1, -9}. The only difference is that upsert's tuple was modified.
As one can see, execution of update operations applied to upsert's tuple
is redundant in the case index already contains tuple with the same key
(i.e. when upserts turns into update). Instead, we are able to
accumulate/squash update operations only. When the last upsert is being
applied, we can either execute all update operation on tuple fetched
from index (i.e. upsert is update) OR on tuple specified in the first
upsert (i.e. first upsert is insert).
Issue #5105 (upsert doesn't follow associative property)
Secondly, current approach breaks associative property: after upsert's
update operations are merged into one array, part of them (related to
one upsert) can be skipped, meanwhile the rest - is applied. For
instance:
-- Index is over second field.
i = s:create_index('pk', {parts={2, 'uint'}})
s:replace{1, 2, 3, 'default'}
s:upsert({2, 2, 2}, {{'=', 4, 'upserted'}})
-- First update operation modifies primary key, so upsert must be ignored.
s:upsert({2, 2, 2}, {{'#', 1, 1}, {'!', 3, 1}})
After merging two upserts we get the next one:
upsert({2, 2, 2}, {{'=', 4, 'upserted'}, {'#', 1, 1}, {'!', 3, 1}}
While we executing update operations, we don't distinguish operations from
different upserts. Thus, if one operation fails, the rest are ignored
as well. As a result, first (in general case - all preceding squashed
upserts) upsert won't be applied, even despite the fact it is
absolutely correct. What is more, user gets no error/warning concerning
this fact.
Issue #1622 (no upsert result validation)
After upsert application, there's no check verifying that result
satisfies space's format: number of fields, their types, overflows etc.
Due to this tuples violating format may appear in the space, which in
turn may lead to unpredictable consequences.
To resolve these issues, let's group update operations of each upsert into
separate array. So that operations related to particular upsert are
stored in single array. In terms of previous example we will get:
upsert({2, 2, 2}, {{{'=', 4, 'upserted'}}, {{'#', 1, 1}, {'!', 3, 1}}}
Also note that we don't longer have to apply update operations on tuple
in vy_apply_upsert() when it comes for two upserts: it can be done once we
face terminal statement; or if there's no underlying statement (i.e. it is
delete statement or no statement at all) we apply all update arrays except
the first one on upsert's tuple. In case one of operations from array
fail, we skip the rest operations from this array and process to the
next array. After successful application of update operations of each
array, we check that the resulting tuple fits into space format. If they
aren't, we rollback applied operations, log error and moving to the next
group of operations.
Finally, arithmetic operations are not longer able to be combined: it is
requirement which is arises from #5105 issue. Otherwise, result of
upserts combination might turn out to be inapplicable to the tuple
stored on disk (e.g. result applied on tuple leads to integer overflow -
in this case only last upsert leading to overflow must be ignored).
Closes #1622
Closes #5105
Closes #5092
Part of #5107
commit 33870e37215bb64af04fb09da8ccc706e41c7f04
Author: Artem Starshov <artemreyt@tarantool.org>
Date: Wed Oct 7 16:09:24 2020 +0300
luacheck: fix warning in tarantoolctl.in
Changed passing global variable arg to function find_instance_name(arg)
instead of passing arg[0] and arg[2] separately. And removed exception
in .luacheckrc for file /extra/dist/tarantoolctl.in.
This change only solves linter warning, nothing else.
Fixed #4929.
Reviewed-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
commit 41ba1479abf630b378c3712ef52aad7a823f1b42
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 01:23:54 2020 +0200
qsync: reset confirmed lsn in limbo on owner change
Order of LSNs from different instances can be any. It means, that
if the synchronous transaction limbo changed its owner, the old
value of maximal confirmed LSN does not make sense. The new owner
means new LSNs, even less than the previously confirmed LSN of an
old owner.
The patch resets the confirmed LSN when the owner is changed. No
specific test for that, as this is a hotfix - tests start fail on
every run after a new election test was added in the previous
commit. A test for this bug will be added later.
Part of #5395
commit 4da30149c3ec9c3584a8a28af71417e72e5ce81e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 00:54:29 2020 +0200
raft: auto-commit transactions of the old leader
According to Raft, when a new leader is elected, it should finish
transactions of the old leader. In Raft this is done via adding a
new transaction originated from the new leader.
In case of Tarantool this can be done without a new transaction
due to WAL format specifics, and the function doing it is called
box_clear_synchro_queue().
Before the patch, when a node was elected as a leader, it didn't
finish the pending transactions. The queue clearance was expected
to be done by a user. There was no any issue with that, just
technical debt. The patch fixes it.
Now when a node becomes a leader, it finishes synchronous
transactions of the old leader. This is done a bit differently
than in the public box.ctl.clear_synchro_queue().
The public box.ctl.clear_synchro_queue() tries to wait for CONFIRM
messages, which may be late. For replication_synchro_timeout * 2
time.
But when a new leader is elected, the leader will ignore all rows
from all the other nodes, as it thinks it is the only source of
truth. Therefore it makes no sense to wait for CONFIRMs here, and
the waiting is omitted.
Closes #5339
commit 43d429697d5ddf0151d0ff95072e3455c79b9359
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 00:46:14 2020 +0200
raft: introduce on_update trigger
Raft state machine now has a trigger invoked each time when any of
the visible Raft attributes is changed: state, term, vote.
The trigger is needed to commit synchronous transactions of an old
leader, when a new leader is elected. This is done via a trigger
so as not to depend on box in raft code too much. That would make
it harder to extract it into a new module later.
The trigger is executed in the Raft worker fiber, so as not to
stop the state machine transitions anywhere, which currently don't
contain a single yield. And the synchronous transaction queue
clearance requires a yield, to write CONFIRM and ROLLBACK records
to WAL.
Part of #5339
commit 6b43f1031137edba070556fd9ea59f5d6490736c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 00:43:30 2020 +0200
raft: new candidate should wait for leader death
When a raft node was configured to be a candidate via
election_mode, it didn't do anything if there was an active
leader.
But it should have started monitoring its health in order to
initiate a new election round when it dies.
The patch fixes this bug. It does not contain a test, because will
be covered by a test for #5339.
Needed for #5339
commit da4998eaab83bfb5333bfd2642445d4942134e2f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 00:35:36 2020 +0200
raft: factor out the code to wakeup worker fiber
Raft has a worker fiber to perform async tasks such as WAL write,
state broadcast.
The worker was created and woken up from 2 places, leading at
least to code duplication. The patch wraps it into a new function
raft_worker_wakeup(), and uses it.
The patch is not need for anything functional, but was created
while working on #5339 and trying ideas. The patch seems to be
good refactoring making the code simpler, and therefore it is
submitted.
commit bd0da66905a93bfa076324fc39afa88a4639168f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Oct 14 00:29:24 2020 +0200
test: add '_stress' suffix to election_qsync test
The test is long, about 10 seconds. But its name is too general.
And it would be better used for a simpler more basic test. This is
going to happen in the next commits.
election_qsync.test.lua will check if the election and qsync work
fine together without any stress cases.
Needed for #5339
commit 77848aa81613174cf194338980a82a5a20420709
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Oct 13 16:15:28 2020 +0300
test: flaky vinyl/gh.test.lua fails on 427 line
Added new checksum for flaky fail on vinyl/gh.test.lua:427 line.
Part of #5141
commit d9d1deaca9ad15e4ff69fb2df88f33c8875fedc2
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Sep 29 12:06:25 2020 +0000
test: flaky replication/replica_rejoin.test.lua
On heavy loaded hosts found the following issue:
[151] --- replication/replica_rejoin.result Tue Sep 29 10:57:26 2020
[151] +++ replication/replica_rejoin.reject Tue Sep 29 10:57:48 2020
[151] @@ -230,7 +230,12 @@
[151] return box.info ~= nil and box.info.replication[1] ~= nil
[151] end)
[151] ---
[151] -- true
[151] +- error: "builtin/box/load_cfg.lua:601: Please call box.cfg{} first\nstack traceback:\n\tbuiltin/box/load_cfg.lua:601:
[151] + in function '__index'\n\t[string \"return test_run:wait_cond(function() ...\"]:1:
[151] + in function 'cond'\n\t/tmp/tnt/151_replication/test_run.lua:411: in function </tmp/tnt/151_replication/test_run.lua:404>\n\t[C]:
[151] + in function 'pcall'\n\tbuiltin/box/console.lua:402: in function 'eval'\n\tbuiltin/box/console.lua:708:
[151] + in function 'repl'\n\tbuiltin/box/console.lua:842: in function <builtin/box/console.lua:828>\n\t[C]:
[151] + in function 'pcall'\n\tbuiltin/socket.lua:1081: in function <builtin/socket.lua:1079>"
[151] ...
[151] test_run:wait_upstream(1, {message_re = 'Missing %.xlog file', status = 'loading'})
[151] ---
[151]
It happened because box.cfg was not ready to provide information. In
real there is no need to use local check for replication information
parts availablity, due to wait_upstream() function used below, do it
itself.
Part of #4985
commit e7203e389165d102930208dccdea05f54afd82ce
Author: Igor Munkin <imun@tarantool.org>
Date: Tue Oct 13 17:22:01 2020 +0300
build: another fix for luajit-tap tests cmake
Fixes the regression from e5039742eba89936bf2f8c5aebc8f21c810710d5
('luajit: bump new version').
Reported-by: Alexander Tikhonov <avtikhon@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
commit 9a8ac59c9aef7a7404a6d553eb2c1fa590d5ac0c
Author: Ilya Kosarev <i.kosarev@tarantool.org>
Date: Tue Oct 13 14:31:56 2020 +0300
key_def: support composite types extraction
key_def didn't support key definitions with array, map, varbinary & any
fields. Thus they couldn't be extracted with
key_def_object:extract_key(). Since the restriction existed due to
impossibility of such types comparison, this patch removes the
restriction for the fields extraction and only leaves it for
comparison.
Closes #4538
commit e5039742eba89936bf2f8c5aebc8f21c810710d5
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue Oct 13 16:14:11 2020 +0300
luajit: bump new version
* misc: add C and Lua API for platform metrics
* core: introduce various platform metrics
commit 3bc455f7dbf5617f09a37d50d84f26c8735be4c2
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Oct 2 15:11:55 2020 +0300
Add flaky tests checksums to fragile 2nd part
Added for tests with issues:
app/socket.test.lua gh-4978
box/access.test.lua gh-5411
box/access_misc.test.lua gh-5401
box/gh-5135-invalid-upsert.test.lua gh-5376
box/hash_64bit_replace.test.lua test gh-5410
box/hash_replace.test.lua gh-5400
box/huge_field_map_long.test.lua gh-5375
box/net.box_huge_data_gh-983.test.lua gh-5402
replication/anon.test.lua gh-5381
replication/autoboostrap.test.lua gh-4933
replication/box_set_replication_stress.test.lua gh-4992
replication/election_basic.test.lua gh-5368
replication/election_qsync.test.lua test gh-5395
replication/gh-3247-misc-iproto-sequence-value-not-replicated.test.lua gh-5380
replication/gh-3711-misc-no-restart-on-same-configuration.test.lua gh-5407
replication/gh-5287-boot-anon.test.lua gh-5412
replication/gh-5298-qsync-recovery-snap.test.lua.test.lua gh-5379
replication/show_error_on_disconnect.test.lua gh-5371
replication/status.test.lua gh-5409
swim/swim.test.lua gh-5403
unit/swim.test gh-5399
vinyl/gc.test.lua gh-5383
vinyl/gh-4864-stmt-alloc-fail-compact.test.lua test gh-5408
vinyl/gh-4957-too-many-upserts.test.lua gh-5378
vinyl/gh.test.lua gh-5141
vinyl/quota.test.lua gh-5377
vinyl/snapshot.test.lua gh-4984
vinyl/stat.test.lua gh-4951
vinyl/upsert.test.lua gh-5398
commit 8bcb6409da9e24365caacc623ce440eddb4c0016
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sun Oct 11 03:27:48 2020 +0300
test: enable flaky tests on FreeBSD 12
Testing on FreeBSD 12 had some tests previously blocked to avoid of
flaky fails. For now we have the ability to avoid of it in test-run
using checksums for fails with opened issues. So adding back 7 tests
to testing on FreeBSD 12.
Closes #4271
commit 389c12b487aeec9d50f33cc5af14fc5e5d7adb17
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Sep 30 12:20:08 2020 +0300
asan: add leak suppressions for flaky test
Met flaky issues on test:
replication/gh-3637-misc-error-on-replica-auth-fail.test.lua
Found memory leaks:
[093] Last 15 lines of Tarantool Log file [Instance "replica_auth"][/builds/DtQXhC5e/0/tarantool/tarantool/test/var/093_replication/replica_auth.log]:
[093] #3 0xa13df8 in coio_on_call /builds/DtQXhC5e/0/tarantool/tarantool/src/lib/core/coio_task.c:264:16
[093] #4 0xfcedbe in eio_execute /builds/DtQXhC5e/0/tarantool/tarantool/third_party/libeio/eio.c:2015:9
[093] #5 0xfcedbe in etp_proc /builds/DtQXhC5e/0/tarantool/tarantool/third_party/libeio/etp.c:373
[093] #6 0x7f8c8260ffa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2)
[093]
[093] Indirect leak of 4 byte(s) in 1 object(s) allocated from:
[093] #0 0x525dfa in calloc (/builds/DtQXhC5e/0/tarantool/tarantool/src/tarantool+0x525dfa)
[093] #1 0xa2eb4a in mh_i64ptr_new /builds/DtQXhC5e/0/tarantool/tarantool/src/lib/salad/mhash.h:408:22
[093] #2 0x8a516d in vy_recovery_new_f /builds/DtQXhC5e/0/tarantool/tarantool/src/box/vy_log.c:2321:23
[093] #3 0xa13df8 in coio_on_call /builds/DtQXhC5e/0/tarantool/tarantool/src/lib/core/coio_task.c:264:16
[093] #4 0xfcedbe in eio_execute /builds/DtQXhC5e/0/tarantool/tarantool/third_party/libeio/eio.c:2015:9
[093] #5 0xfcedbe in etp_proc /builds/DtQXhC5e/0/tarantool/tarantool/third_party/libeio/etp.c:373
[093] #6 0x7f8c8260ffa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2)
To stabilize testing these leaks added as suppressions to asan list.
Part of #5343
commit fa66c2954b37ea27779ba652aa415b3b29421f8c
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Mon Oct 5 12:04:04 2020 +0300
test: move error messages into logs gh-5383
Set error message to log output in test:
vinyl/gc.test.lua
commit e95aec95e864b380f5dd8fcf3d94d818ece01ba4
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sat Oct 3 20:12:48 2020 +0300
test: move error messages into logs gh-4984
Set error message to log output in test:
vinyl/snapshot.test.lua
commit c34d1d6756666687411ff01e38a5978711b210a6
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Sep 30 11:15:43 2020 +0300
test: move error messages into logs gh-5366
Set error message to log output in test:
replication/gh-4402-info-errno.test.lua
commit ca0c279985172f0da824be8dd34aef9ec76eac72
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Sep 29 10:35:17 2020 +0300
test: move error messages into logs gh-4985
Set error message to log output in test:
replication/replica_rejoin.test.lua
commit 1433ed8e8a76660fb92ae3bdbe1de7c188a88638
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Sep 29 10:34:53 2020 +0300
test: move error messages into logs gh-4940
Set error message to log output in test:
replication/gh-3160-misc-heartbeats-on-master-changes.test.lua
commit 69b5759c21b53d6bd9e1ea9e00c9a9a0657bb0b3
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue Oct 13 10:55:23 2020 +0300
test-run: bump new version
* Restart server on each failed test in worker
commit 24974f36a40523adc4aed6a69134e21c5439e27e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Oct 6 22:29:06 2020 +0200
raft: introduce election_mode configuration option
The new option can be one of 3 values: 'off', 'candidate',
'voter'. It replaces 2 old options: election_is_enabled and
election_is_candidate. These flags looked strange, that it was
possible to set candidate true, but disable election at the same
time. Also it would not look good if we would ever decide to
introduce another mode like a data-less sentinel node, for
example. Just for voting.
Anyway, the single option approach looks easier to configure and
to extend.
- 'off' means the election is disabled on the node. It is the same
as election_is_enabled = false in the old config;
- 'voter' means the node can vote and is never writable. The same
as election_is_enabled = true + election_is_candidate = false in
the old config;
- 'candidate' means the node is a full-featured cluster member,
which eventually may become a leader. The same as
election_is_enabled = true + election_is_candidate = true in the
old config.
Part of #1146
commit 0dc72812fb78a192945612f0e954026a0ffe4053
Author: Aleksandr Lyapunov <alyapunov@tarantool.org>
Date: Fri Jul 10 12:03:21 2020 +0300
Introduce fselect - formatted select
space:fselect and index:fselect fetch data like ordinal select,
but formats the result like mysql does - with columns, column
names etc. fselect converts tuple to strings using json,
extending with spaces and cutting tail if necessary. It is
designed for visual analysis of select result and shouldn't
be used stored procedures.
index:fselect(<key>, <opts>, <fselect_opts>)
space:fselect(<key>, <opts>, <fselect_opts>)
There are some options that can be specified in different ways:
- among other common options (<opts>) with 'fselect_' prefix.
(e.g. 'fselect_type=..')
- in special <fselect_opts> map (with or without prefix).
- in global variables with 'fselect_' prefix.
The possible options are:
- type:
- 'sql' - like mysql result (default).
- 'gh' (or 'github' or 'markdown') - markdown syntax, for
copy-pasting to github.
- 'jira' - jira table syntax (for copy-pasting to jira).
- widths: array with desired widths of columns.
- max_width: limit entire length of a row string, longest fields
will be cut if necessary. Set to 0 (default) to detect and use
screen width. Set to -1 for no limit.
- print: (default - false) - print each line instead of adding
to result.
- use_nbsp: (default - true) - add invisible spaces to improve
readability in YAML output. Not applicabble when print=true.
There is also a pair of shortcuts:
index/space:gselect - same as fselect, but with type='gh'.
index/space:jselect - same as fselect, but with type='jira'.
See test/engine/select.test.lua for examples.
Closes #5161
commit ab64b12098aab0f1c30a0ed2eef3090ac34a0c00
Author: Sergey Kaplun <skaplun@tarantool.org>
Date: Tue Oct 6 21:18:31 2020 +0300
fiber: fix build for disabled fiber top
In case when we build without `ENABLE_FIBER_TOP` neither
`struct fiber` contains `clock_stat` field nor `FIBER_TIME_RES`
constant is defined.
This patch adds corresponding ifdef directive to avoid compilation
errors.
commit 05c968ae4b43eff6644cd978a63770e088cb2101
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue Oct 6 09:12:11 2020 +0300
ci: add replace clang8 with clang11
Update Clang 8 testing with most recent version of
Clang: 11. Clang 8 still in the image.
Closes #5386
commit 459a65b7f831df2ed8b1ffaab3124324421e8c97
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue Oct 6 07:25:05 2020 +0300
Remove C99 extension usage from C++ file
File mp_error.cc was using a C99 feature called "designated
initializers" which was catched by Clang v12.
Remove this usage.
commit e7b6df1341c85756847960024398307c1e21f3d3
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Mon Oct 5 18:00:21 2020 +0300
vinyl: fix leak in vy_build_recover_stmt
Variable `delete` wasn't free()-ed in case of
error. Free it properly.
Found by static analyzer.
commit d3f1dd7208f8aee787e18a61d33c132706e1e32a
Author: Igor Munkin <imun@tarantool.org>
Date: Thu Oct 1 21:57:18 2020 +0300
lua: prohibit fiber yield when GC hook is active
While running GC hook (i.e. __gc metamethod) garbage collector engine
is "stopped": the memory penalty threshold is set to LJ_MAX_MEM and
incremental GC step is not triggered as a result. Ergo, yielding the
execution at the finalizer body leads to further running platform with
disabled LuaJIT GC. It is not re-enabled until the yielded fiber doesn't
get the execution back.
This changeset extends <cord_on_yield> routine with the check whether GC
hook is active. If the switch-over occurs in scope of __gc metamethod
the platform is forced to stop its execution with EXIT_FAILURE and calls
panic routine before the exit.
Relates to #4518
Follows up #4727
Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
commit 7e91e27258ca0e5aa3563500e97aab60eac5d9f3
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Fri Oct 2 13:23:49 2020 +0300
raft: add a test with synchronous replication
commit 2711797be8c2e864db874a3f3a8434bc182d2360
Author: Igor Munkin <imun@tarantool.org>
Date: Mon Sep 7 23:19:09 2020 +0300
lua: abort trace recording on fiber yield
Since Tarantool fibers don't respect Lua coroutine switch mechanism, JIT
machinery stays unnotified when one lua_State substitutes another one.
As a result if trace recording hasn't been aborted prior to fiber
switch, the recording proceeds using the new lua_State and leads to a
failure either on any further compiler phase or while the compiled trace
is executed.
This changeset extends <cord_on_yield> routine aborting trace recording
when the fiber switches to another one. If the switch-over occurs while
mcode is being run the platform finishes its execution with EXIT_FAILURE
code and calls panic routine prior to the exit.
Closes #1700
Fixes #4491
Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
commit a390ec55a623627ef86c8776828f300bdfa539fa
Author: Igor Munkin <imun@tarantool.org>
Date: Tue Sep 22 14:07:02 2020 +0300
fiber: introduce a callback for fibers switch-over
Tarantool integrates several complex environments together and there are
issues occurring at their junction leading to the platform failures.
E.g. fiber switch-over is implemented outside the Lua world, so when one
lua_State substitutes another one, main LuaJIT engines, such as JIT and
GC, are left unnotified leading to the further platform misbehaviour.
To solve this severe integration drawback <cord_on_yield> function is
introduced. This routine encloses the checks and actions to be done when
the running fiber yields the execution.
Unfortunately the way callback is implemented introduces a circular
dependency. Considering linker symbol resolving methods for static build
an auxiliary translation unit is added to the particular tests mocking
(i.e. exporting) <cord_on_yield> undefined symbol.
Part of #1700
Relates to #4491
Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
commit 7682d34162be34648172d91008e9185301bce8f6
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 18 12:45:03 2020 +0300
box/vynil: quota -- fix compilation warning
There is a mixture of types and clang prefer
explicit conversion (since @value is a double).
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 7d91ea62d6122f2fa92bde60491d424a817ebbc4
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 18 12:40:31 2020 +0300
box/sql: vdbemem -- fix compilation warning
d is "double" thus placate clang.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 7698abefe95893db75950d2969ea8c28d75ced43
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 18 12:38:37 2020 +0300
box/sql: vdbe -- fix compilation warning
pIn3->u.r is a "double", thus placate clang.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit b9bce9b404f219140d1dafd93b7d458c612bb8ce
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 18 12:35:52 2020 +0300
box/sql: func -- fix compilation warning
The @r is "double" value thus use explicit
conversion to placate clang compiler.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit 0ffba19727af3edfa37d4e7e0526e062161ec902
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 18 12:03:41 2020 +0300
test: unit/guava -- fix compilation warning
Convert to uint64_t explicitly.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit d25ecab48832340dc2ef9c072cf506924c03af52
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Wed Sep 30 17:40:25 2020 +0300
box/raft: fix compilation issue
Without explicit cast we're getting warnings.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
commit cf799645119907d646984ffe52624a8740e6ecbe
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Sep 16 00:53:39 2020 +0200
raft: add tests
Part of #1146
commit 15fc84496f340c643f0bd9c9d2ea3e98e02b715b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Sep 4 00:16:21 2020 +0200
raft: introduce box.info.election
Box.info.election returns a table of form:
{
state: <string>,
term: <number>,
vote: <instance ID>,
leader: <instance ID>
}
The fields correspond to the same named Raft concepts one to one.
This info dump is supposed to help with the tests, first of all.
And with investigation of problems in a real cluster.
The API doesn't mention 'Raft' on purpose, to keep it not
depending specifically on Raft, and not to confuse users who
don't know anything about Raft (even that it is about leader
election and synchronous replication).
Part of #1146
commit 27399889b178747ce773b17a0a25bd811579a8d1
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Sep 2 00:37:27 2020 +0200
raft: introduce state machine
The commit is a core part of Raft implementation. It introduces
the Raft state machine implementation and its integration into the
instance's life cycle.
The implementation follows the protocol to the letter except a few
important details.
Firstly, the original Raft assumes, that all nodes share the same
log record numbers. In Tarantool they are called LSNs. But in case
of Tarantool each node has its own LSN in its own component of
vclock. That makes the election messages a bit heavier, because
the nodes need to send and compare complete vclocks of each other
instead of a single number like in the original Raft. But logic
becomes simpler. Because in the original Raft there is a problem
of uncertainty about what to do with records of an old leader
right after a new leader is elected. They could be rolled back or
confirmed depending on circumstances. The issue disappears when
vclock is used.
Secondly, leader election works differently during cluster
bootstrap, until number of bootstrapped replicas becomes >=
election quorum. That arises from specifics of replicas bootstrap
and order of systems initialization. In short: during bootstrap a
leader election may use a smaller election quorum than the
configured one. See more details in the code.
Part of #1146
commit 67b60f08d8a937c30f0ab7ee52a56c260d9d8dbc
Author: sergepetrenko <sergepetrenko@tarantool.org>
Date: Tue Aug 18 14:49:14 2020 +0300
raft: relay status updates to followers
The patch introduces a new type of system message used to notify the
followers of the instance's raft status updates.
It's relay's responsibility to deliver the new system rows to its peers.
The notification system reuses and extends the same row type used to
persist raft state in WAL and snapshot.
Part of #1146
Part of #5204
commit 1d329f0bbbf98dff878588b2ad754e2dd18404a7
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed Sep 2 00:09:12 2020 +0200
raft: introduce box.cfg.election_* options
The new options are:
- election_is_enabled - enable/disable leader election (via
Raft). When disabled, the node is supposed to work like if Raft
does not exist. Like earlier;
- election_is_candidate - a flag whether the instance can try to
become a leader. Note, it can vote for other nodes regardless
of value of this option;
- election_timeout - how long need to wait until election end, in
seconds.
The options don't do anything now. They are added separately in
order to keep such mundane changes from the main Raft commit, to
simplify its review.
Option names don't mention 'Raft' on purpose, because
- Not all users know what is Raft, so they may not even know it
is related to leader election;
- In future the algorithm may change from Raft to something else,
so better not to depend on it too much in the public API.
Part of #1146
commit 4f0f7c8fc5a54a786d707399fdba466a3f74bdb8
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Aug 13 00:12:49 2020 +0200
raft: introduce persistent raft state
The patch introduces a sceleton of Raft module and a method to
persist a Raft state in snapshot, not bound to any space.
Part of #1146
commit 764a548a00ac37845e7fc10457e6188a8c1848fe
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Sep 3 22:02:19 2020 +0200
replication: track registered replica count
Struct replicaset didn't store a number of registered replicas.
Only an array, which was necessary to fullscan each time when want
to find the count.
That is going to be needed in Raft to calculate election quorum.
The patch makes the count tracked so as it could be found for
constant time by simply reading an integer.
Needed for #1146
commit 403357900583fd1147413a9afa19e6e53bc76321
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Sep 1 23:19:07 2020 +0200
wal: don't touch box.cfg.wal_dir more than once
Relay.cc and box.cc obtained box.cfg.wal_dir value using
cfg_gets() call. To initialize WAL and create struct recovery
objects.
That is not only a bit dangerous (cfg_gets() uses Lua API and can
throw a Lua error) and slow, but also not necessary - wal_dir
parameter is constant, it can't be changed after instance start.
It means, the value can be stored somewhere one time and then used
without Lua.
Main motivation is that the WAL directory path will be needed
inside relay threads to restart their recovery iterators in the
Raft patch. They can't use cfg_gets(), because Lua lives in TX
thread. But can access a constant global variable, introduced in
this patch (it existed before, but now has a method to get it).
Needed for #1146
commit 31dc4fafb2e7e1e98c2864c869f636bec034f6d7
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Sep 1 01:18:22 2020 +0200
box: introduce summary RO flag
An instance is writable if box.cfg.read_only is false, and it is
not orphan. Update of the final read-only state of the instance
needs to fire read-only update triggers, and notify the engines.
These 2 flags were easy and cheap to check on each operation, and
the triggers were easy to use since both flags are stored and
updated inside box.cc.
That is going to change when Raft is introduced. Raft will add 2
more checks:
- A flag if Raft is enabled on the node. If it is not, then Raft
state won't affect whether the instance is writable;
- When Raft is enabled, it will allow writes on a leader only.
It means a check for being read-only would look like this:
is_ro || is_orphan || (raft_is_enabled() && !raft_is_leader())
This is significantly slower. Besides, Raft somehow needs to
access the read-only triggers and engine API - this looks wrong.
The patch introduces a new flag is_ro_summary. The flag
incorporates all the read-only conditions into one flag. When some
subsystem may change read-only state of the instance, it needs to
call box_update_ro_summary(), and the function takes care of
updating the summary flag, running the triggers, and notifying the
engines.
Raft will use this function when its state or config will change.
Needed for #1146
commit 7c14819f3abbd076574fb4a08d14f531f6327884
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Sep 1 00:53:43 2020 +0200
applier: store instance_id in struct applier
Applier is going to need its numeric ID in order to tell the
future Raft module who is a sender of a Raft message. An
alternative would be to add sender ID to each Raft message, but
this looks like a crutch. Moreover, applier still needs to know
its numeric ID in order to notify Raft about heartbeats from the
peer node.
Needed for #1146
commit 90108875a8ceea7bed0a5a3f22102fee0cefad45
Author: Sergey Kaplun <skaplun@tarantool.org>
Date: Mon Sep 21 08:18:04 2020 +0300
httpc: src/httpc.c missed va_end() macro
Found and fixed not closed va_list 'ap' with cppcheck:
[src/httpc.c:190]: (error) va_list 'ap' was opened but not closed by va_end().
commit c5cb8d311514fa8b95716d0050f98b85e29fa765
Author: Roman Khabibov <roman.habibov@tarantool.org>
Date: Fri Sep 11 09:33:39 2020 +0300
box: disallow to alter SQL view
Ban ability to modify view on box level. Since a view is a named
select, and not a table, in fact, altering view is not a valid
operation.
commit 75ba744b394a26a79dd514d3c1bb49ffb08b434c
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Sat Sep 26 15:36:18 2020 +0300
Add flaky tests checksums to fragile
Added for tests with issues:
app/fiber.test.lua gh-5341
app-tap/debug.test.lua gh-5346
app-tap/http_client.test.lua gh-5346
app-tap/inspector.test.lua gh-5346
box/gh-2763-session-credentials-update.test.lua gh-5363
box/hash_collation.test.lua gh-5247
box/lua.test.lua gh-5351
box/net.box_connect_triggers_gh-2858.test.lua gh-5247
box/net.box_incompatible_index-gh-1729.test.lua gh-5360
box/net.box_on_schema_reload-gh-1904.test.lua gh-5354
box/protocol.test.lua gh-5247
box/update.test.lua gh-5247
box-tap/net.box.test.lua gh-5346
replication/autobootstrap.test.lua gh-4533
replication/autobootstrap_guest.test.lua gh-4533
replication/ddl.test.lua gh-5337
replication/gh-3160-misc-heartbeats-on-master-changes.test.lua gh-4940
replication/gh-3247-misc-iproto-sequence-value-not-replicated.test.lua.test.lua gh-5357
replication/gh-3637-misc-error-on-replica-auth-fail.test.lua gh-5343
replication/long_row_timeout.test.lua gh-4351
replication/on_replace.test.lua gh-5344, gh-5349
replication/prune.test.lua gh-5361
replication/qsync_advanced.test.lua gh-5340
replication/qsync_basic.test.lua gh-5355
replication/replicaset_ro_mostly.test.lua gh-5342
replication/wal_rw_stress.test.lua gh-5347
replication-py/multi.test.py gh-5362
sql/prepared.test.lua test gh-5359
sql-tap/selectG.test.lua gh-5350
vinyl/ddl.test.lua gh-5338
vinyl/gh-3395-read-prepared-uncommitted.test.lua gh-5197
vinyl/iterator.test.lua gh-5336
vinyl/write_iterator_rand.test.lua gh-5356
xlog/panic_on_wal_error.test.lua gh-5348
commit 641c1710ccb7eef4001f4b6d5cf73b48b1da1b65
Author: Sergey Kaplun <skaplun@tarantool.org>
Date: Sun Sep 20 20:42:28 2020 +0300
alter: box/alter.cc null pointer dereference
Found and fixed Null pointer dereference with cppcheck:
[src/box/alter.cc:395]: (error) Null pointer dereference
commit 58f84d1a0d39c25de93ae60a8d9f76f2803b9042
Author: Sergey Kaplun <skaplun@tarantool.org>
Date: Mon Sep 21 20:41:45 2020 +0300
fiber: src/lua/fiber.c null pointer dereference
[src/lua/fiber.c:245] -> [src/lua/fiber.c:217]: (warning) Either the condition 'if(func)' is redundant or there is possible null pointer dereference: func.
commit 3cf741b911250220522d0a2ab29e096e37eac4ee
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Sat Sep 26 22:21:32 2020 +0300
test: update test-run
Updated test_run:wait_upstream() and test_run:wait_downstream() to wait
until box will be configured and an instance with given ID will appear
in box.info.replication.
See https://github.com/tarantool/test-run/issues/221
Fixes #5317
Fixes #5329
commit 59dca1898c7c95c0c0c29a926d766ced9ffb0acc
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Sep 25 21:47:47 2020 +0300
test: update test-run
Justify columns in the output.
https://github.com/tarantool/test-run/pull/222
commit 6b98017e63d5239148073c7881c9693f96949b54
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Sep 25 19:04:15 2020 +0300
test: fix mistake in replication/suite.ini
Removed dust line from merge.
commit 74328386da14d461d86aeb06dfada1f9a0410422
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Jul 8 14:49:56 2020 +0300
Enable test reruns on failed fragiled tests
In test-run implemented the new format of the fragile lists based on
JSON format set as fragile option in 'suite.ini' files per each suite:
fragile = {
"retries": 10,
"tests": {
"bitset.test.lua": {
"issues": [ "gh-4095" ],
"checksums": [ "050af3a99561a724013995668a4bc71c", "f34be60193cfe9221d3fe50df657e9d3" ]
}
}}
Added ability to check results file checksum on tests fail and
compare with the checksums of the known issues mentioned in the
fragile list.
Also added ability to set 'retries' option, which sets the number
of accepted reruns of the tests failed from 'fragile' list that
have checksums on its fails.
Closes #5050
commit bb856247cd6573b21bcfdbe9bacef8e4397db2b5
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Aug 19 10:46:08 2020 +0300
test: flaky replication/anon.test.lua test
Found flaky issues multi running replication/anon.test.lua test
on the single worker:
[007] --- replication/anon.result Fri Jun 5 09:02:25 2020
[007] +++ replication/anon.reject Mon Jun 8 01:19:37 2020
[007] @@ -55,7 +55,7 @@
[007]
[007] box.info.status
[007] | ---
[007] - | - running
[007] + | - orphan
[007] | ...
[007] box.info.id
[007] | ---
[094] --- replication/anon.result Sat Jun 20 06:02:43 2020
[094] +++ replication/anon.reject Tue Jun 23 19:35:28 2020
[094] @@ -154,7 +154,7 @@
[094] -- Test box.info.replication_anon.
[094] box.info.replication_anon
[094] | ---
[094] - | - count: 1
[094] + | - count: 2
[094] | ...
[094] #box.info.replication_anon()
[094] | ---
[094]
It happend because replications may stay active from the previous
runs on the common tarantool instance at the test-run worker. To
avoid of it added restarting of the tarantool instance at the very
start of the test.
Closes #5058
