Git log: commit 6332aca655ae7f95d391bdc0109e79915f6e6ad0
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Aug 26 16:44:58 2019 +0300
relay: join new replicas off read view
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.
Closes #1271
commit 6c5d3f060459c943bbba8394e7f1f91f79efb5a6
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Sep 2 13:24:30 2019 +0300
lua: pwd: split data fetch from deserialization
This commit does not change a user visible behaviour. It refactors pwd
module to explicitly divide code that fetches data from passwd / group
databases from code that performs deserialization of the data to Lua
tables.
The idea of splitting of those actions appears when it was observed that
a call of getpw() / getgr() leads to problems on some systems when it is
invoked during passwd / group database traveral.
Now it is more obvious that we don't call getpw() during passwd
traversal and getgr() during group traveral.
Follows up #4428 and #4447.
commit 34d9bd5854d1468137eeb7d9ca898a72daa98dc5
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 30 17:03:21 2019 +0300
readme: add GitLab-CI badge
Aside of that fixed other badges URLs to point to master branch.
(cherry picked from commit 7965aaaa88e6eaf266d2e51776598ac05a4a6cde)
commit 9701200430bb0b83c9bbad1b1b4df306486901a8
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Sep 9 18:06:33 2019 +0300
box: fix compilation with old gcc
LTO build fails on warning message:
In file included from /tarantool/src/lib/core/diag.h:33:0,
from /tarantool/src/box/engine.h:36,
from /tarantool/src/box/memtx_engine.h:40,
from /tarantool/src/box/memtx_engine.c:31:
/tarantool/src/box/memtx_engine.c: In function
'metmx_tuple_chunk_delete':
/tarantool/src/trivia/util.h:201:49: error: initialization from
incompatible pointer type [-Werror]
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
container_of((typeof(tuple_chunk->data) *)data,
^
/tarantool/src/trivia/util.h:201:49: error: (near initialization for
'tuple_chunk') [-Werror]
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
container_of((typeof(tuple_chunk->data) *)data,
Closes #4438
commit 7cf84a54b5c228d37d8e42ce85cc24ee3cea6a94
Author: Mergen Imeev <imeevma@gmail.com>
Date: Tue Jul 16 12:43:51 2019 +0300
sql: test suite for BOOLEAN
This patch provides a test suite that allows us to make sure that
the SQL BOOLEAN type works as intended.
Part of #4228
commit 037bd58cdb4b2aaebeacffddd49e731d311d6638
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Sep 2 11:30:02 2019 +0300
replication: disallow bootstrap of read-only masters
In a configuration with several read-only and read-write instances, if
replication_connect_quorum is not greater than the amount of read-only
instances and replication_connect_timeout happens to be small enough
for some read-only instances to form a quorum and exceed the timeout
before any of the read-write instaces start, all these read-only
instances will choose themselves a read-only bootstrap leader.
This 'leader' will successfully bootstrap itself, but will fail to
register any of the other instances in _cluster table, since it isn't
writeable. As a result, some of the read-only instances will just die
unable to bootstrap from a read-only bootstrap leader, and when the
read-write instances are finally up, they'll see a single read-only
instance which managed to bootstrap itself and now gets a
REPLICASET_UUID_MISMATCH error, since no read-write instance will
choose it as bootstrap leader, and will rather bootstrap from one of
its read-write mates.
The described situation is clearly not what user has hoped for, so
throw an error, when a read-only instance tries to initiate the
bootstrap. The error will give the user a cue that he should increase
replication_connect_timeout.
Closes #4321
@TarantoolBot document
Title: replication: forbid to bootstrap read-only masters.
It is no longer possible to bootstrap a read-only instance in an emply
data directory as a master. You will see the following error trying to
do so:
```
ER_BOOTSTRAP_READONLY: Trying to bootstrap a local read-only instance as master
```
Now if you have a fresh instance, which has
`read_only=true` in an initial `box.cfg` call, you need to set up
replication from an instance which is either read-write, or has your
local instance's uuid in its `_cluster` table.
In case you have multiple read-only and read-write instances with
replication set up, and you still see the aforementioned error message,
this means that none of your read-write instances managed to start
listening on their port before read_only instances have exceeded the
`replication_connect_timeout`. In this case you should raise
`replication_connect_timeout` to a greater value.
commit 435753030b2f2ec69fd83cc6616f2e6ab5698182
Author: Igor Munkin <imun@tarantool.org>
Date: Thu Sep 5 14:28:44 2019 +0300
test: move luajit-tap suite to luajit repo
* All test chunks related to luajit were moved from tarantool source
tree to the luajit repo
* Adjusted CMakeLists via creating a symlink to luajit test directory
to fix out-of-source tests
Closed #4478
commit 4ba57f2f5469dcebede87f164fb8345ba5ccddca
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Tue Sep 10 22:12:37 2019 +0300
luajit: bump new version
commit c7a382af0d35a36f8e2b05a09faeb9d877df947c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Sep 7 15:43:06 2019 +0200
wal: deprecate rows_per_wal in favour of wal_max_size
rows_per_wal does not allow to properly limit size of WAL logs.
A user would need to estimate size of each WAL row to do that.
Now WAL supports option wal_max_size allowing to limit WAL size
much more precise.
Part of #3762
commit 664788a3127632e1b8932999bfaf8524f0500c67
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Aug 19 15:12:03 2019 +0300
lua_cjson: fix segfault on recursive table encoding
The json.encode() used to cause a segfault in case of recursive
table:
tbl = {}
tbl[1] = tbl
json.encode(tbl)
Library doesn't test whether given object on Lua stack parsed
earlier, because it performs a lightweight in-depth traverse
of Lua stack. However it must stop when encode_max_depth is
reached (by design).
Tarantool's lua_cjson implementation has a bug introduced during
porting original library: it doesn't handle some corner cases:
entering into a map correctly increases a current depth, while
entering into an array didn't. This patch adopts author's
approach to check encode_max_depth limit. Thanks to handling this
constraint correctly the segfault no longer occurs.
Closes #4366
commit a1814bb68021504e66747845c4303241a31a204c
Author: Mike Siomkin <msiomkin@mail.ru>
Date: Tue Mar 19 16:35:06 2019 +0300
httpc: add proxy server options for curl
Added support for the following curl options:
* CURLOPT_PROXY
* CURLOPT_PROXYPORT
* CURLOPT_PROXYUSERPWD
* CURLOPT_NOPROXY
@TarantoolBot document
Title: httpc: proxy server options
Use 'proxy' option to specify the proxy server host or IP-address
(optionally may be prefixed with a scheme - e.g. http:// or https://).
'proxy_port' and 'proxy_user_pwd' options may be used to specify the
proxy port (443 for https proxy and 1080 for others by default) and
user credentials (format: [user name]:[password]) respectively.
If 'proxy' option is not set a value from the corresponding
environment variable will be used. Environment variable names are:
'http_proxy', 'https_proxy', 'ftp_proxy' etc. 'all_proxy' variable
is used if no protocol specific proxy was set.
Setting 'proxy' option to an empty string will explicitly disable the
use of a proxy, even if there is an environment variable set for it.
Set 'no_proxy' option to specify a comma separated list of hosts that
do not require a proxy to get reached, even if one is specified by
'proxy' option (or the corresponding environment variable). The only
wildcard available is a single * character, which matches all hosts,
and effectively disables the proxy. 'no_proxy' environment variable
will be used if this option is not set.
Setting 'no_proxy' option to an empty string will explicitly enable
the proxy for all host names, even if there is an environment variable
set for it.
See:
https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html
https://curl.haxx.se/libcurl/c/CURLOPT_PROXYPORT.html
https://curl.haxx.se/libcurl/c/CURLOPT_USERPWD.html
https://curl.haxx.se/libcurl/c/CURLOPT_NOPROXY.html
commit 40c94af74188e5e179536b93a8f3d40f3d857c4d
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Mon Sep 2 08:20:07 2019 +0300
test: avoid of too long running sql-tap tests
The test sql-tap/gh-3083-ephemeral-unref-tuples.test.lua took
too long runtime and failed to finish in timeout limit of 2
minutes when it runs near the end of the running queue:
No output during 120 seconds. Will abort after 120 seconds without output. List of workers not reporting the status:
- 012_sql-tap [sql-tap/gh-3083-ephemeral-unref-tuples.test.lua, vinyl] at var/012_sql-tap/gh-3083-ephemeral-unref-tuples.result:0
Test hung! Result content mismatch:
[File does not exists: sql-tap/gh-3083-ephemeral-unref-tuples.result]
[Main process] No output from workers. It seems that we hang. Send SIGKILL to workers; exiting...
Due to suggestion from Konstantin Osipov at gh-3845:
https://github.com/tarantool/tarantool/issues/3845#issue-385735959
The following tests:
gh-3083-ephemeral-unref-tuples.test.lua
gh-3332-tuple-format-leak.test.lua
where set to run only on memtx configuration.
Also the test:
gh-3083-ephemeral-unref-tuples.test.lua
was removed from the test-run 'fragile' option flaky list.
Closed #4128
commit 51d8e4c35ca1da255b4ebea9850ec2a84365e0a3
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed Aug 28 21:13:22 2019 +0300
box/memtx: strip_core -- Warn on linux only
We know that madvise(MADV_DONTDUMP) is present
on linux based platforms only (macos doesn't
support it at all, freebsd requires MADV_NOCORE
or similar which is unsupported by now) thus
we should not print a warning on other systems
to not spam users.
Fixes #4464
commit d4a7459e794e265432f39392ed5cbcfe14d61d72
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Aug 8 16:31:52 2019 +0300
sql: support user-defined functions in SQL
Closes #2200
Closes #4113
Closes #2233
@TarantoolBot document
Title: The box.internal.sql_function_create is forbidden
Legacy mechanism box.internal.sql_function_create to make some
Lua function available in SQL is forbidden now.
To make some function available in SQL you need to use
box.schema.func.create() mechanism: you need to specify
1) function language and language-specific options(e.g. you
are able to define a persistent Lua function)
2) whether this function is_deterministic or not: deterministic
functions allows to generate more efficient SQL VDBE bytecode
so you better specify it when it is true
3) the function returns type: a Tarantool type string describes
a type of value returned by function
4) param_list - a table of Tarantool's types strings desccribe
function argument types
5) exports - a table of Tarantool's frontends where this function
should be available ('LUA' by default). You need to specify
{'LUA', 'SQL'} to make function available both in SQL requests
and visible in box.func folder
Example:
-- Case1: C function
-- function1.so has int divide() symbol
box.schema.func.create("function1.divide", {language = 'C',
returns = 'number',
param_list = {'number', 'number'},
is_deterministic = true,
exports = {'LUA', 'SQL'}})
box.execute('SELECT "function1.divide"(6, 3)')
- metadata:
- name: '"function1.divide"(6, 3)'
type: number
rows:
- [2]
box.schema.func.drop("function1.divide")
-- Case2: Persistent Lua function
box.schema.func.create("SUMMARIZE", {language = 'LUA',
returns = 'number',
body = 'function (a, b) return a + b end',
param_list = {'number', 'number'},
is_deterministic = true,
exports = {'LUA', 'SQL'}})
box.execute('SELECT summarize(1, 2)')
- metadata:
- name: summarize(1, 2)
type: number
rows:
- [3]
box.schema.func.drop("summarize")
Moreover there is a special predefined Lua function LUA that
allows to evaluate a custom Lua expressions in SQL.
You need to pass a string in form "return ...." to LUA function
that returns more than one value of any type.
Example:
box.execute('SELECT lua(\'return 1 + 1\')')
- metadata:
- name: lua('return 1 + 1')
type: any
rows:
- [2]
box.execute('SELECT lua(\'return box.cfg.memtx_memory\')')
- metadata:
- name: lua('return box.cfg.memtx_memory')
type: any
rows:
- [268435456]
commit 45396002d4eb32453eb2f26e995ad01767bae896
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Aug 7 13:33:48 2019 +0300
sql: get rid of FuncDef function hash
Now it is possible to move all SQL builtin functions to
Tarantool's function hash. An existent FuncDef function
representation was replaced with func_sql_builtin class.
It has a sql-specific method :call and :finalize, while
port API call is not supported and protected with stubs.
This patch removes FuncDef hash and sql_function_create endpoint,
but doesn't introduce something instead. Therefore few affected
tests are disabled. A required functionality would be fixed in
the next patch.
Following tests using sql_create_function are broken now.
They are going to be fixed in the next commit:
sql-tap/alias.test.lua sql-tap/check.test.lua
sql-tap/func5.test.lua sql-tap/lua_sql.test.lua
sql-tap/subquery.test.lua sql-tap/trigger9.test.lua
sql/errinj.result sql/errinj.test.lua
sql/func-recreate.test.lua
Part of #2200, #4113, #2233
@TarantoolBot document
Title: SQL builtins priveleges
All SQL built-ins are executed on SQL privilege level that is
undefined yet.
commit 5b8aebd6973fce8fe6d4183b1bf8f1eb64b5c2ab
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Fri Aug 23 15:34:36 2019 +0300
sql: remove name overloading for SQL builtins
Now all SQL builtins are unique, i.e. each function has the only
hash table entry. This technique requires to remove static argc
checks for some builtins(substr, round, like, trim, count,
group_concat). Now they raise a runtime error in case of invalid
usage.
Updated error messages correspondingly, to provide more
informative messages in such cases. Taking into account upcoming
changes, all built-in functions names are uppercased.
Needed for #2200, #4113, #2233
commit ddc83e0494b7355e50f92ae86603192f6c89c447
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Aug 21 16:11:56 2019 +0300
sql: replace flag MINMAX with flags MIN and MAX
Replaced sql function flag SQL_FUNC_MINMAX with couple new flags
SQL_FUNC_MIN and SQL_FUNC_MAX. This allows to distinguish MIN and
MAX function by flag instead of using user_data context.
This allows to delete user_data field in FundDef object in
further refactoring.
Needed for #2200, #4113, #2233
commit 025f79bd7b5850fa01454fa8ac70492d9bcf2b5f
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Aug 21 13:08:23 2019 +0300
sql: rename sql_vdbe_mem_alloc_region helper
Changed sql_vdbe_mem_alloc_region routine name to the
vdbe_mem_alloc_blob_region because we are going to introduce
a new function with a really similar (but more appropriate) name
vdbemem_alloc_on_region in following patch. Due to the fact that
the function used only in vdbe.c module it has been moved there
and has been marked as "static".
Needed for #2200, #4113, #2233
commit 4125c4ef33414f023d920be216c957d1b80253e9
Author: Nikita Pettik <korablev@tarantool.org>
Date: Tue Aug 27 20:17:04 2019 +0300
sql: make GREATEST/LEAST built-ins accept at least two args
Before a46b52004 SQL implementation featured min()/max() functions
overloading: if one argument was passed, then aggregate version would be
invoked; otherwise - scalar one. We decided to get rid of it and rename
scalar version to LEAST()/GREATEST() correspondingly. However, assertion
inside their implementations has been remained: it verifies that number
of passed arguments is greater than 1. On the other hand, now one can
pass literally any number of arguments to this function, including one
(which results in fired mentioned assertion) and zero (which leads to
NULL dereference in expr.c: these functions are marked with
SQL_FUNC_NEEDCOLL flag, and as a consequence they are assumed to have at
least one argument). Firstly, let's place check that number of passed
arguments more than one. Secondly, let's not assume that functions with
SQL_FUNC_NEEDCOLL must have any arguments.
Closes #4453
commit 165f8ee6f5cd9cf95baf6cffcc4e1d278cf80e4c
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Aug 27 18:02:57 2019 +0300
Set fragile option to flaky tests
Added "fragile" option to the flaky tests that are
not intended to be run in parallel with others.
Option set at the suite.ini file at the appropriate
suites with comments including the issue that stores
the fail.
commit 05477787e5584d4f71a17e40ac5b8d58c065610a
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Aug 27 19:31:15 2019 +0300
travis-ci: remove obsolete jobs on travis
Removed the jobs that is interesting to check on gitlab-ci
instead of travis-ci:
- osx 13 "Sierra"
- all LTO jobs
- ASAN job
Part of #4410
commit ff14626dc5110d0db866a5f4b0e66f1fdae4b43c
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Thu Aug 29 11:17:13 2019 +0300
Destroy port after iproto eval if transaction isn't finished
This is a followup for 7691154af1a51a620db2d3cc80bf8b8d8ca18714
commit 7691154af1a51a620db2d3cc80bf8b8d8ca18714
Author: Maria Khaydich <maria.khaydich@tarantool.org>
Date: Tue Aug 27 19:37:56 2019 +0300
Iproto call won't leak if transaction isn't committed
In case of throwing client error because of inactive function
we did not destroy used port. It could possibly cause huge
memory leaks as could be seen with top or its analogues when
performing net.box test run in a loop.
Closes #4388
commit c80e9416950ac3dff7429a951a1603a3aa15c178
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Aug 29 02:09:21 2019 +0300
Add missing test for _say patch
It turned out that patch d0e38d5 wasn't accompanied
w/ a test. Add proper check.
commit 5a0cfe025735dbb12486971c299803ae9a3ff46b
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Aug 28 11:25:00 2019 +0300
replication: enter orphan mode on manual replication configuration chage
Currently we only enter orphan mode when instance fails to connect to
replication_connect_quorum remote instances during local recovery.
On bootstrap and manual replication configuration change an error is
thrown. We better enter orphan mode on manual config change, and leave
it only in case we managed to sync with replication_connect_quorum
instances.
Closes #4424
@TarantoolBot document
Title: document reaction on error in replication configuration change.
Now when issuing `box.cfg{replication={uri1, uri2, ...}}` and failing to
sync with replication_connect_quorum remote instances, the server will
throw an error, if it is bootstrapping, and just set its state to orphan
in all other cases (recovering from existing xlog/snap files or manually
changing box.cfg.replication on the fly). To leave orphan mode, you may
wait until the server manages to sync with replication_connect_quorum
instances.
In order to leave orphan mode you need to make the server sync with
enough instances. To do so, you may either:
1) set replication_connect_quorum to a lower value
2) reset box.cfg.replication to exclude instances that cannot
be reached or synced with
3) just set box.cfg.replication to "" (empty string)
commit a850acfd02ed8d68b754f1c44f08bd92ddae04a1
Author: Konstantin Osipov <kostja.osipov@gmail.com>
Date: Tue Aug 27 15:42:46 2019 +0300
replication: do not abort replication on ER_UNKNOWN_REPLICA
In 3-node replica-set, registering with the leader node
does not guarantee the registration record arrives to the second
peer immediately. The joining node may bootstrap faster than
the registration record arrives to the second peer, in which
case replication will fail to create a full mesh.
commit 83ef5a17e3ef23aa7a3d95a7929207b10c55363f
Author: Konstantin Osipov <kostja@tarantool.org>
Date: Wed Aug 28 12:53:23 2019 +0300
systemd: Do nothing if NOTIFY_SOCKET env variable is not set.
A follow up on gh-4305, fix failing args.test.py
commit ef14050f0bbd400a0a53c21e35b7574686d00ba4
Author: Konstantin Osipov <kostja@tarantool.org>
Date: Wed Aug 28 12:42:16 2019 +0300
Fix build failure on Linux.
commit 1e509dde3c85052f3118e099925645c959a2d185
Author: Max Melentiev <m.melentiev@corp.mail.ru>
Date: Thu Jul 25 10:05:24 2019 +0300
Enable support for NOTIFY_SOCKET in envs without systemd
To make it possible to develop and test related features on
systems without systemd.
WITH_SYSTEMD cmake flag is used to generate systemd related files:
unit, generator script, etc. To keep this behavior and make it possible
to use NOTIFY_SOCKET without other systemd-related stuff,
I added WITH_NOTIFY_SOCKET cmake flag.
It also required some changes to support other OS:
SOCK_CLOEXEC (not available on macOS) flag for socket()
is replaced with `fcntl(fd, F_SETFD, FD_CLOEXEC)` which has the same effect.
MSG_NOSIGNAL flag for sendmsg is also not available on macOS.
However it has SO_NOSIGPIPE flag for setsockopt which disables SIGPIPE.
So it requires different solution for different OS. Inspired by
https://nwat.xyz/blog/2014/01/16/porting-msg_more-and-msg_nosigpipe-to-osx/
Have to reduce send-buffer size to 4MB because larger values
are not supported on macOS by default. This value should be enough
for all systems because notification messages are usually less than 1KB.
Fixes #4436
commit 2cdfaf3bdfb43ae4c5ada4231ff35eee582f4d2f
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Aug 20 17:06:34 2019 +0300
build: fix linking with static openssl library
System-wide dynamic libraries usually (always?) have NEEDED and RUNPATH
tags in a dynamic section (as `readelf -d /usr/lib/lib<...>.so` shows),
so when we link, say, with libssl.so, which depends on libz.so, a linker
does not complain against unresolved symbols that can be found in Z
library (if it is installed within a system).
Things are different when we linking with a static library. Say, when we
linking with libssl.a, which contains an unresolved symbol from Z
library, a linker reports an error. It is not possible to store an
information where to find unresolved symbols (NEEDED / RUNPATH) in a
static library (AFAIK).
We depend on three libraries that are depend on Z library: libcurl,
libssl and libcrypto (two latter are part of OpenSSL). When one of those
libraries is linked statically we should link with libz.so or libz.a
(depending on BUILD_STATIC flag). The patch doing exactly this.
The patch changes OPENSSL_LIBRARIES variable to fix the issue with
static linking of OpenSSL libraries. It also changes CURL_LIBRARIES in
the same way, however this does not alter any visible behaviour, because
OPENSSL_LIBRARIES is added to CURL_LIBRARIES. The latter change was made
to unify the way to choose libraries to link with: it is pure
refactoring part.
Fixes #4437.
commit 9ee5cf82ae579d62543a4bb335dd1e99b00dacf6
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 29 15:33:46 2018 +0300
tuple: implement update by field name
Tuple fields can be named, accessed by name, indexed by name, but
till this commit field names couldn't be used in update
operations. Now it is possible.
This patch is a teaser of updates by JSON path.
Part of #1261
commit 6d4430d0c1e220d50818e5119016516c127236d5
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Jun 5 21:37:00 2018 +0300
int96: add a missing header
commit 74ae67805943eb91338af54652680437623105e2
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 30 18:17:36 2018 +0300
tuple: relax struct tuple_update dependency on rope
Rope will not be used by tuple_update directly in
next patches.
commit baa4659c1857027b3ae53305baaa147af4803cad
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Apr 28 01:11:19 2018 +0300
rope: make rope library macro template
Rope library stores alloc, split and free functions by pointer,
that is a huge slog at performance, as any other virtual
function call. There is no reason, why the rope library may not
become a template, except historical ones.
The patch not only removes virtual functions, but also makes rope
deletion almost no-op in case if free() function is not defined.
A second motivation point of that patch is that original rope
structure was too big. Again, because it stored several pointers
to functions. In forthcoming patches on #1261 multiple ropes can
be created per each update, so it makes sense to reduce size of
this structure.
commit dba9dba70eebc875bea79e9a1357eb4240350fa1
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Apr 28 00:19:34 2018 +0300
tuple: remove alloc and alloc_ctx args from update()
They are always region_aligned_alloc and region pointer. Lets use
them always inside tuple_update.c, with no necessity to pass
explicitly.
The patch is motivated by forthcoming updates by JSON path, which
will strongly complicate and perhaps slow down the tuple_update.c
code. The present patch as well as some next ones should smooth
these problems.
commit 5a46639f4ebf793840046b50e21e12d1c8feb8d5
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 23 16:07:28 2019 +0300
Use forked tarantool/curl repository
A manual action is needed after pulling of this commit to actually use
the downstream repository instead of the upstream one:
sed -i -e 's@https://github.com/curl/curl.git@https://github.com/tarantool/curl.git@' .git/config
It is part of our processes to use forked repositories for submodules.
Suggested by Konstantin Osipov.
commit 421d440bb23c415810491383014ffd0689b8695a
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Thu Aug 22 09:26:46 2019 +0300
gitlab-ci: clean up .gitlab-ci.yml
Cleaned up the .gitlab-ci.yml file from duplicating
code - added templates that storres the needed
configuration values for different jobs.
Also moved the static_build from 'deploy' tag to
'deploy_test' tag to be sure that tests will not
be run under high load.
commit 24e75aa8e95aed0c2f667c7772e2b8fa725f4947
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Aug 27 11:48:01 2019 +0300
lua: fix decimal comparison with box.NULL
This problem is similar to the one fixed in commit
3c6c1cc96a1a510493c21c472565d4063e403ba2 (lua:fix decimal comparison
with nil)
We should handle box.NULL the same way.
Closes #4454
commit 418ab1722e0e49253ae254c4893d24285708b1c6
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue Aug 27 14:56:49 2019 +0300
test: update test-run
* Added "fragile" option with a list of tests that are not intended to
be run in parallel with others. The format of the list is the same as
for "disabled" option (#187, PR #188).
* Ensured that non-parallel test suites and fragile tests will be run
only when all parallel ones will be finished (PR #188).
* Fixed reporting in case of lack of temporary result (PR #172).
commit 7753d350ffccad4344485bfd6b9ce66fec813bea
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 23 02:58:28 2019 +0300
lua: pwd: fix passwd and group traversal
CentOS 6 and FreeBSD 12 implementations of getpwuid() rewind
setpwent()-{getpwent()}-endpwent() loop to a start that leads to a hang
during pwd.getpwall() invoke. The same is true for a getgrgid() call
during setgrent()-{getgrent()}-endgrent() loop.
The commit modifies pwd module to avoid getpwuid() calls during passwd
database traversal and to avoid getgrgid() calls when traversing groups.
The commit also fixes the important regression on CentOS 6 after
f5d8331e833d6a29dc7a869e681e03fb8e03a5f8 ('lua: workaround
pwd.getpwall() issue on Fedora 29'): tarantool hungs during startup due
to added getpwall() call. This made tarantool unusable on CentOS 6 at
all.
Aside of that the commit fixes another pwd.getgrall() problem: the
function gaves password entries instead of group entries.
Fixes #4428.
Fixes #4447.
Part of #4271.
commit 6c627af390c3fd521ffef77c443aeb3f5cb0b713
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 23 04:13:58 2019 +0300
test: tarantoolctl: verify delayed box.cfg()
app-tap.tarantoolctl.test.lua fails after
17df9edf013fd262be1b2ffd34d5080d20bc716e ('tarantoolctl: allow to start
instances with delayed box.cfg{}').
The commit fixes the test case that did check that an error is reported
if box.cfg() was not called in an instance script.
Follows up #4435.
Fixes #4448.
commit 5b9f207df47442e8cd76d181d0051dd6056487d6
Author: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>
Date: Tue Aug 20 18:37:35 2019 +0300
feedback: unify payload generation logic
This change is related to #4391. The objective was to collect additional
information about modules, but it's hard to do without changing API.
This patch will allow to monkey-patch report generation and achieve the
same results without interfering the daemon behavior.
commit 89aae30c88491fbaca43f7fd417d9b715c0a1800
Author: Max Melentiev <m.melentiev@corp.mail.ru>
Date: Tue Aug 20 14:36:51 2019 +0300
systemd: replace sendmsg with sendto shortcut
There is a problem with calculating .msg_namelen field
of msghdr struct. Instead of
.msg_name = &sa,
.msg_namelen = sizeof(sa.sun_family) + strlen(sd_unix_path),
it must set as
.msg_namelen = sizeof(sa) // larger value than current invalid one
It works on linux but when I tried to enable this feature for macOS
it didn't (maybe because of different order of fields in the struct).
Instead of fixing calculation, I've replaced original sendmsg call
with sendto, because it's a convenient shortcut which
simplifies code and can prevent such mistakes.
Required for #4436
commit 17df9edf013fd262be1b2ffd34d5080d20bc716e
Author: Max Melentiev <m.melentiev@corp.mail.ru>
Date: Mon Aug 19 10:35:55 2019 +0300
tarantoolctl: allow to start instances with delayed box.cfg{}
`tarantoolctl start` patches box.cfg two times:
1) before the init script to set default values and enforce some others,
2) after the init script to prevent changing a pid_file in runtime.
The second patching fails if an init file does not call
box.cfg{} before it's finished. This can take a place in apps with
managed instances which receive configuration from external server.
This patch moves the second patching into the box.cfg
wrapper created during the first patching. So the second patching
is performed only after box.cfg{} was invoked, so it does not fail anymore.
However there is relatively minor flaw for applications that
invoke box.cfg{} after init script is finished:
`tarantoolctl start` goes to background only when box.cfg{} is called.
Though this is not the case for daemon management systems like systemd,
as they handle backgrounding on their side
Fixes #4435
@TarantoolBot document
Title: tarantoolctl allows to start instances without a box.cfg{}
tarantoolctl now works for instances without box.cfg{} or
with delayed box.cfg{} call. This can be managed instances which receive
configuration from external server.
For such instances `tarantoolctl start` goes to background when
box.cfg{} is called, so it will wait until options for box.cfg are received.
However this is not the case for daemon management systems like systemd,
as they handle backgrounding on their side.
commit 72dbeb0e2a92e089ba661fad1dbf10d4b75233ef
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon Aug 5 19:16:19 2019 +0300
Move mp_compare_double_uint64() to trivia/util.h
This function implements common way of precise comparison between
unsigned integer and floating point values (doubles). Currently, it is
used in tuple comparators, but we need the same thing in SQL. Hence,
let's move it to header containing set of utilities.
commit 73a4a525489dc8541864008eb043f6dfd630e28d
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon Aug 5 21:39:34 2019 +0300
sql: use double_compare_uint64() for int<->float cmp
To compare floating point values and integers in SQL functions
compare_uint_float() and compare_int_float() are used. Unfortunately,
they contain bug connected with checking border case: that's not correct
to cast UINT64_MAX (2^64 - 1) to double. Proper way is to use exp2(2^64)
or predefined floating point constant. To not bother fixing function
which in turn may contain other tricky places, let's use instead already
verified double_compare_uint64(). So that we have unified way of
integer<->float comparison.
commit 45c2576d9a6fd10b8e40a07f70fa7465f42e0f7b
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 16 04:32:35 2019 +0300
gitlab-ci: fix building of Debian Buster image
`apt-get update <...>` fails on Debian Buster on docker_bootstrap goal
(see #4331 for the similar issue).
Added a description how to change dependencies in .travis.mk.
commit 1013a744be7890cc8f21a4a2dbc58ab497869964
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon Aug 19 16:26:22 2019 +0300
sql: allow to specify engine in CREATE TABLE stmt
Closes #4422
@TarantoolBot document
Title: Introduce <WITH ENGINE> clause for CREATE TABLE statement
To allow user to specify engine as per table option, CREATE TABLE
statement has been extended with optional <WITH ENGINE = engine_name>
clause. This clause comes at the end of CREATE TABLE statement.
For instance:
CREATE TABLE t_vinyl (id INT PRIMARY KEY) WITH ENGINE = 'vinyl';
Name of engine is considered to be string literal ergo should be
enclosed in single quotation marks and be lower-cased. Note that engine
specified in WITH ENGINE clause overwrites default engine, which is set
via 'pragma sql_default_engine'.
commit 35177fe0e9bdc6888ffc01b214ebf988d3668cca
Author: Nikita Pettik <korablev@tarantool.org>
Date: Sun Aug 18 01:21:47 2019 +0300
Do not log error in engine_find()
Error logging in engine_find() seems to be redundant: error message is
displayed twice (since its callers always push error on the top). Let's
remove this duplication.
commit 8b8b689596a9d54ec5f5250236c4e5e71edbe8e4
Author: Nikita Pettik <korablev@tarantool.org>
Date: Fri Aug 16 21:13:52 2019 +0300
txn: erase old savepoint in case of name collision
Name duplicates are allowed for savepoints (both in our SQL
implementation and in ANSI specification). ANSI SQL states that previous
savepoint should be deleted. What is more, our doc confirms this fact
and says that "...it is released before the new savepoint is set."
Unfortunately, it's not true - currently old savepoint remains in the
list. For instance:
SAVEPOINT t;
SAVEPOINT t;
RELEASE SAVEPOINT t;
RELEASE SAVEPOINT t; -- no error is raised
Let's fix this and remove old savepoint from the list.
commit 0a92ec7e399cda3bd9dc754d3bf277f6fd6f8e38
Author: Nikita Pettik <korablev@tarantool.org>
Date: Sun Aug 4 21:01:47 2019 +0300
sql: use struct txn_savepoint as anonymous savepoint
This allows us to completely remove SQL specific struct Savepoint and
use instead original struct txn_savepoint.
commit 56096ff2dc1404437999746c6b812204aefb984d
Author: Nikita Pettik <korablev@tarantool.org>
Date: Fri Aug 2 19:40:55 2019 +0300
txn: merge struct sql_txn into struct txn
This procedure is processed in several steps. Firstly, we add name
to struct txn_savepoint since we should be capable of operating on named
savepoints (which in turn is SQL feature). Still, anonymous (in the sense
of name absence) savepoints are also valid. Then, we add list (as
implementation of stailq) of savepoints to struct txn: it allows us to
find savepoint by its name. Finally, we patch rollback to/release
savepoint routines: for rollback tail of the list containing savepoints
is cut (but subject of rollback routine remains in the list); for
release routine we cut tail including node being released.
commit 5259274dbf3a293616761005824aa2e0434243df
Author: Nikita Pettik <korablev@tarantool.org>
Date: Fri Aug 2 17:01:35 2019 +0300
txn: move fk_deferred_count from psql_txn to txn
We are going to merge struct psql_txn with struct txn as a part of SQL
integration into NoSQL, so let's move counter of deferred foreign key
violations directly to struct txn.
commit 8317189a9b1f8351dabaecc8216a1c65b175d227
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Fri Aug 9 15:39:33 2019 +0300
box: add support for decimals in update ops
Closes #4413
@TarantoolBot document
Title: update operations on decimal fields.
tuple:update and space:update now support deicmal operands for
arithmetic operations ('+' and '-'). The syntax is as usual:
```
d = box.tuple.new(decimal.new('1'))
---
...
d:update{{'+', 1, decimal.new('0.5')}}
---
- [1.5]
...
```
Insertion ('!') and assignment ('=') are also supported:
```
a = decimal.new('1')
---
...
b = decimal.new('1e10')
---
...
c = decimal.new('1e-10')
---
...
d = box.tuple.new{5, a, 6, b, 7, c, "string"}
---
...
d
---
- [5, 1, 6, 10000000000, 7, 0.0000000001, 'string']
...
d:update{{'!', 3, dec.new('1234.5678')}}
---
- [5, 1, 1234.5678, 6, 10000000000, 7, 0.0000000001, 'string']
...
d:update{{'=', -1, dec.new('0.12345678910111213')}}
---
- [5, 1, 6, 10000000000, 7, 0.0000000001, 0.12345678910111213]
When performing an arithmetic operation ('+', '-'), where either the
updated field or the operand is decimal, the result will be decimal.
When both the updated field and the operand are decimal, the result
will, of course, be decimal.
...
```
commit 8ba72670fa2658b1d28c799f5e927a4780380914
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Aug 7 14:48:58 2019 +0300
decimal: allow to index decimals
Closes #4333
@TarantoolBot document
Title: Document decimal field type.
Decimals may now be stored in spaces. A corresponding field type is
introduced: 'decimal'. Decimal values are also allowed in 'scalar',
'any' and 'number' fields.
'decimal' field type is appropriate for both memtx HASH and TREE
indices, as well as for vinyl TREE index.
```
To create an index 'pk' over a decimal field, say
```
tarantool> box.space.test:create_index('pk', {parts={1, 'decimal'}})
---
- unique: true
parts:
- type: decimal
is_nullable: false
fieldno: 1
id: 0
space_id: 512
type: TREE
name: pk
...
```
Now you can insert some decimal values:
```
tarantool> for i = 1,10 do
> box.space.test:insert{decimal.new((i-5)/10)}
> end
---
...
```
tarantool> box.space.test:select{}
---
- - [-0.4]
- [-0.3]
- [-0.2]
- [-0.1]
- [0]
- [0.1]
- [0.2]
- [0.3]
- [0.4]
- [0.5]
...
```
Decimals may alse be inserted into `scalar` and `number` fields. In this
case all the number values are sorted correctly:
```
tarantool> box.schema.space.create('test')
tarantool> box.space.test:create_index('pk', {parts={1, 'number'}})
tarantool> box.space.test:insert{-1.0001, 'number'}
---
- [-1.0001, 'number']
...
tarantool> box.space.test:insert{decimal.new(-1.00001), 'decimal'}
---
- [-1.00001, 'decimal']
...
tarantool> box.space.test:insert{-1, 'number'}
---
- [-1, 'number']
...
tarantool> box.space.test:insert{decimal.new(-0.999), 'decimal'}
---
- [-0.999, 'decimal']
...
tarantool> box.space.test:insert{-0.998, 'number'}
---
- [-0.998, 'number']
...
tarantool> box.space.test:insert{-0.9, 'number'}
---
- [-0.9, 'number']
...
tarantool> box.space.test:insert{-0.95, 'number'}
---
- [-0.95, 'number']
...
tarantool> box.space.test:insert{decimal.new(-0.92), 'decimal'}
---
- [-0.92, 'decimal']
...
tarantool> box.space.test:insert{decimal.new(-0.971), 'decimal'}
---
- [-0.971, 'decimal']
...
tarantool> box.space.test:select{}
---
- - [-1.0001, 'number']
- [-1.00001, 'decimal']
- [-1, 'number']
- [-0.999, 'decimal']
- [-0.998, 'number']
- [-0.971, 'decimal']
- [-0.95, 'number']
- [-0.92, 'decimal']
- [-0.9, 'number']
...
```
Uniqueness is also preserved between decimals and other number types:
```
tarantool> box.space.test:insert{-0.92}
---
- error: Duplicate key exists in unique index 'pk' in space 'test'
...
tarantool> box.space.test:insert{decimal.new(-0.9)}
---
- error: Duplicate key exists in unique index 'pk' in space 'test'
...
```
You can also set decimal fields in space format:
```
tarantool> _ = box.schema.space.create('test')
---
...
tarantool> _ = box.space.test:create_index('pk')
---
...
tarantool> box.space.test:format{{name='id', type='unsigned'}, {name='balance', type='decimal'}}
---
...
tarantool> box.space.test:insert{1}
---
- error: Tuple field 2 required by space format is missing
...
tarantool> box.space.test:insert{1, 'string'}
---
- error: 'Tuple field 2 type does not match one required by operation: expected decimal'
...
tarantool> box.space.test:insert{1, 1.2345}
---
- error: 'Tuple field 2 type does not match one required by operation: expected decimal'
...
tarantool> box.space.test:insert{1, decimal.new('1337.420')}
---
- [1, 1337.420]
...
```
commit 8d2e7839cf66bf7e20a0dbc818786d36f76f43ec
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Aug 5 15:14:19 2019 +0300
decimal: add conversions to (u)int64_t
Update decNumber library, add methods to convert decimals to uint64_t
and int64_t, add unit tests.
Also replace decimal_round() function with decimal_round_with_mode() to
allow setting rounding mode.
We need to round with mode DEC_ROUND_DOWN in to_int64 conversions in
order to be consistent with double to int conversions.
It will be needed to compute hints for decimal fields.
Prerequisite #4333
commit 485439e33196e26d120e622175f88b4edc7a5aa1
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Thu Jun 20 11:15:12 2019 +0300
decimal: allow to encode/decode decimals as MsgPack
This patch adds the methods necessary to encode and decode decimals to
MsgPack. MsgPack EXT type (MP_EXT) together with a new extension type
MP_DECIMAL is used as a record header.
The decimal MsgPack representation looks like this:
+--------+-------------------+------------+===============+
| MP_EXT | length (optional) | MP_DECIMAL | PackedDecimal |
+--------+-------------------+------------+===============+
The whole record may be encoded and decoded with
mp_encode_decimal() and mp_decode_decimal(). This is equivalent to
performing mp_encode_extl()/mp_decode_extl() on the first 3 fields and
decimal_pack/unpack() on the PackedDecimal field.
It is also possible to decode and encode decimals to msgpack from lua,
which means you can insert decimals into spaces, but only into unindexed
fields for now.
Follow up #692
Part of #4333
commit e2d9f66450298b025ed350a2a40d2682b3b6f17e
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Aug 6 11:40:59 2019 +0300
lua/pickle: fix a typo
Prior to this patch format checking was broken for 'i' (integer) and 'N'
(big-endian integer). pickle.pack() rejected negative integers with
these formats. Fix this
commit fb3a2dfa2993ce2c81792b121e8532b6f811b9fb
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Aug 5 14:14:25 2019 +0300
decimal: fix encoding numbers with positive exponent.
When a number having a positive exponent is encoded, the internal
decPackedFromNumber function returns a negative scale, which differs
from the scale, returned by decimal_scale(). This leads to errors in
decoding. Account for negative scale in decimal_pack() and
decimal_unpack().
Follow-up #692
commit 3c6c1cc96a1a510493c21c472565d4063e403ba2
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Aug 6 17:05:42 2019 +0300
lua: fix decimal comparison with nil
Previously decimal comparison with nil failed with following error:
`expected decimal, number or string as 2 argument`.
Fix this. Throw a more verbose error in case of '>', '<', '>=', '<='
and fix equality check.
Follow-up #692
commit 7e51aebb9cd496b1b50b766ce00bf69a20a6d4b1
Author: Mergen Imeev <imeevma@gmail.com>
Date: Fri Aug 16 04:40:01 2019 +0300
build: link libcurl statically from a submodule
Hold libcurl-7.65.3. This version is not affected by the following
issues:
* #4180 ('httpc: redirects are broken with libcurl-7.30 and older');
* #4389 ('libcurl memory leak');
* #4397 ('HTTPS seem to be unstable').
After this patch libcurl will be statically linked when
ENABLE_BUNDLED_LIBCURL option is set. This option is set by default.
Closes #4318
@TarantoolBot document
Title: Tarantool dependency list was changed
* Added build dependencies: autoconf, automake, libtool, zlib-devel
(zlib1g-dev on Debian).
* Added runtime dependencies: zlib (zlib1g on Debian).
* Removed build dependencies: libcurl-devel (libcurl4-openssl-dev on
Debian).
* Removed runtime dependencies: curl.
The reason is that now we use compiled-in libcurl: so we don't depend on
a system libcurl, but inherit its dependencies.
commit f5d8331e833d6a29dc7a869e681e03fb8e03a5f8
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Aug 12 17:48:52 2019 +0300
lua: workaround pwd.getpwall() issue on Fedora 29
This is a workaround for systemd-nss issue:
https://github.com/systemd/systemd/issues/9585
The following error is observed on app-tap/pwd.test.lua on Fedora 29
(glibc-2.28-26.fc29, systemd-239-12.git8bca462.fc29) when tarantool is
linked with libcurl w/o GSS-API support:
| builtin/pwd.lua:169: getpwall failed [errno 2]: No such file or directory
Such tarantool build lacks of libselinux.so.1 transitive dependency
(tarantool -> libcurl.so.4 -> libgssapi_krb5.so.2 -> libkrb5support.so.0
-> libselinux.so.1) and strace shows the following calls when
pwd.getpwall() is invoked first time:
| openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 7A
| <...>
| access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or directory)
It looks like a part of libselinux initialization code and is invoked
during execution of a last ffi.C.getpwent() call that returns `nil` as a
result and left errno set to ENOENT. Our pwd module set errno to zero
before getpwent() call and expects that it will be preserved if no
unrecoverable errors occur. It seems that this expectation is not meet
due to the systemd-nss issue linked above.
Second and next getpwall() calls will succeed, so the commit adds an
extra getpwall() during pwd module load. This workaround is disabled on
FreeBSD due to another issue: #4428 ('getpwall() hangs on FreeBSD 12').
See also the previous related commit:
efccac691a4609c11f555a11704e91af4cda0836 ('lua: fix error handling in
getpwall and getgrall').
Follows up #3766.
Part of #4318.
commit f7509186dcf064e9b01ded3ec3b59aeab1ea4805
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Aug 20 11:55:18 2019 +0300
gitlab-ci: add static build
Added static build using Dockerfile on Centos 7 for release
commit criteria only. Added the cleanup for cmake generating
CMakeCache.txt files and CMakeFiles directories to avoid of
cmake localy created setup failing inside the docker after
the whole tarantool path was copied into it. Added testing
into the static build, running only when RUN_TESTS environment
variable set to non empty value, used in gitlab-ci job to run
the testing after the build.
Closes #3668
commit d7387ec99083e2132ac1d1a0889ccbf75b498dc5
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Aug 8 18:07:01 2019 +0300
vinyl: don't exempt dropped indexes from dump and compaction
We remove an LSM tree from the scheduler queues as soon as it is
dropped, even though the tree may hang around for a while after
that, e.g. because it is pinned by an iterator. As a result, once
an index is dropped, it won't be dumped anymore - its memory level
will simply disappear without a trace. This is okay for now, but
to implement snapshot iterators we must make sure that an index
will stay valid as long as there's an iterator that references it.
That said, let's delay removal of an index from the scheduler queues
until it is about to be destroyed.
commit 1314b95b6ec85154b80ddb24a4e6b9bf37fe09e1
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Aug 19 19:17:14 2019 +0300
space: get rid of apply_initial_join_row method
There's no reason to use a special method instead of the generic
space_execute_dml for applying rows received from a master during the
initial join stage. Moreover, using the special method results in not
running space.before_replace trigger, which makes it impossible to, for
example, update space engine on a replica, see the on_schema_init test
of the replication test suite.
So this patch removes the special method altogether and makes the code
that used it switch to space_execute_dml.
Closes #4417
commit 6136f84f844307b7df08730090f76e337f43792f
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Aug 8 19:54:57 2019 +0300
memtx: enter small delayed free mode from snapshot iterator
We must enable SMALL_DELAYED_FREE_MODE to safely use a memtx snapshot
iterator. Currently, we do that in checkpoint related callbacks, but if
we want to reuse snapshot iterators for other purposes, e.g. feeding
a read view to a newly joined replica, we better hide this code behind
snapshot iterator constructors.
commit 823312219b9a10225f3664caf5a1e88c3b16041b
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Aug 8 15:13:13 2019 +0300
memtx: use ref counting to pin indexes for snapshot
Currently, to prevent an index from going away while it is being
written to a snapshot, we postpone memtx_gc_task's free() invocation
until checkpointing is complete, see commit 94de0a081b3a ("Don't take
schema lock for checkpointing"). This works fine, but makes it rather
difficult to reuse snapshot iterators for other purposes, e.g. feeding
a consistent read view to a newly joined replica.
Let's instead use index reference counting for pinning indexes for
checkpointing. A reference is taken in a snapshot iterator constructor
and released when the snapshot iterator is destroyed.
commit 96a7ae062f9e86a1dbe6f01bbb84b78dd1c7c1fe
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Aug 19 17:00:53 2019 +0300
vinyl: get rid of vy_env::join_lsn
This fake LSN counter, which is used for assigning LSNs to Vinyl
statements during the initial join stage, was introduced a long time
ago, when LSNs were used as identifiers for lsregion allocations and
hence were supposed to grow strictly monotonically with each new
transaction. Later on, they were reused for assigning unique LSNs to
identify indexes in vylog.
These days, we don't need initial join LSNs to be unique, as we switched
to generations for lsregion allocations while in vylog we now use LSNs
only as an incarnation counter, not as a unique identifier. That said,
let's zap vy_env::join_lsn and simply assign 0 to all statements
received during the initial join stage.
To achieve that, we just need to relax an assertion in vy_tx_commit()
and remove the assumption that an LSN can't be zero in the write
iterator implementation.
commit 02da82ea7e80bba6371a85d8c41b338dbcd85b73
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Aug 8 12:37:47 2019 +0300
vinyl: don't pin index for iterator lifetime
vinyl_iterator keeps a reference to the LSM tree it was created for
until it is destroyed, which may take indefinitely long in case the
iterator is used in Lua. Actually, we don't need to keep a reference to
the index for the whole iterator lifetime, because iterator_next()
wrapper guarantees that iterator->next won't be called for a dropped
index. What we need to do is keep a reference while we are yielding on
disk read, similarly to vinyl_index_get().
Currently, pinning an index for indefinitely long is harmless, because
an LSM tree is exempted from dump/compaction as soon as it is dropped so
we just pin some memory, that's all. However, following patches are
going to enable dump/compaction for dropped but pinned indexes in order
to implement snapshot iterator so we better relax the dependency of an
iterator on an index know.
While we are at it, let's remove env and lsm members of vinyl_iterator
struct: lsm can be accessed via vy_read_iterator embedded in the struct
while env is only needed to access iterator_pool so we better store a
pointer to the pool in vinyl_iterator instead.
commit c33f08049072c060c8896cd5abf86b2f9554f3b1
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Aug 15 17:27:30 2019 +0300
sql: remove SQL_FUNC_SLOCHNG flag
The SQL_FUNC_SLOCHNG flag was useful for datetime function
that are currently not supported. So it could be removed.
Needed for #2200, #4113, #2233
commit 2437b0599f1e3b08503381ab2f680b0d8fe800ad
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Aug 5 17:28:52 2019 +0300
sql: rename OP_Function to OP_BuiltinFunction
Renamed OP_Function opcode to OP_BuiltinFunction to introduce a
new OP_Function operation in a new meaning: a new OP_Function
would call Tarantool's function with new port-based API while
legacy OP_BuiltinFunction is an efficient implementation of
SQL Builtins functions.
Needed for #2200, #4113, #2233
commit 1d69f568d6558b392a5e3588b2d3da6c3f694bfe
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Aug 7 12:56:14 2019 +0300
sql: rework SQL_FUNC_COUNT flag semantics
Tarantool's SQL engine generates a different VDBE bytecode
for ..COUNT(*).. and ..COUNT(fieldname).. operations:
the first one produces a lightweight OP_Count operation that uses
native mechanism to report the count of record in index while
the second one pessimistically opens a space read iterator and
uses Count aggregate function.
A helper routine is_simple_count decides whether such
optimisation is correct. It used to use SQL_FUNC_COUNT flag to
mark a dummy (non-functional) function entry with 0 arguments.
This patch changes SQL_FUNC_COUNT semantics: now it is a marker
of any COUNT function, while is_simple_count relies on count
of arguments to distinguish aggregate and non-aggregate
functions.
Needed for #2200, #4113, #2233
commit 77e4c842c1df157d9d152ae5b1ca0983a3b8235a
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Aug 5 17:02:30 2019 +0300
sql: wrap all trim functions in dispatcher
A new dispatcher function trim_func calls corresponding trim_
function implementation in relation with number of argc - a count
of arguments.
This is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.
Needed for #2200, #4113, #2233
commit a46b52004fd830c5c0183c3e1cf067116b98f59a
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Aug 5 16:39:31 2019 +0300
sql: GREATEST, LEAST instead of MIN/MAX overload
This patch does two things: renames existing scalar min/max
functions and reserves names for them in NoSQL cache.
Moreover it is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.
Closes #4405
Needed for #2200, #4113, #2233
@TarantoolBot document
Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST
The MIN/MAX functions are typically used only as aggregate
functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
Tarantool's SQLite legacy code use them also in meaning
GREATEST/LEAST scalar function. Now it fixed.
commit 79f3bf4bfc4d167aaa6cd570175e3402b70f1833
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Jul 4 10:35:49 2019 +0300
sql: remove SQL_PreferBuiltin flag
The SQL_PreferBuiltin flag is redundant (because builtin names
are forbidden for UDFs) so we may to remove it.
Needed for #4113, #2200, #2233
commit 72279c1cf2c57e1f8811cf7a4a99dfb72c715794
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Aug 12 16:14:55 2019 +0300
sql: improve vdbe_field_ref fetcher
Vdbe field ref is a dynamic index over tuple fields storing
offsets to each field and filling the offset array on demand.
It is highly used in SQL, because it strongly relies on fast and
repetitive access to any field, not only indexed.
There is an optimisation for the case when a requested field
fieldno is indexed, and the tuple itself stores offset to the
field in its own small field map, used by indexes. vdbe_field_ref
then uses that map to retrieve offset value without decoding
anything. But when SQL requests any field > fieldno, then
vdbe_field_ref decodes the tuple from the beginning in the worst
case. Even having previously accessed fieldno. But it could
start decoding from the latter.
An updated vdbe_field_ref fetcher class uses a bitmask of
initialized slots to use pre-calculated offsets when possible.
This speed-ups SQL in some corner case and doesn't damage
performance in general scenarios.
Closes #4267
commit d0e38d59fc93abcbeb3336716e7a0e1d157d67d7
Author: Mons Anderson <mons@cpan.org>
Date: Wed Jul 10 18:20:12 2019 +0300
log: fix segfault on _say without filename
commit 15f12c0b5461bee1b0d44c8fac218674c7792576
Author: rtokarev <rtokarev@corp.mail.ru>
Date: Mon Aug 19 11:01:06 2019 +0300
relay: set `last_row_time' to `now' in `relay_new' and `relay_start'. (#4431)
(cherry picked from commit 507f37215ebcb5df882576d14eb5c135811b5fd9)
commit 6277f48ad8a7213d73e280d5046c5b5872f74e25
Author: Konstantin Osipov <kostja.osipov@gmail.com>
Date: Fri Aug 16 16:13:42 2019 +0300
gc: randomie the next checkpoint time also after a manual box.snapshot().
Before this patch, snapshot interval was set randomly within
checkpoint_interval period. However, after box.snapshot(), the next
snapshot was scheduled exactly checkpoint_interval from the current time.
Many orchestration scripts snapshot entire cluster right after deployment,
to take a backup. This kills randomness, since all instances begin to
count the next checkpoint time from the current time.
Randomize the next checkpoint time after a manual snapshot as well.
Fixes gh-4432
commit 05fb6faacf45cb21101b1e11a8d2dd998243d5c3
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri Aug 16 15:30:21 2019 +0300
test: update test-run
pretest_clean: preserve GREATEST and LEAST built-in functions.
Needed for #4405.
commit 13ed1126bf385f349822ac3f31565e8f13bc0ff0
Author: Roman Khabibov <roman.habibov@tarantool.org>
Date: Fri Aug 2 15:19:04 2019 +0300
sql: remove mask from struct Keyword
Originally, mask in struct Keyword served to reduce set of reserved
keywords for build-dependent features. For instance, it was allowed to
disable triggers as a compilation option, and in this case TRIGGER
wouldn't be reserved word. Nowadays, our build always comprises all
features, so there's no need in this option anymore. Hence, we can
remove mask alongside with related to it macros.
Closes #4155
commit 0894bec230bfb969116731aa8fa82653825a8dfb
Author: Nikita Pettik <korablev@tarantool.org>
Date: Thu Aug 15 18:02:03 2019 +0300
Hotfix for b7d595ac3
It was forgotten to update result file of sql/bind.test.lua
in previous patch. Let's fix that and refresh sql/bind.result with
up-to-date results.
commit b7d595ac3d92aa0977c66d935e1c6eefabb91bec
Author: Nikita Pettik <korablev@tarantool.org>
Date: Wed Jul 24 19:48:56 2019 +0300
sql: fix type in meta for unsigned binding
It was decided that for all integer literals we would return "INTEGER"
type, not "UNSIGNED". Accidentally, after substitution of unsigned
binding value type was set to "UNSIGNED". Let's fix that and set
"INTEGER" type.
commit 03a39c3d38690cd388587bf53301e586bc1d9062
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Aug 15 14:33:58 2019 +0300
luajit: Bump luajit version
commit a634bd7def1df799cfffed0ea8f8c681eeb3ba56
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Aug 15 14:07:34 2019 +0300
luajit: Bump luajit version
commit 2630360420b041bb605e8e0ef8a192a7a2f35a07
Author: Sergey Ostanevich <sergos@tarantool.org>
Date: Wed Jul 24 13:52:50 2019 +0000
test: new test for LuaJIT fold machinery
https://github.com/LuaJIT/LuaJIT/issues/505
commit 2d5e56fffe894c1fb77fe77bcf12b34da772b2a7
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Aug 9 13:08:44 2019 +0300
wal: make wal_sync fail on write error
wal_sync() simply flushes the tx<->wal request queue, it doesn't
guarantee that all pending writes are successfully committed to disk.
This works for now, but in order to implement replica join off the
current read view, we need to make sure that all pending writes have
been persisted and won't be rolled back before we can use memtx
snapshot iterators. So this patch adds a return code to wal_sync():
since now on it returns -1 if rollback is in progress and hence
some in-memory changes are going to be rolled back. We will use
this method after opening memtx snapshot iterators used for feeding
a consistent read view a newly joined replica so as to ensure that
changes frozen by the iterators have made it to the disk.
commit 952d8d1d4ffe81a0be4d7901f063f6c5059d71f5
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Aug 14 08:11:19 2019 +0300
test: app/socket flaky fails at 1118 line
Found that on high loaded hosts the test flaky fails at:
[004] --- app/socket.result Mon Jul 15 07:18:57 2019
[004] +++ app/socket.reject Tue Jul 16 16:37:35 2019
[004] @@ -1118,7 +1118,7 @@
[004] ...
[004] ch:get(1)
[004] ---
[004] -- true
[004] +- null
[004] ...
[004] s:error()
[004] ---
Found that the test in previous was used for testing the
the channel get() function timeout and the error occurred
on it, but later the checking error changed to:
"builtin/socket.lua: attempt to use closed socket" and the
test became not correct. Because for now it passes when the
socket read function runs before the socket closing, but in
this way read call doesn't wait. In the other way on high
loaded hosts the close call may occure before read call and
in this way read call halts and socket get call returns
'null'. As seen both ways are not correct to check the error.
Decided to remove this subtest.
Check commit ba7a4fee73 ("Add tests for socket:close closes #360")
Fixes #4354
commit e687cacdeec31fab464c98e60881b96f6e113f8f
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Aug 9 19:36:39 2019 +0300
xrow: factor out helper for setting REPLACE request body
We will reuse it to relay a snapshot to a newly joined replica.
commit aef84078558f4533e3cd1efbb7e9fc5a44da20a8
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Aug 9 16:42:29 2019 +0300
memtx: allow snapshot iterator to fail
Memtx iterators never fail, that's why the snapshot iterator interface
doesn't support failures. However, once we introduce snapshot iterator
support for vinyl, we will need a way to handle errors in the API.
commit a22f221993153948347224e0d694e166eea5e958
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Aug 8 14:07:41 2019 +0300
memtx: don't store pointers to index internals in iterator
It's pointless as we can always access the index via iterator->index.
commit 7e11dd4f7f72f43f24256e72f263b65c894dcfb3
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Aug 7 15:45:38 2019 +0300
vinyl: move reference counting from vy_lsm to index
Now, as vy_lsm and index are basically the same object, we can implement
reference counting right in struct index. This will allow us to prevent
an index from destruction when a space object it belongs to is freed
anywhere in the code, not just in vinyl.
commit 576611eb7e71ac2a87c73d7040b5671722dbabae
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Aug 7 15:20:33 2019 +0300
vinyl: embed index in vy_lsm
There's no point in having vinyl_engine and vinyl_index wrapper structs
to bind vy_env and vy_lsm to struct engine and index. Instead we can
simply embed engine and index in vy_env and vy_lsm. This will simplify
further development, e.g. this will allow us to move reference counting
from vy_lsm up to struct index so that it can be used in the generic
code.
commit a771c2e5e8876b78d67a1c24fd6711d7dcd3414d
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Aug 7 14:59:13 2019 +0300
vinyl: embed engine in vy_env
There's no point in having vinyl_engine and vinyl_index wrapper structs
to bind vy_env and vy_lsm to struct engine and index. Instead we can
simply embed engine and index in vy_env and vy_lsm. This will simplify
further development, e.g. this will allow us to move reference counting
from vy_lsm up to struct index so that it can be used in the generic
code.
commit 8ea7106ad1ac5def675c3a28d77dad472573a23d
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Thu Aug 8 16:34:35 2019 +0300
decimal: add modulo operator
Part of #4403
@TarantoolBol document
Title: Document decimal modulo operator
There is now a modulo operator for decimal numbers:
```
a = decimal.new(172.51)
a % 1
---
- '0.51'
...
a % 0.3
---
- '0.01'
...
a % 13.27
---
- '0.00'
...
a % 173
---
- '172.51'
...
a % 72
---
- '28.51'
...
720 % a
---
- '29.96'
...
```
commit ef64ee51cc5820da63d66c8ba8baa62695549c2d
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Aug 11 01:37:27 2019 +0200
json: detect a new invalid json path case
JSON paths has no a strict standard, but definitely there is no
an implementation, allowing to omit '.' after [], if a next token
is a key. For example:
[1]key
is invalid. It should be written like that:
[1].key
Strangely, but we even had tests on the invalid case.
Closes #4419
commit 940a673e92ea78a30b8df503a199420048d96332
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Aug 12 00:39:32 2019 +0300
test: update test-run
Disable a check whether yaml responses are well-formed for 'core =
tarantool' tests in test-run. The check was unable to handle complex
(dictionary / list) keys in a dictionary, because pyyaml does not
support them.
See also https://github.com/yaml/pyyaml/issues/88
Fixes #4421.
commit 4b893910aabafc3600b9712b53a8913cd09a7c15
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Jul 30 21:30:10 2019 +0200
test: fix flaky swim/errinj.test.lua
In one place that test sends a packet and expects that it has
arrived two lines below. Under high load it may take more time.
The patch makes the test explicitly wait for the packet arrival.
Closes #4392
commit 2e97c6071ebacd3aa071ccfb270ebac86ec59eba
Author: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>
Date: Thu Aug 8 19:27:52 2019 +0300
Expose tarantool_package value to lua api (#4412)
There is compile time option PACKAGE in cmake to define
current build distribution info. By default it's
"Tarantool" for the community version and "Tarantool Enterprise"
for the enterprise version.
It's displayed in console greeting and in `box.info().package`,
but, unfortunately, it can't be accessed from Lua before `box.cfg`.
This patch exposes `require('tarantool').package`.
Close #4408
@TarantoolBot document
Title: Extend module "tarantool" with the field "package"
Beside from build info and version, module "tarantool" now provides
"package" field. By default it equals string "Tarantool", but
can differ for other distributions like "Tarantool Enterprise".
Example:
```console
tarantool> require('tarantool')
---
- version: 2.3.0-3-g302bb3241
build:
target: Linux-x86_64-RelWithDebInfo
options: cmake . -DCMAKE_INSTALL_PREFIX=/opt/tarantool-install
-DENABLE_BACKTRACE=ON
mod_format: so
flags: ' -fexceptions -funwind-tables -fno-omit-frame-pointer
-fno-stack-protector
-fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra
-Wno-strict-aliasing -Wno-char-subscripts
-Wno-format-truncation -fno-gnu89-inline -Wno-cast-function-type'
compiler: /usr/bin/cc /usr/bin/c++
pid: 'function: 0x40016cd0'
package: Tarantool
uptime: 'function: 0x40016cb0'
...
```
commit 302bb32418924dd9ddf02f7db951443735f3bed8
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Fri Aug 2 15:05:04 2019 +0300
box: make functional index creation transactional
The _func_index space trigger used to reject an insertion of a
tuple that defines an invalid functional index.
As insertion in _index space had been completed before, a garbage
is kept in _index space in such case.
We need to do something with the yelding _func_index trigger(that
rebuilds an index) to wrap all index creation operation in DDL
transaction in further patches (because only the first DDL
operation may yeld now).
This problem could be trivially solved with preparatory
initialization of index_def function ponter: the memtx_tree
index construction would perform all required job in such case.
Therefore the following index rebuild in _func_index trigger
becomes redundant and should be omitted.
In other words, a trivial prefetch operation makes possible
a transactional index creation (with space:create_index operation).
As for index construction during recovery (a lack of function
cache during recovery was the main motivation to introduce
_func_index space), it's workflow is kept unchanged.
Follow up #1250
Needed for #4348
Closes #4401
