Git log: commit e43eeb533d551b85d96f5700cc17e6924ad30fc2
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Jun 20 16:02:08 2019 +0300
box: refactor key_validate_parts to return key_end
The key_validate_parts helper is refactored to return a pointer
to the end of a given key argument in case of success.
This is required to effectively validate a sequence of keys in
scope of functional multikey indexes.
Needed for #1260
commit 2c7c3e44152ba16665393c95c7726951a4a121a4
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Tue Jun 18 18:52:51 2019 +0300
box: introduce key_def->is_multikey flag
Previously only key definitions that have JSON paths were able
to define multikey index. We used to check multikey_path != NULL
test to determine whether given key definition is multikey.
In further patches with functional indexes this rule becomes
outdated. Functional index extracted key definition may be
multikey, but has no JSON paths.
So an explicit is_multikey flag was introduced.
Needed for #1260
commit 53fbb5879f6861e7df980d8e1db44dbef93e4ede
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu Jul 18 07:48:46 2019 +0300
Bump libyaml version
commit 79f876adee1e11961db7840381bea7a18fe18180
Author: Mergen Imeev <imeevma@gmail.com>
Date: Sat Jul 13 13:04:34 2019 +0300
box: increase connection timeout in "net.box.test.lua"
The "box/net.box.test.lua" test contains a check that the error
received contains a 'timed out'. But in cases when testing was
conducted on a slow computer or in the case of a very large load,
it is possible that the connection time-out will be reached
earlier than the mentioned error. In this case, the error "Invalid
argument" will be returned. To prevent this from happening,
this patch will increase the connection timeout.
Closes #4341
commit 0fbfeed932dc7d3a5e88017fa91e6c1ed1eb6acc
Author: Mergen Imeev <imeevma@gmail.com>
Date: Thu Jun 27 17:57:28 2019 +0300
sql: use common registers instead of temp. for constraints data
Prior to this patch, data needed to form tuple to be inserted to
_fk_constraint and _ck_constraint system spaces (to create
corresponding constraints) was stored in the range of temporary
register. After insertion, temporary registers are released. On
the other hand, this data is required for providing clean-up in
case of creation fail (i.e. removing already created constraints
within one CREATE TABLE statement). Hence, instead of using
temporary registers let's use ordinary ones.
Closes #4183
commit c06a3d550fac5fb48345fd5502d2833fa162a49f
Author: Mergen Imeev <imeevma@gmail.com>
Date: Thu Jun 27 16:59:42 2019 +0300
sql: clean-up in case constraint creation failed
This patch makes VDBE to perform a clean-up if the creation of a
constraint fails because of the creation of two or more
constraints of the same type with the same name and in the same
CREATE TABLE statement.
For example:
CREATE TABLE t1(
id INT PRIMARY KEY,
CONSTRAINT ck1 CHECK(id > 1),
CONSTRAINT ck1 CHECK(id < 10)
);
Part of #4183
commit 852bb4d4db939e129f9ef8ca1f6c4f9310d3b4e0
Author: Mergen Imeev <imeevma@gmail.com>
Date: Thu Jun 27 16:28:30 2019 +0300
sql: add OP_SetDiag opcode in VDBE
To separate the error setting and execution halting, a new opcode
OP_SetDiag was created. The only functionality of the opcode is
the execution of diag_set(). It is important to note that
OP_SetDiag does not set is_aborted to true, so we can continue
working with other opcodes, if necessary. This function allows us
to perform cleanup in some special cases, for example, when
creating a constraint failed because of the creation of two or
more constraints with the same name in the same CREATE TABLE
statement.
Since now diag_set() is executed in OP_SetDiag, this functionality
has been removed from OP_Halt.
Needed for #4183
commit c185a3872dc2cd86a383e72d9544ec5b97a9dc8d
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 17 14:41:26 2019 +0300
auth: fix empty password authentication
We are supposed to authenticate guest user without a password. This
used to work before commit 076a842011e0 ("Permit empty passwords in
net.box"), when guest didn't have any password. Now it has an empty
password and the check in authenticate turns out to be broken, which
breaks assumptions made by certain connectors. This patch fixes the
check.
Closes #4327
commit 2ac7f44fd8d002df404bd518ff9b5e555f76f204
Author: Roman Khabibov <roman.habibov@tarantool.org>
Date: Mon May 6 14:30:21 2019 +0300
sql: allow <COLLATE> only for string-like args
Before this patch, user could use COLLATE with non-string-like
literals, columns or subquery results. Disallow such usage.
Closes #3804
commit 2d3f299eb033da7acfbf45ef2e71f34e809eef41
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Mon Jul 15 16:41:03 2019 +0300
test: vinyl/recover fails on dump counter check
On high loaded host the test vinyl/recover failed on waiting loop
for the dumper counter check. It expected that the value should be
equal to "2" exactly, while on the high loaded host the dump could
be already run more times and the counter value found to be "3" or
even bigger values. To fix it the counter value check was changed
from the exact value to the range bigger or equal of the expecting
value. The error message before the fix was:
[003] --- vinyl/recover.result Mon Jul 15 10:46:00 2019
[003] +++ vinyl/recover.reject Mon Jul 15 10:58:10 2019
[003] @@ -517,7 +517,7 @@
[003] ...
[003] test_run:wait_cond(function() return pk:stat().disk.dump.count == 2 end)
[003] ---
[003] -- true
[003] +- false
[003] ...
[003] sk:stat().disk.dump.count -- 1
[003] ---
Closes #4345
commit f266559b167b4643c377724eebde9651877d6cc9
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Jul 5 20:13:16 2019 +0300
ddl: allow to execute non-yielding DDL statements in transactions
The patch is pretty straightforward - all it does is moves checks for
single statement transactions from alter.cc to txn_enable_yield_for_ddl
so that now any DDL request may be executed in a transaction unless it
builds an index or checks the format of a non-empty space (those are the
only two operations that may yield).
There's two things that must be noted explicitly. The first is removal
of an assertion from priv_grant. The assertion ensured that a revoked
privilege was in the cache. The problem is the cache is built from the
contents of the space, see user_reload_privs. On rollback, we first
revert the content of the space to the original state, and only then
start invoking rollback triggers, which call priv_grant. As a result, we
will revert the cache to the original state right after the first
trigger is invoked and the following triggers will have no effect on it.
Thus we have to remove this assertion.
The second subtlety lays in vinyl_index_commit_modify. Before the commit
we assumed that if statement lsn is <= vy_lsm::commit_lsn, then it must
be local recovery from WAL. Now it's not true, because there may be
several operations for the same index in a transaction, and they all
will receive the same signature in on_commit trigger. We could, of
course, try to assign different signatures to them, but that would look
cumbersome - better simply allow lsn <= vy_lsm::commit_lsn after local
recovery, there's actually nothing wrong about that.
Closes #4083
@TarantoolBot document
Title: Transactional DDL
Now it's possible to group non-yielding DDL statements into
transactions, e.g.
```Lua
box.begin()
box.schema.space.create('my_space')
box.space.my_space:create_index('primary')
box.commit() -- or box.rollback()
```
Most DDL statements don't yield and hence can be run from transactions.
There are just two exceptions: creation of a new index and changing the
format of a non-empty space. Those are long operations that may yield
so as not to block the event loop for too long. Those statements can't
be executed from transactions (to be more exact, such a statement must
go first in any transaction).
Also, just like in case of DML transactions in memtx, it's forbidden to
explicitly yield in a DDL transaction by calling fiber.sleep or any
other yielding function. If this happens, the transaction will be
aborted and an attempt to commit it will fail.
commit 626c5fd05e5b3aba5e99ce0f523b190669a111dd
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue Jul 9 17:59:57 2019 +0300
ddl: don't use space_index from AlterSpaceOp::commit,rollback
If there are multiple DDL operations in the same transactions, which is
impossible now, but will be implemented soon, AlterSpaceOp::commit and
rollback methods must not access space index map. To understand that,
consider the following example:
- on_replace: AlterSpaceOp1 creates index I1 for space S1
- on_replace: AlterSpaceOp2 moves index I1 from space S1 to space S2
- on_commit: AlterSpaceOp1 commits creation of index I1
AlterSpaceOp1 can't lookup I1 in S1 by id, because the index was moved
from S1 to S2 by AlterSpaceOp2. If AlterSpaceOp1 attempts to look it up,
it will access a wrong index.
Fix that by caching pointers to old and new indexes in AlterSpaceOp on
construct/prepare instead of using space_index() on commit/rollback to
access them.
commit 0ae5a2d78c649320e0de95455f13d02a48b7caf5
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Jul 5 17:40:56 2019 +0300
memtx: fix txn_on_yield for DDL transactions
Memtx engine doesn't allow yielding inside a transaction. To achieve
that, it installs fiber->on_yield trigger that aborts the current
transaction (rolls it back, but leaves it be so that commit fails).
There's an exception though - DDL statements are allowed to yield.
This is required so as not to block the event loop while a new index
is built or a space format is checked. Currently, we handle this
exception by checking space id and omitting installation of the
trigger for system spaces. This isn't entirely correct, because we
may yield after a DDL statement is complete, in which case the
transaction won't be aborted though it should:
box.begin()
box.space.my_space:create_index('my_index')
fiber.sleep(0) -- doesn't abort the transaction!
This patch fixes the problem by making the memtx engine install the
on_yield trigger unconditionally, for all kinds of transactions, and
instead explicitly disabling the trigger for yielding DDL operations.
In order not to spread the yield-in-transaction logic between memtx
and txn code, let's move all fiber_on_yield related stuff to txn,
export a method to disable yields, and use the method in memtx.
commit 260f63283deff986395d38636978bb6cf90ae4ff
Author: Nikita Pettik <korablev@tarantool.org>
Date: Mon Jul 8 15:53:10 2019 +0300
sql: introduce ADD CONSTRAINT CHECK statement
This patch extends parser's grammar to allow to create CHECK constraints
on already existent tables via SQL facilities.
Closes #3097
@TarantoolBot document
Title: Document ADD CONSTRAINT CHECK statement
Now it is possible to add CHECK constraints to already existent table
via SQL means. To achieve this one must use following syntax:
ALTER TABLE <table> ADD CONSTRAINT <name> CHECK (<expr>);
commit 307133d3289d3d09dac338274ca74fa5ac86fc9f
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Sat Jul 13 17:08:34 2019 +0300
Fix broken build
The argument in func_c_new() is used in Debug mode only.
Mark it w/ MAYBE_UNUSED.
commit 212c57421bd20478cedf4e07ad01bca21815265a
Author: Konstantin Osipov <kostja@tarantool.org>
Date: Sat Jul 13 00:49:49 2019 +0300
func: consolidate func_def checks in func_def_chekc
A follow up on #4182
commit 200a492aa771e50af86a4754b41a5e373fa7a354
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Jun 13 11:54:13 2019 +0300
box: introduce Lua persistent functions
Closes #4182
Closes #4219
Needed for #1260
@TarantoolBot document
Title: Persistent Lua functions
Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"
A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
a. only limited set of Lua functions and modules are available:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8;
b. global variables are forbidden
Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.
The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
<if_not_exists = true|FALSE>, <language = LUA|c>,
<body = string ('')>, <is_deterministic = true|FALSE>,
<is_sandboxed = true|FALSE>, <comment = string ('')>)
This schema change is also reserves names for sql builtin
functions:
TRIM, TYPEOF, PRINTF, UNICODE, CHAR, HEX, VERSION,
QUOTE, REPLACE, SUBSTR, GROUP_CONCAT, JULIANDAY, DATE,
TIME, DATETIME, STRFTIME, CURRENT_TIME, CURRENT_TIMESTAMP,
CURRENT_DATE, LENGTH, POSITION, ROUND, UPPER, LOWER,
IFNULL, RANDOM, CEIL, CEILING, CHARACTER_LENGTH,
CHAR_LENGTH, FLOOR, MOD, OCTET_LENGTH, ROW_COUNT, COUNT,
LIKE, ABS, EXP, LN, POWER, SQRT, SUM, TOTAL, AVG,
RANDOMBLOB, NULLIF, ZEROBLOB, MIN, MAX, COALESCE, EVERY,
EXISTS, EXTRACT, SOME, GREATER, LESSER, SOUNDEX,
LIKELIHOOD, LIKELY, UNLIKELY,
_sql_stat_get, _sql_stat_push, _sql_stat_init, LUA
A new Lua persistent function LUA is introduced to evaluate
LUA strings from SQL in future.
This names could not be used for user-defined functions.
Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('summarize', {body = lua_code,
is_deterministic = true, is_sandboxed = true})
box.func.summarize
---
- aggregate: none
returns: any
exports:
lua: true
sql: false
id: 60
is_sandboxed: true
setuid: false
is_deterministic: true
body: function(a, b) return a + b end
name: summarize
language: LUA
...
box.func.summarize:call({1, 3})
---
- 4
...
@kostja: fix style, remove unnecessary module dependencies,
add comments
commit 77051a11029560eee8ad0adf83c8ac92e64a7d12
Author: Mergen Imeev <imeevma@gmail.com>
Date: Thu Jul 11 16:43:18 2019 +0300
box: do not check state in case of reconnect
Test box/net.box.test.lua checks state of the connection in case
of an error. It should be 'error_reconnect'. But, in cases where
testing was performed on a slow computer or in the case of a very
large load, it is possible that the connection status may change
from the 'error_reconnect' state to another state. This led to the
failure of the test. Since this check is not the main purpose of
the test, it is better to simply delete the check.
Closes #4335
commit f9551660448d91fac86acaf0d08e70ff48b8ee5b
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Fri Jul 5 17:18:52 2019 +0300
sql: move LIKE UConverter object to collation library
Moved UConverter object to collation library. This is required
to get rid of sqlRegisterBuiltinFunctions function in further
patches.
Needed for #4113, #2200, #2233
commit 4e523fa0516c8b93bdf245f17adf6adaadbd42d9
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Jul 3 15:11:12 2019 +0300
sql: replace bool is_derived_coll marker with flag
Introduce a new flag SQL_FUNC_DERIVEDCOLL for function that may
require collation to be applied on its result instead of separate
boolean variable. This is required to get rid of FuncDef in
further patches.
Needed for #4113, #2200, #2233
commit e1a7cb7f030be8f13545c48121dc29fe8adfa51c
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Jul 1 14:03:52 2019 +0300
sql: put analyze helpers to FuncDef cache
Previously analyze functions refer to statically defined
service FuncDef context. We need to change this approach due we
going to rework the builtins functions machinery in following
patches.
Needed for #4113, #2200, #2233
commit 873a3b301bbe10e8839457b2177ce142e72d97f8
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Jul 11 11:55:18 2019 +0300
sql: enable SOUNDEX sql builtin
Needed for #4182
@TarantoolBot document
Title: Introduce SOUNDEX sql function
The SOUNDEX function returns a 4-character code that represents
the sound of the words in the argument. The result can be
compared to the results of the SOUNDEX function of other strings.
The current SOUNDEX function supports only Latin strings.
@kostja: fix test tap count; remove optional invocation; remove
trailing spaces; fix alignment.
commit 9d077bb41824027f4beaa6bd424256c8b56905d2
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Thu Jul 11 20:34:15 2019 +0300
box/memtx: Skip tuple memory from coredump by default
Quoting feature request
| Tarantool is Database and Application Server in one box.
|
| Appserver development process contains a lot of
| lua/luajit-ffi/lua-c-extension code.
|
| Coredump is very useful in case when some part of appserver crashed.
| If the reason is input - data from database is not necessary. If the reason
| is output - data from database is already in snap/xlog files.
|
| Therefore consider core dumps without data enabled by default.
For info: the strip_core feature has been introduced in
549140b390f2db3a29dbba2d44170d215f4c7700
Closes #4337
@TarantoolBot document
Title: Document box.cfg.strip_core
When Tarantool runs under a heavy load the memory allocated
for tuples may be very huge in size and to eliminate this
memory from being present in `coredump` file the `box.cfg.strip_core`
parameter should be set to `true`.
The default value is `true`.
commit 4d936cb250e9906b6d892a0ce1cbfdf39f1a3b87
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu Jul 11 11:52:05 2019 +0300
sql: get rid of MATCH sql builtin
In relation with FuncDef cache rework we need to clean-up
builtins list. The MATCH fucntion is a stub that raises an error
and it could be dropped.
Needed for #4182
@kostja: make the patch actually pass the tests, remove
tap count change in e_expr.test.lua, since it's disabled and was
not run
commit f2876e63f1c49bfc1f63e6e331cab9824d8953d6
Author: Michael Filonenko <filonenko.mikhail@gmail.com>
Date: Tue Feb 26 18:21:14 2019 +0300
Fix travis script to install libiconv on macos using homebrew
commit 9bde340622f41c9d89f1da1cbf08cc984f4e431c
Author: avtikhon <avtikhon@tarantool.org>
Date: Fri Jul 12 00:58:50 2019 +0300
test: box/net.box test flaky fails on grep_log (#4330)
box/net.box test flaky failed on grepping the log file
for 'ER_NO_SUCH_PROC' pattern on high load running hosts,
found that the issue can be resolved by updating the
grep_log to wait_log function to make able to wait the
needed message for some time.
[008] Test failed! Result content mismatch:
[008] --- box/net.box.result Tue Jul 9 17:00:24 2019
[008] +++ box/net.box.reject Tue Jul 9 17:03:34 2019
[008] @@ -1376,7 +1376,7 @@
[008] ...
[008] test_run:grep_log("default", "ER_NO_SUCH_PROC")
[008] ---
[008] -- ER_NO_SUCH_PROC
[008] +- null
[008] ...
[008] box.schema.user.revoke('guest', 'execute', 'universe')
[008] ---
Closes #4329
commit 8dc6486c9ebf25cd1f33dbd7e58aa2dcd224143d
Author: Denis Ignatenko <denis.ignatenko@gmail.com>
Date: Fri Jul 12 00:56:44 2019 +0300
Add python environment for test-run (#4284)
Add python-dev and pip to container to install test-run dependency
Install test-run dependency from requirements.txt of test-run subrepo
commit 366466eb24c11df026f21b29af14702fd480abc8
Author: Denis Ignatenko <denis.ignatenko@gmail.com>
Date: Thu Jun 27 10:57:19 2019 +0300
Add distribution info to box.info
There is compile time option PACKAGE in cmake to define
current build distribution info. For community edition
is it Tarantool by default. For enterprise it is
Tarantool Enterprise
There were no option to check distribution name in runtime.
This change adds box.info.package output for CE and TE.
commit 68dbd154f5017703d04e5a57c82776f7df1de294
Author: avtikhon <avtikhon@tarantool.org>
Date: Fri Jul 12 00:48:34 2019 +0300
travis-ci: apt-get fails to update the APT repos (#4332)
travis-ci APT repository update failed on Debian 10 (Buster)
with command 'apt-get update', like:
Get:1 http://deb.debian.org/debian buster InRelease [118 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [39.1 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [46.8 kB]
Reading package lists... Done
N: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Version' value from '' to '10'
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'testing' to 'stable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '' to '10.0'
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'stable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'testing-updates' to 'stable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
The cause of the issue was:
According to Debian Project News published 1st of July, Debian 10 "Buster"
was scheduled to transition from testing to stable on 2019-07-06.
It looks like the transition was in fact performed as scheduled, and so the
testing distribution should now be catching up witn unstable, to eventually
become Debian 11 "Bullseye". You might be experiencing some temporary side
effects because of this transition of the Debian mirrors.
If you want to stay with Debian 10 "Buster", now would be a good time to switch
your /etc/apt/sources.list to use the release name buster instead of testing.
Otherwise, you'll soon be getting the raw bleeding-edge stuff from unstable,
and you might accidentally get a partial upgrade to proto-"Bullseye".
Also, this is a reminder for anyone using the word stable in their
/etc/apt/sources.list to consider whether to change it to stretch and stay with
the old version, or read the Release Notes and perform the recommended upgrade
steps.
To fix the issue it had to accept interactively the changes in the repositories
NOTE: apt instead of apt-get can accept the changes interactively
apt update -y
either accept only the needed changes for 'version' and 'suite'
apt-get update --allow-releaseinfo-change-version --allow-releaseinfo-change-suite
Seems that the only 'version' and 'suite' accept is better than blind
accept of the all changes.
Closes #4331
commit 9f76bd8602d151d38b2b2e198a8d1dbb03437f20
Author: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>
Date: Fri Jul 12 00:44:20 2019 +0300
tarantoolctl: always initialize notify_socket (#4342)
Notify socket used to be initialized during `box.cfg()`.
There is no apparent reason for that, because we can write tarantool
apps that don't use box api at all, but still leverage the event loop
and async operations.
This patch makes initialization of notify socket independent.
Instance can notify entering event loop even if box.cfg wasn't called.
Closes #4305
commit dc2e7a7b8d6559122bf7e844c62c3ee0c19b2c43
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Jul 8 14:13:48 2019 +0300
sql: rfc for SQL and Lua functions
Part of #4182
commit 8fac697219243b9d4d5d273818e3225c261e024b
Author: Nikita Pettik <korablev@tarantool.org>
Date: Thu May 23 18:02:51 2019 +0300
sql: fix passing FP values to integer iterator
Before this patch it was impossible to compare indexed field of integer
type and floating point value. For instance:
CREATE TABLE t1(id INT PRIMARY KEY, a INT UNIQUE);
INSERT INTO t1 VALUES (1, 1);
SELECT * FROM t1 WHERE a = 1.5;
---
- error: 'Failed to execute SQL statement: Supplied key type of part 0 does not match
index part type: expected integer'
...
That happened due to the fact that type casting mechanism (OP_ApplyType)
doesn't affect FP value when it is converted to integer. Hence, FP value
was passed to the iterator over integer field which resulted in error.
Meanwhile, comparison of integer and FP values is legal in SQL. To cope
with this problem for each equality comparison involving integer field
we emit OP_MustBeInt, which checks whether value to be compared is
integer or not. If the latter, we assume that result of comparison is
always false and continue processing query. For inequality constraints
we pass auxiliary flag to OP_Seek** opcodes to notify it that one of key
fields must be truncated to integer (in case of FP value) alongside with
changing iterator's type: a > 1.5 -> a >= 2.
Closes #4187
commit 0d5e757d0aab57298873f6cd6cb851be6176b4c8
Author: Nikita Pettik <korablev@tarantool.org>
Date: Thu May 23 18:01:03 2019 +0300
sql: remove redundant type derivation from QP
Before value to be scanned in index search is passed to the iterator, it
is subjected to implicit type casting (which is implemented by
OP_ApplyType). If value can't be converted to required type,
user-friendly message is raised. Without this cast, type of iterator may
not match with type of key which in turn results in unexpected error.
However, array of types which is used to provide type conversions is
different from types of indexed fields: it is modified depending on
types of comparison's operands. For instance, when boolean field is
compared with blob value, resulting type is assumed to be scalar. In
turn, conversion to scalar is no-op. As a result, value with MP_BIN
format is passed to the iterator over boolean field. To fix that let's
remove this transformation of types. Moreover, it seems to be redundant.
Part of #4187
commit 3570f366d72930db1747a92ccb55d35008e3bea9
Author: Nikita Pettik <korablev@tarantool.org>
Date: Wed May 1 13:53:36 2019 +0300
sql: remove redundant check of space format from QP
In SQL we are able to execute queries involving spaces only with formats.
Otherwise, at the very beginning of query compilation error is raised.
So, after that any checks of format existence are redundant.
commit db12efaf48f3670a62f62fca39521235abad41aa
Author: Nikita Pettik <korablev@tarantool.org>
Date: Tue Jun 25 13:17:31 2019 +0300
sql: fix antisymmetric boolean comparison in VDBE
There are a few situations when booleans can be compared with values of
other types. To process them, we assume that booleans are always less
than numbers, which in turn are less than strings. On the other hand,
function which implements internal comparison of values -
sqlMemCompare() always returns 'less' result if one of values is boolean
and another one is not, ignoring the order of values. For instance:
... max (false, 'abc') -> 'abc'
... max ('abc', false) -> false
This patch fixes this misbehaviour making boolean values always less
than values of other types.
commit 7bd1dc4bb586d9b72dcbcc9219eff42ea88efe73
Author: Nikita Pettik <korablev@tarantool.org>
Date: Tue Jun 25 03:15:46 2019 +0300
sqltester: don't convert booleans to 0/1 numerics
When there were no booleans in SQL, to represent them numeric values 0
and 1 were involved. However, recently booleans have been introduced in
SQL, so values of result set can take boolean values. Hence, it makes
no sense to continue converting booleans to numeric, so we can use
directly booleans.
commit 2885cf8455ef6ff064bb6462abb4557728df6fad
Author: Mergen Imeev <imeevma@gmail.com>
Date: Sat Jun 15 13:51:58 2019 +0300
sql: ANSI aliases for LENGTH()
This patch creates aliases CHARACTER_LENGTH() and CHAR_LENGTH()
for LENGTH(). These functions are added because they are described
in ANSI.
Closes #3929
@TarantoolBot document
Title: SQL functions CHAR_LENGTH() and CHARACTER_LENGTH()
The SQL functions CHAR_LENGTH() and CHARACTER_LENGTH() work the
same as the LENGTH() function. They take exactly one argument. If
an argument of type TEXT or can be cast to a TEXT value using
internal casting rules, these functions return the length of the
TEXT value that represents the argument. They throw an error if
the argument cannot be cast to a TEXT value.
commit 31a2644865ad678e5f5583a3e2d95bd178b6e1a3
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Jul 6 00:34:58 2019 +0200
swim: optimize struct swim_task layout
Before the patch it was split in two parts by 1.5KB packet, and
in the constructor it was nullifying the whole volume. Obviously,
these were mistakes. The first problem breaks cache locality,
the second one flushes the cache.
commit 837e114edd18396b1b8f1b6068d35be1101cbdbd
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Jul 6 00:24:32 2019 +0200
swim: pool IO tasks
Before the patch each SWIM member had two preallocated task
objects, 3KB in total. It was a waste of memory, because network
load per member in SWIM is ~2 messages per round step regardless
of cluster size.
This patch moves the tasks to a pool, where they can be reused.
Even by different SWIM instances running on the same node.
commit eb0cc50ca2773559bb9d25147a786a6786658b78
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Jul 9 12:16:33 2019 +0300
test: fix net.box occasional failure. Again
The test regarding logging corrupted rows failed occasionally with
```
[016] test_run:grep_log('default', 'Got a corrupted row.*')
[016] ---
[016] -- 'Got a corrupted row:'
[016] +- null
[016] ...
```
The logs then had
```
[010] 2019-07-06 19:36:16.857 [13046] iproto sio.c:261 !> SystemError writev(1),
called on fd 23, aka unix/:(socket), peer of unix/:(socket): Broken pipe
```
instead of the expected message.
This happened, because we closed a socket before tarantool could write a
greeting to the client, the connection was then closed, and execution
never got to processing the malformed request and thus printing the
desired message to the log.
To fix this, actually read the greeting prior to writing new data and
closing the socket.
Follow-up #4273
commit 6e393aca004629341eebe97497c4573a30ecc7b9
Author: Oleg Babin <babinoleg@mail.ru>
Date: Tue Jul 9 12:05:21 2019 +0300
fio: introduce utime function
Closes #4323
@TarantoolBot document
Title: fio.utime
fio.utime (filepath [, atime [, mtime]])
Set access and modification times of a file.
The first argument is the filename, the second argument (atime) is
the access time, and the third argument (mtime) is
the modification time. Both times are provided in seconds since the epoch.
If the modification time is omitted, the access time provided is used;
if both times are omitted, the current time is used.
commit 6ac597db4edda969e3559dc14e56fbab823a2402
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Jul 5 17:08:45 2019 +0300
txn: run on_rollback triggers on txn_abort
When a memtx transaction is aborted on yield, it isn't enough to
rollback individual statements - we must also run on_rollback triggers,
otherwise changes done to the schema by an aborted DDL transaction will
be visible to other fibers until an attempt to commit it is made.
commit 0f9fdd7217445fe3e989452cccdd1e1ec7597247
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Jul 5 14:37:00 2019 +0300
test: net.box: fix case re invalid msgpack warning
The test case has two problems that appear from time to time and lead to
flaky fails. Those fails are look as shown below in a test-run output.
| Test failed! Result content mismatch:
| --- box/net.box.result Mon Jun 24 17:23:49 2019
| +++ box/net.box.reject Mon Jun 24 17:51:52 2019
| @@ -1404,7 +1404,7 @@
| ...
| test_run:grep_log('default', 'ER_INVALID_MSGPACK.*')
| ---
| -- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet body'
| +- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet length'
| ...
| -- gh-983 selecting a lot of data crashes the server or hangs the
| -- connection
'ER_INVALID_MSGPACK.*' regexp should match 'ER_INVALID_MSGPACK: Invalid
MsgPack - packet body' log message, but if it is not in a log file at a
time of grep_log() call (just don't flushed to the file yet) a message
produced by another test case can be matched ('ER_INVALID_MSGPACK:
Invalid MsgPack - packet length'). The fix here is to match the entire
message and check for the message periodically during several seconds
(use wait_log() instead of grep_log()).
Another problem is the race between writing a response to an iproto
socket on a server side and closing the socket on a client end. If
tarantool is unable to write a response, it does not produce the warning
re invalid msgpack, but shows 'broken pipe' message instead. We need
first grep for the message in logs and only then close the socket on a
client. The similar problem (with another test case) is described in
[1].
[1]: https://github.com/tarantool/tarantool/issues/4273#issuecomment-508939695
Closes: #4311
commit 01343264128378369799a34faf69331c26e0b2c1
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Jul 5 21:52:00 2019 +0300
txn: fix execution order of commit triggers
Both commit and rollback triggers are currently added to the list head.
As a result, they are both run in the reverse order. This is correct for
rollback triggers, because this matches the order in which statements
that added the triggers are rolled back, but this is wrong for commit
triggers. For example, suppose we create a space and then create an
index for it in the same transaction. We expect that on success we first
run the trigger that commits the space and only then the trigger that
commits the index, not vice versa. That said, reverse the order of
commit triggers in the scope of preparations for transactional DDL.
commit 27aba00b3221c864e6d08589f7233be7452a1465
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Fri Jul 5 12:54:04 2019 +0300
vinyl: don't sync WAL on space alter if not necessary
Changes done to an altered space while a new index is being built or
the format is being checked are propagated via an on_replace trigger.
The problem is there may be transactions that started before the alter
request. Their working set can't be checked so we simply abort them.
We can't abort transactions that have reached WAL so we also call
wal_sync() to flush all pending WAL requests. This is a yielding
operation and we call it even if there's no transactions that need
to be flushed. As a result, vinyl space alter yields unconditionally,
even if the space is empty and there is no pending transactions
affecting it. This prevents us from implementing transactional DDL.
Let's call wal_sync() only if there's actually at least one pending
transaction affecting the altered space and waiting for WAL.
commit 3ab387a45e52ed01059b727596b35b8efad0b460
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Jun 5 15:58:49 2019 +0300
decimal: expose decimal type to lua.
Add a decimal library to lua.
Part of #692
@TarantoolBot document
Title: Document decimal module in lua.
First of all, you have to require the package via
`decimal = require('decimal')`
Now you can construct decimals via `new` method.
Decimals may be constructed from lua numbers, strings, unsigned and
signed 64 bit integers.
Decimal is a fixed-point type with maximum 38 digits of precision. All
the calculations are exact, so, be careful when constructing decimals
from lua numbers: they may hold only 15 decimal digits of precision.
You are advised to construct decimals from strings, since strings
represent decimals exactly, and vice versa.
```
a = decimal.new(123e-7)
b = decimal.new('123.456')
c = decimal.new('123.456e2')
d = decimal.new(123ULL)
e = decimal.new(2)
```
The allowed operations are addition, subtraction, division,
multiplication and power. If at least one of the operands is decimal,
decimal operations are performed. The other operand may be either
decimal or string, containing a number representation, or a lua number.
Operations only fail on an overflow, i.e. when result exceeds 10^38 - 1.
This includes division by zero. In these cases an error `Operation
failed` is raised.
Underflow is also possible, when precision needed to store the exact
result exceeds 38 digits. Underflow is not an error. When an underflow
happens, the result is rounded to 38 digits of precision.
```
a = decimal.new(123e-7)
b = decimal.new('123.456')
c = decimal.new('123.456e2')
d = decimal.new(123ULL)
e = decimal.new(2)
```
```
tarantool> a + b
---
- '123.456012300000000'
...
tarantool> c - d
---
- '12222.6'
...
tarantool> c / b
---
- '100'
...
tarantool> d * d
---
- '15129'
...
tarantool> d ^ 2
---
- '15129'
...
tarantool> 2 ^ d
---
- '10633823966279326983230456482242756608'...
tarantool> e ^ d
---
- '10633823966279326983230456482242756608'
...
```
The following math functions are also supported:
log10, ln, exp, sqrt. When specified as
`decimal.opname()`, operations may be performed on
strings and lua numbers.
```
f = decimal.new(100)
tarantool> decimal.log10(f)
---
- '2'
...
tarantool> decimal.sqrt(f)
---
- '10'
...
tarantool> e2 = decimal.exp(2)
---
...
tarantool> decimal.ln(e2)
---
- '2.0000000000000000000000000000000000000'
...
There are also `abs` and `tostring` methods, and an unary minus
operator, which are pretty self-explanatory.
```
tarantool> a = decimal.new('-5')
---
...
tarantool> a
---
- '-5'
...
tarantool> decimal.abs(a)
---
- '5'
...
tarantool> -a
---
- '5'
...
tostring(a)
---
- '-5'
...
```
`decimal.precision`, `decimal.scale` and `decimal.round` :
The first two methods return precision, i.e. decimal digits in
number representation, and scale, i.e. decimal digits after the decimal
point in the number representation.
`decimal.round` rounds the number to the given scale.
```
tarantool> a = decimal.new('123.456789')
---
...
tarantool> decimal.precision(a)
---
- 9
...
tarantool> decimal.scale(a)
---
- 6
...
tarantool> decimal.round(a, 4)
---
- '123.4568'
...
```
Comparsions: `>`, `<`, `>=`, `<=`, `==` are also legal and work as
expected. You may compare decimals with lua numbers or strings. In that
case comparsion will happen after the values are converted to decimal
type.
commit 4ca3953728c85c0ba77e411f0e7e8a2e183f1c35
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Jun 19 15:17:38 2019 +0300
lua/utils: add a function to register FFI metatypes.
A ffi metatype has a CTypeID, which can be used to push cdata of the
type on the lua stack, and has an associated metatable, automatically
applied to every created member of the type.
This allows the behavior similar to pushing userdata and assigning a
metatable to it.
Needed for #692
commit f64481de376c3aa0cdf43eb15c5e4036b20a8e29
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Jul 1 14:32:31 2019 +0300
decimal: fix string formatting on construction from double
Use printf "%g" option instead of "%f" to trim traling zeros in such
cases:
decimal_from_double(1) -> '1.000000000000000' -> decimal_from_string()
Now it should be
decimal_from_double(1) -> '1' ->decimal_from_string()
Follow-up 6d62c6c19ed418932ead1bba44fcd7cd84c78a19
commit db27d47075addfc0957bd13e4e9c6eb81fc32c83
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Mon Jul 1 13:37:47 2019 +0300
decimal: diallow infinity and NaN entirely.
While arithmetic operations do not return infinities or NaNs, it is
possbile to construct an invalid decimal value from strings 'Infinity',
'NaN' and similar. Some decimal mathematic functions may also result in
an infinity, say, ln(0) yields '-Infinity'.
So, add checks that the number is not a NaN or infinity after each
operation, so that the operation either returns an error, or a valid
finite decimal number.
Follow-up 6d62c6c19ed418932ead1bba44fcd7cd84c78a19
commit e0d4a5dc03c18e7b30f813b3fbae99cf4f8a30ba
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Jul 2 15:04:11 2019 +0300
decimal: fix ln hang on values between ~ 0.9 and 1.1
Turns out decNumberLn hangs when result is subnormal, according to the
current context settings. To fix this, reset minimal allowed exponent
to a smaller value during the ln operation and round the result afterwards.
Follow-up 6d62c6c19ed418932ead1bba44fcd7cd84c78a19
commit 75dc3e64523c54640805f6d6bd798d961a75fb9e
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Jul 8 16:06:23 2019 +0300
vinyl: fix vy_range_update_compaction_priority hang
Under certain circumstances vy_slice_new() may create an empty slice,
e.g. on range split:
|------------------ Slice ---------------|
|---- Run -----|
+
split key
|---- Slice 1 ----||------ Slice 2 ------|
^^^^^^^
Empty
vy_range_update_compaction_priority() uses the size of the last slice in
a range as a base for LSM tree level sizing. If the slice size happens
to be 0, it will simply hang in an infinite loop. Fix this potential
hang by using 1 if the last slice size is 0.
commit 1c5482745c4326255271b331b585011847bcbbf4
Author: Konstantin Osipov <kostja.osipov@gmail.com>
Date: Mon Jul 8 12:22:32 2019 +0300
alter: add a comment explaining the purpose of AlterSpaceLock
commit b37959bea5df343ece490a843fb9139fcaab5c47
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Fri Jul 5 15:05:00 2019 +0300
gitlab-ci: merge test and deploy stages
Current results have some tests with flaky results, which
blocks the deploy stage - decided to merge deploy stage into
test stage temporary to fix it.
Follows up #4156
commit 3fb2b875f7946f0dc6a26cb60f50dbd657456d6f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Jul 4 00:29:17 2019 +0200
swim: push-pull dissemination
SWIM in the original paper says, that dissemination time of an
event is O(log(N)), where N is size of the cluster. It is true,
when both ping and ack messages carry dissemination and
anti-entropy. Before this patch it wasn't so - only regular
pings were carrying something.
After this patch the SWIM module has true exponential
dissemination speed.
Closes #4253
commit f30309dee8ed4536a77610e4b033cd6a61fba73b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Jul 4 01:07:48 2019 +0200
swim: speed-up empty payloads cluster bootstrap
One another place consuming most of the tests start up time is
useless dissemination of an empty payload, which can be skipped
in fact.
Consider a cluster of 300 nodes. Each one of them are
interconnected manually, and now a test wants to wait for a
stabilization, when there are no events. On such a cluster it
happens for ~200 round steps till there are no any single event.
This is not about big packets, or log() TTD. There may be a few
events, may be more, but when a test wants the cluster to be
clean, it needs to wait for all the events being done.
This patch abuses the fact, that empty payloads can be compared
for free, no any single memcmp. If both new and the old payload
are empty, then nothing to disseminate.
It could help in a real cluster too, if initially there are no
payloads.
Needed for #4253
commit 7446ed219210ee745ea18018a22956e10eafb145
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Jul 4 00:18:09 2019 +0200
swim: speed-up tests
With following patches some of the tests will work much slower
due to significantly increased size of the most of packets.
This commit tries to smooth it by
* Turning off verbose logs in unit tests;
* Using much more light version of UUID comparator.
According to the profiler these places increase speed in a
couple of times, and at the same time they are simple.
Needed for #4253
commit a0d6ac29567bfab86ba74c1a22ca6ec97407748c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Jul 2 23:29:56 2019 +0200
test: redo some swim tests using error injections
There were tests relying on certain content of SWIM messages.
After next patches these conditions won't work without an
explicit intervention with error injections.
The patchset moves these tests to separate release-disabled
files.
Part of #4253
commit 679dea4ecd7c9d59f31efb2e8ef6de53e5797ac8
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Jul 2 21:56:57 2019 +0200
swim: sadly remove cache
SWIM sends basically the same message during a round. There was
a microoptimization so as not to reassemble the message on each
step. Now it is getting harder to support that island of
perfectionism, because
* Soon all the messages will carry all the sections,
including indirect messages. Their body is smaller, so it
is not possible to maintain one cached message without
reducing its maximal size;
* In big-clusters even without any changes a cached message
would need to be rebuilt. This is because anti-entropy
section won't help much unless it is being changed
frequent enough;
* In big clusters changes happen often enough to invalidate
the cached message constantly, unless SWIM would had
maintained what members are included into the cache, and
which are not. Then change of a member, not included into
the message, would not affect the cache. But it would
complicate the code too much.
Part of #4253
commit 30f91257dd73cd00cc7742ccf9e6472db6b0589b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Jul 4 00:34:19 2019 +0200
swim: fix flaky 'indirect pings' test
With a certain random seed sometimes a member was checked for a
status in a wrong moment of time.
commit 506d18785c07e1e365627a73e5d60c3e5b41af6a
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Jun 30 20:58:31 2019 +0200
swim: be suspicious when add new member
The previous commit solves one important problem with too long
event dissemination. Events could for too long time occupy the
whole UDP packet. Now they live log() time, but 'dead' and 'left'
members were bound to TTD. Such members were deleted after TTD
is 0.
Now they are deleted to early. Cluster nodes too early forget
about dead ones, and nodes not aware of death of the latters, can
accidentally resurrect them via anti-entropy. Cluster nodes need
to be suspicious when someone tells them to add a new not dead
member.
This patch makes SWIM add a new member in two cases only: manually
and if an ACK was received from it. A new member can't be added
indirectly via events and anti-entropy anymore. Instead, a ping is
sent to the members who are said to be new and alive. If ACK is
received directly from them, then they are added.
The patch does not affect updates. They are still indirect,
because if something has updated in an existing member, then it
is definitely alive.
Part of #4253
commit 0ec29b2f4fb21a84ffc88958ce54780f3d01003f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Jun 30 19:39:04 2019 +0200
swim: disseminate event for log(cluster_size) steps
Before the patch there was a problem of events and anti-entropy
starvation, when a cluster generates so many events, that they
consume the whole UDP packet. A packet fits up to 26 events. If
during the event storm something important happens, that event is
likely to be lost, and not disseminated until the storm is over.
Sadly, there is no way to prevent a storm, but it can be made
much shorter. For that the patch makes TTD of events logarithmic
instead of linear of cluster size.
According to the SWIM paper and to experiments the logarithm is
really enough. Linear TTD was a redundant overkill.
When events live shorter, it does not solve a problem of the
events starvation - still some of them can be lost in case of a
storm. But it frees some space for anti-entropy, which can finish
dissemination of lost events.
Experiments in a simulation of a cluster with 100 nodes showed,
that a failure dissemination happened in ~110 steps if there is
a storm. Basically, no dissemination at all.
After the patch it is ~20 steps. So it is logarithmic as it
should be, although with a bigger constant than without a storm.
Part of #4253
commit 2e880af097a548036278f43b3d6b0f6534a66bac
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Wed Jul 3 10:34:55 2019 +0300
travis-ci: freeze curl version on 7.65.0 on OS X
Homebrew now contains curl-7.65.1 which affected by curl/curl#3995 (this
problem leads to segfaults). The next version is not released yet. The
current commit downgrades the curl version to 7.65.0.
Close #4288
commit febacc4ba87d1cde30f1fe4e53d32daa2c2fc090
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Tue Jun 25 11:12:10 2019 +0300
lua/trigger: cleanup lua stack after trigger run
This patch adds a stack cleanup after a trigger is run and its return
values, if any, have been read.
This problem was found in a case when on_schema_init trigger set an
on_replace trigger on a space, and the trigger ran during recovery.
This lead to Lua stack overflows for the aforementioned reasons.
Closes #4275
commit e5c4ce75f86d279e6a044adffa3fb99cced0ded4
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Thu Jul 4 19:49:00 2019 +0300
Replace schema lock with fine-grained locking
Now, as we don't need to take the schema lock for checkpointing, it is
only used to synchronize concurrent space modifications (drop, truncate,
alter). Actually, a global lock is a way too heavy means to achieve this
goal, because we only care about forbidding concurrent modifications of
the same space while concurrent modifications of different spaces should
work just fine. So this patch replaces the global schema lock with per
space locking.
A space lock is held while alter_space_do() is in progress so as to make
sure that while AlterSpaceOp::prepare() is performing a potentially
yielding operation, such as building a new index, the space struct
doesn't get freed from under our feet. Note, the lock is released right
after index build is complete, before the transaction is committed to
WAL, so if the transaction is non-yielding it can modify the space again
in the next statement (this is impossible now, but will be done in the
scope of the transactional DDL feature).
If alter_space_do() sees that the space is already locked it bails out
and throws an error. This should be fine, because long-lasting operation
involving schema change, such as building an index, are rare and only
performed under the supervision of the user so throwing an error rather
than waiting seems to be adequate.
Removal of the schema lock allows us to remove latch_steal() helper and
on_begin_stmt txn trigger altogether, as they were introduced solely to
support locking.
This is a prerequisite for transactional DDL, because it's unclear how
to preserve the global schema lock while allowing to combine several DDL
statements in the same transaction.
commit d9fc5dc12ba97754710a7a66e342366303156d4f
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Sun Jun 30 17:55:29 2019 +0300
vinyl: don't yield while logging index creation
Currently, we always log a vinyl index creation in the vylog file
synchronously, i.e. wait for the write to complete successfully. This
makes any index creation a yielding operation, even if the target space
is empty. To implement transactional DDL for non-yielding statements, we
need to eliminate yields in this case. We can do that by simply using
vy_log_try_commit() instead of vy_log_commit() for logging index
creation, because we can handle a missing VY_LOG_PREPARE_INDEX record
during recovery - the code was left since before commit dd0827ba7948
("vinyl: log new index before WAL write on DDL") which split index
creation into PREPARE and COMMIT stages so all we need to do is slightly
modify the test.
The reason why I'm doing this now, in the series removing the schema
lock, is that removal of the schema lock without making space truncation
non-yielding (remember space truncation basically drops and recreates
all indexes) may result in a failure while executing space.truncate()
from concurrent fibers, which is rather unexpected. In particular, this
is checked by engine/truncate.test.lua. So to prevent the test failure
once the schema lock is removed (see the next patch), let's make empty
index creation non-yielding right now.
commit 94de0a081b3a8760bc14984f55b77116b804c31b
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Sat Jun 29 23:41:46 2019 +0300
Don't take schema lock for checkpointing
Memtx checkpointing proceeds as follows: first we open iterators over
primary indexes of all spaces and save them to a list, then we start
a thread that uses the iterators to dump space contents to a snap file.
To avoid accessing a freed tuple, we put the small allocator to the
delayed free mode. However, this doesn't prevent an index from being
dropped so we also take the schema lock to lock out any DDL operation
that can potentially destroy a space or an index. Note, vinyl doesn't
need this lock, because it implements index reference counting under
the hood.
Actually, we don't really need to take a lock - instead we can simply
postpone index destruction until checkpointing is complete, similarly
to how we postpone destruction of individual tuples. We even have all
the infrastructure for this - it's delayed garbage collection. So this
patch tweaks it a bit to delay the actual index destruction to be done
after checkpointing is complete.
This is a step forward towards removal of the schema lock, which stands
in the way of transactional DDL.
commit ce623a23416eb192ce70116fd14992e84e7ccbbe
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Mon Jun 10 10:57:01 2019 +0300
Enable GitLab CI testing
Implemented GitLab CI testing process additionally to existing Travis
CI. The new testing process is added to run tests faster. It requires to
control a load of machines to avoid flaky fails on timeouts. GitLab CI
allows us to run testing on our machines.
Created 2 stages for testing and deploying packages.
The testing stage contains the following jobs that are run for all
branches:
* Debian 9 (Stretch): release/debug gcc.
* Debian 10 (Buster): release clang8 + lto.
* OSX 14 (Mojave): release.
* FreeBSD 12: release gcc.
And the following jobs that are run of long-term branches (release
branches: for now it is 1.10, 2.1 and master):
* OSX 13 (Sierra): release clang.
* OSX 14 (Mojave): release clang + lto.
The deployment stage contains the same jobs as we have in Travis CI.
They however just build tarballs and packages: don't push them to S3 and
packagecloud.
In order to run full testing on a short-term branch one can name it with
'-full-ci' suffix.
The additional manual work is needed when dependencies are changed in
.travis.mk file ('deps_debian' or 'deps_buster_clang_8' goals):
| make GITLAB_USER=foo -f .gitlab.mk docker_bootstrap
This command pushes docker images into GitLab Registry and then they are
used in testing. Pre-built images speed up testing.
Fixes #4156
commit 1a3d4fd8bc94f00f710838aaa9800d9aef7ccea3
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Sun Jun 30 21:47:42 2019 +0300
test: make vinyl/replica_rejoin more stable
The test checks that files left after rebootstrap are removed by the
garbage collector. It does that by printing file names to the result
file. This is inherently unstable, because should timing change, and
we can easily get an extra dump or compaction resulting in a different
set of files and hence test failure. Let's rewrite the test so that
it checks that files are actually removed using fio.path.exists().
commit 3d5da41c50974e387e14c677ef8e8526df8e83f8
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Sat Jun 29 22:51:09 2019 +0300
Replace ERRINJ_SNAP_WRITE_ROW_TIMEOUT with ERRINJ_SNAP_WRITE_DELAY
Timeout injections are unstable and difficult to use. Injecting a delay
is much more convenient.
commit ef25d66e8c5acf2d5fde2830b65856959573323a
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Sat Jun 29 21:33:10 2019 +0300
Add ERROR_INJECT_YIELD and ERROR_INJECT_SLEEP helpers
ERROR_INJECT_YIELD yields the current fiber execution by calling
fiber_sleep(0.001) while the given error injection is set.
ERROR_INJECT_SLEEP suspends the current thread execution by calling
usleep(1000) while the given error injection is set.
commit 92321d4ca1657f673f4593dad9ede1aec64da4b6
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 3 21:22:12 2019 +0300
ddl: don't use txn_last_stmt on _cluster commit/rollback
When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.
Note, replicas are still registered/unregisterd in the on_commit
trigger, but that's okay, as we don't really need _cluster space to
be in sync with the replica set.
commit 80eaa94de3df8b11d384b3e28296273586bf9dc4
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 3 20:27:50 2019 +0300
ddl: don't use txn_last_stmt on _ck_constraint commit/rollback
When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.
While we are at it, let's also make sure that changes are propagated
to Lua on replace, not on commit, by moving on_alter_space trigger
invocation appropriately.
commit ef5dfaf42de2e6ff1ebabb3d869b17262e9756d3
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 3 18:40:41 2019 +0300
ddl: don't use txn_last_stmt on _trigger commit/rollback
When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.
commit ff2575e9aa83e73cfc8c3f26d5db0a38c3ca0db5
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 3 15:36:59 2019 +0300
ddl: don't use txn_last_stmt on _collation commit/rollback
When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.
commit f430623846fdbd04b565f62a3a95e02d6d7be089
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed Jul 3 22:10:29 2019 +0300
ddl: restore sequence value if drop is rolled back
A sequence isn't supposed to roll back to the old value if the
transaction it was used in is aborted for some reason. However,
if a sequence is dropped, we do want to restore the original
value on rollback so that we don't lose it on an unsuccessful
attempt to drop the sequence.
commit 644c20b24fe3f87b9701cc2282a6808bdf379343
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue Jul 2 21:20:09 2019 +0300
ddl: fix _space_sequence rollback
_space_sequence changes are not rolled back properly. Fix it keeping in
mind that our ultimate goal is to implement transactional DDL, which
implies that all changes to the schema should be done synchronously,
i.e. on_replace, not on_commit.
commit 4be73c92a6462b80da5a6cefb141cc1ce70c28ff
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue Jul 2 18:17:49 2019 +0300
ddl: synchronize sequence cache with actual data state
To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.
Note, to do this in case of the sequence cache, we have to rework the
way sequences are exported to Lua - make on_alter_sequence similar to
how on_alter_space and on_alter_func triggers are implemented.
commit 01972ca14136687494753db5530109d2399dfa0d
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Jul 1 19:57:25 2019 +0300
ddl: synchronize func cache with actual data state
To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.
commit 9c1839ff98463125441b386ac1309efea2709927
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Jul 1 19:10:08 2019 +0300
ddl: synchronize user cache with actual data state
To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.
See also commit 22bedebe715c ("ddl: synchronize privileges cache with
actual data state").
commit 37f837e13a8c8e27cd3b827afcd564dd46f57409
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Jul 1 13:34:24 2019 +0300
ddl: unreference view on space drop synchronously
Do it on_replace rather than on_commit. This is required to implement
transactional DDL.
Note, this is the only place where on_replace_dd_space() postpones
schema update until after commit. Other than that, space updates are
already transactional DDL friendly.
commit 5046069bc6641b137c994f63b6c464e48a2ad257
Author: Serge Petrenko <sergepetrenko@tarantool.org>
Date: Wed Jun 26 18:27:21 2019 +0300
test: fix box/on_shutdown flakiness
Replace prints that indicate on_shutdown trigger execution with
log.warn, which is more reliable. This eliminates occasional test
failures. Also instead of waiting for the server to start and executing
grep_log, wait for the desired log entries to appear with wait_log.
Closes #4134
commit 1df9c3d2ab0f6e386e0b7c2a889a99a195052aa7
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed Jul 3 16:45:25 2019 +0300
test: update test-run
Fixed app/strict.test.lua fail after a test that sets 'a' global
variable. It was due to the way how 'strict' module tracks declared
global variables and was fixed in pretest_clean.lua test-run's module.
commit 59de57d23bbee02f2ac3cdecf83cb06eee5b2eb2
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Wed Jun 26 19:25:30 2019 +0300
box: introduce VARBINARY field type
A new VARBINARY field type would be useful for SQL type system.
Closes #4201
Needed for #4206
@TarantoolBot document
Title: new varbinary field type
Introduced a new field type varbinary to represent mp_bin values.
The new type varbinary may be used in format or index definition.
Example:
s = box.schema.space.create('withdata')
s:format({{"b", "varbinary"}})
pk = s:create_index('pk', {parts = {1, "varbinary"}})
commit 123c5aa5e061478e53a7c6b35cb82bff9bd78239
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon Jul 1 14:16:40 2019 +0300
schema: fix box.func cleanup after a function is dropped
Fixes commit abf5ef4fead9 ("box: export registered functions in box.func
folder").
commit d81d7b1caa36a5c2ef69d3b0013a63ef8454d25b
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Jul 1 13:40:37 2019 +0300
test: update test-run
* Implement SQL driver (#4123).
* Support line carrying with backslash.
* Support new result file format (for new tests).
See the link below for a full description of the new features.
https://github.com/tarantool/test-run/commit/a04b5b096c607172ce4fc86a84e3531c9f3a7304
Fixes #4123.
commit b6b72013137a74ab779e9c0c3d23c041819a3071
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Jun 28 01:13:25 2019 +0200
swim: default generation is timestamp
Generation is supposed to be a persistent counter to distinguish
between different installations of the same SWIM instance. By
default it was set to 0, which was quite unsafe.
Kostja proposed an easy and bright solution - generation could be
set to timestamp by default. In such a case on each restart it is
almost 100% will be different.
Follow up #4280
commit ea1e9192ee9e1fafec206aa74081b4451fac24c4
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Jun 28 01:04:41 2019 +0200
swim: fix inability to set generation only
swim.new() is declared as allowed to be called before swim:cfg().
But in fact swim.new({generation = ...}) didn't work because
after generation extraction the empty config {} was passed to
swim:cfg() and led to an error.
The patch allows to call swim.new() with generation only, as well
as without parameters at all.
Follow up #4280
commit 98f296454d4c716aa92a7e0a80e41886960d7f01
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Jun 28 00:52:34 2019 +0200
swim: fix a dangerous yield in ffi.gc
FFI can't survive yields. A yield in ffi.C.func() leads to a
crash; yield in ffi.gc is not documented as allowed. Yield in any
GC function leads to garbage collector stuck until the yield is
finished.
This patch makes SWIM GC callback non-yielding. Now yielding
swim_delete() is called in a separate fiber created in GC
callback, but started at the end of event loop only.
Follow up #3234
commit eb40359821020d34662cf4edad186976e5f4bee4
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri Jun 28 00:40:23 2019 +0200
swim: fix a leak when a trigger is installed
SWIM wraps user triggers to prepare arguments. The wrapper
function kept a reference to SWIM object, and prevented its
automatic deletion at GC.
The patch makes this reference weak.
Follow up #4250
commit 7418c373935158582cce175990a5dde4873c0eeb
Author: Mergen Imeev <imeevma@gmail.com>
Date: Sat May 18 13:15:58 2019 +0300
sql: allow to use vectors as left value of IN operator
In SQL, it is allowed to use vector expressions, that is, an
operation that uses vectors as operands. For instance, vector
comparison:
SELECT (1,2,3) < (1,2,4);
Accidentally, routines handling IN operator contained a bug: in
cases where we used a vector as the left value in the IN operator,
we received an assertion in debug build or a segmentation fault in
release. This was due to some legacy code in which it was assumed
that the left value of the IN operator can have only one column in
case it is a vector. Let's fix this by allowing vectors of the
other sizes as the left value of the IN operator and providing
check which verifies that both sides of IN operator have the same
dimension.
Closes #4204
commit 3f5806ab17ae706281c88a4244ba59aa9ad35a93
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Sat Jun 22 00:48:21 2019 +0300
test: fix flaky test
This test fails sporadically
commit 0529e2f0ef7dd4c19283b9e8e6b0e7e43bf400a3
Author: Mergen Imeev <imeevma@gmail.com>
Date: Sat Jun 22 12:07:15 2019 +0300
netbox: optimize netbox requests
This patch optimizes some netbox requests that were slowed down in
commit 665fc3a19c.
Follow-up #2978
commit 8c84932ade1b70971890dfd27d72b3145834594b
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Fri Jun 7 22:24:20 2019 +0300
applier: apply transaction in parallel
Applier use asynchronous transaction to batch journal writes. All
appliers share the replicaset.applier.tx_vclock which means the vclock
applied but not necessarily written to a journal. Appliers use a trigger
to coordinate in case of failure - when a transaction is going to
be rolled back.
Closes: #1254
commit 18ea440aba1216d61086dfd3e2bf2bd4055ffe74
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Wed Apr 10 20:44:58 2019 +0300
txn: introduce asynchronous txn commit
This commit implements asynchronous transaction processing using
txn_write. The method prepares a transaction and sends it to an journal
without an yield until the transaction was finished. The transaction
status could be controlled via on_commit/on_rollback triggers.
In order to support asynchronous transaction journal_write method turned
to an asynchronous one and now a transaction engine controls journal status
using journal entry finalization callback.
Prerequisites: #1254
commit a16ff8693b64e74a10e508055abe34e49c700d55
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Wed Jun 19 00:29:05 2019 +0300
wal: introduce a journal entry finalization callback
Finalize a transaction thorough a journal entry callback. So transaction
processing doesn't rely on fiber schedule.
Also allow to steal locked latch ownership for fiber which isn't owner
of the latch. This is required to process transaction triggers
asynchronously.
Prerequisites: #1254
commit 68404cfd662ec26fda33ee740904432df023223c
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Wed Apr 10 17:22:42 2019 +0300
txn: get rid of fiber_gc from txn_rollback
Refactoring: don't touch a fiber gc storage on a transaction rollback
explicitly. This relaxes dependencies between fiber and transaction
life cycles.
Prerequisites: #1254
commit e070cc4d91a45be4aaeebda440b5374268dad753
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Thu Apr 4 18:11:45 2019 +0300
txn: get rid of autocommit from a txn structure
Move transaction auto start and auto commit behavior to the box level.
From now a transaction won't start and commit automatically without
txn_begin/txn_commit invocations. This is a part of a bigger transaction
refactoring in order to implement detachable transactions and a parallel
applier.
Prerequisites: #1254
commit 399e2b653c27662559f795d941ed171b0703e80d
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Fri Jun 21 13:44:19 2019 +0300
txn: unref statement at txn_free
Refactoring: put txn statement unref code into transaction free function.
commit 564ba89aceae35311beebaf4c8c3042cda60ac00
Author: Stanislav Zudin <szudin@tarantool.org>
Date: Wed Jun 19 12:28:02 2019 +0300
ddl: No replication for temp and local spaces
Do not spread the space:truncate() to replicas if the
affected space is local or temporary.
Closes #4263
commit a758bced7c81e3a162a4f8e9c1bb3844db684825
Author: Konstantin Osipov <kostja@tarantool.org>
Date: Tue Jun 25 11:00:23 2019 +0300
Revert "box: introduce Lua persistent functions"
This reverts commit 4e3470ceb0b164479ffd29750b83173bbb936813.
The RFC did not pass review yet.
