Git log: commit d07e3ac2facfc5c5cec43ff138e2bdbd1fab1011
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed May 22 22:05:11 2019 +0300
test/luajit-tap: Add table_chain_bug_LuaJIT_494.test.lua
Backport of openresty/luajit2-test-suite
commit ce2c916d5582914edeb9499f487d9fa812632c5c
To test hash chain bug.
Part-of #4171
commit 925f923f598d7e05a3429cea6955f031fa456b90
Author: Kirill Yukhin <kyukhin@tarantool.org>
Date: Thu May 23 13:10:15 2019 +0300
Bump luajit submodule
commit b3e924de48c8029995ab4ca08d2dac49ccab7057
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Thu May 23 10:09:40 2019 +0300
box: ban multikey index with multiple root field
The test for multikey index prefix compatibility was insufficient
because JSON path is relative for some fieldno. Those root field
identifiers also must coincide.
Follow up #1257
commit c0a7556cb1453895b77dcb1f368721e1820e126c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 22 20:33:30 2019 +0300
swim: allow to set codec before cfg
One another problem discovered with UDP broadcast test is that
it can affect other tests, even after termination. Doing
swim:broadcast() on one test a programmer can't be sure, who will
listen it, answer, and break the test scenario.
This commit reduces probability of such a problem by
* allowance to set a codec before swim:cfg(). It allows to
protect SWIM nodes of different tests from each other -
they will not understand messages from other tests. By the
way, the same problem can appear in real applications too;
* do not binding again a URI passed by test-run into the
test and closed here. If a test closes a URI given to it,
it can't be sure, that next bind() will be successful -
test-run could already reuse it.
Follow up #3234
commit 7a5ac3d74e944e8cec146d92a74fdf7e7dd9e8d7
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 22 19:45:08 2019 +0300
swim: be ready to idle round steps when net is slow
First of all, the problem in a nutshell was that ev_timer with
non-zero 'repeat' field in fact is a ev_periodic. It is restarted
*automatically*, even if a user does not write ev_timer_again()
nor ev_timer_start().
This led to a situation, that a round message send is scheduled,
and next round step timer alarm happens before the message is
actually sent. It, in turn, led to an assertion on attempt to
schedule a task twice.
This patch fixes the swim test harness to behave like ev_timer
with 'repeat' > 0, and on first idle round step stops the timer -
it will be restarted once the currently hanging task will be
finally sent.
Follow up #3234
commit cda363f3522d463dd38901b29ec2b17b4674d516
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 22 20:00:56 2019 +0300
swim: fix flaky parts in swim/swim.test.lua
They are caused by
* too slow network, when SWIM tests are run under high load;
* UDP packets late arrival or drop.
Follow up #3234
commit 8a20035facc71cd409d927190bac9e9b8d1493c3
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 22 12:15:57 2019 +0300
swim: fix an obvious use-after-free
Follow up #3234
commit bae689017c5b1eaae4fde3de4de5b09899199c8d
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 22 04:07:24 2019 +0300
swim: fix an obvious leak in swim_delete()
Follow up #3234
commit aad779db4511f48845cb82f4c73dcb9dfe3b854b
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 22 18:09:32 2019 +0300
tuple format: remove invalid assertion from tuple_format_iterator_next
It's too early to assert msgpack type as an array when a multikey field
is encountered - we haven't checked the field type yet so the type might
as well be a map, in which case we will raise an error just a few lines
below. Remove the assertion and add a test case.
commit e1d3fe8ab8eed65394ad17409401a93b6fcdc435
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 22 16:30:15 2019 +0300
tuple format: don't allow null where array/map is expected
If an indexed field expects array/map, it shouldn't be allowed to insert
null instead, because this might break expectations of field accessors.
For unikey indexes inserting null instead of array/map works fine though
somewhat confusing: for a non-nullable field you get a wrong error
message ("field is missing" instead of "array/map expected, got nil");
for a nullable field, this silently works, just looks weird as there's a
clear type mismatch here. However, for a multikey field you get a crash
as tuple_multikey_count() doesn't expect to see null where an array
should be according to the format:
tuple_raw_multikey_count: Assertion `mp_typeof(*array_raw) == MP_ARRAY' failed.
This issue exists, because we assume all fields are nullable by default
for some reason. Fix that and add some tests.
Note, you can still omit nullable fields, e.g. if field "[2].a[1]" is
nullable you may insert tuple [1, {a = {}}] or [1, {b = 1}] or even [1],
you just can't pass box.NULL instead of an array/map.
commit b1828dd4051d14257e7a70bfbda800e126cc8df4
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Tue May 21 18:25:45 2019 +0300
box: fix assert with multikey hybrid index
Tarantool used to assume that offset_slot has an extension
iff field_map_get_offset is called with multikey_idx >= 0.
In fact, when some part of the index contains a multikey index
placeholder, tuple_compare_* routines pass a tuple_hint in
meaning of multikey index for each tuple_field_raw_by_part call,
even for regular key_part that doesn't have array index
placeholder (and, correspondingly, field_map extension).
Thus this assumption is invalid.
This patch uses the fact that field_map slots that have extension
store negative offset to distinguish multikey and normal usage
of the field_map_get_offset routine.
Closes #4234
commit eca5c292ddbade17ded53710c1411df69d593329
Author: Konstantin Osipov <kostja@tarantool.org>
Date: Wed May 22 00:21:45 2019 +0300
Update README.md
commit a1ec38f3e0541742fc1c7a78f66df9bc10077beb
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 21 19:59:56 2019 +0300
swim: interpret no-payload as nil
Empty string as a no-payload-flag was not a good idea, because
then a user can't write something like:
if not member:payload() then
...
Follow up #3234
commit 2dc1af756d07a29f3cc208d1b0cead6175e5a0e3
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu May 16 20:53:27 2019 +0300
swim: swim:set_codec() Lua API
Encryption with an arbitrary algorithm and any mode with a
configurable private key.
Closes #3234
commit 6137c197f27d8e9a23fbbc8b69dab898f7f02974
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon Apr 29 19:08:29 2019 +0300
swim: implement and expose transport-level encryption
SWIM is going to be used in and between datacenters, which means,
that its packets will go through public networks. Therefore raw
SWIM packets are vulnerable to attacks.
An attacker can do any and all of the following things:
1) Extract secret information from member payloads, like
credentials to Tarantool binary ports;
2) Change UUIDs and addresses in the packets and break a
topology;
3) Catch the packets and pretend being a Tarantool instance,
which could lead to undefined behaviour depending on an
application logic.
SWIM packets need a protection layer. This commit introduces it.
SWIM transport level allows to choose an encryption algorithm
with a private key to encrypt each packet with that key.
Besides, each packet is encrypted using a random public key
prepended to the packet.
SWIM now provides a public API to choose an encryption algorithm
and a private key.
Part of #3234
commit f77f4b9e51ea94203dc848cbadff00e2a6961164
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon Apr 29 18:49:51 2019 +0300
swim: split send/recv into phases
At this moment swim_scheduler_on_output() is a relatively simple
function. It takes a task, builds its meta and flushes a result
into the network. But soon SWIM will be able to encrypt messages.
It means, that in addition to regular preprocessing like building
meta headers a new phase will appear - encryption. What is more -
conditional encryption, because a user may want to do not encrypt
messages.
All the same is about swim_scheduler_on_input() - if a SWIM
instance uses encryption, it should decrypt incoming messages
before forwarding them into the SWIM core logic.
The chosen strategy - lets reuse on_output/on_input virtuality
and create two version of on_input/on_output functions:
swim_on_plain_input() | swim_on_encrypted_input()
swim_on_plain_output() | swim_on_encrypted_output()
One of these pairs is chosen depending on if the instance uses
encryption.
To make these 4 functions as simple and short as possible this
commit creates two sets of functions, doing all the logic except
encryption:
swim_begin_send()
swim_do_send()
swim_complete_send()
swim_begin_recv()
swim_do_recv()
swim_complete_recv()
These functions will be used by on_input/on_output functions with
different arguments.
Part of #3234
commit 39dd852da7f8448607aadd5f753073c646838f2e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 19:32:06 2019 +0300
swim: cache members in Lua member table
Each time a member was returned from a SWIM instance object, it
was wrapped by a table with a special metatable, cached payload.
But next the same lookup returned a new table. It
- created garbage as a new member wrapper;
- lost cached decoded payload.
This commit caches in a private table all wrapped members and
returns an existing wrapper on a next lookup. A microbenchmark
showed, that cached result retrieval is 10 times faster, than
each time create a new table.
Cache table keeps week references - it means, that when a member
object looses all its references in a user's application, it is
automatically dropped from the table.
Part of #3234
commit 8ae88a3fe7c81200b76aae65dd6627aeb54c2792
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 19:09:53 2019 +0300
swim: cache decoded payload in the Lua module
Users of Lua SWIM module likely will use Lua objects as a
payload. Lua objects are serialized into MessagePack
automatically, and deserialized back on other instances. But
deserialization of 1.2Kb payload on each member:payload()
invocation is quite heavy operation. This commit caches decoded
payloads to return them again until change.
A microbenchmark showed, that cached payload is returned ~100
times faster, than it is decoded each time. Even though a tested
payload was quite small and simple:
s:set_payload({a = 100, b = 200})
Even this payload is returned 100 times faster, and does not
affect GC.
Part of #3234
commit 70e993232e2f3c8bafc08d41e8ad5b317dbc4872
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 17:21:31 2019 +0300
swim: allow to use cdata struct tt_uuid in Lua API
Sometimes, especially in tests, it is useful to make something
like this:
s:add_member({uuid = member:uuid(), uri = member:uri()})
But member:uuid() is cdata struct tt_uuid. This commit allows
that.
Part of #3234
commit 248f2425e09c3bae7033253bb478d1270108764a
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 13 17:29:28 2019 +0300
swim: pairs() function to iterate over member table
Expose iterators API to be able to iterate over a member table in
a 'for' loop like it would just be a Lua table.
Part of #3234
commit 4b8c872351dee06e13f443bb56e877a6df960fb7
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 8 21:21:51 2019 +0300
swim: Lua bindings to access individual members
Expose API to search members by UUID, to read their attributes,
to set payload.
Part of #3234
commit 1203bf90b7ede74ae178da586745aada8eba675c
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 8 19:37:59 2019 +0300
swim: Lua bindings to manipulate member table
Expose methods to add, remove, probe members by uri, uuid. Expose
broadcast method to probe multiple members by port.
Part of #3234
commit 407adf4006e5ec69efbf39879d7cacaeb9fc245a
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 6 16:19:55 2019 +0300
swim: introduce Lua interface
SWIM as a library can be useful not only for server internals,
but for users as well. This commit exposes Lua bindings to SWIM
C API. Here only basic bindings are introduced to create, delete,
quit, check a SWIM instance. With sanity tests.
Part of #3234
commit ae651ab400b2774e5e93650041725bdca5d1ca6b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 20:45:58 2019 +0300
swim: validate URI in swim_probe_member()
Similar methods validate their arguments: add_member,
remove_member. Validate here as well for consistency.
Part of #3234
commit dcf4fbdf1eabf4d50bf22ea1a31a76dd66606525
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 18:46:50 2019 +0300
swim: make swim_new_round() void
Firstly, I thought that there is an error - swim_begin_step()
does not reschedules round timer, when new_round() fails. But
then new_round() appeared never failing. This commit makes it
void to eliminate confusion.
Probably it is a legacy since the shuffled members array was
allocated and freed in new_round().
Part of #3234
commit 10ad9706f0a1a1b63568983abe2f3add4f4e05b5
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 15 18:44:02 2019 +0300
swim: fix an assertion on attempt to chage timeouts
Appeared, that libev does not allow to change ev_timer values in
flight. A timer, reset via ev_timer_set(), should be restarted,
because the function changes 'ev_timer.at', which in turn is used
internally by timer routines.
Part of #3234
commit a2c19aa812754451f6c828ce074bf7443847496e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun May 19 01:00:38 2019 +0300
buffer: replace all ffi.new(type[1]) with cached union
Lua, which suffers from lack of ability to pass values by
pointers into FFI functions, nor has an address operator '&' to
take an address of integer or char or anything. Because of that
a user need to either use ffi.new(type[1]) or use static buffer,
but for such small allocations they are both too expensive and
aggravate GC problem.
Now buffer module provides preallocated basic types to use in FFI
functions. The commit is motivated by one another place where
ffi.new('int[1]') appeared, in SWIM module, to obtain payload
size as an out parameter of a C function.
commit 7fd6c8096343ca254fec10c2ce8a8233ebe47028
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat May 18 22:34:33 2019 +0300
buffer: port static allocator to Lua
Static allocator gives memory blocks from cyclic BSS memory
block of 3 pages 4096 bytes each. It is much faster than
malloc, when a temporary buffer is needed. Moreover, it does not
complicate GC job. Despite being faster than malloc, it is still
slower, than ffi.new() of size <= 128 known in advance (according
to microbenchmarks).
ffi.new(size<=128) works either much faster or with the same
speed as static_alloc, because internally FFI allocator library
caches small blocks and can return them without malloc().
A simple micro benchmark showed, that ffi.new() vs
buffer.static_alloc() is ~100 times slower on allocations of
> 128 size, on <= 128 when size is not inlined.
To better understand what is meant as 'inlined': this
ffi.new('char[?]', < <=128 >)
works ~100 times faster than this:
local size = <= 128
ffi.new('char[?]', size)
ffi.new() with inlined size <= 128 works faster than light, and
even static allocator can't beat it.
commit 9566f14ccda537e2e4e4f9bbeb1bf69d1e2cb7f0
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 21 14:36:43 2019 +0300
vinyl: fix assertion while recovering dumped statement
Certain kinds of DML requests don't update secondary indexes, e.g.
UPDATE that doesn't touch secondary index parts or DELETE for which
generation of secondary index statements is deferred. For such a request
vy_is_committed(env, space) may return false on recovery even if it has
actually been dumped: since such a statement is not dumped for secondary
indexes, secondary index's vy_lsm::dump_lsn may be less than such
statement's signature, which makes vy_is_committed() assume that the
statement hasn't been dumped. Further in the code we have checks that
ensure that if we execute a request on recovery, it must not be dumped
for the primary index (as the primary index is always dumped after
secondary indexes for the sake of recovery), which fires in this case.
To fix that, let's refactor the code basing on the following two facts:
- Primary index is always updated by a DML request.
- Primary index may only be dumped after secondary indexes.
Closes #4222
commit 3cf75bc8060f5f73bcbb9fe342cd21bed4ea63f2
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue May 21 13:41:34 2019 +0300
build: fix strip_core with in-source build
Yet another fix for building of small library as part of tarantool.
Before this commit slab_arena test fails:
| [019] Test failed! Result content mismatch:
| [019] --- small/slab_arena.result Mon May 20 21:37:46 2019
| [019] +++ small/slab_arena.reject Mon May 20 21:47:01 2019
| [019] @@ -23,3 +23,4 @@
| [019] arena->maxalloc = 2000896
| [019] arena->used = 0
| [019] arena->slab_size = 65536
| [019] +ERROR: Expected dd flag on VMA address 0x7f3ec2080000
See the corresponding commit in the small submdoule for more info.
commit a6a33e9f344a8e798720f699ff79720e35c03120
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 15 16:39:48 2019 +0300
box: fix autoincrement for json path indexes
The autoincrement code was written when there were no nested field.
Now, it isn't enough to just skip to the autoincrement field - we also
need to descend deeper if key_part->path is set.
Note, the code expects the nested field to be present and set to NULL.
That is, if field path is [1].a.b, the tuple must have all intermediate
fields set: {{a = {b = box.NULL}}} (usage of box.NULL is mandatory to
create a tuple like that in Lua).
Closes #4210
commit 60111894457e30cec17e5df0ea1b57fbf80c7898
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 14 15:44:21 2019 +0300
schema: allow to set sequence for any index part, not just the first
Closes #4009
@TarantoolBot document
Title: Sequence can now be set for an index part other than the first
Initially one could attach a sequence (aka autoincrement) only to the
first index part. Now it's possible to attach a sequence to any primary
index part. The part still must be integer though.
Syntax:
```
box.schema.space.create('test')
box.space.test:create_index('primary', {
parts = {{1, 'string'}, {2, 'unsigned'}, {3, 'unsigned'}},
sequence = true, sequence_part = 2
})
box.space.test:insert{'a', box.null, 1} -- inserts {'a', 1, 1}
```
Note, `sequence_part` option is 1-base.
If `sequence_part` is omitted, 1 is used, which assures backward
compatibility with the original behavior.
One can also attach a sequence to another index part using
`index.alter` (the code below continues the example above):
```
box.space.test.index.primary:alter{sequence_part = 3}
box.space.test:insert{'a', 1, box.null, 'x'} -- inserts {'a', 1, 2, 'x'}
```
commit 7d778de63cf1e0d30fa4767c15a18f880b4e107b
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 14 21:01:06 2019 +0300
schema: fix error while altering index with sequence
A check was missing in index.alter. This resulted in an attempt to drop
the sequence attached to the altered index even if the sequence was not
modified.
Closes #4214
commit f47f2004abc75cc5911fd797071ccea87304c7b7
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 14 18:19:12 2019 +0300
schema: use tuple field names in Lua
When schema.lua was introduced, there was no such thing as space format
and we had to access tuple fields by no. Now we can use human readable
names. Let's do it - this should improve code readability.
A note about box/alter.test.lua: for some reason it clears format of
_space and _index system spaces, which apparently breaks our assumption
about field names. Let's zap those pointless test cases.
commit 612f05b27ba1d467dd7180e0c6812a85ca3685a9
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon May 20 21:28:57 2019 +0300
build: fix out of source build
Updated small submodule with the corresponding fix.
commit b6466ac7dc225ed9c70322eaf8c7eca746b54130
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri May 17 17:36:24 2019 +0300
coio: fix getaddrinfo assertion on 0 timeout
Background. Coio provides a way to schedule arbitrary tasks
execution in worker threads. A task consists of a function to
execute, and a custom destructor.
To push a task the function coio_task_post(task, timeout) was
used. When the function returns 0, a caller can obtain a result
and should free the task manually. But the trick is that if
timeout was 0, the task was posted in a detached state. A
detached task frees its memory automatically despite
coio_task_post() result, and does not even yield. Such a task
object can't be accessed and so much the more freed manually.
coio_getaddrinfo() used coio_task_post() and freed the task when
the latter function returned 0. It led to double free when
timeout was set 0. The bug was introduced here
800cec7311955e1d9dfd2b9f881712d002090f3a in an attempt to do not
yield in say_logrotate, because it is not fiber-safe.
Now there are two functions: coio_task_execute(task, timeout),
which never detaches a task completed successfully, and
coio_task_post(task), which posts a task in a detached state.
Closes #4209
commit 9c4f1c8ae401dae1962f94c0f5d0cfd7b691d476
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri May 17 15:17:35 2019 +0300
coio: make hints in coio_getaddrinfo optional
According to the standard by Open Group, getaddrinfo() hints
argument is optional - it can be NULL. When it is NULL, hints
is assumed to have 0 in ai_flags, ai_socktype, and ai_protocol;
AF_UNSPEC in ai_family.
See The Open Group Base Specifications.
commit 632e3160a9841633c57bafa39ab5256320b9f9b0
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon May 20 17:42:34 2019 +0300
test: fix a test after box.cfg.strip_core addition
The result file of the test app-tap/init_script.test.lua was not updated
in 549140b390f2db3a29dbba2d44170d215f4c7700 ('box/memtx: Allow to skip
tuple memory from coredump').
Follow up #3509.
commit e55396c87e2560540bc46f3b8935bee8d8b057d7
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue May 14 15:46:02 2019 +0300
travis-ci: fix LTO and clang
Made fixes:
- Added CMAKE_EXTRA_PARAMS environment to docker's container
runs to enable -DENABLE_LTO=ON/OFF cmake option.
- Added CC/CXX environment to docker's container runs to set
clang for cmake. Also the additional environment variables
{CC,CXX}_FOR_BUILD were postponed, because we didn't
run cross-compilation at the moment, for more info check:
https://docs.travis-ci.com/user/languages/cpp/#choosing-compilers-to-test-against
- Changed LTO docker's image to 'debian-buster' due to LTO needed
higher versions of packages, check for more information commit:
f9e28ce4602aff3f9bb4e743b0d6167b0f8df88d ('Add LTO support')
- Fixed sources to avoid of failures on builds by GCC with LTO:
1) src/box/memtx_rtree.c: In function ‘mp_decode_rect’:
src/box/memtx_rtree.c:86:24: error: ‘c’ may be used uninitialized
in this function [-Werror=maybe-uninitialized]
rect->coords[i * 2] = c;
^
src/box/memtx_rtree.c:74:10: note: ‘c’ was declared here
coord_t c;
^
2) src/box/sql/func.c: In function ‘quoteFunc’:
src/box/sql/func.c:1103:3: error: ‘b’ may be used uninitialized
in this function [-Werror=maybe-uninitialized]
sql_result_text(context, sql_value_boolean(argv[0]) ?
^
src/box/sql/vdbeapi.c:217:7: note: ‘b’ was declared here
bool b;
^
3) src/box/tuple_update.c: In function ‘update_read_ops’:
src/box/tuple_update.c:1022:4: error: ‘field_no’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
diag_set(ClientError, ER_NO_SUCH_FIELD_NO, field_no);
^
src/box/tuple_update.c:1014:11: note: ‘field_no’ was declared here
int32_t field_no;
^
4) src/httpc.c: In function ‘httpc_set_verbose’:
src/httpc.c:267:2: error: call to ‘_curl_easy_setopt_err_long’
declared with attribute warning: curl_easy_setopt expects a long
argument for this option [-Werror]
curl_easy_setopt(req->curl_request.easy, CURLOPT_VERBOSE, curl_verbose);
^
5) src/lua/httpc.c: In function ‘luaT_httpc_request’:
src/lua/httpc.c:128:64: error: ‘MEM[(int *)&parser + 20B]’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
lua_pushinteger(L, (parser.http_minor > 0) ? parser.http_minor: 0);
^
src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 20B]’ was declared here
struct http_parser parser;
^
src/lua/httpc.c:124:64: error: ‘MEM[(int *)&parser + 16B]’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
lua_pushinteger(L, (parser.http_major > 0) ? parser.http_major: 0);
^
src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 16B]’ was declared here
struct http_parser parser;
^
Close #4215
commit 549140b390f2db3a29dbba2d44170d215f4c7700
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Fri May 17 00:45:59 2019 +0300
box/memtx: Allow to skip tuple memory from coredump
In case if there are huge amount of tuples the whole
memory goes to coredump file even if we don't need it
for problem investigation. In result coredump may
blow up to gigabytes in size.
Lets allow to exclude this memory from dumping via
box.cfg::strip_core boolean parameter.
Note that the tuple's arena is used not only for tuples
themselves but for memtx->index_extent_pool and
memtx->iterator_pool as well, so they are affected
too.
Fixes #3509
@TarantoolBot document
Title: Document box.cfg.strip_core
When Tarantool runs under a heavy load the memory allocated
for tuples may be very huge in size and to eliminate this
memory from being present in `coredump` file the `box.cfg.strip_core`
parameter should be set to `true`.
The default value is `false`.
commit 10873f164d2abe8c6ff2d8491f0b2df578b80142
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Fri May 17 14:42:19 2019 +0300
msgpack: validate msgpack.decode() cdata size argument
Negative size led to an assertion. The commit adds a check if
size is negative.
Closes #4224
commit 539aee3d07c9abe654108cd9ed960efc6182c856
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed May 15 23:07:35 2019 +0300
iproto: init coio watcher before join/subscribe
box_process_join() and box_process_subscribe() use coio_write_xrow(),
which calls coio_writev_timeout() under hood. If a socket will block at
write() the function calls ev_io_start() to wake the fiber up when the
socket will be ready to write. This code assumes that the watcher
(struct ev_io) is initialized as coio watcher, i.e. coio_create() has
been called.
The reason why the code works before is that coio_write_xrow() in
box_process_{join,subscribe}() writes a small piece of data and so the
situation when a socket write buffer has less free space then needed is
rare.
Fixes #4110.
commit e4fc22d00d1a9088631fe8eae0d08157834d94d8
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Sat May 18 23:25:08 2019 +0300
Update small library
commit afc5f92d27c4cf8197e8a3dd2463232021f42895
Author: Mergen Imeev <imeevma@gmail.com>
Date: Sat Apr 27 18:03:32 2019 +0300
sql: replace schema_find_id() by box_space_id_by_name()
This patch replaces schema_find_id() with box_space_id_by_name()
in SQL. The box_space_id_by_name() is more specialized. In
addition, it checks if the user has sufficient rights, unlike
schema_find_id().
Closes #3570
commit d039fc05348b5268d700c41736406152f39a1714
Author: Mergen Imeev <imeevma@gmail.com>
Date: Wed May 15 19:56:05 2019 +0300
sql: stop parser on error
This patch stops the parser if any error occurs. Prior to this
patch, it was possible to replace the error with another one,
since the parser may continue to work, even if an error occurred.
For example:
box.execute("insert into not_exist values(1) a")
The first error is "Space 'NOT_EXIST' does not exist", but "Syntax
error near 'a'" is displayed.
After this patch, the first error will be displayed.
Closes #3964
Closes #4195
commit e6869dd2e834c4437ddc49734a33ad171915ebb9
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Tue May 14 23:49:14 2019 +0300
travis-ci: add fedora 30
Fixes #4194.
commit c783c8b705839ef8dbb97a0d24cdd95400ba9ba1
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Fri May 17 09:32:30 2019 +0300
Remove schema latch relock
As we enforced applier row order so we don't need to reacquire schema
latch after a ddl statement.
Followup for: 056deb2cf0488486fd667c6161eda8f7bdbf2096
commit 46e7d24a2b88f5c5173be3f24ab54816124baa8f
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu May 16 04:31:15 2019 +0300
test: update test-run
Support more then 60 parallel jobs (#82, PR #171).
commit 544d648c9926cd70de2a19feca569cc5246bad16
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 7 22:04:20 2019 +0300
crypto: use crypto library in crypto.lua
crypto.lua is a public module using OpenSSL directly. But now
lib/crypto encapsulates OpenSSL with additional checks and
similar but more conforming API. It allows to replace OpenSSL
cipher in crypto.lua with lib/crypto methods.
commit 4467fdac4857d4510758cacbc7d6cc4d84a7caa5
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Apr 25 21:19:43 2019 +0300
crypto: implement crypto libary
OpenSSL API is quite complex and hard to follow, additionally it
is very unstable. Encoding/decoding via OpenSSL methods usually
consists of multiple calls of a lot of functions. This patch
wraps OpenSSL API with one more easy to use and conforming
Tarantool code style in scope of crypto library.
The traditional OpenSSL API is wrapped as well in a form of
crypto_stream object, so OpenSSL API is not cut off.
Besides struct crypto_stream the library provides struct
crypto_codec which encapsulates all the steps of encryption logic
in two short functions:
crypto_codec_encrypt/decrypt(iv, in, in_size, out, out_size)
A caller can create a needed codec via crypto_codec_new, which
now supports all the same algorithms as crypto.lua module.
Needed for #3234
commit 0c7183450dc05dc353bf28544fa730178f65eef2
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Apr 25 14:31:50 2019 +0300
crypto: make exported methods conform code style
Tarantool has a strict rule for naming methods of libraries - use
the library name as a prefix. For crypto lib methods it should be
'crypto_', not 'tnt_'.
commit 1b9d3c6b5fe13353730fa064128b1c9d1239604f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Thu Apr 25 13:49:23 2019 +0300
crypto: move crypto business into a separate library
Crypto in Tarantool core was implemented and used very poorly
uintil now. It was just a one tiny file with one-line wrappers
around OpenSSL API. Despite being small and simple, it provided a
powerful interface to the Lua land used by Lua 'crypto' public
and documented module.
Now the time comes when OpenSSL crypto features are wanted on
lower level and with richer API, in core library SWIM written
in C. This patch moves crypto wrappers into a separate library
in src/lib, and drops some methods from the header file because
they are never used from C, and are needed for exporting only.
Needed for #3234
commit 0891b159f3372568cac2dd6761acfe7683c4debb
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 14 15:23:09 2019 +0300
Drop an unused function and class
commit 453fff2b215e0c2e2d4c5951fae711329eff7b48
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 14 17:04:22 2019 +0300
msgpack: allow to pass 'const char *' into decode()
msgpack.decode() internally uses 'const char *' variable to
decode msgpack, but somewhy expects only 'char *' as input.
This commit allows to pass 'const char *' as well.
commit 7d98a3cfac330a1f32b388a3725c75566d1e0878
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 14 15:00:30 2019 +0300
msgpack: allow to pass 'struct ibuf *' into encode()
Before the patch msgpack Lua module provided a method encode()
able to take a custom buffer to encode into. But it should be of
type 'struct ibuf', what made it impossible to use
buffer.IBUF_SHARED as a buffer, because its type is
'struct ibuf *'. Strangely, but FFI can't convert these types
automatically.
This commit allows to use 'struct ibuf *' as well, and moves this
functionality into a function in utils.h. Now both msgpack and
merger modules can use ibuf directly and by pointer.
commit 8cb0206bf0db182843f733b1441c6f17c9f75c3b
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 14 20:38:02 2019 +0300
swim: set 'left' status in self on swim_quit()
swim_quit() notifies all the members that this instance has left
the cluster. Strangely, except self. It is not a real bug, but
showing 'left' status in self struct swim_member would be more
correct than 'alive', obviously.
It is possible, that self struct swim_member was referenced by a
user - this is how 'self' can be available after SWIM instance
deletion.
Part of #3234
commit 2149496b6cf65065ed1785ce751b5c9bfab9ff1f
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 13 19:27:29 2019 +0300
swim: do not rebind when new 'port' is 0
SWIM internally tries to avoid unnecessary close+socket+bind
calls on reconfiguration if a new URI is the same as an old one.
SWIM transport compares <IP, port> couples and if they are
equal, does nothing.
But if a port is 0, it is not a real port, but a sign to the
kernel to find any free port on the IP address. In such a case
SWIM transport after bind() retrieves and saves a real port.
When the same URI is specified again, the transport compares two
addresses: old <IP, auto found port>, new <IP, 0>, sees they are
'different', and rebinds. It is not necessary, obviously, because
the new URI covers the old one.
This commit avoids rebind, when new IP == old IP, and new port is
0.
Part of #3234
commit 513c196a10d0f67a52a69fc3b490e5f8545dc4dc
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 13 18:04:01 2019 +0300
swim: encapsulate 'uint16' payload size inside swim.c
uint16 was used in public SWIM C API as a type for payload size
to emphasize its small value. But it is not useful in Lua,
because Lua API should explicitly check if a number overflows
uint16 maximal value, and return the same error as in case it is
< uint16_max, but > payload_size_max.
So main motivation of the patch is to avoid unnecessary checks in
Lua and error message duplication. Internally payload size is
still uint16.
commit 3fc4000142983bae63b3ced46b8692b071c1b1d5
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Mon May 13 17:38:10 2019 +0300
swim: drop swim_info() function
Swim_info() was a function to dump SWIM instance info to a Lua
table without explicit usage of Lua. But now all the info can be
taken from 1) self member and member API, 2) cached cfg options
as a Lua table in a forthcoming Lua API - this is how
box.cfg.<index> works.
commit 42b7f6d84611233eb3471fcd35ff51a3e6b573c3
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed May 15 05:52:27 2019 +0300
test: update test-run
- Fix killing of servers at crash (PR #167).
- Show logs for a non-default server failed at start (#159, PR #168).
- Fix TAP13 hung test reporting (#155, PR #169).
- Fix false positive internal error detection (PR #170).
commit d11b552e3f7266e7dfaddd9ead6fc7dd53a9cae1
Author: Ilya Konyukhov <runsfor@gmail.com>
Date: Sat Mar 23 15:46:37 2019 +0300
httpc: add MAX_TOTAL_CONNECTIONS option binding
Right now there is only one option which is configurable for http
client. That is CURLMOPT_MAXCONNECTS. It can be setup like this:
> httpc = require('http.client').new({max_connections = 16})
Basically, this option tells curl to maintain this many connections in
the cache during client instance lifetime. Caching connections are very
useful when user requests mostly same hosts.
When connections cache is full and all of them are waiting for response
and new request comes in, curl creates a new connection, starts request
and then drops first available connection to keep connections cache size
right.
There is one side effect, that when tcp connection is closed, system
actually updates its state to TIME_WAIT. Then for some time resources
for this socket can't be reused (usually 60 seconds).
When user wants to do lots of requests simultaneously (to the same
host), curl ends up creating and dropping lots of connections, which is
not very efficient. When this load is high enough, sockets won't be able
to recover from TIME_WAIT because of timeout and system may run out of
available sockets which results in performance reduce. And user right
now cannot control or limit this behaviour.
The solution is to add a new binding for CURLMOPT_MAX_TOTAL_CONNECTIONS
option. This option tells curl to hold a new connection until
there is one available (request is finished). Only after that curl will
either drop and create new connection or reuse an old one.
This patch bypasses this option into curl instance. It defaults to -1
which means that there is no limit. To create a client with this option
setup, user needs to set max_total_connections option like this:
> httpc = require('http.client').new({max_connections = 8,
max_total_connections = 8})
In general this options usually useful when doing requests mostly to
the same hosts. Other way, defaults should be enough.
Option CURLMOPT_MAX_TOTAL_CONNECTIONS was added from 7.30.0 version, so
if curl version is under 7.30.0, this option is simply ignored.
https://curl.haxx.se/changes.html#7_30_0
Also, this patch adjusts the default for CURLMOPT_MAX_CONNECTS option to
0 which means that for every new easy handle curl will enlarge its max
cache size by 4. See this option docs for more
https://curl.haxx.se/libcurl/c/CURLMOPT_MAXCONNECTS.html
Fixes #3945
commit 14e9b43dc92e73b7600b499c069b11e8a7297173
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue May 14 02:10:27 2019 +0300
small: fix build of static allocator
See details in the small repository commit. In the summary, looks
like a GCC bug. Fixed with a workaround.
commit 6f20fb7b75a95b658e0c91fc5a5e42a9095ac621
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Apr 28 17:35:48 2019 +0300
sio: optimize sio_strfaddr() for the most common case
SIO library provides a wrapper for getnameinfo able to stringify
Unix socket addresses. But it does not care about limited
Tarantool stack and allocates buffers for getnameinfo() right on
it - ~1Kb. Besides, after successful getnameinfo() the result is
copied onto another static buffer.
This patch optimizes sio_strfaddr() for the most common case -
AF_INET, when 32 bytes is more than enough for any IP:Port pair,
and writes the result into the target buffer directly.
The main motivation behind this commit is that SWIM makes active
use of sio_strfaddr() for logging - for each received/sent
message it writes a couple of addresses into a log. It does it in
verbose mode, but the say() function arguments are still
calculated even when the active mode is lower than verbose.
commit b6455e6a3a3b43de4ce92ffd93149cf5d06a5c12
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sun Apr 28 17:15:40 2019 +0300
Use static_alloc() instead of 'static char[]' where possible
This patch harnesses freshly introduced static memory allocator
to eliminate wasteful usage of BSS memory. This commit frees
11Kb per each thread.
commit 890068fc8bd396bf77aa1a4264385ec75a8e211a
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Sat Apr 27 00:20:42 2019 +0300
small: introduce small/static
Before the patch Tarantool had a thread- and C-file- local array
of 4 static buffers, each 1028 bytes. It provided an API
tt_static_buf() allowing to return them one by one in a cycle.
Firstly, it consumed totally 200Kb of BSS memory in summary over
all C-files using these buffers. Obviously, it was a bug and was
not made intentionally. The buffers were supposed to be a one
process-global array.
Secondly, even if the bug above had been fixed somehow, sometimes
it would have been needed to obtain a bit bigger buffer. For
example, to store a UDP packet - ~1.5Kb.
This commit replaces these 4 buffers with small/ static allocator
which does basically the same, but in more granulated and
manoeuvrable way. This commit frees ~188Kb of BSS section.
A main motivation for this commit is a wish to use a single
global out-of-stack buffer to read UDP packets into it in the
SWIM library, and on the other hand do not pad out BSS section
with a new SWIM-special static buffer. Now SWIM uses stack for
this and in the incoming cryptography SWIM component it will need
more.
commit e639ec4b930bd0e096d733e669f206de61b1ca6d
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon May 13 22:19:07 2019 +0300
tuple_format_iterator: don't set multikey_idx for multikey array itself
Currently, we set multikey_idx to multikey_frame->idx for the field
corresponding to the multikey_frame itself. This is wrong, because
this field doesn't have any indirection in the field map - we simply
store offset to the multikey array there. It works by a happy
coincidence - the frame has index -1 and we treat -1 as no-multikey
case, see MULTIKEY_NONE. Should we change MULTIKEY_NONE to e.g. -2
or INT_MAX, we would get a crash because of it. So let's move the
code setting multikey_idx before initializing multikey_frame in
tuple_format_iterator_next().
commit 361e218b360dc6bec021325f6ad9d2c751b280ae
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon May 13 19:11:15 2019 +0300
Use MULTIKEY_NONE instead of -1
Solely to improve code readability. No functional changes.
Suggested by @kostja.
commit 64704bb3b7aeb72a468ea126063895b0719e9092
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 8 17:05:32 2019 +0300
vinyl: implement multikey index support
In case of multikey indexes, we use vy_entry.hint to store multikey
array entry index instead of a comparison hint. So all we need to do is
patch all places where a statement is inserted so that in case the key
definition is multikey we iterate over all multikey indexes and insert
an entry for each of them. The rest will be done automatically as vinyl
stores and compares vy_entry objects, which have hints built-in, while
comparators and other generic functions have already been patched to
treat hints as multikey indexes.
There are just a few places we need to patch:
- vy_tx_set, which inserts a statement into a transaction write set.
- vy_build_insert_stmt, which is used to fill the new index on index
creation and DDL recovery.
- vy_build_on_replace, which forwards modifications done to the space
during index creation to the new index.
- vy_check_is_unique_secondary, which checks a secondary index for
conflicts on insertion of a new statement.
- vy_tx_handle_deferred_delete, which generates deferred DELETE
statements if the old tuple is found in memory or in cache.
- vy_deferred_delete_on_replace, which applies deferred DELETEs on
compaction.
Plus, we need to teach vy_get_by_secondary_tuple to match a full
multikey tuple to a partial multikey tuple or a key, which implies
iterating over all multikey indexes of the full tuple and comparing
them to the corresponding entries to the partial tuple.
We already have tests that check the functionality for memtx. Enable and
tweak it a little so that it can be used for vinyl as well.
commit e40a79582022a58529662dae33ac67a5f8ca644d
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 8 13:15:32 2019 +0300
vinyl: use multikey hints while writing runs
Currently, we completely ignore vy_entry.hint while writing a run file,
because they only contain auxiliary information for tuple comparison.
However, soon we will use hints to store multikey offsets, which is
mandatory for extracting keys and hence writing secondary run files.
So this patch propagates vy_entry.hint as multikey offset to tuple_bloom
and tuple_extract_key in vy_run implementation.
commit 35be5c1c2edca40a7f23cd348c0ff9919ba8dc5e
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 7 17:03:46 2019 +0300
vinyl: use field_map_builder for constructing stmt field map
Currently, we construct a field map for a vinyl surrogate DELETE
statement by hand, which works fine as long as field maps don't have
extents. Once multikey indexes are introduced, there will be extents
hence we must switch to field_map_builder.
commit ca0f750c297477b6940c1346407db0b63dab6ece
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 8 12:53:34 2019 +0300
Make tuple_bloom support multikey indexes
Just like in case of tuple_extract_key, simply pass multikey_idx to
tuple_bloom_builder_add and tuple_bloom_maybe_has. For now, we always
pass -1, but the following patches will pass offset in multikey array
if the key definition is multikey.
commit 9118d408136b826269719ca6d3298b8f93409f6f
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 8 12:16:52 2019 +0300
Make tuple_extract_key support multikey indexes
Add multikey_idx argument to tuple_extract_key and forward it to
tuple_field_by_part in the method implementation. For unikey indexes
pass -1. We need this to support multikey indexes in Vinyl.
We could of course introduce a separate set of methods for multikey
indexes (something like tuple_extract_key_multikey), but that would
look cumbersome and hardly result in any performance benefits, because
passing -1 to a relatively cold function, such as key extractor, isn't
a big deal. Besides, passing multikey_idx unconditionally is consistent
with tuple_compare.
commit 6c44e11ebf17938473ba2e95e56be4a13e94cfda
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Wed May 8 11:08:05 2019 +0300
Get rid of tuple_field_by_part_multikey
Always require multikey_idx to be passed to tuple_field_by_part.
If the key definition isn't multikey, pass -1. Rationale: having
separate functions for multikey and unikey indexes doesn't have
any performance benefits (as all those functions are inline), but
makes the code inconsistent with other functions (e.g. tuple_compare)
which don't have a separate multikey variant. After all, passing -1
when the key definition is known to be unikey doesn't blow the code.
While we are at it, let's also add a few assertions ensuring that
the key definition isn't multikey to functions that don't support
multikey yet.
commit f732ec1b4d87e6c5a0e90bf9f9fbd4af39989681
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon May 13 18:44:26 2019 +0300
Fix compilation
Follow-up 14c529dfe863 ("Make tuple comparison hints mandatory").
commit 14c529dfe86395d45e6257fd79245ae32b2e177e
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 7 20:09:53 2019 +0300
Make tuple comparison hints mandatory
There isn't much point in having separate versions of tuple comparators
that don't take tuple comparison hints anymore, because all hot paths
have been patched to use the hinted versions. Besides un-hinted versions
don't make sense for multikey indexes, which use hints to store offsets
in multikey arrays. Let's strip the _hinted suffix from all hinted
comparators and zap un-hinted versions. In a few remaining places in the
code that still use un-hinted versions, let's simply pass HINT_NONE.
commit c7915eccdb48e7b59e676ab4ef9a39113c82c084
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Mar 4 08:46:50 2019 +0300
Add merger for tuple streams (Lua part)
Fixes #3276.
@TarantoolBot document
Title: Merger for tuple streams
The main concept of the merger is a source. It is an object that
provides a stream of tuples. There are four types of sources: a tuple
source, a table source, a buffer source and a merger itself.
A tuple source just return one tuple. However this source (as well as a
table and a buffer ones) supports fetching of a next data chunk, so the
API allows to create it from a Lua iterator:
`merger.new_tuple_source(gen, param, state)`. A `gen` function should
return `state, tuple` on each call and then return `nil` when no more
tuples available. Consider the example:
```lua
box.cfg({})
box.schema.space.create('s')
box.space.s:create_index('pk')
box.space.s:insert({1})
box.space.s:insert({2})
box.space.s:insert({3})
s = merger.new_tuple_source(box.space.s:pairs())
s:select()
---
- - [1]
- [2]
- [3]
...
s = merger.new_tuple_source(box.space.s:pairs())
s:pairs():totable()
---
- - [1]
- [2]
- [3]
...
```
As we see a source (it is common for all sources) has `:select()` and
`:pairs()` methods. The first one has two options: `buffer` and `limit`
with the same meaning as ones in net.box `:select()`. The `:pairs()`
method (or `:ipairs()` alias) returns a luafun iterator (it is a Lua
iterator, but also provides a set of handy methods to operate in
functional style).
The same API exists to create a table and a buffer source:
`merger.new_table_source(gen, param, state)` and
`merger.new_buffer_source(gen, param, state)`. A `gen` function should
return a table or a buffer on each call.
There are also helpers that are useful when all data are available at
once: `merger.new_source_fromtable(tbl)` and
`merger.new_source_frombuffer(buf)`.
A merger is a special kind of a source, which is created from a key_def
object and a set of sources. It performs a kind of the merge sort:
chooses a source with a minimal / maximal tuple on each step, consumes
a tuple from this source and repeats. The API to create a merger is the
following:
```lua
local key_def_lib = require('key_def')
local merger = require('merger')
local key_def = key_def_lib.new(<...>)
local sources = {<...>}
local merger_inst = merger.new(key_def, sources, {
-- Ascending (false) or descending (true) order.
-- Default is ascending.
reverse = <boolean> or <nil>,
})
```
An instance of a merger has the same `:select()` and `:pairs()` methods
as any other source.
The `key_def_lib.new()` function takes a table of key parts as an
argument in the same format as box.space.<...>.index.<...>.parts or
conn.space.<...>.index.<...>.parts (where conn is a net.box connection):
```
local key_parts = {
{
fieldno = <number>,
type = <string>,
[ is_nullable = <boolean>, ]
[ collation_id = <number>, ]
[ collation = <string>, ]
},
...
}
local key_def = key_def_lib.new(key_parts)
```
A key_def can be cached across requests with the same ordering rules
(typically requests to a same space).
commit ec12395f6a19b2c267ebfcbffb1c990f3f46c153
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Mon Mar 4 08:43:59 2019 +0300
Add merger for tuples streams (C part)
Needed for #3276.
commit 1aaf6378708fc9f52446b1335457ba098450846a
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu Jan 31 01:59:18 2019 +0300
net.box: add skip_header option to use with buffer
Needed for #3276.
@TarantoolBot document
Title: net.box: skip_header option
This option instructs net.box to skip {[IPROTO_DATA_KEY] = ...} wrapper
from a buffer. This may be needed to pass this buffer to some C function
when it expects some specific msgpack input.
Usage example:
```lua
local net_box = require('net.box')
local buffer = require('buffer')
local ffi = require('ffi')
local msgpack = require('msgpack')
local yaml = require('yaml')
box.cfg{listen = 3301}
box.once('load_data', function()
box.schema.user.grant('guest', 'read,write,execute', 'universe')
box.schema.space.create('s')
box.space.s:create_index('pk')
box.space.s:insert({1})
box.space.s:insert({2})
box.space.s:insert({3})
box.space.s:insert({4})
end)
local function foo()
return box.space.s:select()
end
_G.foo = foo
local conn = net_box.connect('localhost:3301')
local buf = buffer.ibuf()
conn.space.s:select(nil, {buffer = buf})
local buf_str = ffi.string(buf.rpos, buf.wpos - buf.rpos)
local buf_lua = msgpack.decode(buf_str)
print('select:\n' .. yaml.encode(buf_lua))
-- {48: [[1], [2], [3], [4]]}
local buf = buffer.ibuf()
conn.space.s:select(nil, {buffer = buf, skip_header = true})
local buf_str = ffi.string(buf.rpos, buf.wpos - buf.rpos)
local buf_lua = msgpack.decode(buf_str)
print('select:\n' .. yaml.encode(buf_lua))
-- [[1], [2], [3], [4]]
local buf = buffer.ibuf()
conn:call('foo', nil, {buffer = buf})
local buf_str = ffi.string(buf.rpos, buf.wpos - buf.rpos)
local buf_lua = msgpack.decode(buf_str)
print('call:\n' .. yaml.encode(buf_lua))
-- {48: [[[1], [2], [3], [4]]]}
local buf = buffer.ibuf()
conn:call('foo', nil, {buffer = buf, skip_header = true})
local buf_str = ffi.string(buf.rpos, buf.wpos - buf.rpos)
local buf_lua = msgpack.decode(buf_str)
print('call:\n' .. yaml.encode(buf_lua))
-- [[[1], [2], [3], [4]]]
os.exit()
```
commit b1b2e7b0aaef0253b2597205322d116df86a93fc
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Thu Jan 31 01:45:22 2019 +0300
lua: add non-recursive msgpack decoding functions
Needed for #3276.
@TarantoolBot document
Title: Non-recursive msgpack decoding functions
Contracts:
```
msgpack.decode_array_header(buf.rpos, buf:size()) -> arr_len, new_rpos
msgpack.decode_map_header(buf.rpos, buf:size()) -> map_len, new_rpos
```
These functions are intended to be used with a msgpack buffer received
from net.box. A user may want to skip {[IPROTO_DATA_KEY] = ...} wrapper
and an array header before pass the buffer to decode in some C function.
See https://github.com/tarantool/tarantool/issues/2195 for more
information re this net.box's API.
commit 13238539e8a57cd17af560aba507dc3238f0207b
Author: avtikhon <avtikhon@tarantool.org>
Date: Tue Apr 16 08:43:49 2019 +0300
test: fix flaky show_error_on_disconnect failures
It is needed to wait for upstream/downstream status, otherwise
error occurs:
[025] --- replication/show_error_on_disconnect.result Fri Apr 12 14:49:26 2019
[025] +++ replication/show_error_on_disconnect.reject Tue Apr 16 07:35:41 2019
[025] @@ -77,11 +77,12 @@
[025] ...
[025] box.info.replication[other_id].upstream.status
[025] ---
[025] -- stopped
[025] +- sync
[025] ...
[025] box.info.replication[other_id].upstream.message:match("Missing")
[025] ---
[025] -- Missing
[025] +- error: '[string "return box.info.replication[other_id].upstrea..."]:1: attempt to
[025] + index field ''message'' (a nil value)'
[025] ...
[025] test_run:cmd("switch master_quorum2")
[025] ---
[025]
Close #4161
commit 31ed449f02cb21f618122f11885c839534d9ee7c
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Fri May 10 20:36:06 2019 +0300
test: update test-run
Added test_run:wait_upstream() and test_run:wait_downstream() functions
to wait for certain box.info.replication values (#158).
commit 132bb3b1f007873fa703a50d0294c6585fd86870
Author: Stanislav Zudin <szudin@tarantool.org>
Date: Thu Apr 11 17:18:00 2019 +0300
Feature request for a new collation
Adds more tests for collations
Marks unstable collation tests.
Removes a duplicate test
Closes #4007
@TarantoolBot document
Title: New collations
The recent commit includes a wide variety of collations.
The naming of the new collations have the following principles:
unicode_<locale>_<strength>
Three strengths are used:
Primary - "s1
Secondary - "s2" and
Tertiary - "s3".
The following list contains a so called "stable" collations -
the ones whose sort order doesn't depend on the ICU version:
unicode_am_s3
unicode_fi_s3
unicode_de__phonebook_s3
unicode_haw_s3
unicode_he_s3
unicode_hi_s3
unicode_is_s3
unicode_ja_s3
unicode_ko_s3
unicode_lt_s3
unicode_pl_s3
unicode_si_s3
unicode_es_s3
commit e66ab61fc81cba9450b37865706ecb8a7e0cac7e
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 8 19:35:09 2019 +0300
swim: check broadcast interfaces more rigorously
Appeared, that getifaddrs() standard function can return
addresses having IFF_BROADCAST flag, but at the same time not
having struct sockaddr *ifa_broadaddr pointer (NULL).
It led to a crash. The patch does additional check if the address
is NULL.
commit 55e1a140db245b5eb1f50ff9de0568c6f402316e
Author: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Tue May 7 14:51:06 2019 +0300
box: zap field_map_get_size
Turns out we don't really need it as we can use data_offset + bsize
(i.e. the value returned by tuple_size() helper function) to get the
size of a tuple to free. We only need to take into account the offset
of the base tuple struct in the derived struct (memtx_tuple).
There's a catch though:
- We use sizeof(struct memtx_tuple) + field_map_size + bsize for
allocation size.
- We set data_offset to sizeof(struct tuple) + field_map_size.
- struct tuple is packed, which makes its size 10 bytes.
- memtx_tuple embeds struct tuple (base) at 4 byte offset, but since
it is not packed, its size is 16 bytes, NOT 4 + 10 = 14 bytes as
one might expect!
- This means data_offset + bsize + offsetof(struct memtx_tuple, base)
doesn't equal allocation size.
To fix that, let's mark memtx_tuple packed. The only side effect it has
is that we save 2 bytes per each memtx tuple. It won't affect tuple data
layout at all, because struct memtx_tuple already has a packed layout
and so 'packed' will only affect its size, which is only used for
computing allocation size.
My bad I overlooked it during review.
Follow-up f1d9f2575c11 ("box: introduce multikey indexes in memtx").
commit f1d9f2575c11afcf3b0368d38f44ae399cd7e045
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Fri Mar 15 15:24:36 2019 +0300
box: introduce multikey indexes in memtx
- In the case of multikey index arises an ambiguity: which key
should be used in the comparison. The previously introduced
comparison hints act as an non-negative numeric index of key
to use,
- Memtx B+ tree replace and build_next methods have been
patched to insert the same tuple multiple times by different
logical indexes of the key in the array,
- Map fields have been expanded service areas "extent" that
contain an offset of multikey index keys by additional logical
index.
Part of #1257
@TarantoolBot document
Title: introduce multikey indexes in memtx
Any JSON index in which at least one partition contains "[*]"
- array index placeholder sign is called "Multikey".
Such indexes allows you to automatically index set of documents
having same document structure.
Multikey indexes design have a number of restrictions that must
be taken into account:
- it cannot be primary because of the ambiguity arising from
it's definition (primary index requires the one unique key
that identify tuple),
- if some node in the JSON tree of all defined indexes contains
an array index placeholder [*], no other JSON path can use an
explicit JSON index on it's nested field.
- it support "unique" semantics, but it's uniqueness a little
different from conventional indexes: you may insert a tuple
in which the same key occurs multiple times in a unique
multikey index, but you cannot insert a tuple when any of its
keys is in some other tuple stored in space,
- the unique multikey index "duplicate" conflict occurs when
the sets of extracted keys have a non-empty logical
intersection
- to identify the different keys by which a given data tuple is
indexed, each key is assigned a logical sequence number in
the array defined with array index placeholder [*] in index
(such array is called multikey index root),
- no index partition can contain more than one array index
placeholder sign [*] in it's JSON path,
- all parts containing JSON paths with array index placeholder
[*] must have the same (in terms of json tokens) prefix
before this placeholder sign.
Example 1:
s = box.schema.space.create('clients')
s:format({{name='name', type='string'}, {name='phone', type='array'}})
name_idx = s:create_index('name_idx', {parts = {{'name', 'string'}}})
phone_idx = s:create_index('phone_idx', {parts = {{'phone[*]',
'string'}}})
s:insert({"Jorge", {"911", "89457609234"}})
s:insert({"Bob", {"81239876543"}})
phone_idx:get("911")
---
- ['Jorge', ['911', '89457609234']]
...
Example 2:
s = box.schema.space.create('withdata')
pk = s:create_index('pk')
parts = {
{2, 'str', path = 'data[*].name'},
{2, 'str', path = 'data[*].extra.phone'}
}
idx = s:create_index('idx', {parts = parts})
s:insert({1, {data = {{name="A", extra={phone="111"}},
{name="B", extra={phone="111"}}},
garbage = 1}}
idx:get({'A', '111'})
commit 771c00d4907e3ed920f9079d016fd54f4562fad6
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Wed Apr 10 21:03:16 2019 +0300
Use mempool to alloc wal messages
Don't use fiber gc region to alloc wal messages. This relaxes friction
between fiber life cycle and transaction processing.
Prerequisites: #1254
(cherry picked from commit bedc2e06521c2f7a4a6d04510c8f72fa57a44f96)
commit 6f6af986baa9aa221c730a086d45735e64c80507
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Thu Apr 18 18:12:28 2019 +0300
Alloc journal entry on a txn memory region
Use txn memory to allocate a journal entry structure.
This relaxes a dependency between a journal entry and a fiber.
Prerequisites: #1254
(cherry picked from commit 92e68deb21ab17aacf43d8ca409f587b9da86c07)
commit 06c73f1dfa34e8b38afe38da67ae24e7a2665eb8
Author: Georgy Kirichenko <georgy@tarantool.org>
Date: Tue Apr 2 14:45:17 2019 +0300
Introduce a txn memory region
Attach a separate memory region for each txn structure in order to store
all txn internal data until the transaction finished. This patch is a
preparation to detach a txn from a fiber and a fiber gc storage.
Prerequisites: #1254
(cherry picked from commit c1486242445ebf82b8644c21ac7434d89ddeb3b1)
commit 22db9c264c9119d19a3130e6ad8e7fded0e16c6d
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Jan 7 19:12:50 2019 +0300
lua: add key_def lua module
Needed for #3276.
Fixes #3398.
Fixes #4025.
@TarantoolBot document
Title: lua: key_def module
It is convenient to have tuple compare function into lua-land for the
following case:
- exporting key from tuple to iterate over secondary non-unique index
and delete tuples from space
- support comparing a tuple with a key / a tuple, support merging
key_defs from Lua
- factor out key parts parsing code from the tuples merger
A key_def instance has the following methods:
- :extract_key(tuple) -> key (as tuple)
Receives tuple or Lua table. Returns tuple representing extracted
key.
- :compare(tuple_a, tuple_b) -> number
Receives tuples or Lua tables.
Returns:
- a value > 0 when tuple_a > tuple_b,
- a value == 0 when tuple_a == tuple_b,
- a value < 0 otherwise.
- :compare_with_key(tuple, key) -> number
- a value > 0 when key(tuple) > key,
- a value == 0 when key(tuple) == key,
- a value < 0 otherwise.
- :merge(another_key_def) -> new key_def instance
Constructs a new key definition with a set union of key parts
from first and second key defs
- :totable() -> table
Dump key_def object as a Lua table (needed to support __serialize
method)
The root key_def library exports all instance methods directly.
The format of `parts` parameter in the `key_def.new(parts)` call is
compatible with the following structures:
* box.space[...].index[...].parts;
* net_box_conn.space[...].index[...].parts.
Example for extract_key():
```lua
-- Remove values got from a secondary non-unique index.
local key_def_lib = require('key_def')
local s = box.schema.space.create('test')
local pk = s:create_index('pk')
local sk = s:create_index('test', {unique = false, parts = {
{2, 'number', path = 'a'}, {2, 'number', path = 'b'}}})
s:insert{1, {a = 1, b = 1}}
s:insert{2, {a = 1, b = 2}}
local key_def = key_def_lib.new(pk.parts)
for _, tuple in sk:pairs({1})) do
local key = key_def:extract_key(tuple)
pk:delete(key)
end
```
commit 20681ebd8d0b95fabf8b1ba055ee41ce26c2ddd3
Author: avtikhon <avtikhon@tarantool.org>
Date: Mon May 6 17:45:05 2019 +0300
test: errinj* tests need fixes for stress runs
Tests failed with the following issues:
[010] --- vinyl/errinj.result Tue Mar 19 17:52:48 2019
[010] +++ vinyl/errinj.reject Tue Mar 19 19:07:58 2019
[010] @@ -81,7 +81,7 @@
[010] -- fails due to scheduler timeout
[010] box.snapshot();
[010] ---
[010] -- error: Error injection 'vinyl dump'
[010] +- ok
[010] ...
[010] fiber.sleep(0.06);
[010] ---
[010]
Decided to remove the current check for box snapshot fail on
scheduler timeout.
[035] --- vinyl/errinj_stat.result Mon May 6 13:11:27 2019
[035] +++ vinyl/errinj_stat.reject Mon May 6 17:58:48 2019
[035] @@ -250,7 +250,7 @@
[035] ...
[035] box.snapshot()
[035] ---
[035] -- ok
[035] +- error: Error injection 'vinyl dump'
[035] ...
[035] i:compact()
[035] ---
[035]
Decided to add the wait condition checker of the scheduler
tasks_completed number instead of the fiber delay.
Close #4058
(cherry picked from commit d64e4c95f1b72d1f32f9689a33319545506e1c58)
commit 63c2ade6e6f034229b23ec508a7c19071fdc2ea7
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Tue Apr 30 16:55:45 2019 +0300
salad: introduce bps_tree_delete_identical routine
A new routine bps_tree_delete_identical performs an element
deletion if and only if the found element is identical
to the routine argument.
Needed for #1257
commit 4d567b1218addc57f0ab3deaa728a49abd8d073f
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Mon Mar 18 10:56:42 2019 +0300
box: introduce field_map_builder class
The new field_map_builder class encapsulates the logic associated
with field_map allocation and initialization. In the future it
will be extended to allocate field_map that has extensions.
Needed for #1257
commit 203f1a07199b1697b104ab790832268ae494475b
Author: Kirill Shcherbatov <kshcherbatov@tarantool.org>
Date: Fri Mar 29 15:31:04 2019 +0300
box: introduce tuple_format_iterator class
The similar code in tuple_field_map_create and
vy_stmt_new_surrogate_delete_raw that performs tuple decode
with tuple_format has been refactored as reusable
tuple_format_iterator class.
Being thus encapsulated, this code will be uniformly managed and
extended in the further patches in scope of multikey indexes.
Extended engine/json test with vy_stmt_new_surrogate_delete_raw
corner case test. There was no problem before this patch, but
small bug appeared during tuple_format_iterator_next
implementation was not covered.
Needed for #1257
commit 69a6365ef3634a8c7b2afbe9a6ba3ecc88db916a
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 6 13:22:21 2019 +0300
test: app/fio -- Fixup modes for open
There were typos -- we should use octal base, othervise
numbers are treated as decimals.
commit 09b8a604e150fc66129e28308b9cf53365b3b53f
Author: avtikhon <avtikhon@tarantool.org>
Date: Mon Apr 15 15:02:07 2019 +0300
test: enable parallel testing for vinyl suite
Need to switch on is_parallel flag for vinyl suite
to be able to run subtests in parallel using:
./test_run.py -j'<threads>'
Test vinyl/throttle.test.lua temporary disabled due
to it tests the performance which fails in parallel
runs because of highloaded hardware.
Fix #4158
commit 7b378bc6f7b1edffeeec1f5f77603ef11f9a7f75
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Mon May 6 12:59:02 2019 +0300
core/coio_file: copyfile -- Make it behave as regular cp
Traditional cp utility opens destination with O_TRUNC
flag, iow it drops old content of the target file if
such exists.
Fixes #4181
commit 5f87a3a311ccb91d13253ab5587f7f0cb55a34df
Author: Alexander V. Tikhonov <avtikhon@tarantool.org>
Date: Tue Mar 19 16:20:11 2019 +0300
travis-ci: set jobs not to stop on failed tests
Added --force flag to travis-ci jobs not to stop on failed tests.
Due to any found failed test breaks the testing it masks the other
fails and in the following ways it's not good:
- flaky test masks real problem
- release testing needs overall result to fix it fast
- parallel testing may produce flaky test
Close: #4131
commit 35b5095ab0f437df6e78b1bacf4d41c3737a540e
Author: avtikhon <avtikhon@tarantool.org>
Date: Tue Apr 30 16:48:51 2019 +0300
test: fix replication/gc flaky failures
Two problems are fixed here. The first one is about correctness of the
test case. The second is about flaky failures.
About correctness. The test case contains the following lines:
| test_run:cmd("switch replica")
| -- Unblock the replica and break replication.
| box.error.injection.set("ERRINJ_WAL_DELAY", false)
| box.cfg{replication = {}}
Usually rows are applied and the new vclock is sent to the master before
replication will be disabled. So the master removes old xlog before the
replica restart and the next case tests nothing.
This commit uses the new test-run's ability to stop a tarantool instance
with a custom signal and stops the replica with SIGKILL w/o dropping
ERRINJ_WAL_DELAY. This change fixes the race between applying rows and
disabling replication and so makes the test case correct.
About flaky failures. They were look like so:
| [029] --- replication/gc.result Mon Apr 15 14:58:09 2019
| [029] +++ replication/gc.reject Tue Apr 16 09:17:47 2019
| [029] @@ -290,7 +290,12 @@
| [029] ...
| [029] wait_xlog(1) or fio.listdir('./master')
| [029] ---
| [029] -- true
| [029] +- - 00000000000000000305.vylog
| [029] + - 00000000000000000305.xlog
| [029] + - '512'
| [029] + - 00000000000000000310.xlog
| [029] + - 00000000000000000310.vylog
| [029] + - 00000000000000000310.snap
| [029] ...
| [029] -- Stop the replica.
| [029] test_run:cmd("stop server replica")
| <...next cases could have induced mismathes too...>
The reason of the fail is that a replica applied all rows from the old
xlog, but didn't sent an ACK with a new vclock to a master, because the
replication was disabled before that. The master stops relay and keeps
the old xlog. When the replica starts again it subscribes with the
vclock value that instructs a relay to open the new xlog.
Tarantool can remove an old xlog just after a replica's ACK when
observes that the xlog was fully read by all replicas. But tarantool
does not remove xlogs when a replica is subscribed. This is not a big
problem, because such 'stuck' xlog file will be removed with a next xlog
removal.
There was the attempt to fix this behaviour and remove old xlogs at
subscribe, see the following commits:
* b5b4809cf2e6d48230eb9e4301eac188b080e0f4 ('replication: update replica
gc state on subscribe');
* 766cd3e1015f6f76460a748c37212fb4c8791500 ('Revert "replication: update
replica gc state on subscribe"').
Anyway, this commit fixes this flaky failures, because stops the replica
before applying rows from the old xlog. So when the replica starts it
continues reading from the old xlog and the xlog file will be removed
when will be fully read.
Closes #4162
commit de2906a8dc037db6f6b1d95a0488e6599f4989eb
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Wed May 1 00:27:41 2019 +0300
swim: explicitly stop old ev_io input/output on rebind
When an input/output file descriptor was changed, SWIM did this:
swim_ev_io_set(io, new_fd)
swim_ev_io_start(io)
It worked in an assumption that libev allows to change fd on fly,
and the tests were passing because fake event loop was used, not
real libev.
But it didn't work. Libev requires explicit ev_io_stop() called
on the old descriptor. This patch makes this:
swim_ev_io_stop(io)
//do bind ...
swim_ev_io_set(io, new_fd)
swim_ev_io_start(io)
Part of #3234
commit 5320d8f005ee2c7e7f87affef7b6767ccd429993
Author: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Date: Tue Apr 30 23:44:06 2019 +0300
swim: allow to omit host in URI
Before the patch swim.cfg() was returning an error on an attempt
to use URI like 'port', without a host. But it would be useful,
easy, and short to allow that, and use '127.0.0.1' host by
default. Compare:
swim:cfg({uri = 1234})
vs
swim:cfg({uri = '127.0.0.1:1234'})
It is remarkable that box.cfg{listen} also allows to omit host.
Note, that use '0.0.0.0' (INADDR_ANY) is forbidden. Otherwise
1) Different instances interacting with this one via not the
same interfaces would see different source IP addresses.
It would mess member tables;
2) This instance would not be able to encode its IP address
in the meta section, because it has no a fixed IP. At the
same time omission of it and usage of UDP header source
address is not possible as well, because UDP header is not
encrypted and therefore is not safe to look at.
Part of #3234
