# Console completion

## Architecture overview

Completion engine is based on `readline` library which provides hooks so we can provide own callbacks and fill completion list to be shown in console upon `TAB` key press.

The core functions are sitting in `src/box/lua/console.c` file.

When `console` Lua module is registered we define a closure to point which completion handler to use for `readline` library bindings.

void
tarantool_lua_console_init(struct lua_State *L)
{
    // ...
	/* readline() func needs a ref to completion_handler (in upvalue) */
    lua_getfield(L, -1, "completion_handler");
    lua_pushcclosure(L, lbox_console_readline, 1);
    lua_setfield(L, -2, "readline");
    // ...
}

Then `readline` will read text input from a terminal

static int
lbox_console_readline(struct lua_State *L)
{
    // ...
    rl_attempted_completion_function = console_completion_handler;
    // ...
    rl_callback_handler_install(prompt, console_push_line);
    top = lua_gettop(L);
    while (top == lua_gettop(L)) {
	    while (coio_wait(STDIN_FILENO, COIO_READ,
                         TIMEOUT_INFINITY) == 0) {
            luaL_testcancel(L);
        }
        rl_callback_read_char();
    }
    // ...
}

Every time the user hits `TAB` key the `console_completion_handler` function is executed to provide a list of possible completions to the console output.

For example if we have a few spaces (_space1_ and _space2_) we may type only a few characters of their name and hit `TAB` to list all spaces.

tarantool> box.space.space<TAB>
box.space.space1  box.space.space2  

When `TAB` is pressed we enter

static char **
console_completion_handler(const char *text, int start, int end)
{
    // ...
    if (lua_equal(readline_L, -1, lua_upvalueindex(1))) {
        lua_pop(readline_L, 1);
        res = lua_rl_complete(readline_L, text, start, end);
        goto done;
    }
done:
    return res;
}

The helper `lua_rl_complete` is executed with text we proved with `TAB` upon. For example above, when we type

box.space.space<TAB>

we enter `lua_rl_complete(readline_L, "box.space.space", 0, 15)`. This function does a number of tricky things, so better put detailed comments right into the body. Its purpose is to return an array of pointers of possible words matching the prefix provided in arguments.

static char **
lua_rl_complete(lua_State *L, const char *text,
                int start, int end)
{
    dmlist ml;
    const char *s;
    size_t i, n, dot, items_checked;
    int loop, savetop, is_method_ref = 0;

    //
    // Exit early if there is nothing to complete.
    if (!(text[0] == '\0' || isalpha(text[0]) || text[0] == '_'))
        return NULL;

    //
    // This is a dynamic list where completion
    // results are accumulated.
    ml.list = NULL;
    ml.idx = ml.allocated = ml.matchlen = 0;

    //
    // Main lookup loop
    // - save current top Lua stack pointer to restore later
    // - push LUA_GLOBALSINDEX to the stack so that the lookup
    //   will be traversing all global objects
    savetop = lua_gettop(L);
    lua_pushglobaltable(L);
    
    //
    // Traverse incoming @text left to right, once `.`
    // or `:` found we call lua_rl_getfield helper. For
    // example when `box.space.space` traversed we do
    // 
    //  - lua_rl_getfield(`box.space.space`)
    //  - lua_rl_getfield(`space.space`)
    //
    //  The lua_rl_getfield helper takes each entry and
    //  verifies that prefix objects are valid, ie there are
    //  global `box` object and `box.space` object. The @dot
    //  remains pointing to longest prefix. We will use it to
    //  enumerate _all_ keys associated with `box.space` when
    //  we gonna fetch all keys and find matching ones.
    //  
    for (n = (size_t)(end-start), i = dot = 0; i < n; i++) {
        if (text[i] == '.' || text[i] == ':') {
            is_method_ref = (text[i] == ':');
            if (!lua_rl_getfield(L, text+dot, i-dot))
                goto error; /* Invalid prefix. */
            dot = i+1;
            /* Points to first char after dot/colon. */
        }
    }

    //
    // If no `.` or `:` been found in the text to complete,
    // we traverse Lua keywords dictionary and try to match
    // on of reserved keywords, for example `ret` can be
    // completed to `return`.
    if (dot == 0) {
        for (i = 0; (s = lua_rl_keywords[i]) != NULL; i++) {
            if (n >= KEYWORD_MATCH_MIN &&
                !strncmp(s, text, n) &&
                lua_rl_dmadd(&ml, NULL, 0, s, ' ')) {
                goto error;
            }
        }
    }

    /*
     * Add all valid matches from all tables/metatables.
     * It is basically the core of completions engine.
     */
    loop = 0;
    items_checked = 0;
    lua_pushglobaltable(L);
    lua_insert(L, -2);
    do {
        if (!lua_istable(L, -1) ||
            (loop != 0 && lua_rawequal(L, -1, -2)))
            continue;

        for (lua_pushnil(L); lua_next(L, -2); lua_pop(L, 1)) {
            /* Beware huge tables */
            if (++items_checked > ITEMS_CHECKED_MAX)
                break;

            if (lua_type(L, -2) != LUA_TSTRING)
                continue;

            s = lua_tostring(L, -2);
            /*
             * Only match names starting with '_'
             * if explicitly requested.
             * 
             * For example with `box.space.space<TAB>` it
             * filters out another objects such as
             * `box.space._sequence_data`,
             * `box.space._vpriv`.
             * 
             * The @s for box.space.spac<TAB>
             * will be enumerating
             * ...
             * _sequence_data
             * _trigger
             * space2
             * space1
             * _space_sequence
             * ...
             */
            if (strncmp(s, text+dot, n-dot) ||
                !valid_identifier(s) ||
                (*s == '_' && text[dot] != '_'))
                continue;

            int suf = 0; /* Omit suffix by default. */
            int type = lua_type(L, -1);
            switch (type) {
            case LUA_TTABLE:
            case LUA_TUSERDATA:
                /*
                 * For tables and userdata omit a
                 * suffix, since all variants, i.e.
                 * T, T.field, T:method and T()
                 * are likely valid.
                 */
                break;
            case LUA_TFUNCTION:
                /*
                 * Prepend '(' for a function. This
                 * helps to differentiate functions
                 * visually in completion lists. It is
                 * believed that in interactive console
                 * functions are most often called
                 * rather then assigned to a variable or
                 * passed as a parameter, hence
                 * an ocasional need to delete an
                 * unwanted '(' shouldn't be a burden.
                 */
                suf = '(';
                break;
            }
        
            /*
             * If completing a method ref, i.e
             * foo:meth<TAB>, show functions only.
             */
            if (!is_method_ref || type == LUA_TFUNCTION) {
		        if (lua_rl_dmadd(&ml, text, dot, s, suf))
                    goto error;
            }
        }
    } while (++loop < METATABLE_RECURSION_MAX &&
             lua_rl_getmetaindex(L));

    lua_pop(L, 1);

    //
    // Finally convert all found matching words to Lua's
    // array of words and provide them back to a caller which
    // in turn spit them out to console.
    if (ml.idx == 0) {
error:
        lua_rl_dmfree(&ml);
        lua_settop(L, savetop);
        return NULL;
    } else {
        ml.list[0] = malloc(sizeof(char)*(ml.matchlen+1));
        if (!ml.list[0])
            goto error;
        memcpy(ml.list[0], ml.list[1], ml.matchlen);
        ml.list[0][ml.matchlen] = '\0';
        if (lua_rl_dmadd(&ml, NULL, 0, NULL, 0))
            goto error;
    }

    lua_settop(L, savetop);
    return ml.list;
}

From routine above it is obvious that such completion may be huge in number of entries, so the only up to `ITEMS_CHECKED_MAX=500` fetched from tables.

## Network streams

Lua interface for network streams sits in `src/box/lua/net_box.lua` file. New streams are created from a connection object

local remote_methods = {}
local remote_mt = {
    __index            = remote_methods,
    __serialize        = remote_serialize,
    __metatable        = false
}

function remote_methods:new_stream()
    self._last_stream_id = self._last_stream_id + 1
    local stream = setmetatable({
        new_stream     = stream_new_stream,
        begin          = stream_begin,
        commit         = stream_commit,
        rollback       = stream_rollback,
        _stream_id     = self._last_stream_id,
        space          = setmetatable({
            _stream_space_cache = {},
            _stream             = nil,
        }, stream_spaces_mt),
        _conn          = self,
        _schema_version= self.schema_version,
    }, {
        __index        = self,
        __serialize    = stream_serialize
    })
    stream.space._stream = stream
    return stream
end

Note that `space` member is wrapped with own metatable thus completion won't work. This is done to shrink memory usage and don't copy space references here.

Instead when one type some existing space directly (without completion) the `stream_spaces_mt::__index` is involved and fetches the space requested and put it into the local stream cache.

local stream_spaces_mt = {
    __index = function(self, key)
        local stream = self._stream
        --
        -- Validate the schema version and
        -- clear the cached entries on mismatch
        if stream._schema_version ~= stream._conn.schema_version then
            stream._schema_version = stream._conn.schema_version
            self._stream_space_cache = {}
        end
        
        --
        -- Request the cached version first
        if self._stream_space_cache[key] then
            return self._stream_space_cache[key]
        end

        --
        -- Otherwise fetch a fresh one and cache it
        local src = stream._conn.space[key]
        if not src then
            return nil
        end

        local res = stream_wrap_space(stream, src)
        self._stream_space_cache[key] = res
        
        return res
    end,
    __serialize = stream_spaces_serialize
}

Obviously our current completion engine can't do anything about it because it simply fetches everything it can via `LuaC` interface and filter words implicitly and `space` is zapped with own methods.

## Enhancing completion handler

Worth to mention that if we create a connection it still has access to all existing spaces (but not new ones, only those which were already created at moment of connection creation). And stream still can enumerate them all via internal connection object

c = require('net.box').connect('3301')
-- tarantool> c.space.space<TAB>
-- c.space.space1  c.space.space2  c.space.space3
strm = c:new_stream()
-- tarantool> strm._conn.space.space<TAB>
-- strm._conn.space.space1  strm._conn.space.space2  strm._conn.space.space3

Here I provide console output in comments.

Thus we already have spaces accessible the only thing left is to enumerate them inside our completion handler.

And here we have a few options as far as I see. Mark that I didn't try to implement any PoC so these are just raw ideas, maybe even non implementable ideas.

### Special method for completion on Lua level

We could provide some kind of `__completion` table which will be enumerated together with `__index`, where we collect all entries to be looked up (similar option has already been proposed in former bug comment). Note that this option doesn't provide any filtering on itself, the completion engine's C code will simply jump in and fetch *all* possible keys.

### Traverse completions inside Lua own handler

Another option is to defined internal `__completion` function inside Lua objects. The C engine call this `obj.__completion` where arguments will be text and its length user typed so far, which means the `__completion` handler inside Lua code will have to enumerate and filter words by itself, which I don't like actually.