# scpi-parser: Introduction of `HELP?`

## Links

Issue: https://github.com/j123b567/scpi-parser/issues/129

PR: https://github.com/j123b567/scpi-parser/pull/135

## Motivation

First standadized three dacades ago in 1992, the Standard Commands for Programmable Instruments (SCPI) have been, and continue to be, a common way to communicate with lab instruments of all sorts. With the exception of less-frequently used block data, SCPI commands and reponses are human-readable both in terms of mnemonics and parameters.

Instruments supporting SCPI usually come with detailed `Programming Guide` documents that spell out and explain commands and parameter types, parameter ranges and the corresponding return values.

In essence, there is a long-standing tradition how to implement and document SCPI, which has historically been the domain of big corporations with the sophistication of technical documentation a hallmark of their overall dedication to quality.

Through the rise of open source hardware and a broadening offering of lower-cost lab equipment (e.g. [EastTester](https://budgetlightforum.com/node/73303)), things have changed a bit.

Some problems can therefore arise through current and future use of SCPI:

- SCPI Programming Guides for older devices, especially exotic lab equipment, can be hard to find or only found in print form. Manufacturers that had gone out of business may be unreachable.
- Open source project documentation may often be non-existent or out-of-date.
- Higher release cadence increases the chance for documentation to become incongruent with the feature set.
- Documentation may not be properly archived and become hard to find / lost when an open source project is updated.
- Archiving of documentation for hardware that cannot be updated in a running setup more often than not becomes a private matter.
- Implementation of instrument control libraries and debugging still relies on the use of the Programming Guide (or source code).
- SCPI as per its old standard may feel complex and tedious to new users and hinder continued adption, as most of what makes implementing and debugging systems with SCPI efficient is the memorization or access to command documentation and test cases.

A cursory glance at [scpi-parser forks](https://techgaun.github.io/active-forks/index.html#j123b567/scpi-parser) and repositories by the same people, as well as some online search hints at the reality of the use of this library:

- [scpi_sx127x](https://os.mbed.com/teams/Semtech/code/scpi_sx127x/file/8767be3c1b7f/scpi-def.cpp/) essentially takes the example code and tacks on the application-specific commands. The documentation is succinct and would immediately translate to description strings as described below.
- [RedPitaya](https://github.com/RedPitaya/RedPitaya/blob/master/scpi-server/src/scpi-commands.c) is probably one of the scpi-parser use cases with the longest list of commands and an actual separate, complete [command list](https://github.com/RedPitaya/RedPitaya/tree/master/scpi-server/doc).
- In [mini project: PICO-PI programmable Lab Switch - 2: add SCPI parser](https://community.element14.com/technologies/test-and-measurement/b/blog/posts/mini-project-pico-pi-programmable-lab-switch---2-add-scpi-parser-669206140), one can see best the expectations towards an open source scpi parser library and "starter pack" with a set of usable examples: **"The complexity of what you and I have to do is low. We have to integrate an existing SCPI parser. But the yield is high: you get the (operational!) skeleton of a programmable lab device."**. It is however also clear that the library and implementation that interfaces with it is stricly a means to an end, and it's hard to see someone take the time to produce a legitimate interface documentation.
- ...

Good documentation is a task in its own right, and not everyone has the love to spare for it. One could argue that the quality of the outcomes of lower-effort open source projects is a function of the examples and paradigms provided to get people started. Along with better code examples, there is thus an opportunity to add a low-threshold means to add documentation.

In a [Hackaday article](https://hackaday.com/2021/11/17/scpi-on-teaching-your-devices-the-lingua-franca-of-laboratories/) on SCPI, it also became evident that some programmers end up re-inventing the wheel or seek to implement an "easier" (albeit non-compliant) version of SCPI or something radically different altogether. Apart from promoting `scpi-parser`, it won't hurt to make the experience, or at least the outcomes of using the library more user-friendly.

## Proposed Enhancement

Modern microcontrollers, even lower-end ones, tend to offer enough memory to store a set of strings to describe commands and data types on top of the command patterns which are already in place for command parsing.

It is therefore possible to:

- introduce a `HELP?` command to query all supported commands,
- pass a search string via `HELP? "<string>"` to narrow down the search, and
- append command descriptions to `_scpi_command_t`.

Return type:

- A comma-separated list of arbitrary block data.
- Block data format is: `#1A1234...xyz`, `#2AB1234...xyz` or `#3ABC1234...xyz` where the first digit specifies the number of data length (in Bytes) digits A, AB or ABC.
- Line break characters are absorbed into the data block. This provides conformity and human readability in a console. PC software processing the block data should trim these extra characters.

The mnemonic `HELP?` is chosen as the SCPI version of `--help` (see Gnu Coding Standards, [4.8 Standards for Command Line Interfaces](https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html#Command_002dLine-Interfaces)). In scpi-99 and later, `IEEE-488.2 Common Commands` are decorated with a leading asterisk. When reading this as a way to identify Common Command as required by the standard, it is probably not recommended (although not explicitly discouraged) to mimic them in the form of e.g. `*HLP?`.

Application code still retains full control over the extent to which these features are included, up to removal of the new `HELP?` handler function `SCPI_HelpQ` when found to be unused during optimization.

On the PC side, applications can parse returned block data to display commands in a drop-down list, or to show suggestions as the user types commends to directly interact with the SCPI device.

## Example

Shown below are the two types of queries with the full set of features in place:

- `HELP?` returning the full list of supported commands highlighting optional and mandatory parts, as well as plain text descriptions of each function. Parameters can be explained on a per-command basis.
- `HELP? "*"` returning a reduced list of entries matching the search string `*` (case insensitive).


**Note:**

When compiled with `USE_COMMAND_DESCRIPTIONS` disabled, the user is presented the list of supported commands which are already stored as strings. One would still need to know (or guess) the number and type of input values. At the small overhead this bare-bones implementation comes (especially with `USE_HELP_FILTER` disabled and `strncasestrn` optimized away), it's hard to find reasons other than aesthetics / needless minimalism against the addition of a `HELP?` command.

## Limitations

- Currently, only command strings are being searched when a string is passed after `HELP?`. If needed, the boolean expression in `SCPI_HelpQ(scpi_t * context)` can be expanded accordingly after checking `USE_COMMAND_DESCRIPTIONS` (see below).
- It is not recommended to add a `HELP` command without `?`, as this would not comply with standard requirements. The user is expected to have this degree of familiarity with SCPI to know that queries are built with a question mark.

## Code Contribution

### Options

The size and structure of `_scpi_command_t` entries is parameterized through `USE_COMMAND_TAGS` and `USE_COMMAND_DESCRIPTIONS` options.

In `inc/scpi/types.h`:

    struct _scpi_command_t {
        const char * pattern;
        scpi_command_callback_t callback;
#if USE_COMMAND_DESCRIPTIONS
        const char * description;
#endif
#if USE_COMMAND_TAGS
        int32_t tag;
#endif /* USE_COMMAND_TAGS */
    };

/* Helper macros for _scpi_command_t items. Usage: 
    _scpi_command_t cmd = {
        .pattern = ":SOME:PATTern" , 
        .callback = SCPI_StubQ , 
        SCPI_CMD_DESC("\t - a command") SCPI_CMD_TAG(0) };
*/
#if USE_COMMAND_DESCRIPTIONS
#define SCPI_CMD_DESC(S) (S),
#else
#define SCPI_CMD_DESC(S)
#endif
#if USE_COMMAND_TAGS
#define SCPI_CMD_TAG(T) (T),
#else
#define SCPI_CMD_TAG(T)
#endif

Their defaults are given below.

In `inc/scpi/config.h`:

#ifndef USE_COMMAND_TAGS
#define USE_COMMAND_TAGS 1
#endif

#ifndef USE_COMMAND_DESCRIPTIONS
#define USE_COMMAND_DESCRIPTIONS 0
#endif

//...

#ifndef USE_HELP_FILTER
#define USE_HELP_FILTER 1
#endif

To override defaults, create and include `scpi_user_config.h` containing the corresponding definitions. When using cmake, als add

add_definitions(-DSCPI_USER_CONFIG=1)

to `CMakeLists.txt`.

### SCPI\_HelpQ Default Handler

In `libscpi/inc/scpi/minimal.h`:

    scpi_result_t SCPI_HelpQ(scpi_t * context);

In `libscpi/src/minimal.c`:

With `USE_HELP_FILTER 0` and `USE_COMMAND_DESCRIPTIONS 0`, the `HELP?` command would reduce to the function below (given for illustration purposes only):

scpi_result_t SCPI_HelpQ(scpi_t * context) {
    for(int i = 0; context->cmdlist[i].pattern != NULL; i++) {
        size_t pattern_len = strlen(context->cmdlist[i].pattern);
        size_t block_len = 1 + pattern_len + strlen(SCPI_LINE_ENDING);

        SCPI_ResultArbitraryBlockHeader(context, block_len);
        SCPI_ResultArbitraryBlockData(context, "\t", 1);
        SCPI_ResultArbitraryBlockData(context, context->cmdlist[i].pattern, 
                pattern_len);
        SCPI_ResultArbitraryBlockData(context, SCPI_LINE_ENDING, 
                strlen(SCPI_LINE_ENDING));
    }
    return SCPI_RES_OK;
}

`SCPI_ResultArbitraryBlockDataHeader` produces a header of numbers preceded by '#' and variable length, so to achieve proper horizontal alignment, a tab character is used as the first data byte, followed by the i-th pattern string, a `SCPI_LINE_ENDING`sequence and a comma. The comma ends up leading the next line, separating consecutive arbitrary data blocks. Neither the comma nor the header characters are included in `block_len`.

A search string passed as `HELP? "<string>""` is optional (`narrowed_down` is `True` if given, and only patterns containing the search string are returned).

The full `SCPI_HelpQ` handler also outputs description strings:

/**
 * HELP? [<string>]
 * @param context
 * @return
 */
scpi_result_t SCPI_HelpQ(scpi_t * context) {
#if USE_HELP_FILTER
    size_t search_string_len = 0;
    const char * search_string = NULL;
    scpi_bool_t narrowed_down = SCPI_ParamCharacters(
        context, &search_string, &search_string_len, false);
#endif

    for(int i = 0; context->cmdlist[i].pattern != NULL; i++) {
        size_t pattern_len = strlen(context->cmdlist[i].pattern);
#if USE_HELP_FILTER
        if(narrowed_down && (NULL == strncasestrn(context->cmdlist[i].pattern, 
            pattern_len, search_string, search_string_len))){
	        continue;
        }
#endif
        size_t block_len = 1 + pattern_len + strlen(SCPI_LINE_ENDING);
#if USE_COMMAND_DESCRIPTIONS
        size_t description_len = context->cmdlist[i].description ? 
            strlen(context->cmdlist[i].description) : 0;
        if(description_len > 0){
            block_len = 1 + pattern_len + 1 + description_len + 
                strlen(SCPI_LINE_ENDING);
        }
#endif
        SCPI_ResultArbitraryBlockHeader(context, block_len);
        SCPI_ResultArbitraryBlockData(context, "\t", 1);
        SCPI_ResultArbitraryBlockData(context, context->cmdlist[i].pattern, 
                pattern_len);
#if USE_COMMAND_DESCRIPTIONS
        if(description_len > 0){
            SCPI_ResultArbitraryBlockData(context, " ", 1);
            SCPI_ResultArbitraryBlockData(context, 
                context->cmdlist[i].description, description_len);
        }
#endif
        SCPI_ResultArbitraryBlockData(context, SCPI_LINE_ENDING, 
                strlen(SCPI_LINE_ENDING));
    }
    return SCPI_RES_OK;
}

### Utils

A case-insensitive `strnstrn` implementation is not part of libc or newlib and must be provided separately (seek first character match, then invoke `strncasecmp`):

In `libscpi/src/utils_private.h`:

    char * strncasestr  (const char *s, const char *find, size_t slen);
    char * strncasestrn (const char *s, size_t slen, const char *find, size_t findlen);
	
#ifndef strncasestrn_s
#define strncasestrn_s(s,s_len,lit) strncasestrn(s, s_len, lit, sizeof(lit)-1)
#endif

In `libscpi/src/utils.c`:

#define CHAR_TO_UPPER(c) ((c)>96 && (c)<123 ? (char)((c) ^ 0x20) : (c))
#define CHAR_TO_LOWER(c) ((c)>64 && (c)< 91 ? (char)((c) | 0x20) : (c))
 
/**
 * @brief Locate a binary substring within a binary string (case-insensitive `strnstrn`).
 * @param[in]  s  binary string
 * @param[in]  slen  length of binary string s
 * @param[in]  find  binary substring
 * @param[in]  findlen  length of binary substring find
 * @return  Pointer to first match in s if found, otherwise `NULL`.
 * @author  Alvaro Lopez Ortega <[e-mail]>
 */
char * strncasestrn (const char *s, size_t slen, const char *find, size_t findlen)
{
    char first;
    char cursor_chr;

    if ((find == NULL) || (findlen == 0))
        return (char *)s;

    if ((*find == '\0'))
        return (char *)s;

    first = CHAR_TO_LOWER(*find);
    find++;
    findlen--;

    do {
        do {
            if (slen-- < 1 || (cursor_chr = *s++) == '\0')
                return NULL;
        } while (CHAR_TO_LOWER(cursor_chr) != first);
        if (findlen > slen) {
            return NULL;
        }
    } while (strncasecmp (s, find, findlen) != 0);

    s--;
    return (char *)s;
}

/**
 * @brief Locate a substring in a binary string (case-insensitive `strnstr`).
 * @param[in]  s  binary string
 * @param[in]  find  substring (zero-terminated)
 * @param[in]  slen  length of binary string s
 * @return  Pointer to first match in s if found, otherwise `NULL`.
 */
char * strncasestr (const char *s, const char *find, size_t slen)
{
    return strncasestrn (s, slen, find, strlen(find));
}

(Adaptation the implementation found in utils.c of the [Cherokee Web Server](https://github.com/cherokee/webserver/blob/5b1dbdb4dd68872014874ac05f8af0f833343d0f/cherokee/util.c) project.)

### Adding SCPI `HELP?` To Your Project

A `HELP?` command pattern should be the first item in the command list. General use of `SCPI_CMD_DESC?` is recommended even when `USE_COMMAND_DESCRIPTIONS` excludes the strings from the data structure, as it adds a level of documentation and functionality written at a time of best understanding of the intent regarding the corresponding handler. Later on, `SCPI_CMD_DESC` can still become effective when code is re-used or when the project is ported to a device with sufficient memory.

const scpi_command_t scpi_commands[] = {
    /* Optional help command */
#if USE_HELP_FILTER
    {"HELP?", SCPI_HelpQ, 
         SCPI_CMD_DESC("[<string>] - list supported commands [containing \"<string>\"] (multiple block data)") SCPI_CMD_TAG(0)},
#else 
    {"HELP?", SCPI_HelpQ, 
         SCPI_CMD_DESC("\t - list supported commands (multiple block data)") SCPI_CMD_TAG(0)},
#endif
    
    {"*CLS",  SCPI_CoreCls,  
         SCPI_CMD_DESC("\t - clear all Event Status registers, errors, output queue") SCPI_CMD_TAG(0)},
    {"*ESE",  SCPI_CoreEse,  
         SCPI_CMD_DESC("<0..255> - set Standard Event Status Enable / event mask") SCPI_CMD_TAG(0)},
    {"*ESE?", SCPI_CoreEseQ, 
         SCPI_CMD_DESC("\t - read ESE (0..255)") SCPI_CMD_TAG(0)},
    {"*ESR?", SCPI_CoreEsrQ, 
         SCPI_CMD_DESC("\t - read+clear Standard Event Status register (0..255)") SCPI_CMD_TAG(0)},

Exemplary descriptions are included in the `Examples` common definitions. When no parameters are expected, it could look like:

    "\t - description (return value or range)"

Note that in `SCPI_HelpQ`, a space character is inserted after the pattern if a description string is available, so when parameters are expected, mandatory and optional parameters can be listed immediately:

    "<parameter>[,<optional param>] - description (return value or range)"

No space character is needed, as it will automatically be added between the command pattern and command description strings.

Arbitrary block data is fully specified through the header and not searched for special characters, line endings or other. Therefore, no particular limitations to the nature of command description strings exist.