# Using binary formats in OpenWRT In some of LibreMesh issues, we are in the need of having efficient storage of information. For now we have been using JSON as a way to persist and exchange information as a swiss army tool, but there are some cases were this is not the best approach. JSON is great when the user needs to read and write the information often, but in the cases were computers are the one that read and write the information more ofen, binary formats are the way to go. Binary formats are formats that are expressed in space efficient form, in formats that resemble as much as possible machine-readable formats. These formats have the benefit of: * they are space efficient: as they store space efficient representations of the values, they use less space than the string counterpart that JSON uses * can separate schema from data: this is another space improvement, as the structure of the data can be shared by a different channel than the information itself, making it more efficient in the exchange of information * faster read/writes: because the information is stored in machine-readable formats, there are no steps for parsing them. This increases the speed of the processes by orders of magnitude by removing the parsing step, that requires both memory. This document briefs the reasons why to use these formats: https://medium.com/@icex33/beyond-json-introduction-to-flatbuffers-fba1dfd0dcfe The requirements that we have for our problems are: * must: allow to split schema from data * must: compatible with OpenWRT * must Binary format * may: zero copy (fast serializing-deserializing) There are many format around, like: * CBOR * FlatBuffers * Cap'n Proto * Protocol Buffers * MessagePack * ... https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats Almost none of them have been implemented in OpenWRT. For those that have implementations in lua, support for lua 5.1 is also lacking. After going through some research about those that are supported in OpenWRT, I realized that CBOR was unnecesarily verbose, FlatBuffers only supports lua 5.3, and so on... Amongst those that are verbose you also have messagepack, that has full lua support: https://github.com/kengonakajima/lua-msgpack Aside from those, luckily one of the compact ones, [Apache Avro](https://avro.apache.org/), is [supported in OpenWRT](https://github.com/openwrt/packages/tree/master/libs/avro): avro-c. ## Apache Avro TODO: Now need to try lua-avro-c, the avro-c binding for lua5.1: https://github.com/adqio/lua-avro-c https://github.com/esha-/lua-avro-c https://github.com/LuaDist2/lua-avro Or pure lua: https://github.com/moteus/lua-avro This might also be useful: https://github.com/tarantool/avro-schema ## Protocol Buffers Other option: https://github.com/starwing/lua-protobuf ## MessagePack https://luarocks.org/modules/fperrad/lua-messagepack https://luarocks.org/modules/antirez/lua-cmsgpack https://github.com/kengonakajima/lua-msgpack ## CBOR https://github.com/spc476/CBOR ## Apache Thrift https://thrift.apache.org/ https://github.com/BixData/lua-thrift ======================================= https://eng.uber.com/trip-data-squeeze/ https://luarocks.org/modules/calio/lua-capnproto https://luarocks.org/modules/akhaustov/lua-pb https://luarocks.org/modules/djungelorm/protobuf https://luarocks.org/modules/xavier-wang/lua-protobuf https://luarocks.org/modules/zash/lua-cbor https://luarocks.org/modules/spc476/org.conman.cbor https://github.com/openwrt/packages/blob/master/libs/protobuf/Makefile https://github.com/openwrt/packages/blob/master/libs/avro/Makefile https://github.com/openwrt/packages/blob/master/libs/fbthrift/Makefile https://github.com/openwrt/packages/blob/master/libs/libcoap/Makefile https://github.com/openwrt/packages/blob/master/libs/libtasn1/Makefile