OpenSearch Community Meeting - June 14 2021

Agenda Items

• Release Candidate -> GA
• Beats & Logstash with OpenSearch

Feel free to comment on the agenda before the meeting if you want to add an item or have a question. During the meeting the agenda will be unlocked for collaborative editing / note taking. After the meeting the agenda will be set to read-only mode.

Previous meeting agenda/notes

QA Log
Longer strategic idea. Completely understand the importance of Beats + Logstash for immediate compatibility (and the plugins you are noting), but long term it should be expected that OpenSearch will eventually diverge from compatibility with these two products. Not due to OpenSearch causing an incompatibility, although it may, but because of hostility from maintainers of the Beats+Logstash. Would it make more sense to integrate more tightly with other communities such as Fluentd/Fluentbit with compatible philosophies? Something along the lines of an OFO Stack (OpenSearch, Fluentd/Bit, OpenSearch Dashboard).

Brian Grabau to Everyone (11:16 AM)
I would recommend beats => kafka

Dawn Foster to Everyone (11:17 AM)
Are you concerned that Elastic will find other ways to prevent people from using OpenSearch with Logstash & Beats? I worry that this could get into a long-term game of whack-a-mole between OpenSearch & Elastic.

Erin Verbeck-Lane to Everyone (11:17 AM)
Agreed with what Kyle said - I don’t think alignment of values should mean we should remove compatibility with other products

Justin Weeks (ATS) to Everyone (11:17 AM)
That is the Major concern Dawn.

Robert Cowart to Everyone (11:17 AM)
Beat -> Kafka or even Beats -> Redis still requires you go Kafka -> ES or Redis -> ES so you still have connection to ES issue, but maybe with other options.

Brian Grabau to Everyone (11:18 AM)
Egh, the stuff Elastic co. is add is not that cool

Robert Cowart to Everyone (11:18 AM)
Telegraf is also an option. It has an ES output already and is much more community driven than Beats.

stek0v to Everyone (11:19 AM)
nifi

Ryan Paras to Everyone (11:19 AM)
with Beats being less modular, less open, and more hostile from its inception - should advocate new deployments not really use it

Brian Grabau to Everyone (11:20 AM)
Python lol

nean to Everyone (11:20 AM)
RSyslog + elastic output modul outperformes logstash/fluentd and co. incl. very good caching option, but its complex to configure

Carl Meadows to Everyone (11:21 AM)
Job one IMO is to help make sure folks aren’t blocked. Where we want to invest and innovate in the future is separate - Looks like Elastic is moving their efforts to their unified agent anyway over Beats

Ryan Paras to Everyone (11:22 AM)
reminder that opensearch docs start to speak to Compatibility - https://docs-beta.opensearch.org/clients/agents-and-ingestion-tools/index/#compatibility-matrices

Carl Meadows to Everyone (11:22 AM)
Question

Daniel Doubrovkine to Everyone (11:22 AM)
We’ve also made a PR into logstash-output-elasticsearch to restore backwards compatibility with Elasticsearch OSS (last version is 7.10.2), https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1018, and are going to have a Beats PR up today or tomorrow.

Carl Meadows to Everyone (11:22 AM)
Who here has developed a custom beat? Is that something you need?

Robert Cowart to Everyone (11:24 AM)
@Carl, the new ElastiFlow collector is written in Go, but intentionally does not use libbeat as it doesn’t provide the performance that we achieved using other methods. Avoiding libbeat and its limited Elasticsearch output is the main reason we achieve over 4 times more throughput.

Carl Meadows to Everyone (11:25 AM)
Sweet good to know Robert

Eli Fisher to Everyone (11:25 AM)
@Robert is it a fully custom collector then or based on some of project/tool?

Daniel Doubrovkine to Everyone (11:25 AM)
Correction: the setting is going to be cluster-wide setting.

Brian Grabau to Everyone (11:26 AM)
What about remote clusters

Robert Cowart to Everyone (11:27 AM)
100% new development. I mean we do use go-elasticsearch (plus our secret sauce) and handful of other packages, like Uber’s Zap logger, but the main collection and data processing is 100% new code.

Madhusudhan Konda to Everyone (11:27 AM)
Are we suggesting we go with the option of overriding using Docker too?
Andrew Hopp to Everyone (11:27 AM)
https://github.com/opensearch-project/OpenSearch/pull/814

Brian Grabau to Everyone (11:28 AM)
My team has question about grph

Daniel Doubrovkine to Everyone (11:28 AM)
I think remote clusters will seed from a local cluster, including this setting. But we should test.

Brian Grabau to Everyone (11:28 AM)
Yup

stek0v to Everyone (11:30 AM)
What about section on site for company-partners?

Erin Verbeck-Lane to Everyone (11:30 AM)
Our remote clusters are standalone Daniel,
so each cluster would need that override opensearch.yml

Daniel Doubrovkine to Everyone (11:31 AM)
Brian, Erin: could you please describe what configuration you use in https://github.com/opensearch-project/OpenSearch/pull/814 and how it will/will not work for you?

Sokratis Papadopoulos to Everyone (11:31 AM)
Is there any response to the EUI and Elastic Charts move towards SSPL/Elastic2?

Daniel Doubrovkine to Everyone (11:32 AM)
Erin: a cluster setting can be set via API, too, however in order to have the setting you need a node that knows about the setting (it needs to be declared in code), so if you do a rolling upgrade, you’d effectively want it in opensearch.yml.

Brian Grabau to Everyone (11:33 AM)
We were looking at Graph and could not figure out if it supported authentication, Krishna could not figure out if it supported external auth or internal or any
none

Krishnanand Singh to Everyone (11:34 AM)
Can opensearch/opendistro be used with eland (python lib for machine learning with elasticsearch)?

Brian Grabau to Everyone (11:37 AM)
if I can figure out how to unmute
KS you on

Krishnanand Singh to Everyone (11:39 AM)
No haven't tried yet

Ryan Paras to Everyone (11:39 AM)
eland is an elasticsearch library - so like everything - be careful :/
Krishnanand Singh to Everyone (11:39 AM)
It's xpack though

Brian Grabau to Everyone (11:41 AM)
We are moving into advanced searching so maybe we can support each other

Sokratis Papadopoulos to Everyone (11:41 AM)
is there an ETA for Logstash OpenSearch output plugin?
ok thanks

Ryan Paras to Everyone (11:42 AM)
if we could please get the meetups updated sooner, rather than later. eg. https://www.meetup.com/Open-Distro-for-Elasticsearch-Meetup-Group/events/thmcwryccjblc/ still shows june 28th 10am pdt

Madhusudhan Konda to Everyone (11:43 AM)
Is there a meeting/regular catchup for developers?

Sokratis Papadopoulos to Everyone (11:45 AM)
thanks Kyle!

Henrique Vicente de Oliveira Pinto to Everyone (11:45 AM)
Thank you!

Paul Borgermans to Everyone (11:45 AM)
Thanks Kyle & co!

Brian Grabau to Everyone (11:45 AM)
Thanks

Abdelmoumene to Everyone (11:45 AM)
Thanks

Justin Weeks to Everyone (11:45 AM)
thank you!

nean to Everyone (11:45 AM)
Thanks Kyle!

Eli Fisher to Everyone (11:45 AM)
Thx!