OpenSearch Community Meeting - Early Nov (Mon Nov 1)

Agenda:

• Indexing Backpressure
• Data Prepper Log Ingestion Demo

Feel free to comment on the agenda before the meeting if you want to add an item or have a question. During the meeting the agenda will be unlocked for collaborative editing / note taking. After the meeting the agenda will be set to read-only mode.

Late Oct Agenda

Chat log

10:00:10 From Alberto Paro to Everyone:
Hi everyone
10:02:00 From Andrew Hopp (he/him) to Everyone:
Sounds good on my end!
10:02:01 From Sean Neumann to Everyone:
I can hear you great.
10:02:07 From Daniel Doubrovkine to Everyone:
All good, thanks!
10:02:20 From Alberto Paro to Everyone:
Yes Audio is good
10:02:58 From Ryan Paras to Everyone:
https://bit.ly/opensearch-nov-1
10:05:39 From Kyle Davis to Everyone:
Thanks Ryan!
10:07:47 From Hasan Asfoor to Everyone:
I would like to thank the teams/individuals who posted tutorials in the blog such as the one about developing plugins. I also appreciate if one of the developers can share tutorials on how to build the OpenSearch and all its plugins (together or individually). This would really help get started to contribute to the project.
10:09:26 From Kyle Davis to Everyone:
Great feedback, Hasan! I know we’re working on a few tutorials
10:18:42 From Ryan Paras to Everyone:
i imagine this will impact some of the line protocol and the usage of 'HTTP 429' which is currently used for some backpressure. is much expected to change to line protocol in advance of 2.x ?
10:19:56 From Brian Grabau to Everyone:
Looks good
10:20:59 From Hasan Asfoor to Everyone:
Is Data Prepper a one-to-one alternative to Logstash?
10:22:03 From Ryan Paras to Everyone:
https://github.com/opensearch-project/data-prepper
10:22:06 From Kyle Davis to Everyone:
@hasan - this session will go over that to some degree 🙂
10:27:02 From Brian Grabau to Everyone:
Flow Logs Stitched and deduplicated - My team is on board to contribue
10:29:04 From David Venable to Everyone:
@Brian Grabau, I’m on the Data Prepper team and we are happy to discuss ideas and receive contributions. You can engage with in the GitHub project. https://github.com/opensearch-project/data-prepper
10:29:05 From Ryan Paras to Everyone:
are their any plans to support a DLQ functionality like that in logstash? this is extremely valuable for catching messages which may have mapping conflicts. https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html
10:31:30 From David Venable to Everyone:
@Ryan Paras. There are no current plans for a dead-letter queue. But, we are happy to discuss further.
10:31:41 From Ryan Paras to Everyone:
let me know the time and place :)
10:32:03 From David Venable to Everyone:
GitHub. Anytime. ;)
10:32:11 From David Venable to Everyone:
Demo link: https://discuss.opendistrocommunity.dev/t/data-prepper-log-ingestion-demo-guide/7497
10:32:39 From David Venable to Everyone:
Links for Data Prepper

​- Learn more about Data Prepper by accessing the code, examples, and documentation in the Data Prepper repository: https://github.com/opensearch-project/data-prepper 
​
​
​	- Checkout our roadmap: https://github.com/opensearch-project/data-prepper/projects/1 
​
​
​	- Connect with us and provide feedback in our forum: https://discuss.opendistrocommunity.dev/c/data-prepper/61
​
​
​	- Contribute by creating issues, RFC’s and pull-requests in the Data Prepper Repository
​
​
​	- We're hiring engineers at all levels. If interested, contact Phill at treddeni@amazon.com

10:33:58 From Saurabh Singh to Everyone:
https://github.com/opensearch-project/OpenSearch/issues/1446
10:34:30 From Brian Grabau to Everyone:
Can't we just pull over out the outputs and input and filter plugins from Logstash then add magic to that
10:35:43 From JENOVAI, GARY to Everyone:
Was Data Prepper build from ground up, or is it a forked version of Logstash?
10:37:03 From Madhusudhan Konda to Everyone:
Prepper seems developed in Java
10:38:41 From Brian Grabau to Everyone:
Joining and deduplicate and inline enrichment is super cool
10:39:02 From JENOVAI, GARY to Everyone:
Any plans to support inputs from Elastics’ beat suite?
10:40:22 From GSmith to Everyone:
Data Prepper seems a bit of a misnomer…it seems less about prep'ing data and seems more like the start of an observability solution. But I joined late, so maybe I misunderstood?
10:40:27 From DETERT, JON to Everyone:
I thought open search was planning to create its own Logstash compatible product. Is that true? If so, how do you see Data Prepper in comparison to the open search equivalent?
10:42:52 From David Venable to Everyone:
Logstash output plugin for OpenSearch: https://github.com/opensearch-project/logstash-output-opensearch
10:44:12 From Brian Grabau to Everyone:
You can start/test with my configs
10:45:01 From Kris Freedain to Everyone:
https://opensearch.org/community_projects
10:45:17 From Brian Grabau to Everyone:
We can provide log examples
10:46:18 From Kris Freedain to Everyone:
https://github.com/orgs/opensearch-project/projects/1
10:48:38 From Saurabh Singh to Everyone:
Link to Indexing Backpressure. We are looking for comments and feedback, please try out the feature (in 1.x branch already) and let us know any improvements or inputs you would like to have.
Meta Issue : https://github.com/opensearch-project/OpenSearch/issues/1446
PRs : https://github.com/opensearch-project/OpenSearch/pull/1336
10:50:28 From Sokratis Papadopoulos to Everyone:
maybe any news on eui fork? I've seen discussions on how it'll actually get integrated into Dashboards, but seems on-hold for now
10:50:53 From Madhusudhan Konda to Everyone:
Developer sessions please! :)
10:52:40 From Tommy Markley to Everyone:
I am making an RFC label for Dashboards now to make finding those easier :)
10:52:54 From Andrew Hopp (he/him) to Everyone:
https://github.com/opensearch-project/OpenSearch-Dashboards/issues/695

​We'll be closing this this week but please take a look!

10:53:04 From Sokratis Papadopoulos to Everyone:
thanks!
10:53:20 From Andrew Hopp (he/him) to Everyone:
And this is the design doc Kyle mentioned on Drag & Drop! https://github.com/opensearch-project/OpenSearch-Dashboards/issues/896
10:54:47 From Andrew Hopp (he/him) to Everyone:
Thanks everyone!
10:54:52 From Charlotte to Everyone:
wave!
10:54:59 From Hasan Asfoor to Everyone:
Thanks all
10:54:59 From Henrique Vicente de Oliveira Pinto to Everyone:
Thank you. Bye!