Guest: Mukul Karnik (LinkedIn)
Company: AWS (Twitter)
Show: Let’s Talk

AWS’s search services have evolved through innovations like Amazon OpenSearch Service and Cloud Search, fostering improved scalability, performance, and community engagement. In this show, Mukul Karnik, GM of Search Services at Amazon Web Services (AWS), discusses the evolution of search services at AWS, along with OpenSearch’s transition to the Linux Foundation and its expanding contributor community. Karnik also highlights the integration of generative AI (GenAI) tools and outlines the 2024-2025 roadmap, emphasizing upcoming innovations in search and the benefits of open-source development.

Currently, the project has over 1,000 contributors and more than 200 maintainers. Interestingly, more than half of these contributors come from outside of Amazon. For example, SAP is actively contributing to the log ingestion system, while ByteDance is enhancing the vector database capabilities in OpenSearch. “We’re seeing a diverse range of community members adding new features and capabilities. Our goal has been to drive this community participation,” says Karnik.

Introduction to Karnik’s role and responsibilities at AWS

• Karnik leads the team responsible for AWS search services, including Amazon OpenSearch Service, which also includes Amazon’s major contributions to the open-source OpenSearch project.
• Karnik explains that another key responsibility is managing Cloud Search, a service that Amazon has maintained for over 10 years, providing customers with search capabilities for a variety of use cases.

The progression of search technology and its impact on AWS’s offerings

• Karnik talks about the background of search technologies, particularly the development of Elasticsearch and its impact on AWS’s offerings.
• Karnik explains that Elasticsearch became open-source in 2014, and AWS began offering a fully managed Elasticsearch service in 2015 to meet the increasing demand for search and log analytics.
• Karnik highlights how the growing popularity of these services has led to continuous innovation in areas like scalability, flexibility, and performance enhancements within AWS’s search services.

Key components of OpenSearch and its diverse use cases

• Karnik explains the core components of AWS search services. Karnik likens the search capabilities in OpenSearch to a book’s index, powered by the inverted index, which uses Lucene to enable efficient search and log analysis.
• Karnik elaborates on how OpenSearch is versatile enough to handle use cases beyond search, including log analytics and security analytics, making it an essential tool for various applications.

Transitioning OpenSearch to a neutral foundation for growth and collaboration

• Karnik explains the decision to create OpenSearch in 2021 and move it to the Linux Foundation.
• Karnik tells us that the main objective was to ensure the project’s long-term growth and diversity of contributions by opening it up to multiple companies, reducing its reliance solely on Amazon.
• The advantages of joining the Linux Foundation include access to training resources, support for global events, and the ability to foster greater collaboration within the OpenSearch ecosystem.

Building the OpenSearch community and growing contributions

• Karnik discusses the OpenSearch community’s size and how AWS is working to further expand it.
• Karnik emphasizes that the OpenSearch project has attracted more than 1000 contributors, with over 200 maintainers, and many of them coming from outside Amazon, including large organizations like SAP and ByteDance.
• OpenSearch’s Slack community has also grown to over 2000 members, actively participating in discussions ranging from technical issues to upcoming releases, showing strong engagement across the board.

Benefits and organizational structure of open-source projects

• Karnik feels that open-source contributions are an opportunity to innovate rather than lose control, as they invite new ideas from other companies while still maintaining project direction.
• Karnik outlines the role of the Technical Steering Committee, which ensures the project stays on course, and the support provided by the Linux Foundation, which helps with training and collaboration between contributors.

OpenSearch’s roadmap and focus on innovative technologies

• Karnik shares details of the recently published 2024-2025 roadmap on GitHub, which details AWS’s focus on strengthening search, log analytics, and security analytics capabilities.
• Karnik discusses the integration of GenAI and vector databases into search services, with the goal of improving accuracy, performance, and scalability, as well as ongoing efforts to reduce the cost of log analytics for customers.

Combining observability and security analytics for improved insights

• Karnik explains how observability and security analytics are increasingly overlapping. While these use cases were once distinct, they now share much of the same data and tools, with observability and security professionals using overlapping workflows.
• Karnik mentions AWS’s work in developing more tailored user interfaces to meet the needs of both observability and security analysts, aiming to simplify how these tools and data are managed.

Open Search adoption and the expanding list of use cases

• Karnik highlights OpenSearch’s versatility, noting that it powers a wide range of applications, from external websites to internal AWS projects, as well as being used by partners like SAP to enhance their own products.
• Karnik believes that OpenSearch’s flexibility and permissive license make it particularly appealing for businesses, as it can be easily customized to meet specific needs and integrated into various services.

Open-source search technology’s role in generative AI growth

• Open-source search technologies like OpenSearch are playing an increasingly important role in generative AI developments.
• Search technology is at the core of GenAI applications, such as conversational agents, making OpenSearch a crucial tool in powering these AI-driven solutions.
• Karnik points out OpenSearch’s integration with popular open-source toolkits like LangChain and Langflow, which streamline AI development and ensure seamless deployment for companies working on GenAI  projects.

AWS’s evolving commitment to open-source contributions

• Karnik talks about AWS’s broader approach to open source saying the company is increasingly contributing to open source. Karnik highlights major projects like Valkyrie and OpenSearch, where AWS has taken a leading role in the community.
• Karnik explains how AWS employs an open-source-first development model, with contributions made first on GitHub and then integrated into Amazon-managed services, benefiting both the community and AWS customers alike.

This summary was written by Emily Nicholls.