Posts

Open source logging tool has been adopted by over 5,000 community users

SAN FRANCISCO, Calif. – April 11, 2019 – The Cloud Native Computing Foundation® (CNCF®), which sustains open source technologies like Kubernetes® and Prometheus™, today announced that Fluentd is its sixth project to graduate, following Kubernetes, Prometheus, Envoy, CoreDNS and containerd. To move from the maturity level of incubation to graduation, projects must demonstrate thriving adoption, a documented, structured governance process, and a strong commitment to community sustainability and inclusivity.

“In the two and a half years since Fluentd became a part of CNCF, we’ve seen it grow beyond a project to a full ecosystem of integrations and third party components,” said Chris Aniszczyk, CTO/COO of the Cloud Native Computing Foundation. “With an active community of contributors and users, the CNCF TOC felt Fluentd was ready for graduation and we look forward to cultivating their community.”

Fluentd was created in 2011 by Sadayuki “Sada” Furuhashi, co-founder of Treasure Data, Inc., as an open source data collector for building a Unified Logging Layer, which unifies the data collection and consumption for a better use and understanding of data. In November 2016, Fluentd was accepted as CNCF’s sixth hosted project after Kubernetes, Prometheus and OpenTracing.

Fluentd solves logging problems at scale, not just for standalone applications, but it streamlines components of distributed architectures, making it an integral tool for any cloud native organization” said Masahiro Nakagawa, Principal Engineer at Arm®Treasure Data and core Fluentd Maintainer. “We’re thrilled to have had the opportunity to collaborate with the CNCF community and to grow our user base, and are honored to graduate.”

With more than 900 plugins connecting to its many data sources and data outputs; including Docker, Google Cloud, Kubernetes, etc., Fluentd was the 2016 Bossie Awards winner for the best open source datacenter and cloud software. More than 5,000 data-driven community users rely on Fluentd to differentiate their products and services through a better use and understanding of their log data, including Atlassian, Amazon Web Services, Backplane, Change.org, CyberAgent, DeNa, Drecom, GREE, GungHo, LINE Corp, Nintendo, Microsoft, Slideshare and many others.

“Fluentd has earned its place as the industry standard for log collection and shipping, and I am excited to see it as a graduated CNCF project,” said Gabe Monroy, Lead Program Manager for Containers, Microsoft Azure. “At Microsoft, we are proud to use Fluentd to power our cloud native logging subsystems and we look forward to working with the growing the open source community around Fluentd.”

Fluentd now has 4 active maintainers, over 160 contributors and more than 4,400 commits. It also has 189 official releases, more than 48 Million Docker Hub pulls, 941 plugins available made by the community and over 7500 Github Stars.

“As one of the longest standing incubating projects within CNCF, Fluentd has demonstrated impressive growth, adoption, and numerous integrations within the broader community,” said Brian Grant, Principal Software Engineer at Google and CNCF TOC representative. “We have no doubt that Fluentd is ready to move to this next stage, and are excited to see what the project continues to accomplish.”

To officially graduate from incubating status, the project also adopted the CNCF Code of Conduct, earned a Core Infrastructure Initiative Best Practices Badge. Completed in August 2017, the CII badge shows an ongoing commitment to code quality and security best practices.  

Fluentd Background

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. All components are available under the Apache 2 License, and are overseen by a self-selected team of active contributors to the project. For downloads, documentation, and how to get involved, visit https://github.com/fluent/fluentd, https://www.fluentd.org/ and https://twitter.com/fluentd.

The Linux Foundation is developing an elearning course titled Cloud Native Logging with Fluentd, which will be available in May 2019. This self-paced, hands-on course is designed to introduce individuals with a technical background to the Fluentd log forwarding and aggregation tool for use in Cloud Native Logging and provide them with the skills necessary to deploy Fluentd in a wide range of production settings.

Additional Resources

About Cloud Native Computing Foundation

Cloud native computing uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. The Cloud Native Computing Foundation (CNCF) hosts critical components of cloud native software stacks, including Kubernetes and Prometheus. CNCF serves as the neutral home for collaboration and brings together the industry’s top developers, end users and vendors – including the world’s largest public cloud and enterprise software companies as well as dozens of innovative startups. CNCF is part of The Linux Foundation, a nonprofit organization. For more information about CNCF, please visit www.cncf.io.

###

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

Media Contact

Natasha Woods

The Linux Foundation

nwoods@linuxfoundation.org

Container orchestration system widely deployed at scale with numerous global organizations 

SONOMA, Calif., March 6, 2018 – Open Source Leadership Summit – The Cloud Native Computing Foundation® (CNCF®), which sustains and integrates open source technologies like Kubernetes® and Prometheus™, today announced that Kubernetes is the first project to graduate. To move from incubation to graduate, projects must demonstrate thriving adoption, a documented, structured governance process, and a strong commitment to community success and inclusivity.

“Kubernetes led to the creation of the CNCF as the first project accepted by the Technical Operating Committee (TOC) a little over two years ago,” said Chris Aniszczyk, COO of the Cloud Native Computing Foundation. “With the project’s rapid growth, broad participation from numerous organizations, cloud providers and users, and proven ability to operate at scale, the TOC readily endorsed Kubernetes moving on from incubation to graduate. It signals that Kubernetes is mature as an open source project and resilient enough to manage containers at scale across any industry in companies of all sizes.”

For more Kubernetes milestones and its tenure at CNCF, read this blog.

Established, global organizations like Uber, Bloomberg, Blackrock, BlaBlaCar, The New York Times, Lyft, eBay, Buffer, Squarespace, Ancestry, GolfNow, Goldman Sachs and many others use Kubernetes in production at massive scale. Furthermore, according to Redmonk, 71 percent of the Fortune 100 use containers and more than 50 percent of Fortune 100 companies use Kubernetes as their container orchestration platform.

To officially graduate from incubating status, the project also adopted the CNCF Code of Conduct, defined its own governance structure and established a Steering Committee. Today committers come from multiple companies and lists who has designated responsibility over different parts of the Kubernetes codebase.

Additionally, Kubernetes also had to earn (and maintain) a Core Infrastructure Initiative Best Practices Badge. Completed in August of 2016, the CII badge shows an ongoing commitment to code quality and security best practices.

“The Kubernetes project is proud to graduate into a full CNCF project as well as to have helped the organization launch and grow into the industry-leading position it has today. This project and company ecosystem has changed the face of infrastructure in the cloud. The Kubernetes community looks forward to maturing the impact cloud native development has had in the industry as a whole,” said Sarah Novotny, Open Source Strategy Lead, Google Cloud.

Kubernetes has become transcendent in its ability to shatter assumptions about what is possible in an open source project. Ascending from incubation to a full-fledged CNCF project puts the finishing touches on a lighthouse built with care to guide other communities toward conscientious success,” said Jaice Singer DuMars, Kubernetes Project Ambassador, Microsoft

Availability and Oversight

The open source container orchestration system includes apps, services, network storage, cluster management and performance and stability features. Kubernetes is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. The 13-person Steering Committee guides the project’s day-to-day management, including community development and technical processes. For downloads, documentation, and how to get involved, visit https://github.com/kubernetes, https://kubernetes.io/ and https://twitter.com/kubernetesio.

Additional Resources

About Cloud Native Computing Foundation

Cloud native computing uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. The Cloud Native Computing Foundation (CNCF) hosts critical components of cloud native software stacks, including Kubernetes and Prometheus. CNCF serves as the neutral home for collaboration and brings together the industry’s top developers, end users and vendors – including the world’s largest public cloud and enterprise software companies as well as dozens of innovative startups. CNCF is part of The Linux Foundation, a nonprofit organization. For more information about CNCF, please visit www.cncf.io.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

Media Contact

Natasha Woods

The Linux Foundation

(415) 312-5289

PR@CNCF.io

Today, the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC) voted to accept Vitess as the 16th hosted project, alongside Kubernetes, Prometheus, OpenTracing, Fluentd, Linkerd, gRPC, CoreDNS, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF and Rook. Vitess has been accepted as an incubation-level project, under the CNCF Graduation Criteria v1.0.

The TOC’s Graduation Criteria provides every CNCF project an associated maturity level of either inception, incubating or graduated. As an incubating project, Vitess must document it is being used successfully in production by at least three independent end users, have a healthy number of committers and demonstrate a substantial ongoing flow of contributions.

Originally created as an internal solution by YouTube to handle scaling for massive amounts of storage, Vitess is a database orchestration system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, organizations can even split and merge shards as needs grow, with an atomic cutover step that takes only a few seconds. Companies like BetterCloud, Flipkart, Quiz of Kings, Slack, Square Cash, Stitch Labs and YouTube are using Vitess across various stages of production and deployment. Organizations including Booking.com, GitHub, HubSpot, Slack, and Square are also active contributors to the project.

“Slack is in the midst of a major migration of the MySQL infrastructure at the core of our service, driven by the need for an architecture that scales to meet the growing demands of our largest customers and features under the pressure to maintain a stable and performant service that executes billions of MySQL transactions per hour, said Michael Demmer, Senior Staff Engineer at Slack. “We needed a solution that would offer a familiar full featured SQL interface, and wanted to continue to use MySQL as the backing store to maintain our operations knowledge and comfort level. Vitess is a natural choice for this purpose and has served us well so far.”

Vitess has been under development since 2010. The earliest version was a connection proxy that helped to buy some headroom, but over time the features evolved, while the tools and servers grew to be more efficient, fault tolerant, and manageable. This iterative journey led to what Vitess has become today: a distributed, cloud-based storage solution that exhibits some of the best properties of a relational database.

“Faced with rapid organic and internal growth at YouTube, we had to come up with something that would leap ahead of the curve instead of just fighting fires. When we finally built the initial feature list for Vitess, it was obvious that we were addressing problems that are common to all growing organizations,” said Sugu Sougoumarane, CTO at PlanetScale Data and Co-creator of Vitess. “Our collaboration with Kubernetes over the last two years means anyone can now run Vitess the way YouTube does: dynamically scaled and scheduled in a container cluster. We’re excited to work with the CNCF to expand the capabilities of Vitess even further.”

Main features:

  • Combines important MySQL features with the scalability of a NoSQL database
  • Enables MySQL to run in the cloud
  • Cloud-native functionality such as support for automatic failover/recovery, replication and rolling upgrades
  • Vertical and horizontal sharding support, and virtually seamless dynamic re-sharding
  • Multiple sharding schemes, with the ability to plug-in custom ones
  • Query routing, rewriting and sanitization, blacklisting, streaming, and de-duping
  • Master management tools (handles reparenting)
  • Performance analysis tools

Technical Specs:

  • Backend components implemented in Go
  • Continuously tested against Ubuntu 14.04 (Trusty) and Debian 8 (Jessie). Other Linux distributions should work as well
  • Supports MySQL 5.6, MariaDB 10.0, and any newer versions
  • VTGate server is the main entry point applications use to connect to Vitess
  • Supports data backups to either a network mount (e.g. NFS) or to a blob store

“To build successful cloud native applications, developers have to consider new ways of managing storage,” said Chris Aniszczyk, COO of Cloud Native Computing Foundation and Vice President of Developer Relations at The Linux Foundation. “Vitess is a fantastic example of bridging the existing popular MySQL database and scaling it in a cloud native fashion.”

The open source Vitess project runs best in a containerized environment. With Kubernetes and Google Kubernetes Engine as the container cluster manager, it’s now a lot easier to get started. In this environment, Vitess provides a MySQL storage layer with improved reliability, scalability, and manageability. Kubernetes handles scheduling onto nodes in a compute cluster, actively manages workloads on those nodes, and groups containers comprising an application for easy management and discovery. Also with Kubernetes, Vitess even becomes agnostic to the underlying choice of cloud platform, providing cloud portability with no vendor lock-in.

Notable Milestones:

  • 105 contributors
  • 5,413 GitHub stars
  • 15 releases
  • 13,733 commits
  • 707 forks

“A big advantage of cloud solutions is easy horizontal scalability, but when it comes to stateful systems such as databases, the problem is that application complexity and operational overhead tend to scale up along with the number of instances,” said Brian Grant, TOC representative and project sponsor. “A cloud-native solution like Vitess abstracts the complexity of horizontal scalability from both application developers and database operators. We’re thrilled to welcome Vitess as a CNCF project as we continue to enable more classes of workloads in cloud-native environments.”

As a CNCF hosted project, Vitess is part of a neutral cloud native foundation aligned with its technical interests, as well as the larger Linux Foundation, which provide the project with with project governance, marketing support and community outreach.

For more on Vitess, read DZone’s Launching Vitess: How to Run Youtube’s MySQL Sharding Engine, or watch the the recording of the Vitess beyond YouTube session from Percona Live Open Source Database Conference 2017. You can also read Vitess’ guide to getting started on Kubernetes.