Posts

Open source logging tool has been adopted by over 5,000 community users

SAN FRANCISCO, Calif. – April 11, 2019 – The Cloud Native Computing Foundation® (CNCF®), which sustains open source technologies like Kubernetes® and Prometheus™, today announced that Fluentd is its sixth project to graduate, following Kubernetes, Prometheus, Envoy, CoreDNS and containerd. To move from the maturity level of incubation to graduation, projects must demonstrate thriving adoption, a documented, structured governance process, and a strong commitment to community sustainability and inclusivity.

“In the two and a half years since Fluentd became a part of CNCF, we’ve seen it grow beyond a project to a full ecosystem of integrations and third party components,” said Chris Aniszczyk, CTO/COO of the Cloud Native Computing Foundation. “With an active community of contributors and users, the CNCF TOC felt Fluentd was ready for graduation and we look forward to cultivating their community.”

Fluentd was created in 2011 by Sadayuki “Sada” Furuhashi, co-founder of Treasure Data, Inc., as an open source data collector for building a Unified Logging Layer, which unifies the data collection and consumption for a better use and understanding of data. In November 2016, Fluentd was accepted as CNCF’s sixth hosted project after Kubernetes, Prometheus and OpenTracing.

Fluentd solves logging problems at scale, not just for standalone applications, but it streamlines components of distributed architectures, making it an integral tool for any cloud native organization” said Masahiro Nakagawa, Principal Engineer at Arm®Treasure Data and core Fluentd Maintainer. “We’re thrilled to have had the opportunity to collaborate with the CNCF community and to grow our user base, and are honored to graduate.”

With more than 900 plugins connecting to its many data sources and data outputs; including Docker, Google Cloud, Kubernetes, etc., Fluentd was the 2016 Bossie Awards winner for the best open source datacenter and cloud software. More than 5,000 data-driven community users rely on Fluentd to differentiate their products and services through a better use and understanding of their log data, including Atlassian, Amazon Web Services, Backplane, Change.org, CyberAgent, DeNa, Drecom, GREE, GungHo, LINE Corp, Nintendo, Microsoft, Slideshare and many others.

“Fluentd has earned its place as the industry standard for log collection and shipping, and I am excited to see it as a graduated CNCF project,” said Gabe Monroy, Lead Program Manager for Containers, Microsoft Azure. “At Microsoft, we are proud to use Fluentd to power our cloud native logging subsystems and we look forward to working with the growing the open source community around Fluentd.”

Fluentd now has 4 active maintainers, over 160 contributors and more than 4,400 commits. It also has 189 official releases, more than 48 Million Docker Hub pulls, 941 plugins available made by the community and over 7500 Github Stars.

“As one of the longest standing incubating projects within CNCF, Fluentd has demonstrated impressive growth, adoption, and numerous integrations within the broader community,” said Brian Grant, Principal Software Engineer at Google and CNCF TOC representative. “We have no doubt that Fluentd is ready to move to this next stage, and are excited to see what the project continues to accomplish.”

To officially graduate from incubating status, the project also adopted the CNCF Code of Conduct, earned a Core Infrastructure Initiative Best Practices Badge. Completed in August 2017, the CII badge shows an ongoing commitment to code quality and security best practices.  

Fluentd Background

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. All components are available under the Apache 2 License, and are overseen by a self-selected team of active contributors to the project. For downloads, documentation, and how to get involved, visit https://github.com/fluent/fluentd, https://www.fluentd.org/ and https://twitter.com/fluentd.

The Linux Foundation is developing an elearning course titled Cloud Native Logging with Fluentd, which will be available in May 2019. This self-paced, hands-on course is designed to introduce individuals with a technical background to the Fluentd log forwarding and aggregation tool for use in Cloud Native Logging and provide them with the skills necessary to deploy Fluentd in a wide range of production settings.

Additional Resources

About Cloud Native Computing Foundation

Cloud native computing uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. The Cloud Native Computing Foundation (CNCF) hosts critical components of cloud native software stacks, including Kubernetes and Prometheus. CNCF serves as the neutral home for collaboration and brings together the industry’s top developers, end users and vendors – including the world’s largest public cloud and enterprise software companies as well as dozens of innovative startups. CNCF is part of The Linux Foundation, a nonprofit organization. For more information about CNCF, please visit www.cncf.io.

###

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

Media Contact

Natasha Woods

The Linux Foundation

nwoods@linuxfoundation.org

Logging is a critical component on production environments that allow us to perform monitoring and data analysis. While applications runs at scale, the Logging layer also needs to scale and year over year we see new challenges that needs to be solved from different angles such as parsing, performance, log enrichment (metadata), filtering and so on.

Fluentd was born to solve Logging problems as a whole, not only for standalone applications but also for distributed architectures where each running application and system have their own way to solve logging, integration between all components and the ability to move data from one place to another in a secure and reliable way was a requirement from day one.. and it continue to be as of today. That’s why it has been adopted by thousands of companies and thanks to it growing community around the world demonstrate that Fluentd became the Industry Standard for Logging.

Fast iterations and solving the problem better than yesterday is our mantra, that’s why the global team of maintainers is proud to announce that Fluentd v1.0 has been released!, this is a big milestone for everyone around the Fluentd Ecosystem which means much more that a number, it’s a maturity resulting of years of work in the community where feedback, ideas and general testing have been the roots for it growth. So thank all you who have been involved on this process!

What’s Hot in Fluentd v1.0 ?

Fluentd v1.0 built on top of v0.14-stable, some of the biggest changes on this new series are:

  • Core:
    • Multiprocess Workers: take advantage of SMP systems
    • Sub-second time resolution: all log records now have a granular time resolution
  • Portability: Windows Support has finally arrived to Fluentd.
  • New plugins API: Our biggest contributions to Fluentd ecosystem is through plugins, with more than 700 plugins available made by the community we have focused into improve the developer experience.
  • Data Management: new internal buffers can optionally enable compressions to save disk space.
  • Networking:
    • New Fluentd Forward Protocol v1: includes authentication using shared keys and authorizations through username/password.
    • Native Transport-Layer-Security (TLS) support

These changes are not the only ones, there many improvements around performance, portability and flexibility for data management.

Monitoring and Data Streaming

Fluentd is more than a project, it’s a full ecosystem and integration with third party components is fundamental, that’s why as part of our Fluentd v1.0 release we are proud to announce also the continuous investment in integration in Prometheus (monitoring) and Apache Kafka (data streaming) within many others.

Prometheus makes easier monitoring and having Fluentd provide native support for it has been in high demand in the last time and we are happy to announce that fluent-plugin-prometheus is now officially part of Fluentd Ecosystem hosted on CNCF Fluent organization on Github.

Data streaming is such important as monitoring, and companies around are looking for ways to integrate more components where Logging can be a critical compoment of the data pipeline. Fluentd is getting better and better at it: Fluentd and Apache Kafka can talk each other smoothly and securely.

What’s Next

Next year we will continue working towards performance improvements and connectors that makes easier to hook Fluentd on any place. As well from an ecosystem perspective, Fluent Bit, our lightweight log processor will keep growing in terms of capabilities for cloud native environments such as like load balancing, persistent queues and monitoring within others.

2018 will be an exciting journey, don’t hesitate to be part of it!

Connect at CloudNativeCon + KubeCon

If you are attending KubeCon + CloudNativeCon North America this week, please join us December 6th for a Fluentd Salon, our community will be presenting the following lightning talks:

  • “Moving Data at Scale” by Sadayuki Furuhashi (Fluentd Creator)
  • “Fluentd and Kafka” by Masahiro Nakagawa (Fluentd Maintainer)
  • “Fluentd and Prometheus” by Yuki Ito (Fluentd Maintainer)
  • “Fluent Bit” by Eduardo Silva (Fluent Bit Maintainer)
  • “Plugging into fluent-bit: how to use plugin templates to customize fluent-bit to serve your needs” by Yeni Capote, Samsung SDS

we will also discuss Fluentd v1.0, roadmap and tools around the Fluent ecosystem plus an open space for discussions. Additionally, do not miss the additional sessions on Fluentd:

Closing Thoughts

Reaching v1.0 for us it’s a new begginning, we will continue working together with our end-users, community and companies around to make Fluentd better for 2018. If you are around CloudNativeCon+KubeCon don’t hesitate to contact us in our sessions or during the break ours, come and join us!, you can also help to solve the problem better than yesterday !

Let’s keep improving the Logging Standard…