How this Telecom Network Testing Company Automated Emailing of Hotspot Reports from Kibana and Grafana to 3,000 Technicians

Executive Summary

 
This case study will show how Enghouse Systems, a telecom network testing solution provider to the leading service providers in North America, automated emailing of daily and weekly test reports in PDF format from its Elasticsearch-Kibana-Grafana data lake to three thousand field technicians and their managers.
 
Automated test reports helped the Enghouse technicians and managers to address the most critical issues in their customer switches and maintain the SLAs for their telecom networks. Enghouse was able to accomplish this goal even though they did not have the enterprise subscription for Elastic Stack or Grafana.
 
In this case study, we’ll show you how reporting is critical for Enghouse’s telecom testing business and how Enghouse added reporting to Kibana and Grafana using Skedler to implement this key business requirement.
 

The Secret to Keeping Telecom Networks Humming

 
Testing and monitoring telecom service providers’ networks for failure and hotspots is one of the key lines of business at Enghouse, a global company with business ranging from software development to integration. Enghouse got into the Telecom network testing business through the acquisition of Tollgrade which developed both hardware and software solutions for Telecommunications. Tollgrade’s solution will test the service provider’s phone lines and if the customer has a metallic connection, it will also provide information based on Digital Subscriber Line Access Multiplexer(DSLAM) or Gigabit Passive Optical Networks(GPON).
 
Enghouse has hardware in different central offices across the various states in the US. This hardware will test metallically the lines of the switch. It will test whether the measurement heads are testing properly. Testing is broken down by switch clusters in specific states. Over 3000 technicians across the US monitor the data coming from these tests.
 

Migrating Test Data Platform from Legacy Infrastructure to Elasticsearch, Kibana and Grafana

 
Enghouse had developed their own system for providing testability information to the telecom companies that use their software. Since it is not user friendly and not accessible via browser, they migrated to a browser based solution that was built using Elasticsearch as the backend database with Kibana and Grafana as the frontend dashboard applications.
 
Depending on the type of visualization needed, either Kibana or Grafana is used to visualize the test data captured in Elasticsearch.
 
The powerful combination of Elasticsearch, Kibana and Grafana enabled customers and technicians to easily access the data and visualize the information that they need to monitor.

Check out the complete video of John Hirth speaking about his use case.
 

Email Daily Reports to 3000 Technicians

 
The legacy product line had the ability to send out reports to the field technicians and managers. The first set of reports is sent to technicians in a particular state. It will provide a daily summary of the metallic test data from the lines off the switch clusters in that state. The technicians will come to know if the measurement heads are testing properly. More than 3000 technicians receive the daily summary reports that are critical for their day-to-day work. The reports help the technicians identify the switches that require immediate attention and fix them in a timely manner.
 

Weekly Hotspot Reports to Managers

 
The second report is a hotspot report that is delivered to the managers. This report gives the managers an overview of what is going on in their network. Since the technician’s time is valuable, the managers use this report to direct their technicians to address issues in critical central offices.
 

 
The technician report goes out Monday through Friday and the hotspot report is sent out every Tuesday.
 

Missing Reporting Capability in Kibana and Grafana

 
Enghouse uses the open source version of Elasticsearch, Kibana and Grafana. Since Kibana and Grafana don’t offer reporting unless you purchase their enterprise subscriptions, John Hirth, the Program Manager responsible for the monitoring system, searched the internet for an affordable tool that generates reports from Kibana and Grafana. That’s how he came across Skedler, the Kibana and Grafana reporting solution.
 

Kibana and Grafana Reports to 3000 users in
Just 2 months

 
John got in touch with the Skedler team, saw a demonstration, and ran a proof of concept. After a successful PoC that involved automating sample reports for technicians and managers, he purchased Skedler license and activated the license in his PoC environment. He used Skedler to build additional technician reports and hotspot reports off his Kibana and Grafana dashboards.
 

 
With management being able to have an email sent to them that says, “Hey, something’s going wrong” is a tremendous value. “They need to work on the critical issues and having the ability to send the report or alert saying something’s wrong is very valuable” says John.
 

John Hirth’s Word of Wisdom for Kibana and Grafana Users

 
Last, but not least, John Hirth recommends companies to check out Skedler if they want to have an application that they can quickly bring up and satisfy their Kibana and Grafana reporting needs. “It’s a great fit,” says John.
 
If you found this case study inspiring, we’d really appreciate it if you would share the story on Twitter/LinkedIn/Facebook or other forums.
 

How this Managed Detection and Response Team Automated Cybersecurity Reports to Customers from Elasticsearch Kibana

Executive Summary

 
This case study will show how Cynet, a cybersecurity platform and service provider from Israel, automated emailing of security reports to its customers in PDF format from its Elasticsearch-Kibana security data lake and freed up more than 20 hours per month of employee time.
 
Visual security reports help the Cynet Managed Threat Detection and Response(MDR) service team to drive value to customers about their unique solution and meet the customer SLA requirements. Cynet was able to easily accomplish their goal of automating insightful cybersecurity reports to their major customers within one month and freed up tens of hours per month for its Operations team.
 
In this case study, we’ll show you how reporting helps the Cynet MDR Cybersecurity Operations Team and how Cynet accomplished this key business requirement.
 

Cynet’s Unique Approach to Cybersecurity

 
More than ever, Cybersecurity is at the top of mind for all organizations. The recent Solarwinds hack not only highlights the sophistication of the attacks, but also brings the need for an equally sophisticated cybersecurity strategy and operations for every organization to the forefront. And this is exactly the forte of Cynet, a cybersecurity platform and services provider based in Israel.
 
Cynet was started in Israel five years ago to focus on endpoint detection and protection while delivering included security service on top of their dedicated Cynet 360 platform. The unique advantage of Cynet is that their Managed Threat Detection and Response(MDR) service is included free of cost with their Cynet 360 platform to all their customers. Its customers include large and medium sized organizations such as Allianz, Costa Crociere, and East Boston Neighborhood Health Center.
 

The Secret Sauce of Cynet Managed Detection & Response Service

 
Cynet’s MDR service is offered by the Cyber Operations(CyOps) team which is led by Shiran Grinberg, CyOps Manager and Sivan Chachashvili, CyOps Team Leader. The CyOps team is a 24×7 SWAT team of seasoned professionals focused on identifying threats and vulnerabilities in customer environments and providing the human oversight necessary to detect and respond in a timely manner.
 
On any given day, the CyOps team goes through tons of data and alerts coming from the customer environment. They use Elasticsearch to capture and analyze the customer environment security data. Kibana is used to visualize the data. The CyOps team analyzes the alerts and decides if the alerts are traces of malicious activity. If they are malicious, a chain of events called Incident Response is initiated to mitigate and remediate the threat in the customer environment.
 

Using Cybersecurity Reports to Drive
Customer Value

 
The CyOps team needed to present the findings from their analysis to clients in a visual manner. While their Cynet360 platform is used for monitoring the activities, customers needed monthly and quarterly reports of the security posture of their environments. These reports not only provide insights to customers about their cybersecurity readiness but also help Cynet accentuate the value delivered by its unique combination of MDR platform and CyOps service. The reports increase customer retention and also differentiate Cynet from its competitors.
 
Check out the complete video of Shiran and Sivan speaking about their company and how they use Skedler for MDR Reporting from Kibana.
 

Manual Reporting from Kibana was
Time-Consuming

 
Since the open source version of Kibana lacked reporting capability, the CyOps team had to manually create these reports for each of its customers. They would first create the visualizations in Kibana, take screenshots, paste them into a report, format the report, email it to their customer and repeat it for each and every customer.
 
Needless to say, this was a cumbersome and time consuming process. A CyOps team member was spending several days per month to create these customer reports. It became unsustainable and forced the CyOps team to look for tools that could automate the report creation and distribution.
 

Cynet Automates Export of Kibana Reports
with Skedler

 
The CyOps team evaluated three potential solutions for Kibana reporting and ultimately chose Skedler as their Kibana reporting solution. “We just took Skedler as it was the best solution for us”, says Sivan Chachashvili. For CyOps team, it’s the manpower and the time saved every month by not spending countless hours to take screenshots and manually create reports for customers.
 

 
“Within one month, we have produced 20 reports for most major customers using Skedler and we’ve already started to gain traction”, says Shiran Grinberg, CyOps Team Manager at Cynet.
 

Last Word, But Not the Least, from CyOps Team

Skedler’s technical prowess was not the only factor that impressed Cynet. Shiran and team were pleased with the sales and after-sales support provided by the Skedler team. “Sales and support team is a part of the overall picture or the overall product. Once you have people who you can communicate with, they understand you and you understand them, it makes everything way easier.” concludes Shiran.
 
If you found this case study inspiring, we’d really appreciate it if you would share the story on Twitter/LinkedIn/Facebook or other forums.
 

How this India’s Publishing Group is Winning at Digital with an Early Warning System for its Online Newspaper

The Hindu and many other popular news outlets in India are running smoothly thanks to Skedler.

Client profile: The Hindu Group

• One of the oldest publishing companies in India. Publisher of The Hindu, a daily newspaper that began in 1878 and has been steadily growing in circulation ever since.
• The Hindu’s independent editorial stand and its reliable and balanced presentation of the news have won the serious attention and regard of the people who matter in India and abroad.
• The Hindu uses modern facilities for news gathering, page composition and printing. It is printed in seventeen centers across India that are connected with high-speed data lines.

The increasing use of mobile devices over the last decade has fuelled a desire on the part of online news outlets to provide readers with an improved mobile experience across all kinds of browsers. This effort is sometimes hampered by unforeseen browser incompatibilities, resulting in browsing anomalies for some readers. Because smooth user experience is so essential in attracting and retaining audiences, most news websites are continuously striving to detect and deal with such problems quickly and efficiently.

Poonkuyilan V is one of the often unsung, behind-the-scenes heroes that keep the online reading experience bug-free. He works at India’s national newspaper – The Hindu – where he leads the IT department’s infrastructure team. His team is responsible for maintaining the smooth operation of the website.

Things are a bit chaotic before Skedler

Initially, Poonkuyilan and his team used Elastic Stack for the access logs only, but as they became more comfortable with it, they began using it for all of the logs: “We feed the logs into Elasticsearch to analyze them. The visualization tool we like to use is Kibana, a very user-friendly tool that is part of Elasticsearch. These visualizations make it straightforward for us to not just solve any issues that may come up, but to also find what’s causing them much easier compared to the traditional way of log monitoring.” In addition to monitoring, Poonkuyilan’s team is responsible for the availability and integrity of the websites. Once they get an alert that something is wrong, they have to react based on that alert. However, the IT team was becoming very aware that most of the alerts weren’t coming from the Elastic Stack tool’s Elastalert, but rather from the end-users. Many readers were not having a smooth experience using The Hindu Group’s websites.

“Indeed, the readers would call or email us, asking how to fix a problem with the website pages or why a certain anomaly keeps happening. So then we would go into the system to look at the numbers and to solve the issue,” remembers Poonkuyilan. Anyone can see the problem here. Relying on end-users to signal about problems in your product is not the optimal way to attract and retain readers. It was clear to everyone in the IT team that a more proactive approach was required. That’s when Poonkuyilan and his colleagues started actively searching for viable alternatives to the Elastalert: “We were looking at online forums and that’s how we discovered the existence of Skedler. Once we downloaded it and connected it to the Elasticsearch, Skedler started generating alerts right away. The whole process was very intuitive and provided us with a way to easily configure alerts right in the dashboard.”

Check out the complete video of Poonkuyilan speaking about his use case.

The team’s favorite Skedler features

Poonkuyilan admits he finds other alerting systems for the Elastic Stack a bit too difficult to use: “You need to understand the data and you also need to understand the tool you are trying to implement. They often require some development effort to work correctly, so it’s not that easy for some of the team members to create an alert.

One of The Hindu Group’s IT team’s favorite Skedler features is the ability to create alerts in the console itself within two or three minutes: “We have Skedler directly integrated in the Elasticsearch, which is great. Creating alerts directly in the console itself is a fantastic feature. We also found all the available pre-defined templates to be a really nice touch as well.”

Another Skedler feature that has recently become a team favorite is the way Skedler generates weekly reports: “Some of the top-level management don’t really need daily alerts; they just want to receive a high-level, weekly report that outlines what’s happened in the website during the week,” says Poonkuyilan, “Sometimes all they want is a monthly report. In these weekly and monthly reports where we are including data about any 503 errors that may have occurred, as well as stats on the responsiveness of the websites and the pages. This is where Skedler really helps us with how easy and intuitive it is to understand how the pages have performed”

Poonkuyilan says that Skedler has definitely helped his team in fine-tuning the monitoring process: “Thanks to its sensitive alerting system, we started pinpointing what the anomalies and issues were going to be before they even occur. We truly appreciate Skedler’s ease of use. We can set alarms quickly and then concentrate on the day-to-day work and other processes. We don’t have to fiddle with the alarm settings at all. In fact, since we began using the Skedler tool, there haven’t been any major incidents with any of The Hindu Group online publications!”

If you found this case study inspiring, we’d really appreciate it if you would share the story on Twitter/LinkedIn/Facebook or other forums.

Elastic Stack SIEM Use Case – Why this company uses ELK to detect network threats

Thousands of organizations all around the world use Elastic Stack, also known as ELK Stack (which stands for Elasticsearch, Logstash, Kibana), to manage, monitor and analyze logs. The open source tools are flexible and can be applied to multiple different use cases. In this post, we will highlight one such application: Elastic Stack for SIEM.

What is SIEM and why do you need it?

SIEM stands for Security Information and Event Management. In a nutshell, SIEM is a combination of technologies that give an overall look at a system’s infrastructure as well as analysis (and more), in order to keep the system safe.

If your company handles sensitive data, you need to have an SIEM. Companies employ SIEM tools for compliance, certifications, log management and monitoring, case management, and enforcing and identifying policy violations.

Elastic Stack + SIEM

Chris Rock at DEFCON

To get a real-world example, we went directly to an expert. Chris Rock is the CEO and founder of Kustodian and SIEMonster. SIEMonster is an SIEM tool to detect threats and risks in a company’s network. They focus on detecting risks for companies using the ELK framework as a base. For example, detecting unauthorized access attempts, brute force logins, network parameters for safety, and more.

With SIEMonster, an open source alternative to Shield or Marvel, organizations have global, real-time security monitoring without the development headaches, documentation integration and price tag of other SIEM solutions.  

Pros & Cons

When deciding which systems to use to build SIEMonster, Rock looked for a solution that was mature. He considered Cisco Open Framework (now Apache Metron). But “companies like Netflix and Ebay were already using it, so ELK as a backend was perfect for us to build an SIEM around it,” he explained.

Many security vendors have general solutions. SIEMonster customers need something to show them what is happening on their specific network.

Rock also likes ELK for its scalability. Elastic Stack is the perfect Database and Log parsing suite because it will scale indefinitely and grow with their SIEM tools.

Elastic Stack’s versatility allowed Rock to develop a solution for monitoring different metrics for each customer. Using ELK, Rock built out plugins, dashboards and parses to collect, process, and visualize data, then automatically detect risks, create tickets, and send alerts and reports. There are almost no limits to what you can build.

“The metrics we are after depend on what the customer wants to see. For example, the top 10 offending IP addresses, countries performing the most amount of attacks, etc.”

And each customer can get exactly what they need.

The biggest drawback for Elastic Stack from Rock’s perspective are the compression rates. Compared to commercial solutions, like Splunk, which compresses data to 20% of its original size, ELK has room for improvement. ELK compresses data closer to about 80%, so that is better than nothing, but worth considering.

Finally, Rock rounds out the SIEMonster solution with Skedler. To send automated Kibana reports like those mentioned earlier, as a PDF, Excel file or PNG, “We needed a reporting tool for SIEMonster, Skedler is perfect for this situation.”

If SIEMonster detects any issues for their clients, they can immediately send a report with all the pertinent information. Additionally, they can send summary reports regularly, such as weekly or monthly, to give customers a status update for peace of mind.

Conclusion

The Elastic Stack – Elasticsearch, Logstash, Kibana – is a powerful suite of tools for centralized logging, system monitoring, business intelligence, monitoring and security, and more. There is no shortage of examples of the applications of ELK.

Here we explored how one SIEM solution was built using ELK stack, including the benefits and some drawbacks. Rock and the SIEMonster team found ELK tools can help them deliver a better product for their customers. Hopefully, this will help to give you inspiration and ideas for implementing it yourself.

Get future tech posts and use cases delivered directly to you by subscribing to our newsletter.

How This Managed Security Provider Blew Their Customers Away with Customized ELK Reports

Ben, a cybersecurity analyst at Dynetics, found himself in a tough spot. He was tasked with finding a scalable solution to finally fix his company’s Kibana reporting.

The solution they had built to export Kibana dashboard visualizations using open source tools “was horrible.” It consumed valuable resources with maintenance and didn’t work well. X-Pack was an option, but the price point was too high and features were overkill.

He needed something else – something that was easy to deploy, affordable and delivered relevant information.

That’s when Ben turned to Skedler.

Skedler Reports saved him thousands of dollars and countless hours. Plus, his customers were happier with the “eye candy” the reports provide. If you work with ELK stack, you can also impress your customers and increase market competitiveness with customized ELK reports using Skedler.

Engineers using ELK stack need a way to export Kibana data for managers, customers, analysts and other stakeholders. Capturing structured logs is very important for a variety of reasons. They need a solution to allow periodic reports on business metrics contained in Kibana. Ideally, it will automate delivery and allow customization.

In many cases, generating periodic reports is part of service level agreements (SLAs) with customers. Reports help with identifying threat activity or security incidents, staying compliant, keeping customers informed about activity and becoming more competitive in the market. Unfortunately, getting data from the ELK platform into reports for stakeholders can be a challenge.

How do you easily and affordably send relevant reports to various stakeholders?

Before Skedler: Wrestling with Open Source Tools

With more and more customers requiring periodic data reports, Ben’s company, Dynetics, knew they needed to automate the reporting process. Like Ben, others have tried a variety of open source tools to build a solution that would take a screenshot of the Kibana dashboards and export them.

However, they didn’t scale well, required technical expertise to create reports and queries, and didn’t allow you to format or personalize the output. In most cases, it took a huge amount of resources and time to maintain and actually get to work.

After a lot of frustration, they decided to find a web-based tool that provided the ability to quickly set up reports on a schedule, send them via email and was cost-effective.

How to Evaluate ELK Reporting Solutions

When it comes to finding the right solution, DevOps managers, engineers and business intelligence or security analysts have several different considerations. The key to a reporting solution is that it gives the recipient what they need, it doesn’t consume the entire budget and it’s easy to use.

Give the Recipient What They Need

Think of the report in the terms of its value for recipients.

First, what are the customer requirements? Primarily, as Logi Analytics point out in Reporting in BI, “Reporting means collecting and presenting data so that it can be analyzed.” It’s vital to consider the recipient in terms of the format, frequency and delivery method of the report.

Log reporting requirements range from preventing financial and reputational loss on behalf of clients, to gauging the operational health of the platform, to measuring key performance indicators. In each instance, the file type, frequency and ease of access differ.

How frequently do stakeholders need to receive information? Frequency changes depending on the the stakeholder and/or type of information.

Customers or managers may want a weekly or monthly printable PDF report. Analysts may require an hourly or daily export of raw data for further analysis. In other cases, VPs only require monthly KPI reports. Additionally, there are ad-hoc reports.

For analysts, the best format is a .csv or .xls file to continue analysis. But other stakeholders need something that looks good and is printable. You need to provide professional reports, customized with your company logo and branding, to increase credibility.

Your reporting tool must deliver the necessary relevant information that your customers need, when and how they need it. While some recipients prefer email, others want delivery via productivity tools, like Slack, or in other applications.

Watch Your Budget

In addition to creating relevant information, the reporting solution must be affordable and easy to implement.

As ELK users, many engineers consider X-Pack for a reporting solution. But in many cases, the price comes as a shock. One Skedler customer saw X-Pack as the obvious solution, believing it would be free like the rest of ELK stack.

Then they saw the price tag.

At the other end of the budget spectrum, you have some open source solutions.

While certainly affordable, the cost of open source options is in extra bandwidth from your team. Open source tools can be built to your specific requirements. But they also require technical expertise to build, configure, maintain, scale and, in many cases, run the actual reports.

A few years ago, open source could have been your only option. But these days, you can rely on the skills of someone else and focus on your core competencies instead.

Make it Easy to Use

If you have the resources to spare for maintaining a solution to produce the bare minimum, then open source might solve the issue for you. But when they needed an out-of-the-box solution to automate reports simply, Ben turned to Skedler.

“Skedler was extremely easy to setup and is very simple to use. We hooked it up to our application, configured the email, and we were ready to send out reports to customers in less than an hour,” said Ben.

Don’t mistake simple to use for lacking functionality.

As mentioned earlier, engineers often have a variety of stakeholders with differing needs. Skedler has taken those requirements into account when building Reports. The result is a lot of flexibility and customization within a straightforward user interface. This allows users to email a PDF report of ELK data to customer or attach a CSV file to a Slack message to internal stakeholders.

DevOps managers and BI analysts can add branding and customize ELK reports, so they look great for recipients. Producing a sophisticated, professional report goes beyond satisfying SLA requirements to build trust with the customer.

In Summary

Reports are a necessary part of business. ELK engineers need an affordable, easy solution to provide relevant information.

Finding the best tool to generate reports will take some research. You need to understand the needs of the recipient as well as internal restrictions on budget, team resources and tool integrations.

The reports Skedler creates for Dynetics and others have made their jobs much more streamlined. They allow organizations to stay competitive, even with many large players in the industry, keep their customers happy and free up more time for other priorities. What makes Skedler Reports so great is that it is simple to set up and use, cost-effective and flexible.

Are you looking for a way to automatically export and email reports from Kibana to your customers and analysts? Start a free trial of Skedler today.

Kibana Reporting in Action: A Kane LPI Case Study

Every company carries valuable data, whether it’s relevant to a specific client’s private information, statistics, or finances. Alongside the fear that you might lose precious data or incur a security breach with faulty programs, data needs to be filed and exported accurately in order to meet certain deadlines and practical standards; which is why Kibana reporting has proven to be an imperative tool for loss prevention within any given company.

The Company

Kane LPI Solutions is a prime example of how Kibana reporting achieved higher marketability within the cognoscente program Skedler. A trusted provider of Third Party Administration services for more than 15 years, Kane LPI has issued over US$11 billion of offshore annuity and investment products for an extensive global client base. They needed a robust solution to help prevent financial and reputation losses as well as boost KPIs. As it stood, competition was fierce — there were many much larger players out there on the field.

The Challenge

The company’s practical processes specifically involved sending sensitive and accurate post-trading files on a timely basis. Kane LPI’s clients had strict operational requirements for control and compliance: if the time window for sending files was missed, financial loss was borne both for them and the company. Delays like this could incur strict penalties by settlement and clearing corporations such as The Depository Trust and Clearing Corporation (DTCC). With so much at stake, our challenge was to fundamentally satisfy operational concerns to prevent those losses, as well as improve pragmatic business flow.

Kibana Reporting: A Means of Automation

Kibana reporting allowed Kane LPI to send out daily scheduled reports to both clients and internal users from thousands of lines of log entries within multiple systems, serving as a key monitoring tool and satisfying auditing requirements in the process. As a result, Skedler became a critical go-to ELK stack tool for this company and its clients, allowing Kane LPI to send out reports to their clients and internal users, which the company couldn’t do before.

Developers were able to receive automated error reports at the beginning of the day, allowing them to ensure information is sent to regulatory and settlement organizations to meet deadlines. Simultaneously, managers received daily and weekly reports on SLA performance and non-compliance, enabling them to take remedial steps to prevent recurrence and to protect time-sensitive transactions. Vendors also received daily and weekly reports on SLA performance and non-compliance of their product in KPI solution, which fundamentally reused the investment of energy and time in an ELK stack based solution.

As a precautionary measure, Skedler introduced a critical line of defence to errors by inaugurating manager and software vendors with automated reporting sent at the beginning of the day, which described potential errors during transfer of files, allowing future errors to be minimized. We then added another layer of security by administering any issues during batch runs from the previous 24 hours. 

Statistically, all of these enhancements enabled Kane LPI’s clients to avoid up to $5 million per month in trade and exchange losses, as well as protect their well-nurtured reputation as a high-quality provider.

Ready to start saving time by creating, scheduling and distributing Kibana reports automatically? Try Skedler for free.