Tag: aiops for devops services in usa

Augmented Analytics with SAP Analytics Cloud

Posted on by ZIF

Augmented Analytics

In 2017, Gartner coined the term ‘augmented analytics’ and claimed it would be the future of data analytics. They predicted it would be a dominant driver of new purchases of analytics and business intelligence as well as data science and machine learning platforms, and of embedded analytics.

Here is the why and how.

Most organizations depend on data to back up its decision-making and strategy. Organizations collect data on all accounts of processes and events; thus, analyzing and effectively managing the breadth of this data is challenging yet significant for mining it for business insights.

Traditional business intelligence tools have given way to a new generation of business intelligence tools – Augmented Analytics technology.

Augmented Analytics is an approach of data analytics that employs machine learning (ML) and natural language processing (NLP) to automate and improve data access and data quality, uncover hidden patterns and correlations in data, pinpoint what’s driving results, predict future results and suggest actions to maximize or minimize desirable or undesirable outcomes.

Augmented Analytics is designed to conduct analyses and generate business insights automatically with little to no supervision and can be used without needing the assistance of a business analyst or data scientist. However, the focus of Augmented Analytics stays in its assistive role, where technology does not replace humans but supports them.

Evolution of Analytics

Business Intelligence (BI) and Analytics has evolved, increasing the demand for decision making through data analytics. It drives to unfold from traditional mirror reporting into self-service Business Intelligence and analytics.

Despite the advances in self-service analytics with agile discovery, many businesses demand assistance to uncover insights in data.

The next generation of BI and analytics products are augmented with artificial intelligence (AI) including ML, which automates complex analytics processes, and NLP makes it easier for users without knowledge of data science or query languages to obtain insights.

best aiops solutions in usa

Augmented analytics offer starting-point suggestions and guidance to the users. It also empowers businesses to leverage more of their data to make better decisions when compared to the traditional and self-service Business Intelligence.

SAP Analytics Cloud

SAP Analytics Cloud (SAC) is an analytical solution that features all the analytics functionalities like business intelligence, augmented analytics, predictive analytics, enterprise planning, and application building in one intuitive user interface. It is empowered with ML and built-in AI that helps discover in-depth insights, simplify access to critical information and enable adequate decision-making.

aiops for devops services in usa

Augmented SAP Analytics Cloud

Augmented analytics capabilities offered by SAP Analytics Cloud empowers business intelligence to reap the benefits of AI and ML.

SAP Analytics Cloud facilitates users to interact with the system using natural language to gather automatic insights, where Predictive Scenarios offer an accessible way into Predictive Analytics using the past data to foresee the future.

Let’s look at the Analytics features, and capabilities offered by SAP Analytics Cloud

AI Devops Automation Service Tools

Search to Insight – Query search in Natural Language

The Search to Insight feature enables query search through natural language through conversational AI and NLP. No knowledge of query languages like SQL, R, or Python is required. Asking questions just like in a search engine or digital personal assistant fetches insightful answers represented by visualization or numeric values tailored based on the question type.

Search to Insight provides auto-complete suggestions to match words or phrases in questions for measures and dimensions in the data and includes auto spell-check.

AIOps Artificial Intelligence for IT Operations

Smart Insights – Instant explanations

The Smart Insights feature facilitates digging deeper into the data points. It analyzes the underlying dataset and runs various statistical algorithms to offer insights based on the current user context.

It helps to understand top contributors of specific data points without having to manually pivot or slice and dice the data. When a data point is selected, ML calculations run on information that is of the same nature as the selected data point. For example, if the selected data point is ‘Total Revenue’, the top contributors are based on ‘Total Revenue’. It analyzes the dimension in the selected data and looks for members in these dimensions that influence the selected value.

Smart Discovery – Easily reveal insights

The Smart Discovery feature identifies hidden patterns and statically relevant relationships in the data to discover how business factors influence performance. It helps to understand the business drivers behind the core KPIs.

Based on the selection of measure or dimension, smart discovery automatically generates interactive story pages as below –

Overview: It explains the data distribution, summary of trends, and the detected patterns for the target dimension or measure.

Key Influencers:  It explains the influence of the dimensions for the value of the target measures in the context of the selected model using classification and regression techniques, where the classification techniques are used to identify dimensions that segregate results into different groups of results and the regression techniques identify relationships between data points to predict future results.

Unexpected Values: It displays the details about outliers, where the actual values differ greatly from what the predictive model would expect. If an actual value diverges from the regression line it is categorized as unexpected.

Simulation: The simulation facilitates the ‘what-if’ analysis, users can change the values of the measures and dimensions to see the predicted change positively, negatively, or neutrally in the target measure.

Smart Predict – Answers the toughest questions

Smart Predict feature predicts the likelihood of different outcomes based on the historical data using techniques such as data mining, statistics, machine learning, and artificial intelligence.

Smart Predict, also referred as Predictive forecasting, considers different values, trends, cycles, and/or fluctuations in the data to make predictions that can be leveraged to aid business planning processes.

Smart Predict provides 3 different predictive scenario options for selection

Classification: It can be used to generate predictions for a binary event. For example, whether individual customers would be likely to buy the target product or not.

Time Series: It can be used to forecast values over a set period. For example, forecasting the sales of product by month or week, using historical data.

Regression: It can be used to predict values and explore key values behind them. For example, predicting the price of an imported product based on projected duties or shipping charges.

In the modern world of business Intelligence, SAP Analytics cloud’s ML technology augments the analytic process which assists from insights to actions and enables avoiding the agenda-driven and biased decision making by revealing the accurate patterns which drives the business.

References

MF Kashif

About the Author –

Kashif is a SAP Business objects consultant and a business analytics enthusiast. He believes that the “Ultimate goal is not about winning, but to reach within the depth of capabilities and to compete against yourself to be better than what you are today.”

Why is AIOps an Industrial Benchmark for Organizations to Scale in this Economy?

Posted on by ZIF

Business Environment Overview

In this pandemic economy, the topmost priorities for most companies are to make sure the operations costs and business processes are optimized and streamlined. Organizations must be more proactive than ever and identify gaps that need to be acted upon at the earliest.

The industry has been striving towards efficiency and effectivity in its operations day in and day out. As a reliability check to ensure operational standards, many organizations consider the following levers:

  1. High Application Availability & Reliability
  2. Optimized Performance Tuning & Monitoring
  3. Operational gains & Cost Optimization
  4. Generation of Actionable Insights for Efficiency
  5. Workforce Productivity Improvement

Organizations that have prioritized the above levers in their daily operations require dedicated teams to analyze different silos and implement solutions that provide the result. Running projects of this complexity affects the scalability and monitoring of these systems. This is where AIOps platforms come in to provide customized solutions for the growing needs of all organizations, regardless of the size.

Deep Dive into AIOps

Artificial Intelligence for IT Operations (AIOps) is a platform that provides multilayers of functionalities that leverage machine learning and analytics.  Gartner defines AIOps as a combination of big data and machine learning functionalities that empower IT functions, enabling scalability and robustness of its entire ecosystem.

These systems transform the existing landscape to analyze and correlate historical and real-time data to provide actionable intelligence in an automated fashion.

AIOps platforms are designed to handle large volumes of data. The tools offer various data collection methods, integration of multiple data sources, and generate visual analytical intelligence. These tools are centralized and flexible across directly and indirectly coupled IT operations for data insights.

The platform aims to bring an organization’s infrastructure monitoring, application performance monitoring, and IT systems management process under a single roof to enable big data analytics that give correlation and causality insights across all domains. These functionalities open different avenues for system engineers to proactively determine how to optimize application performance, quickly find the potential root causes, and design preventive steps to avoid issues from ever happening.

AIOps has transformed the culture of IT war rooms from reactive to proactive firefighting.

Industrial Inclination to Transformation

The pandemic economy has challenged the traditional way companies choose their transformational strategies. Machine learning powered automation for creating an autonomous IT environment is no longer a luxury. he usage of mathematical and logical algorithms to derive solutions and forecasts for issues have a direct correlation with the overall customer experience. In this pandemic economy, customer attrition has a serious impact on the annual recurring revenue. Hence, organizations must reposition their strategies to be more customer centric in everything they do. Thus, providing customers with the best-in-class service coupled with continuous availability and enhanced reliability has become an industry-standard.

As reliability and scalability are crucial factors for any company’s growth, cloud technologies have seen a growing demand. This shift of demand for cloud premises for core businesses has made AIOps platforms more accessible and easier to integrate. With the handshake between analytics and automation, AIOps has become a transformative technology investment that any organization can make.

As organizations scale in size, so does the workforce and the complexity of the processes. The increase in size often burdens organizations with time-pressed teams having high pressure on delivery and reactive housekeeping strategies. An organization must be ready to meet the present and future demands with systems and processes that scale seamlessly. This why AIOps platforms serve as a multilayered functional solution that integrates the existing systems to manage and automate tasks with efficiency and effectivity. When scaling results in process complexity, AIOps platforms convert the complexity to effort savings and productivity enhancements.

Across the industry, many organizations have implemented AIOps platforms as transformative solutions to help them embrace their present and future demand. Various studies have been conducted by different research groups that have quantified the effort savings and productivity improvements.

The AIOps Organizational Vision

As the digital transformation race has been in full throttle during the pandemic, AIOps platforms have also evolved. The industry did venture upon traditional event correlation and operations analytical tools that helped organizations reduce incidents and the overall MTTR. AIOps has been relatively new in the market as Gartner had coined the phrase in 2016.  Today, AIOps has attracted a lot of attention from multiple industries to analyze its feasibility of implementation and the return of investment from the overall transformation. Google trends show a significant increase in user search results for AIOps during the last couple of years.

ai automated root cause analysis solution

While taking a well-informed decision to include AIOps into the organization’s vision of growth, we must analyze the following:

  1. Understanding the feasibility and concerns for its future adoption
  2. Classification of business processes and use cases for AIOps intervention
  3. Quantification of operational gains from incident management using the functional AIOps tools

AIOps is truly visioned to provide tools that transform system engineers to reliability engineers to bring a system that trends towards zero incidents.

Because above all, Zero is the New Normal.

About the Author –

Ashish Joseph

Ashish Joseph is a Lead Consultant at GAVS working for a healthcare client in the Product Management space. His areas of expertise lie in branding and outbound product management.

He runs a series called #BizPective on LinkedIn and Instagram focusing on contemporary business trends from a different perspective. Outside work, he is very passionate about basketball, music and food.

Customize Business Outcomes with ZIFTM

Posted on by ZIF

Zero Incident Framework™ (ZIF) is the only AIOps platform that is powered with true machine learning algorithms with the capability to self-learn and adapt to today’s modern IT infrastructure.

ZIF’s goal has always been to deliver the right business outcomes for the stakeholders. Return on investment can be measured based on the outcomes the platform has delivered. Users get to choose what business outcomes are expected from the platform and the respective features are deployed in the enterprise to deliver the chosen outcome.

Single Pane of Action – Unified View across IT Enterprise

The biggest challenge IT Operations teams have been trying to tackle over the years is to get a bird’s eye view on what is happening across their IT landscape. The more complex the enterprise becomes the harder it becomes for the IT Operations team to understand what is happening across their enterprise. ZIF solves this issue with ease.

digital transformation company in usa

The capability to ingest data from any source monitoring or ITSM tool has helped IT organizations to have a real-time view of what is happening across their landscape. Enormous time can be saved by the IT engineers with ZIF’s unified view, who would otherwise be traversing between multiple monitoring tools.

ZIF can integrate with 100+ tools to ingest (static/dynamic) data in real-time via ZIF Universal Connector. This is a low code component of ZIF and dataflows within the connector can also be templatized for reuse. 

AIOps based Analytics Platform

Intelligence – Reduction in MTTR – Correlation of Alerts/Events

Approximately 80% of the time is lost by IT engineers in identifying the problem statement for an incident. This has been costing billions of dollars for enterprises. ZIF, with the help of Artificial Intelligence, can reduce the mean time to identify the probable root cause of the incident within seconds. The high-performance correlation engine that runs under the hood of the platform process millions of patterns that the platform has learned from the historical data and correlates the sequences that are happening in real-time and creates cases. These cases are then assigned to IT engineers with the probable root cause for them to fix the issue. This increases the productivity of the IT engineers resulting in better revenue for organizations.

best aiops solutions in usa

best aiops products tools and products

Intelligence – Predictive Analytics

AIOps platforms are incomplete without the Predictive Analytics capability. ZIF has adopted unsupervised machine learning algorithms to perform predictive analytics on the utilization data that is ingested into the platform. These algorithms can learn trends and understand the symptoms of an incident by analyzing tons of data that the platform had consumed over a period. Based on the analysis, the platform generates opportunity cards that help IT engineers take proactive measures on the forecasted incident. These opportunity cards are generated a minimum of 60 minutes in advance which gives the engineers a lead time to fix an issue before it strikes the landscape.

Visibility – Auto-Discovery of IT Assets & Applications

ZIF agentless discovery is a seamless discovery component, that helps in identifying all the IP assets that are available in an enterprise. Just not discovering the assets, but the component also plots a physical topology & logical map for better consumption of the IT engineers. This gives a very detailed view of every asset in the IT landscape. The logical topology gives in-depth insights into the workload metrics that can be utilized for deep analytics.

predictive analytics using ai applications

ai data analytics monitoring tools

Visibility – Cloud Monitoring

ai devops platform management services

In today’s digital transformation journey, cloud is inevitable. To have a better control over the cloud orchestrated application, enterprises must depend on the monitoring tools provided by cloud providers. The lack of insights often leads to the unavailability of applications for end-users. More than monitoring, insights that help enterprises take better-informed decisions are the need of the hour.

ZIF’s cloud monitoring components can monitor any cloud instance. Data that are generated from the provider provided monitoring tools are ingested into ZIF to further analyze the data. ZIF can connect to Azure, AWS & Google Cloud to derive data-driven insights.

Optimization – Remediation – Autonomous IT Operations

ZIF does not stop by just providing insights. The platform deploys the right automation bot to remediate the incident.

ZIF has 250+ automation bots that can be deployed to fast-track the resolution process by a minimum of 90%. Faster resolutions result in increased uptime of applications and better revenue for the enterprise.

Sample ZIF bots:

  • Service Restart / VM Restart
  • Disk Space Clean-up
  • IIS Monitoring App Pool
  • Dynamic Resource Allocation
  • Process Monitoring & Remediation
  • DL & Security Group Management
  • Windows Event Log Monitoring
  • Automated phishing control based on threat score
  • Service request automation like password reset, DL mapping, etc.
best aiops solutions in usa

For more information on ZIF, please visit www.zif.ai

About the Author –

Anoop Aravindakshan

An evangelist of Zero Incident FrameworkTM, Anoop has been a part of the product engineering team for long and has recently forayed into product marketing. He has over 14 years of experience in Information Technology across various verticals, which include Banking, Healthcare, Aerospace, Manufacturing, CRM, Gaming, and Mobile.

Generative Adversarial Networks (GAN)

Posted on by ZIF

In my previous article (zif.ai/inverse-reinforcement-learning/), I had introduced Inverse Reinforcement Learning and explained how it differs from Reinforcement Learning. In this article, let’s explore Generative Adversarial Networks or GAN; both GAN and reinforcement learning help us understand how deep learning is trying to imitate human thinking.

With access to greater hardware power, Neural Networks have made great progress. We use them to recognize images and voice at levels comparable to humans sometimes with even better accuracy. Even with all of that we are very far from automating human tasks with machines because a tremendous amount of information is out there and to a large extent easily accessible in the digital world of bits. The tricky part is to develop models and algorithms that can analyze and understand this humongous amount of data.

GAN in a way comes close to achieving the above goal with what we call automation, we will see the use cases of GAN later in this article.

This technique is very new to the Machine Learning (ML) world. GAN is a deep learning, unsupervised machine learning technique proposed by Ian Goodfellow and few other researchers including Yoshua Bengio in 2014. One of the most prominent researcher in the deep learning area, Yann LeCun described it as “the most interesting idea in the last 10 years in Machine Learning”.

What is Generative Adversarial Network (GAN)?

A GAN is a machine learning model in which two neural networks compete to become more accurate in their predictions. GANs typically run unsupervised and use a cooperative zero-sum game framework to learn.

The logic of GANs lie in the rivalry between the two Neural Nets. It mimics the idea of rivalry between a picture forger and an art detective who repeatedly try to outwit one another. Both networks are trained on the same data set.

A generative adversarial network (GAN) has two parts:

  • The generator (the artist) learns to generate plausible data. The generated instances become negative training examples for the discriminator.
  • The discriminator (the critic) learns to distinguish the generator’s fake data from real data. The discriminator penalizes the generator for producing implausible results.

GAN can be compared with Reinforcement Learning, where the generator is receiving a reward signal from the discriminator letting it know whether the generated data is accurate or not.

Generative Adversarial Networks

During training, the generator tries to become better at generating real looking images, while the discriminator trains to be better classify those images as fake. The process reaches equilibrium at a point when the discriminator can no longer distinguish real images from fakes.

Generative Adversarial Networks

Here are the steps a GAN takes:

  • The input to the generator is random numbers which returns an image.
  • The output image of the generator is fed as input to the discriminator along with a stream of images taken from the actual dataset.
  • Both real and fake images are given to the discriminator which returns probabilities, a number between 0 and 1, 1 meaning a prediction of authenticity and 0 meaning fake.

So, you have a double feedback loop in the architecture of GAN:

  • We have a feedback loop with the discriminator having ground truth of the images from actual training dataset
  • The generator is, in turn, in a feedback loop along with the discriminator.

Most GANs today are at least loosely based on the DCGAN architecture (Radford et al., 2015). DCGAN stands for “deep, convolution GAN.” Though GANs were both deep and convolutional prior to DCGANs, the name DCGAN is useful to refer to this specific style of architecture.

Applications of GAN

Now that we know what GAN is and how it works, it is time to dive into the interesting applications of GANs that are commonly used in the industry right now.

Generative Adversarial Networks

Can you guess what’s common among all the faces in this image?

None of these people are real! These faces were generated by GANs, exciting and at the same time scary, right? We will focus about the ethical application of the GAN in the article.

GANs for Image Editing

Using GANs, appearances can be drastically changed by reconstructing the images.

GANs for Security

GANs has been able to address the concern of ‘adversarial attacks’.

These adversarial attacks use a variety of techniques to fool deep learning architectures. Existing deep learning models are made more robust to these techniques by GANs by creating more such fake examples and training the model to identify them.

Generating Data with GANs

The availability of data in certain domains is a necessity, especially in domains where training data is needed to model learning algorithms. The healthcare industry comes to mind here. GANs shine again as they can be used to generate synthetic data for supervision.

GANs for 3D Object Generation

GANs are quite popular in the gaming industry. Game designers work countless hours recreating 3D avatars and backgrounds to give them a realistic feel. And, it certainly takes a lot of effort to create 3D models by imagination. With the incredible power of GANs, wherein they can be used to automate the entire process!

GANs are one of the few successful techniques in unsupervised machine learning and it is evolving quickly and improving our ability to perform generative tasks. Since most of the successful applications of GANs have been in the domain of computer vision, generative model sure has a lot of potential, but is not without some drawbacks.

About the Author –

Naresh B

Naresh is a part of Location Zero at GAVS as an AI/ML solutions developer. His focus is on solving problems leveraging AI/ML.
He strongly believes in making success as a habit rather than considering it as a destination.
In his free time, he likes to spend time with his pet dogs and likes sketching and gardening.

Lambda (λ), Kappa (κ) and Zeta (ζ) – The tale of three musketeers (Part-2)

Posted on by ZIF

In my previous article https://bit.ly/2T7DO9r, we saw the brief introduction and terminologies of Lambda Architecture. Let’s jump on to its various implementation patterns in the enterprises.

Lambda data processing architecture can be implemented in three ways,

  1. GenericLambdaλArchitecture.
  2. Unified LambdaλArchitecture.
  3. Multi-Agent Lambdaλ Architecture (MALA)

Generic Lambda λ Architecture

The three layers of Generic λ

  • Batch Layer-The The master data is managed here, and the batch views are precomputed.
  • Speed Layer-This layer serves recent data only and increments the real-time views
  • Serving Layer-This layer is responsible for indexing and exposing the views so that they can be queried.

How does Generic Lambdaλ Architecture work?

The new information collected, or ingested data is sent simultaneously to both Batch and Speed/Streaming layers for processing. The batch layer, called ‘Data Lake’, handles two vital tasks.

1) Managing the master data set (Data Lake), which is an immutable append-only raw data.

2) Precomputing the batch views on business-relevant aggregations and metrics.

The computation from Batch Layer is fed into Serving Layer which indexes the batch views, for a low latency query.

In the Speed layer or Streaming layer, the views are transient in nature, since only new data is considered to compensate for the high latency of the writes.

A serving layer can be a presentation side/reporting layer aimed to handle both batch reporting as well as real-time reporting. At the presentation side, queries are answered by merging both batch and real-time views. 

Generic Lambdaλ is technology agnostic

 The data pipeline can be broken down into layers with clear delineation of roles and responsibilities and at each layer, we can choose from several technologies. For instance, in the speed layer any of Apache Storm or Apache Spark Streaming, or spring ‘XD’ (eXtreme Data)could be employed.

Speed Layer Components

The following table represents some of the Stream Processing Frameworks, that are well suited for the speed components.

Apache Storm

Apache Storm is an open-source, distributed, and advanced Big Data processing engine that processes the real-time streaming data at an unprecedented speed, way faster than Apache Hadoop. What Hadoop does for batch processing, Apache Storm does for unbounded streams of data in a reliable manner.

  • Apache Storm can process over a million jobs on a node in a fraction of a second.
  • It is integrated with Hadoop to harness higher throughputs.
  • It is easy to implement and can be integrated with any programming language.

Apache Spark Streaming

Spark Streaming was added to Apache Spark in 2013, an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data is ingested from varied sources like Apache Kafka, Flume, Kinesis and can be processed using complex algorithms expressed with high-level functions like map, reduce, join, and window. The processed data can be pushed out to filesystems, databases, and live dashboards. Spark’s machine learning and graph processing algorithms can be applied to data streams.

Spring XD (eXtreme Data)

Spring XD (eXtreme Data) is a unified, distributed, and extensible service for data ingestion, real-time analytics, batch processing, and data export.

The Spring Data team has via Spring XD has provided support for NoSQL datastores and has also simplified the development experience with Hadoop. Spring XD is built on the fundamental blocks of Apache Hadoop. Also, it uses various pre-existing Spring technologies. For instance, Spring Data supports NoSQL/Hadoop work, Spring Batch is employed to support the workflow orchestration with job state management and retry/restart capabilities, and Spring Integration manages the event-driven data ingestion stream processing and the various Enterprise Application Integration patterns. Spring Reactor provides simplified API for developing asynchronous applications using the LMAX Disruptor.

Batch layer Components

Similarly, in Batch Layer frameworks like Apache Pig, Apache MapReduce and Apache spark can be employed. The Processing Frameworks that are commonly used in the batch layer are outlined below.

Apache Hadoop MapReduce

Hadoop MapReduce is a paradigm and software framework for writing applications that process large amounts of data on large clusters of commodity hardware in a parallel, reliable, fault-tolerant manner. MapReduce programs are written in various languages like Java, Ruby, Python, and C++ can be run in Apache Hadoop platform.

Apache Pig

Apache Pig is an abstraction over MapReduce and a tool/platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. To address the problem of programs generating series of Map and Reduce stages in MapReduce, Apache Pig creates an abstraction over them. The most noticeable property of Pig programs is that their structure is amenable to substantial parallelization, which in turn enables them to handle very large data sets.

Apache Spark

Apache Spark is the largest open source project in Big data processing. It is a blazing-fast cluster computing technology, designed for fast computation. It is based on Hadoop MapReduce and it extends the MapReduce model to efficiently use it for more types of computations, which includes interactive queries and stream processing. The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application.

Serving Layer Components

Technologies or Merge/Low-Latency Databases like Druid, Apache HBase, Elephant DB, Apache SOLR, Elasticsearch, Azure Cosmos DB, MongoDB, VoltDBcan be employed for speed-layer output.

Limitations of GenericLambda λ architecture

  1. Write everything twice

    2. In the generic lambda architecture, the data must be written twice i.e. data is sent to both the speed layer and the batch layer as it is created. Any logic is duplicated and implemented twice. The batch layer takes a while to produce results so the speed layer does the same work so it can answer questions about in-flight events and recent activities.

  2. Two execution paths

    3. There are always two separate execution paths for streaming and batch. It’s a maintenance nightmare, where dealing with a plethora of frameworks, components, and clusters.

  3. Two programming models

    4. Typically, an undesirable effect of Generic Lambda is that the codebases tend to diverge since the code that executes in a batch world works on a large but finite data set while a real-time stream processing system works on an infinite event stream.

  4. Diverse skill sets

    5. Just to manage the platform, more developers with diverse skill sets are needed than focusing on core business problems.

Conclusion

It is evident that the Generic Lambda λ fits best for the system that has fast data i.e. high velocity of data and Data Lake i.e. system that involves complex processing of both historical (re-computational) and real-time (incremental) aggregated view with nearly unlimited memory capacity and data storage space. Use cases like login gestion (syslog’s, application logs, weblogs), which are one-way data pipelines are some of the areas where the Generic Lambda λ shines the brightest. But having known the limitations of the Generic lambda λ, there is always a constant search to address its limitations. Let’s continue to explore to find solutions in the next part.

Happy Learning!

About the Author:

Bargunan Somasundaram


Bargunan is a Big Data Engineer and a programming enthusiast. His passion is to share his knowledge by writing his experiences about them. He believes “Gaining knowledge is the first step to wisdom and sharing it is the first step to humanity.”

Automating IT ecosystems with ZIF Remediate

Posted on by ZIF
Alwinking N Rajamani

Alwinking N Rajamani


Zero Incident FrameworkTM (ZIF) is an AIOps based TechOps platform that enables proactive detection and remediation of incidents helping organizations drive towards a Zero Incident Enterprise™. ZIF comprises of 5 modules, as outlined below.

This article’s focus is on the Remediate function of ZIF. Most ITSM teams envision a future of ticketless ITSM, driven by AI and Automation.

Remediate being a key module ofZIF, has more than 500+ connectors to various ITSMtools, Monitoring, Security and Incident management tools, storage/backup tools and others.Few of the connectors are referenced below that enables quick automation building.

Key Features of Remediate

  • Truly Agent-less software.
  • 300+ readily available templates – intuitive workflow/activity-based tool for process automation from a rich repository of pre-coded activities/templates.
  • No coding or programming required to create/deploy automated workflows. Easy drag & drop to sequence activities for workflow design.
  • Workflow execution scheduling for pre-determined time or triggering from events/notifications via email or SMS alerts.
  • Can be installed on-premise or on the cloud, on physical or virtual servers
  • Self Service portal for end-users/admins/help-desk to handle tasks &remediation automatically
  • Fully automated service management life cycle from incident creation to resolution and automatic closure
  • Has integration packs for all leading ITSM tools

Key features for futuristic Automation Solutions

Although the COVID pandemic has landed us in unprecedented times, we have been able to continue supporting our customers and enabled their IT operations with ZIF Remediate.

  • Self-learning capability to deliver Predictive/Prescriptive actionable alerts.
  • Access to multiple data sources and types – events, metrics, thresholds, logs, event triggers e.g. mail or SMS.
  • Support for a wide range of automation
    • Interactive Automation – Web, SMS, and email
    • Non-interactive automation – Silent based on events/trigger points
  • Supporting a wide range of advanced heuristics.

Benefits of AIOPS driven Automation

  • Faster MTTR
  • Instant identification of threats and appropriate responses
  • Faster delivery of IT services
  • Quality services leading to Employee and Customer satisfaction
  • Fulfillment and Alignment of IT services to business performance

Interactive and Non-interactive automation

Through our automation journey so far, we have understood that the best automation empowers humans, rather than replacing them. By implementing ZIF Remediate, organizations can empower their people to focus their attention on critical thinking and value-added activities and let our platform handle mundane tasks by bringing data-driven insights for decision making.

  • Interactive Automation – Web portal, Chatbot and SMS based
  • Non-interactive automations – Event or trigger driven automation

Involved decision driven Automations

ZIF Remediate has its unique, interactive automation capabilities, where many automation tools do not allow interactive decision making. Need approvals built into an automated change management process that involves sensitive aspects of your environment? Need numerous decision points that demand expert approval or oversight? We have the solution for you. Take an example of Phishing automation, here a domain or IP is blocked based on insights derived by mimicking an SOC engineer’s actions – parsing the observables i.e. URL, suspicious links or attachments in a phish mail and have those observables validated for threat against threat response tools, virus total, and others.

Some of the key benefits realized by our customers which include one of the largest manufacturing organizations, a financial services company, a large PR firm, health care organizations, and others.

  • Reduction of MTTR by 30% across various service requests.
  • Reduction of 40% of incidents/tickets, thus enabling productivity improvements.
  • Ticket triaging process automation resulting in a reduction of time taken by 50%.
  • Reclaiming TBs of storage space every week through snapshot monitoring and approval-driven model for a large virtualized environment.
  • Eliminating manual threat analysis by Phishing Automation, leading to man-hours being redirected towards more critical work.
  • Reduction of potential P1 outages by 40% through self-healing automations.

For more detailed information on ZIF Remediate, or to request a demo please visit https://zif.ai/products/remediate/

About the Author:

Alwin leads the Product Engineering for ZIF Remediate and zIrrus. He has over 20 years of IT experience spanning across Program & Portfolio Management for large customer accounts of various business verticals.

In his free time, Alwin loves going for long drives, travelling to scenic locales, doing social work and reading & meditating the Bible.

Lambda (λ), Kappa (κ) and Zeta (ζ) – The Tale of 3 AIOps Musketeers (PART-1)

Posted on by ZIF

Bargunan Somasundaram


Architecture inspires people, no wonder so many famous writers, artists, politicians, and designers have such profound and fascinating observations about architecture. Whether embracing minimalism or adoring resplendence, everyone has experiences and tastes that shape the way they interact with the world. The Greek architectural beauties have captured the imagination of many. The crown jewel of their architecture is the “post and lintel” which was used for their grand, large, open-air structures that could accommodate 20,000 spectators.

Greeks are also famous for their Alphabets. When the Greek Architecture and Alphabets are merged, the state-of-the-art overarching “Big Data Processing Architecture” is produced; Lambda λ, kappa κ, and Zeta ζ.

Big Data Architectural patterns

The evolution of the technologies in Big Data in the last decade has presented a history of battles with growing data volume. An increasing number of systems are being built to handle the Volume, Velocity, Variety, Veracity, Validity, and Volatility of Big Data and help gain new insights and make better business decisions. A well-designed big data architecture must handle the 6 V’s of Big Data, save your company money, and help predict future trends.

Lambda (λ) Architecture

The Lambda Architecture λis an emerging paradigm in Big Data computing. The name lambda architecture is derived from a functional point of view of data processingi.e. all data processing is understood as the application of a function to all data.

Lambda architecture is popular for its data processing technique of handling huge amounts of data by taking advantage of both a batch layer and a speed/stream-processing layer. This specific approach attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data. The outputs from both batch and speed layers can be merged before the presentation.

The efficiency of this architecture becomes evident in the form of increased throughput, reduced latency, and negligible errors, thus resulting in a linearly scalable architecture that scales out rather than scaling up.

Basic tenets of Lambda Architecture

The Lambda Architecture achieves high scalability and low latency due to the following principles,

  • Immutability of Data
  • Data Denormalization
  • Precomputed Views

Immutability of Data

The Big Data immutability is based on similar principles as the immutability in programming data structures. The goal being the same – do not change the data in-place and instead create a new one. The data can’t be altered and deleted. This rule can be defined for eternity or for a specified time period.   

Immutable data is fundamentally simpler than mutable data. The idea here is not to change the data in-place i.e. no updating or deleting of records but creating new ones. Now, this could be time-bound or for the eternity. Thus, write operations only add new data units. In CRUD parlance only CR (Create & Read) and no UD (Update & Delete).

This approach makes data handling highly scalable because it is very easy to distribute and replicate data. This immutable model makes the data aggregation kind of a logging system. With the attributes like “data creation timestamp”, the old and the most recent version can be distinguished. Apache Kafka – an append-only distributed log system is a great example of an immutable data store.

As a drawback, even more, data is generated, and answering queries becomes more difficult. For example, to find the current owner of a brand, the owner for that brand with the latest timestamp must be found.

In the mutable data model, it is no longer possible to find out that the brand Jaguar was once owned by Ford. This is different when using an immutable data model which is achieved by adding a timestamp to each data record.

Now it is possible to get both bits of information: the fact that Jaguar is now owned by Tata Motors (latest timestamp) and the fact it was formerly owned by Ford. It is also much easier to recover from errors because the old information is not deleted.

Data Denormalization

The traditional database systems are named for their storage efficiency and data integrity. It is possible due to the Normalization process like 1NF, 2NF, 3NF, BCNF, 4NF, and 5NF. Due to efficient normalization strategy, data redundancy is eliminated. The same data need not be saved in multiple places (tables) and any updates (partial or full) on the same, need not be done at multiple places (tables). But this makes the traditional databases poor at scaling their read performance since data from multiple places (tables) need to be brought together by complex and costly join operations.

For the sake of performance, Big data systems accept denormalization and duplication of data as a fact of life with the data schema such that data stored in-representation is equivalent to that after performing joins on normalized tables.

In this way, the knowledge about the schema is not necessary, and joins can be avoided, and the query results are faster. This also motivates the query-driven data modeling methodology. Albeit the data exists in multiple places after denormalization, the consistency of the data is ensured via strong consistency, timeline consistency, and eventual consistency models in the event of partial or full updates. This is often acceptable, especially when denormalized representations are used as precomputed views.

Precomputed Views

To give fast and consistent answers to queries on huge amounts of data, precomputed views are prepared both in the batch layer and in the speed layer. In the batch layer, these are constructed by applying a batch function to all the data. This leads to a transformation of the data into a more compact form suitable for answering a pre-defined set of queries. This idea is essentially the same as what is done in data warehousing.

Layers of Lambda

The Lambda Architecture solves the problem of computing arbitrary functions on arbitrary data in real-time by decomposing the problem into three layers,

  1. Batch Layer or Cold Path
  2. Speed Layer or Hot path
  3. Serving Layer

Batch layer or Cold path

The nub of the λ is the master dataset. The master dataset is the source of truth in Lambda Architecture.  The Master dataset must hold the following three properties,

  1. Data is raw.
  2. Data is immutable.
  3. Data is eternally true.

This gives the Lambda architecture ability to reconstruct the application from the master data even if the whole serving layer data set is lost. The batch layer pre-computes results using a distributed processing system that can handle very large quantities of data. The batch layer aims at perfect accuracy by being able to process all available data when generating views. 

The batch layer prefers re-computation algorithms over incremental algorithms. The problem with incremental algorithms is the failure to address the challenges faced by human mistakes. The re-computational nature of the batch layer creates simple batch views as the complexity is addressed during precomputation. Additionally, the responsibility of the batch layer is to historically process the data with high accuracy. Machine learning algorithms take time to train the model and give better results over time. Such naturally exhaustive and time-consuming tasks are processed inside the batch layer.

The problem with the batch layer is high latency. The batch jobs must be run over the entire master dataset. These jobs can process data that can be relatively old as they cannot keep up with the inflow of stream data. This is a serious limitation for real-time data processing. To overcome this limitation, the speed layer is very significant.

Frameworks and solutions such as Hadoop MapReduce, Spark core, Spark SQL, GraphX, and MLLib are the widely adopted big-data tools using batch mode. Batch schedulers include Apache Oozie, Spring Batch, and Unix crontab which, invoke the processing at a periodic interval.

Speed layer or Streaming layer or Hot path

The real-time data processing is realized in the speed layer. The speed layer achieves up-to-date query results and compensates for the high latency of the batch layer.

To create real-time views of the most recent data, this layer sacrifices throughput and decreases latency substantially. The real-time views are generated immediately after the data is received but are not as complete or precise as the batch layer.  In contrast to the re-computation approach of the batch layer, the speed layer adopts incremental computational algorithms. Since the data is not complete i.e less data so less computation. The incremental computation is more complex, but the data handled in the speed layer is vastly smaller and the views are transient.

Most operations on streams are windowed operations operating on slices of time such as moving averages for the stock process every hour, top products sold this week, fraud attempts in banking, etc. Popular choices for stream-processing tools include Apache Kafka, Apache Flume, Apache Storm, Spark Streaming, Apache Flink, Amazon Kinesis, etc.

Serving Layer

The output from both the batch and speed layers are stored in the serving layer.pre-computed batch views are indexed in this layer for faster retrieval. All the on-demand queries from the reporting or presentation layer are served by merging the batch and real-time views and outputs a result.

Query = λ (Complete data) = λ (live streaming data) * λ (Stored data)

The must-haves of the serving layer are,

  • Batch writable

The batch views for a serving layer are produced from scratch. When a new version of a view becomes available, it must be possible to completely swap out the older version with the updated view.

  • Scalable

A serving layer database must be capable of handling views of arbitrary size. As with the distributed filesystems and batch computation framework previously discussed, this requires it to be distributed across multiple machines.

  • Random reads

A serving layer database must support random reads, with indexes providing direct access to small portions of the view. This requirement is necessary to have low latency on queries.

  • Fault-tolerant

Because a serving layer database is distributed, it must be tolerant of machine failures.

This is how Lambda Architecture λ handles humongous amounts of data with low latency queries in a fault-tolerant manner. Let’s see the various implementation of lambda architecture and its applications in the next part.

To be continued…

About the Author:

Bargunan is a Big Data Engineer and a programming enthusiast. His passion is to share his knowledge by writing his experiences about them. He believes “Gaining knowledge is the first step to wisdom and sharing it is the first step to humanity.”

Assess Your Organization’s Maturity in Adopting AIOps

Posted on by ZIF

Artificial Intelligence for IT operations (AIOps) is adopted by organizations to deliver tangible Business Outcomes. These business outcomes have a direct impact on companies’ revenue and customer satisfaction.

A survey from AIOps Exchange 2019, reports that 84% of Business Owners who attended the survey, confirmed that they are actively evaluating AIOps to be adopted in their organizations.

So, is AIOps just automation? Absolutely NOT!!

Artificial Intelligence for IT operations implies the implementation of true Autonomous Artificial Intelligence in ITOps, which needs to be adopted as an organization-wide strategy. Organizations will have to assess their existing landscape, processes, and decide where to start. That is the only way to achieve the true implementation of AIOps.

Every organization trying to evaluate AIOps as a strategy should read through this article to understand their current maturity, and then move forward to reach the pinnacle of Artificial Intelligence in IT Operations.

The primary Success Factor in adopting AIOps is derived from the Business Outcomes the organization is trying to achieve by implementing AIOps –that is the only way to calculate ROI.

There are 4 levels of Maturity in AIOps adoption. Based on our experience in developing an AIOps platform and implementing the platform across multiple industries, we have arrived at these 4 levels. Assessing an organization against each of these levels helps in achieving the goal of TRUE Artificial Intelligence in IT Operations.

Level 1: Knee-jerk

Events, logs are generated in silos and collected from various applications and devices in the infrastructure. These are used to generate alerts that are commissioned to command centres to escalate as per the SOPs (standard operating procedures) defined. The engineering teams work in silos, not aware of the business impact that these alerts could potentially create. Here, operations are very reactive which could cost the organization millions of dollars.

Level 2: Unified

Have integrated all events, logs, and alerts into one central locale. ITSM process has been unified. This helps in breaking silos and engineering teams are better prepared to tackle business impacts. SOPs have been adjusted since the process is unified, but this is still reactive incident management.

Level 3: Intelligent

Machine Learning algorithms (either supervised or unsupervised) have been implemented on the unified data to derive insights. There are baseline metrics that are calibrated and will be used as a reference for future events. With more data, the metrics get richer. IT operations team can correlate incidents/events with business impacts by leveraging AI & ML. If Mean Time To Resolve (MTTR) an incident has been reduced by automated identification of the root cause, then the organization has attained level 3 maturity in AIOps.

Level 4: Predictive & Autonomous

The pinnacle of AIOps is level 4. If incidents and performance degradation of applications can be predicted by leveraging Artificial Intelligence, it implies improved application availability. Autonomousremediation bots can be triggered spontaneously based on the predictive insights, to fix incidents that are prone to happen in the enterprise. Level 4 is a paradigm shift in IT operations – moving operations entirely from being reactive, to becoming proactive.

Conclusion:

As IT operations teams move up each level, the essential goal to keep in mind is the long-term strategy that needs to be attained by adopting AIOps. Artificial Intelligence has matured over the past few decades, and it is up to AIOps platforms to embrace it effectively. While choosing an AIOps platform, measure the maturity of the platform’s artificial intelligent coefficient.

About the Author:

Anoop Aravindakshan (Principal Consultant Manager) at GAVS Technologies.


An evangelist of Zero Incident FrameworkTM, Anoop has been a part of the product engineering team for long and has recently forayed into product marketing. He has over 14 years of experience in Information Technology across various verticals, which include Banking, Healthcare, Aerospace, Manufacturing, CRM, Gaming, and Mobile.

Prediction for Business Service Assurance

Posted on by ZIF

Artificial Intelligence for IT operations or AIOps has exploded over the past few years. As more and more enterprises set about their digital transformation journeys, AIOps becomes imperative to keep their businesses running smoothly. 

AIOps uses several technologies like Machine Learning and Big Data to automate the identification and resolution of common Information Technology (IT) problems. The systems, services, and applications in a large enterprise produce volumes of log and performance data. AIOps uses this data to monitor the assets and gain visibility into the behaviour and dependencies among these assets.

According to a Gartner publication, the adoption of AIOps by large enterprises would rise to 30% by 2023.

ZIF – The ideal AIOps platform of choice

Zero Incident FrameworkTM (ZIF) is an AIOps based TechOps platform that enables proactive detection and remediation of incidents helping organizations drive towards a Zero Incident Enterprise™.

ZIF comprises of 5 modules, as outlined below.

At the heart of ZIF, lies its Analyze and Predict (A&P) modules which are powered by Artificial Intelligence and Machine Learning techniques. From the business perspective, the primary goal of A&P would be 100% availability of applications and business processes.

Let us understand more about thePredict module of ZIF.

Predictive Analytics is one of the main USP of the ZIF platform. ZIF encompassesSupervised, Unsupervised and Reinforcement Learning algorithms for realization of various business use cases (as shown below).

How does the Predict Module of ZIF work?

Through its data ingestion capabilities, the ZIF platform can receive and process all types of data (both structured and unstructured) from various tools in the enterprise. The types of data can be related to alerts, events, logs, performance of devices, relations of devices, workload topologies, network topologies etc. By analyzing all these data, the platform predicts the anomalies that can occur in the environment. These anomalies get presented as ‘Opportunity Cards’ so that suitable action can be taken ahead of time to eliminate any undesired incidents from occurring. Since this is ‘Proactive’ and not ‘Reactive’, it brings about a paradigm shift to any organization’s endeavour to achieve 100% availability of their enterprise systems and platforms. Predictions are done at multiple levels – application level, business process level, device level etc.

Sub-functions of Prediction Module

How does the Predict module manifest to enterprise users of the platform?

Predict module categorizes the opportunity cards into three swim lanes.

  1. Warning swim lane – Opportunity Cards that have an “Expected Time of Impact” (ETI) beyond 60 minutes.
  2. Critical swim lane – Opportunity Cards that have an ETI within 60 minutes.
  3. Processed / Lost– Opportunity Cards that have been processed or lost without taking any action.

Few of the enterprises that realized the power of ZIF’s Prediction Module

  • A manufacturing giant in the US
  • A large non-profit mental health and social service provider in New York
  • A large mortgage loan service provider in the US
  • Two of the largest private sector banks in India

For more detailed information on GAVS’ Analyze, or to request a demo please visithttps://zif.ai/products/predict/

References:https://www.gartner.com/smarterwithgartner/how-to-get-started-with-aiops/

About the Author:

Vasudevan Gopalan

Vasu heads Engineering function for A&P. He is a Digital Transformation leader with ~20 years of IT industry experience spanning across Product Engineering, Portfolio Delivery, Large Program Management etc. Vasu has designed and delivered Open Systems, Core Banking, Web / Mobile Applications etc.

Outside of his professional role, Vasu enjoys playing badminton and focusses on fitness routines.

Discover, Monitor, Analyze & Predict COVID-19

Posted on by ZIF

“Uber, the world’s largest taxi company, owns no vehicles. Facebook, the world’s most popular media owner, creates no content. Alibaba, the most valuable retailer, has no inventory. Netflix, the world’s largest movie house, own no cinemas. And Airbnb, the world’s largest accommodation provider, owns no real estate. Something interesting is happening.”

– Tom Goodwin, an executive at the French media group Havas.

This new breed of companies is the fastest growing in history because they own the customer interface layer. It is the platform where all the value and profit is. “Platform business” is a more wholesome termfor this model for which data is the fuel; Big Data & AI/ML technologies are the harbinger of new waves of productivity growth and innovation.

With Big data and AI/ML is making a big difference in the area of public health, let’s see how it is helping us tackle the global emergency of coronavirus formally known as COVID-19.

“With rapidly spreading disease, a two-week lag is an eternity.”

DISCOVERING/ DETECTING

Chinese technology giant Alibaba has developed an AI system for detecting the COVID-19 in CT scans of patients’ chests with 96% accuracy against viral pneumonia cases. It only takes 20 seconds for the AI to decide, whereas humans generally take about 15 minutes to diagnose the illness as there can be upwards of 300 images to evaluate.The system was trained on images and data from 5,000 confirmed coronavirus cases and has been tested in hospitals throughout China. Per a report, at least 100 healthcare facilities are currently employing Alibaba’s AI to detect COVID-19.

Ping An Insurance (Group) Company of China, Ltd (Ping An) aims to address the issue of lack of radiologists by introducing the COVID-19 smart image-reading system. This image-reading system can read the huge volumes of CT scans in epidemic areas.

Ping An Smart Healthcare uses clinical data to train the AI model of the COVID-19 smart image-reading system. The AI analysis engine conducts a comparative analysis of multiple CT scan images of the same patient and measures the changes in lesions. It helps in tracking the development of the disease, evaluation of the treatment and in prognosis of patients.Ultimately it assists doctors to diagnose, triage and evaluate COVID-19 patients swiftly and effectively.

Ping An Smart Healthcare’s COVID-19 smart image-reading system also supports AI image-reading remotely by medical professionals outside the epidemic areas.Since its launch, the smart image-reading system has provided services to more than 1,500 medical institutions. More than 5,000 patients have received smart image-reading services for free.

The more solutions the better. At least when it comes to helping overwhelmed doctors provide better diagnoses and, thus, better outcomes.

MONITORING

  • AI based Temperature monitoring & scanning

In Beijing, China, subway passengers are being screened for symptoms of coronavirus, but not by health authorities. Instead, artificial intelligence is in-charge.

Two Chinese AI giants, Megvii and Baidu, have introduced temperature-scanning. They have implemented scanners to detect body temperature and send alerts to company workers if a person’s body temperature is high enough to constitute a fever.

Megvii’s AI system detects body temperatures for up to 15 people per second andup to 16 feet. It monitors as many as 16 checkpoints in a single station. The system integrates body detection, face detection, and dual sensing via infrared cameras and visible light. The system can accurately detect and flag high body temperature even when people are wearing masks, hats, or covering their faces with other items. Megvii’s system also sends alerts to an on-site staff member.

Baidu, one of the largest search-engine companies in China, screens subway passengers at the Qinghe station with infrared scanners. It also uses a facial-recognition system, taking photographs of passengers’ faces. If the Baidu system detects a body temperature of at least 99-degrees Fahrenheit, it sends an alert to the staff member for another screening. The technology can scan the temperatures of more than 200 people per minute.

  • AI based Social Media Monitoring

An international team is using machine learning to scour through social media posts, news reports, data from official public health channels, and information supplied by doctors for warning signs of the virus across geographies.The program is looking for social media posts that mention specific symptoms, like respiratory problems and fever, from a geographic area where doctors have reported potential cases. Natural language processing is used to parse the text posted on social media, for example, to distinguish between someone discussing the news and someone complaining about how they feel.

The approach has proven capable of spotting a coronavirus needle in a haystack of big data. This technique could help experts learn how the virus behaves. It may be possible to determine the age, gender, and location of those most at risk quicker than using official medical sources.

PREDICTING

Data from hospitals, airports, and other public locations are being used to predict disease spread and risk. Hospitals can also use the data to plan for the impact of an outbreak on their operations.

Kalman Filter

Kalman filter was pioneered by Rudolf Emil Kalman in 1960, originally designed and developed to solve the navigation problem in the Apollo Project. Since then, it has been applied to numerous cases such as guidance, navigation, and control of vehicles, computer vision’s object tracking, trajectory optimization, time series analysis in signal processing, econometrics and more.

Kalman filter is a recursive algorithm which uses time-series measurement over time, containing statistical noise and produce estimations of unknown variables.

For the one-day prediction Kalman filter can be used, while for the long-term forecast a linear model is used where its main features are Kalman predictors, infected rate relative to population, time-depended features, and weather history and forecasting.

The one-day Kalman prediction is very accurate and powerful while a longer period prediction is more challenging but provides a future trend.Long term prediction does not guarantee full accuracy but provides a fair estimation following the recent trend. The model should re-run daily to gain better results.

GitHub Link: https://github.com/Rank23/COVID19

ANALYZING

The Center for Systems Science and Engineering at Johns Hopkins University has developed an interactive, web-based dashboard that tracks the status of COVID-19 around the world. The resource provides a visualization of the location and number of confirmed COVID-19 cases, deaths and recoveries for all affected countries.

The primary data source for the tool is DXY, a Chinese platform that aggregates local media and government reports to provide COVID-19 cumulative case totals in near real-time at the province level in China and country level otherwise. Additional data comes from Twitter feeds, online news services and direct communication sent through the dashboard. Johns Hopkins then confirms the case numbers with regional and local health departments. This kind of Data analytics platform plays a pivotal role in addressing the coronavirus outbreak.

All data from the dashboard is also freely available in the following GitHub repository.

GitHub Link:https://bit.ly/2Wmmbp8

Mobile version: https://bit.ly/2WjyK4d

Web version: https://bit.ly/2xLyT6v

Conclusion

One of AI’s core strengths when working on identifying and limiting the effects of virus outbreaks is its incredibly insistent nature. AIsystems never tire, can sift through enormous amounts of data, and identify possible correlations and causations that humans can’t.

However, there are limits to AI’s ability to both identify virus outbreaks and predict how they will spread. Perhaps the best-known example comes from the neighboring field of big data analytics. At its launch, Google Flu Trends was heralded as a great leap forward in relation to identifying and estimating the spread of the flu—until it underestimated the 2013 flu season by a whopping 140 percent and was quietly put to rest.Poor data quality was identified as one of the main reasons Google Flu Trends failed. Unreliable or faulty data can wreak havoc on the prediction power of AI.

References:

About the Author:

Bargunan Somasundaram

Bargunan Somasundaram

Bargunan is a Big Data Engineer and a programming enthusiast. His passion is to share his knowledge by writing his experiences about them. He believes “Gaining knowledge is the first step to wisdom and sharing it is the first step to humanity.”