Lambda (λ), Kappa (κ) and Zeta (ζ) – The Tale of 3 AIOps Musketeers (PART-3)

“Data that sit unused are no different from data that were never collected in the first place.” – Doug Fisher

In the part 1 (https://bit.ly/3hDChCH), we delved into Lambda Architecture and in part 2 (https://bit.ly/3hDCg1B) about Generic Lambda. Given the limitations of the Generic lambda architecture and its inherent complexity, the data is replicated in two layers and keeping them in-sync is quite challenging in an already complex distributed system.There is a growing interest to find the simpler alternative to the Generic Lambda, that would bring just about the same benefits and handle the full problem set. The solution is Unified Lambda (λ) Architecture.

Unified Lambda (λ) Architecture

The unified approach addresses the velocity and volume problems of Big Data as it uses a hybrid computation model. This model combines both batch data and instantaneous data transparently.

There are basically three approaches:

  1. Pure Streaming Framework
  2. Pure Batch Framework
  3. Lambdoop Framework

1. Pure streaming framework

In this approach, a pure streaming model is adopted and a flexible framework like Apache Samza can be employed to provide unified data processing model for both stream and batch processing using the same data flow structure.

Pure streaming framework

To avoid the large turn-around times involved in Hadoop’s batch processing, LinkedIn came up with a distributed stream processing framework Apache Samza. It is built on top of distributed messaging bus; Apache Kafka, so that it can be a lightweight framework for streaming platform. i.e. for continuous data processing. Samza has built-in integration with Apache Kafka, which is comparable to HDFS and MapReduce. In the Hadoop world, HDFS is the storage layer and MapReduce, the processing layer. In the similar way, Apache Kafka ingests and stores the data in topics, which is then streamed and processed by Samza. Samza normally computes results continuously as and when the data arrives, thus delivering sub-second response times.

Albeit it’s a distributed stream processing framework, its architecture is pluggable i.e. can be integrated with umpteen sources like HDFS, Azure EventHubs, Kinensis etc. apart from Kafka. It follows the principle of WRITE ONCE, RUN ANYWHERE; meaning, the same code can run in both stream and batch mode. Apache Samza’s streams are re-playable, ordered partitions.

Unified API for Batch & Streaming in pure Streaming

Apache Samza offers a unified data processing model for both real-time as well as batch processing.  Based on the input data size, bounded or unbounded the data processing model can be identified, whether batch or stream.Typically bounded (e.g. static files on HDFS) are Batch data sources and streams are unbounded (e.g. a topic in Kafka). Under the bonnet, Apache Samza’s stream-processing engine handles both types with high efficiency.

Unified API for Batch & Streaming in pure Streaming

Another advantage of this unified API for Batch and Streaming in Apache Samza, is that makes it convenient for the developers to focus on the processing logic, without treating bounded and unbounded sources differently. Samza differentiates the bounded and unboundeddata by a special token end-of-stream. Also, only config change is needed, and no code changes are required, in case of switching gears between batch and streaming, e.g. Kafka to HDFS.Let us take an example of Count PageViewEvent for each mobile Device OS in a 5-minute window and send the counts to PageViewEventPerDeviceOS

Pure Batch framework

This is the reverse approach of pure streaming where a flexible Batch framework is employed, which would offer both the batch processing and real-time data processing ability. The streaming is achieved by using mini batches which is small enough to be close to real-time, with Apache Spark/Spark Streaming or Storm’s Trident. Under the hood, Spark streaming is a sequence of micro-batch processes with the sub-second latency. Trident is a high-level abstraction for doing streaming computation on top of Storm. The core data model of Trident is the “Stream”, processed as a series of batches.

Apache Spark achieves the dual goal of Batch as well as real-time processing by the following modes.

  • Micro-batch processing model
  • Continuous Processing model

Micro-batch processing model

Micro-batch processing is analogous to the traditional batch processing in that data are usually processed as a group. The primary difference is that the batches are smaller and processed more often. In spark streaming, the micro-batches are created based on the time rather than on the accumulated data size. The smaller the time to trigger a micro-batch to process, lesser the latency.

Continuous Processing model

Apache Spark 2.3, introduced Low-latency Continuous Processing Mode in Structured Streaming whichenables low (~1 ms) end-to-end latency with at-least-once fault-tolerance guarantees. Comparing this with the default micro-batch processing engine which can achieve exactly-once guarantees but achieve latencies of ~100 ms at best. Without modifying the application logic i.e. DataFrame/Dataset operations mini-batching or continuous streaming can be chosen at runtime. Spark Streaming also has the abilityto work well with several data sources like HDFS, Flume or Kafka.

Example of Micro-batching and Continuous Batching

3. Lambdoop Approach

In many places, capability of both batch and real time processing is needed.It is cumbersome to develop a software architecture of such capabilities by tailoring suitable technologies, software layers, data sources, data storage solutions, smart algorithms and so on to achieve the good scalable solution. This is where the frameworks like Spring “XD”, Summingbird or Lambdoop comes in, since they already have a combined API for batch and real-time processing.

Lambdoop

Lambdoop is a software framework based on the Lambda architecture which provides an abstraction layer to the developers. This feature makes the developers life easy to develop any Big Data applications by combining real time and batch processing approaches. Developers don’t have to deal with different technologies, configurations, data formats etc. They can use the Lambdoop framework as the only needed API. Also, Lambdoop includes other interesting tools such as input/output drivers, visualization tools, cluster management tools and widely accepted AI algorithms.

The Speed layer in Lambdoop runs on Storm and the Batch layer on Hadoop, Lambdoop (Lambda-Hadoop, with HBase, Storm and Redis) also combines batch/real-time by offering a single API for both processing models.

Summingbird

Summingbird aka‘Streaming MapReduce’ is a hybrid computational system where both the batch/streaming computations can be run at the same time and the results can be merged automatically. In Summingbird, the developer can write the code/job logic once and change the backend as and when needed. Following are the modes in which Summingbird Job/code can be executed.

  • batch mode (using Scalding on Hadoop)
  • real-time mode (using Storm)
  • hybrid batch/real-time mode (offers attractive fault-tolerance properties)

If the model assumes streaming, one-at-a-time semantics, then the code can be run in real-time e.g. Strom or in offline/batch mode e.g. Hadoop, spark etc. It can operate in a hybrid processing mode, when there is a need to transparently integrate batch and online results to efficiently generate up-to-date views over long time spans.

Conclusion

The volume of any Big Data platform is handled by building a batch processing application which requires, MapReduce, spark development, Use of other Hadoop related tools like Sqoop, Zookeeper, HCatalog etc. and storage systems like HBase, MongoDB, HDFS, Cassandra. At the same time the velocity of any Big Data platform is handled by building a real-time streaming application which requires, stream computing development using Storm, Samza, Kafka-connect, Apache Flink andS4, and use of temporal datastores like in-memory data stores, Apache Kafka messaging system etc.

The Unified Lambda handles the both Volume and Velocity if any Big Data platform by the intermixed approach of featuring a hybrid computation model, where both batch and real-time data processing are combined transparently. Also, the limitations of Generic Lambda like Dual execution mode, Replicating and maintaining the data sync between different layers are avoided and in the Unified Lambda, there would be only one system to learn and maintain.

About the Author:

Bargunan Somasundaram

Bargunan Somasundaram

Bargunan is a Big Data Engineer and a programming enthusiast. His passion is to share his knowledge by writing his experiences about them. He believes “Gaining knowledge is the first step to wisdom and sharing it is the first step to humanity.”

Modern IT Infrastructure

Infrastructure today has grown beyond the physical confines of the traditional data center, has spread its wings to the cloud, and is increasingly distributed, virtual, and abstract. With the cloud gaining wide acceptance, most enterprises have their workloads spread across data centers, colocations, multi-cloud, and edge locations. On-premise infrastructure is also being replaced by Hyperconverged Infrastructure (HCI) where software-defined, virtualized compute, storage, and network are in one single system, greatly simplifying IT operations. Infrastructure is also becoming increasingly elastic, scales & shrinks on demand and doesn’t have to be provisioned upfront.

Let’s look at a few interesting technologies that are steering the modern IT landscape.

Containers and Serverless

Traditional application deployment on physical servers comes with the overhead of managing the infrastructure, middleware, development tools, and everything in between. Application developers would rather have this grunt work be handled by someone else, so they could focus on just their applications. This is where containers and serverless technologies come into picture. Both are cloud-based offerings and provide different levels of abstraction, in a way that hides layers beyond the front end, from the developer. They typically deploy smaller components of monolithic applications, microservices, and functions.

A Container is like an all-in-one-box, containing the app, and all its dependencies like libraries, executables & config files. The containerized application is highly portable, will run anywhere the container runtime is installed, and behave the same regardless of the OS or hardware it is deployed on. Containers give developers great flexibility and control since they cater to specific application requirements like the OS, S/W versions. The flip side is that there is still a need for manual maintenance of the runtime environment, like security patches, software updates, etc. Secondly, the flexibility it affords translates into high operational costs, since it lacks agility in scaling.

Serverless technologies provide much greater abstraction of the OS and infrastructure. ‘Serverless’ though, does not imply that there are no servers, it just means application developers do not have to worry about the underlying OS, the server environment, or the infra that their applications will be deployed on. Serverless is event-driven and is based on the premise that the application is split into functions that get executed based on events. The developer only needs to deploy function code and define the event(s) that will trigger them! The rest of the magic is done by the cloud service provider (with the help of third parties). 

The biggest advantage of serverless is that consumers are billed only for the running time of the function instances or the number of times the function gets executed, depending on the provider. Since it has zero administrative overhead, it guarantees rapid iterative deployment and faster time to market. Since the architecture is intrinsically auto-scaling, it is a perfect fit for applications with undefinable usage patterns. The other side of the coin is that developers need to deal with a black box back-end environment, so, holistic testing, debugging of the application becomes a challenge. Vendor lock-in is a real problem since the consumer is restricted by the technology stack supported by the vendor. Since serverless best practices dictate light, isolated functions with limited scope, building complex applications can get difficult. Function as a Service (FaaS) is a subset of serverless computing.

Internet of Things (IoT)

IoT is about connecting everyday things – beyond just computing devices or smartphones – to the internet. It is possible to convert practically anything into an IoT device, with a computer chip installation & internet access, and have it communicate independently with the internet – without any human intervention. But why would we want everyday things like for instance a watch or a light bulb, to become IoT devices? It’s in a bid to bridge the chasm between the physical and digital worlds and make the environment around us more intelligent, communicative, and responsive to our needs.

IoT’s use cases are just about everywhere; in personal devices, self-driving cars, smart homes, smart workspaces, smart cities, and industries across all verticals. For instance, live data from sensors in products while in use, gives good visibility into their operations on the ground, helps remediate issues proactively & aids improvements in design/manufacturing processes.

The Industrial Internet of Things (IIoT) is the use of IoT data in business, in tandem with Big Data, AI, Analytics, Cloud, and High-speed networks, with the primary goal of finding efficient business models to improve productivity & optimize expenditure. The need for real-time response to sensor data and advanced analytics to power insights has increased the demand for 5G networks for speed, cloud technologies for storage and computing, edge computing to reduce latency, and hyper-scale data centers for rapid scaling.

With IoT devices extending an organization’s infrastructure landscape, and the likelihood that IT staff may not even be aware of all the IoT devices in it is a security nightmare that could open corporate networks & sensitive data for attacks. Global standards and regulations for IoT device security are in the works. Until then, it is up to the enterprise security team to safeguard against IoT-related vulnerabilities.

Hyperscaling

The ability of infrastructure to rapidly scale out on a massive level is called hyperscaling.

Unprecedented needs for high-power computing and on-demand massive scalability has given rise to a new breed of hyperscale computing architectures, where traditional elements are replaced by hyper-converged, software-defined infrastructure with a high degree of virtualization. These hyperscale environments are characterized by high-density server racks, with software designed and specifically built for scale-out environments. Since high-density implies heavy power consumption, heating problems need to be handled by specialized cooling solutions like liquid cooling. Hyperscale data centre operators usually look for renewable energy options to save on power & cooling.

Today, there are several hundred hyperscale data centers in the world, with the dominant players being Microsoft, Google, Apple, Amazon & Facebook.

Edge Computing

Edge computing as the name indicates means moving data processing away from distant servers or the cloud, closer to the source of data.  This is to reduce latency and network bandwidth used for back & forth communication between the data source and the server. Edge, also called the network edge refers to where the data source connects to the internet. The explosive growth of IoT and applications like self-driving cars, virtual reality, smart cities for instance, that require real-time computing and analytics are paving the way for edge computing. Most cloud providers now provide geographically distributed edge servers. As with IoT devices, data at the edge can be a ticking security time bomb necessitating appropriate security mechanisms.

The evolution of IT technologies continuously raises the bar for the IT team. IT personnel have been forced to move beyond legacy practices and mindsets & constantly up-skill themselves to be able to ride the wave. For customers pampered by sophisticated technologies, round the clock availability of systems and immersive experiences have become baseline expectations. With more & more digitalization, there is increasing reliance on IT infrastructure and hence lesser tolerance for outages. The responsibilities of maintaining a high-performing IT infrastructure with near-zero downtime fall on the shoulders of the IT operations team.

This has underscored the importance of AI in IT operations since IT needs have now surpassed human capabilities. Gavs’ AI-powered Platform for IT operations, ZIF, caters to the entire ITOps spectrum, right from automated discovery of the landscape, monitoring, to predictive and prescriptive analytics that proactively drive the organization towards zero incidents. For more details, please visit https://zif.ai

About the Author:

Padmapriya Sridhar

Priya is part of the Marketing team at GAVS. She is passionate about Technology, Indian Classical Arts, Travel, and Yoga. She aspires to become a Yoga Instructor someday!

Can automation manage system alerts?

System alerts and critical alerts

One of the most important and critical roles of an IT professional is to handle incoming alerts efficiently and effectively. This will ensure a threat-free environment and reduce the chances of system outages. Now, not all incoming alerts are critical; an alert can pop up on a window screen for a user to act on, blocking the underlying webpage. One can configure the setting to automatic alert resolution where an alert will be closed automatically after a number of days.

Can automation manage system alerts?

Gradually, many companies are incorporating automation in the field of managing system alerts. The age-old technology of monitoring system for both, internal and external alerts is not effective in streamlining the actual process of managing these incoming alerts. Here, IT process automation (ITPA) can take incident management to a whole new level. Automation in collaboration with monitoring tools can identify, analyze and finally prioritize incoming alerts while sending notification to fix the issue. Such notifications can be customized depending on the selected mode of preference. Also, it is worth mentioning here that automated workflows can be created to open, update and close tickets in the service desk, minimizing human intervention while electronically resolving issues.

Integration of a monitoring system with automation

Automation of system alerts happen with the following workflow. It highly improved the incident management system, reducing human intervention and refining the quality of monitoring.

  1. The monitoring system detects an incident within the IT infrastructure and triggers an alert.
  2. The alert is addressed by automation software and a trouble ticket is generated thereafter in service desk.
  3. Then the affected lot is notified via preferred method of communication.
  4. Network admin is then notified by ITPA to address the issue and recover.
  5. The service ticket is accordingly updated through implementation of automation.

Benefits of automation to manage system alerts

Relying on a process that is manually performed especially, while dealing with critical information in a workflow can be difficult. In such a scenario, automation of monitoring critical data in business systems like accounting, CRM, ERP or warehousing can improve on consistency. It can also recognize significant or critical data changes immediately triggering notification for the same. With this 360-degree visibility of critical information, decision making can happen a lot faster which in the long run can forestall serious crisis. It also improves the overall performance of the company and customer service and reduces financial risk due to anomalies and security threats. Hence, it can be aptly mentioned that automation of system alerts can effectively reduce response and resolution time. It can also lessen system downtime and improve MTTR.

BPA platform’s role to manage system alerts

The business process automation (BPA) platform enables multi-recipient capabilities so that notification can be sent to employees across different verticals. This will increase their visibility on real-time information that is relevant to their organizational role. This platform also provides escalation capabilities where notification will be sent to higher management if an alert is not addressed on time.

Conclusion

For large-scale organizations, the number of alerts spotted by detection tools are growing in number with time. This inspired IT enterprises to automate security control configurations and implement responsive security analysis tasks. Through automation of security control and processes, a new firewall rule can be automatically created or deleted based on alerts. Once a threat is detected, automated response is created. We can conclude that automation can manage system alerts efficiently and effectively. And a pre-built workflow often helps to jump-start an automation process of addressing a system alert.

READ ALSO OUR NEW UPDATES

AIOps Trends in 2019

Adoption of AIOps by organizations

Artificial Intelligence in IT operations (AIOps) is rapidly pacing up with digital transformation. Over the years, there has been a paradigm shift of enterprise application and IT infrastructure. With a mindset to enhance flexibility and agility of business processes, organizations are readily adopting cloud platforms to provision their on-premise software. Implementation of technologies like AIOps and hybrid environment has facilitated organizations to gauge the operational challenges and reduced their operational costs considerably. It helps enterprises in:

  • Resource utilization
  • Capacity planning
  • Anomaly detection
  • Threat detection
  • Storage management
  • Cognitive analysis

Infact, if we look at Gartner’s prediction, by 2022, 40% of medium and large-scale enterprises will adopt artificial intelligence (AI) to increase IT productivity.

AIOps Market forecast

According to Infoholic Research, the AIOps market is expected to reach approximately $14 billion by 2024, growing at a CAGR of 33.08% between 2018–2024. The companies that will provide AIOps solutions to enhance IT operations management in 2019 include BMC Software, IBM, GAVS Technologies, Splunk, Fix Stream, Loom System and Micro Focus. By end of 2019, US alone is expected to contribute over 30% of growth in AIOps and it will also help the global IT industry reach over $5,000 billion by the end of this year. Research conducted by Infoholic also confirmed that AIOps has been implemented by 60% of the organizations to reduce noise alerts and identify real-time root cause analysis.

Changes initiated by enterprises to adopt AIOps

2019 will be the year to reveal the true value of AIOps through its applications. By now, organizations have realized that context and efficient integrations with existing systems are essential to successfully implement AIOps.

1. Data storage

Since AIOps need to operate on a large amount of data, it is essential that enterprises absorb data from reliable and disparate sources which, then, can be contextualized for use in AI and ML applications. For this process to work seamlessly, data must be stored in modern data lakes so that it can be free from traditional silos.

2. Technology partnership

Maintaining data accuracy is a constant struggle and in order to overcome such complexity, in 2019, there will be technology partnership between companies to deal with customer demands for better application program interface (APIs).

3. Automation of menial tasks

Organizations are trying to automate menial tasks to increase agility by freeing up resources. Through automation, organizations can explore a wide range of opportunities in AIOps that will increase their efficiency.

4. Streamling of people, process and tools

Although multi-cloud solutions provide flexibility and cost-efficiency, however, without proper tools to monitor, it can be challenging to manage them. Hence, enterprises are trying to streamline their people, process and tools to create a single, siloed-free overview to benefit from AIOps.

5. Use of real-time data

Enterprises are trying to ingest and use real-time data for event correlation and immediate anomaly detection since, with the current industrial pace, old data is useless to the market.

6. Usage of self-discovery tools

Organizations are trying to induce self-discovery tools in order to overcome the challenge of lack of data scientists in the market or IT personnel with coding skills to monitor the process. The self-discovery tools can operate without human intervention.

Conclusion

Between 2018 to 2024, the global AIOps market value of real time analytics and application performance management is expected to grow at a rapid pace. Also, it is observed that currently only 5% of large IT firms have adopted AIOps platforms due to lack of knowledge and assumption about the cost-effectiveness. However, this percentage is expected to reach 40% by 2022. Companies like CA Technologies, GAVS Technologies, Loom Systems and ScienceLogic has designed tools to simplify AIOps deployment and it is anticipated that over the next three years, there will be sizable progress in the AIOps market.

READ ALSO OUR NEW UPDATES

Pivotal Role of AI and Machine Learning in Industry 4.0 and Manufacturing

Industry 4.0 is a name given to the current trend of automation and data exchange in manufacturing technologies. It includes cyber-physical systems, the Internet of things, cloud computing and cognitive computing.Industry 4.0 is commonly referred to as the fourthindustrial revolution.

Industry 4.0 is the paving the path for digitization of the manufacturing sector, where artificial intelligence (AI) and machine-learning based systems are not only changing the ways we interact with information and computers but also revolutionizing it.

Compelling reasons for most companies to shift towards Industry 4.0 and automate manufacturing include;

  • Increase productivity
  • Minimize human / manual errors
  • Optimize production costs
  • Focus human efforts on non-repetitive tasks to improve efficiency

Manufacturing is now being driven by effective data management and AI that will decide its future. The more data sets computers are fed, the more they can observe trends, learn and make decisions that benefit the manufacturing organization. This automation will help to predict failures more accurately, predict workloads, detect and anticipate problems to achieve Zero Incidence.

GAVS’ proprietary AIOps based TechOps platform – Zero Incident Framework TM (ZIF) can successfully integrate AI and machine learning into the workflow allowing manufacturers to build robust technology foundations.

To maximize the many opportunities presented by Industry 4.0, manufacturers need to build a system with the entire production process in mind as it requires collaboration across the entire supply chain cycle.

Top ways in which ZIF’s expertise in AI and ML are revolutionizing manufacturing sector:

  • Asset management, supply chain management and inventory management are the dominant areas of artificial intelligence, machine learning and IoT adoption in manufacturing today. Combining these emerging technologies, they can improve asset tracking accuracy, supply chain visibility, and inventory optimization.
  • Improve predictive maintenance through better adoption of ML techniques like analytics, Machine Intelligence driven processes and quality optimization.
  • Reduce supply chain forecasting errors and reduce lost sales to increase better product availability.
  • Real time monitoring of the operational loads on the production floor helps in providing insights into the production schedule performances.
  • Achieve significant reduction in test and calibration time via accurate prediction of calibration and test results using machine learning.
  • Combining ML and Overall Equipment Effectiveness (OEE), manufacturers can improve yield rates, preventative maintenance accuracy and workloads by the assets. OEE is a universally used metric in manufacturing as it combines availability, performance, and quality, defining production effectiveness.
  • Improving the accuracy of detecting costs of performance degradation across multiple manufacturing scenarios that reduces costs by 50% or more.

Direct benefits of Machine Learning and AI for Manufacturing

The introduction of AI and Machine Learning to industry 4.0 represents a big change for manufacturing companies that can open new business opportunities and result in advantages like efficiency improvements among others.

  • Cost reduction through Predictive Maintenance that leads to less maintenance activity, which means lower labor costs, reduced inventory and materials wastage.
  • Predicting Remaining Useful Life (RUL): Keeping tabs on the behavior of machines and equipment leads to creating conditions that improve performance while maintaining machine health. By predicting RUL, it reduces the scenarios which causes unplanned downtime.
  • Improved supply chain management through efficient inventory management and a well monitored and synchronized production flow.
  • Autonomous equipment and vehicles: Use of autonomous cranes and trucks to streamline operations as they accept containers from transport vehicles, ships, trucks etc.
  • Better Quality Control with actionable insights to constantly raise product quality.
  • Improved human-machine collaboration while improving employee safety conditions and boosting overall efficiency.
  • Consumer-focused manufacturing: Being able to respond quickly to changes in the market demand.

Touch base with GAVS AI experts here: https://www.gavstech.com/reaching-us/ and see how we can help you drive your manufacturing operation towards Industry 4.0.