IoT Resources

DZone's Featured IoT Resources

Real-Time Analytics

By Tim Spann CORE

This is an article from DZone's 2023 Data Pipelines Trend Report.For more: Read the Report We live in an era of rapid data generation from countless sources, including sensors, databases, cloud, devices, and more. To keep up, we require real-time analytics (RTA), which provides the immediacy that every user of data today expects and is based on stream processing. Stream processing is used to query a continuous stream of data and immediately process events in that stream. Ideally, your stream is processing events in subseconds as they occur. RTA enables organizations to process and analyze data in real-time, providing valuable insights and driving timely decisionmaking. The key factor is that we must process each event as it arrives, and time matters. Unlike traditional batch processing, where data is analyzed in predetermined intervals, RTA provides an immediate and continuous understanding of events and data as they arrive. Why is this important for data pipelines? Let's discuss. How RTA Is Relevant to Data Pipelines RTA needs to be fed by a continuous flow of data from various sources. Data pipelines are responsible for ingesting data from sources — like sensors, logs, databases, and APIs — and delivering the data to an RTA platform as a stream. In the case of a real-time streaming data pipeline fed from a tool such as Apache NiFi, we receive each chunk of data as an event from Apache Kafka. There are many data pipelines that are batched by being fed in large chunks at regular intervals or on demand. These batches yield bursts of data when translated to a stream, and the bursts can cause a backup, like going from local roads to a superhighway. A tool like NiFi that lets you transition batches to streams without code is ideal. There are several architecture decisions based on how to keep batches and streams flowing and available to the same consumers and often having the same sources. An important part is that we need a real-time lakehouse based on Apache Iceberg or Apache Paimon to permanently store large data that will be needed later for LLM, data training, and lookups. Stream Processing and RTA RTA is built on stream processing, which includes frameworks like Apache Flink, Apache Kafka Streams, RisingWave, and Faust Streaming. Stream processing is often done by writing code in Java, Scala, or Python. The most exciting form of stream processing is done with SQL, and it can be often done without implicit compiling or deploying in an IDE. The main feature of stream processing is ingesting a continuous event stream to filter, enrich, join, transform, convert, route, and analyze data as it arrives in near real time. Real time has a lot of definitions, but for stream processing in most contexts, let us assume time means subsecond. The Importance of Real-Time Analytics RTA is important for timely decision making since the faster data arrives and is processed, the faster we can act. The immediate usage of data streams can make new types of applications possible, such as detecting fraud while it occurs. With data arriving constantly from thousands of devices, RTA is critical for receiving, processing, handling, and potentially alerting on this data. RTA allows for predictive analytics on real-world machinery that could be alerting to upcoming failures based on current and historical data and machine learning (ML) analysis. RTA enables processing not just for each event as it arrives, but also for time windows of data, frequently in one-, five-, or 15-minute chunks. Batch vs. Real Time for Data Pipelines I often want to make every data pipeline a real-time one for the previously mentioned reasons. But there are a number of reasons batches make sense for certain use cases. Batch is good for collecting data at predefined times and chunks. This often occurs when dealing with mainframes, legacy systems, and old data warehouses. These batches tend to be easier to scale and more predictable. This can allow for more affordable data processing at slower, timed intervals. Batch workloads can be run at cheaper times on lesser hardware and allow for longer time to process. You can optimize and break up batches to use less resources (in exchange for time). Real time lowers latency to seconds or less at the cost of being more expensive to scale, less predictable, burstable, and using many resources in a short period of time. Real time lends itself better to things like Kubernetes with the ability to scale on demand or automatically. If you need results immediately, then you cannot choose batch processing for your pipeline. One great feature of a tool like Apache Nifi is that converting from batch to stream or stream to batch is merely choosing a continuous interval or setting a cron schedule. For many applications, you may need a combination of batch and real-time processing on your data, aka the famous Lambda architecture. For a variation of this, we send each event immediately to Apache Kafka, while we set an interval before we push chunks of data to Apache Iceberg. We often set chunks to be 250 megabytes, 500,000 records, or five minutes of data — whichever comes first. Figure 1: Real-time architecture layout Implementing a Real-Time Analytics Pipeline One of the great features of RTA pipelines is that if you plan your architecture, each piece is straightforward to implement. We will be using the most popular open-source frameworks to build a pipeline and highlight alternatives available in the community. The most important thing is to determine your data sources, time and data size requirements, user requirements, and platform. The first implementation step is to choose how you are ingesting data sources. I suggest utilizing Apache NiFi, which is a great way to start the ingestion process and explore that data as it arrives. This is also when you start building data schemas, determine the enrichments and transformations needed, and tour intermediate and final data formats. Often, there will be multiple data destinations, including real-time OLAP datastores like Apache Pinot, vector databases like Milvus, analytic data lake tables like Apache Iceberg, and column-oriented data stores like Apache Kudu. For many use cases, we often write a raw copy of the incoming data into storage like Apache Ozone or Hadoop Distributed File System (HDFS). This may be used later for validation, reloads, or training ML models. Once the data meets the minimum clean format and has a basic schema, we stream it into Apache Kafka or Pulsar for stream processing. We usually choose JSON or Apache Avro as the format for this data. Now that our data is streaming into one or more topics within Kafka, we can start consuming it with our real-time stream processing and analytics tools. There are a number of ways to create RTA applications with Flink, but I recommend starting with Flink SQL. This allows you to build streaming applications with just SQL. You will be able to join streams, do windowing, and handle multiple sources and sinks. When Flink SQL has completed its flows, we often sink it to another Kafka topic, database table, or other datastore. For most architectures, I sink it to Kafka. Exploring RTA Use Cases There are many interesting use cases out there for RTA. I will highlight a couple that I have worked on, but this is not an exhaustive list. You can do much more; your data dreams are the only limit. Monitoring IoT Devices and Systems A use case that I love due to personal experience is monitoring IoT devices and systems. This use case is constantly expanding as every real-world object is adding sensors, networks, cameras, GPUs, computers, and even edge AI systems. One of the main difficulties with IoT is that often, existing systems are not designed to stream data and may not make sensor endpoints available at all or in non-standard formats. This has been challenging in the past but is improving. Often, we add additional devices alongside built-in systems so we can control the data types, speed, and operation. This is enabled by adding a small smart agent such as Apache MiNiFi. Real-Time Fraud Detection This is probably one of the most quintessential use cases for protecting people from bad transactions, and most major credit cards have been implementing this for some time. With real time, we can see not just every transaction and all the metadata surrounding them, but we can see trends over windows of time. The ability of stream processing to discover duplicate, anomalous, and overly frequent charges in unusually small windows of event time helps detect fraud. Stream processing makes it possible to compare new data against existing data profiles of people and credit cards. This is also supercharged by real-time pipelines continuously training ML models on new data so that they recognize typical and atypical patterns for every credit card and its user. RTA Data Pipeline Guiding Principles Designing an RTA data pipeline with Apache NiFi, Apache Kafka, and Apache Flink involves careful consideration of the data sources, processing logic, and how each component fits into the overall architecture. Here's a step-by-step approach to help you decide how to process different data sources: Discover your data sources such as databases, APIs, files, logs, messages, documents, images, videos, and sensors. Start collecting them with NiFi to see what the data looks like, how large the stream is, and if you need to use a different tool. This is where you can do basic cleanup, transformations, enrichment, and field definitions. Then you can route to Kafka for additional processing. For simple use cases, you may have a NiFi stream to a datastore directly. Before arriving in Kafka, you must plan out your topics, schemas of data, and run some tests to see the amount of data. The next step is where you determine which RTA you need. This could be as simple as detecting alert conditions; joining data streams like weather and transit on latitude and longitude; transforming data further; or making computations, aggregations, and anything you can do with advanced SQL using Flink SQL. Often, you will want this data stored for future usage. Flink SQL can store in an underlying streaming table format in Apache Paimon or in an open lakehouse format like Apache Iceberg. Finally, to move this to production, you need to run on an enterprise data platform that provides things like monitoring, fault tolerance, auto-scaling, data security, data governance, and control of the underlying hybrid cloud environment. Remember that the specific implementation details will depend on your use case, data volume, processing requirements, and technical constraints. It's crucial to thoroughly understand the capabilities and limitations of each framework to design an efficient and robust RTA pipeline. Conclusion All data pipelines need to embrace real-time data streams to harness the value of data as it is created, and it's also time to embrace both real-time analytics and batch processing as the two processes converge. It is critical to be ready for the future, which will push more data, more speed, more requirements, and more challenges, including: As edge computing and edge AI continue to grow, they will produce ever increasing amounts of data and require faster processing. New frameworks, updated cloud computing architectures, more integrated AI/ML workloads, and new paradigms are bound to intercept with RTA and data pipelines. The next couple of years should be very exciting. Additional Resources: "Harnessing the Power of NiFi" by Timothy Spann "Real-Time Stream Processing With Hazelcast and StreamNative" by Timothy Spann and Fawaz Ghali, PhD Getting Started With Apache Iceberg Refcard by Ted Gooch Apache Kafka Patterns and Anti-Patterns Refcard by Abhishek Gupta Apache Kafka Essentials Refcard by Sudip Sengupta "Flink + Iceberg," Alibaba Cloud, Community Blog GitHub RTA examples This is an article from DZone's 2023 Data Pipelines Trend Report.For more: Read the Report More

Understanding Europe's Cyber Resilience Act and What It Means for You

By Carsten Rhod Gregersen

IoT manufacturers in every region have a host of data privacy standards and laws to comply with — and Europe is now adding one more. The Cyber Resilience Act, or CRA, has some aspects that are simply common sense and others that overlap with already existing standards. However, other aspects are entirely new and could present challenges to IoT manufacturers and providers. Let’s explore the act and consider how it will change the world of connected devices. The Basics of the Cyber Resilience Act The CRA lays out several specific goals that the act is intended to fulfill: Goal #1: Ensuring fewer vulnerabilities and better protection in IoT devices and products Goal #2: Bringing more responsibility for cybersecurity to the manufacturer Goal #3: Increasing transparency Goal #2 leads directly to the obligations laid out for manufacturers: Cybersecurity must be an aspect of every step of the device or software development life cycle. Risks need to be documented. Manufacturers must report, handle, and patch vulnerabilities for any devices that are sold for the product’s expected lifetime or for five years, whichever comes first. The manufacturer must provide clear and understandable instructions for any products with digital elements. So, to whom does the CRA apply? The answer is anyone who deals in IoT devices, software, manufacturing, development, etc. The standard lays the responsibility for vulnerabilities squarely at the foot of the manufacturers of the IoT device or software product in a way most other standards have yet to stipulate. However, the CRA does not affect all devices and manufacturers equally. Three main categories will dictate how manufacturers apply the standard. The first is the default category, which covers roughly 90 percent of IoT products. Devices and products in this category are non-critical, like smart speakers, non-essential software, etc. The default category doesn’t require a third-party assessment of adherence to the standard, so the category just provides a basis for self-assessment that allows manufacturers to establish best practices for product security. The second and third categories are Critical Class I and Critical Class II, which apply to roughly 10 percent of IoT products. Class I includes password managers, network interfaces, firewalls, microcontrollers, and more. In other words, the vendors and designers of MCUs and the other components included in Class I will have to comply with all of the requirements for that category. Class II is for operating systems, industrial firewalls, MPUs, etc. Again, that means the vendors who develop these operating systems and microprocessors will need to ensure they meet the specific requirements for Class II. The criteria that divide the classes are based on intended functionality (like whether the software will be used for security/access management), intended use (like whether it is for an industrial environment), and breach or vulnerability likelihood, among other criteria. Both Critical Class categories require a third-party assessment for compliance purposes. Importantly, there are penalties for non-compliance, which include the possibility of a full ban on the problematic product, as well as fines of €15 million or 2.5 percent of the annual turnover of the offending company, whichever is higher. Why This Act Matters The Cyber Resilience Act is part of a longstanding and ongoing endeavor by EU governing bodies to ensure a deeper level of cybersecurity in the EU. This endeavor is largely in response to a marked increase in ransomware and denial of service attacks since the pandemic and especially since the start of the Russia-Ukraine War. Still, the CRA overlaps with some other standards including the upcoming NIS2 Directive, which is the EU’s blanket cybersecurity legislation. Because of that, it’s easy to think that the CRA doesn’t have much to add, but it actually does. The act is broader than a typical IoT security standard because it also applies to software that is not embedded. That is to say, it applies to the software you might use on your desktop to interact with your IoT device, rather than just applying to the software on the device itself. Since non-embedded software is where many vulnerabilities take place, this is an important change. A second important change is the requirement for five years of security updates and vulnerability reporting. Few consumers who buy an IoT device expect regular software updates and security patches for that type of time range, but both will be a requirement under the CRA. The third important point of the standard is the requirement for some sort of reporting and alerting system for vulnerabilities so that consumers can report vulnerabilities, see the status of security and software updates for devices, and be warned of any risks. The CRA also requires that manufacturers notify the European Union Agency for Cybersecurity (ENISA) of a vulnerability within 24 hours of discovery. These requirements are intended to keep consumers’ data safe, but they will also allow manufacturers to avoid costly breaches. Prepare for Compliance Today The Cyber Resilience Act is in its early stages and, even when it is approved, manufacturers will have two years to comply. So, full compliance will probably not be obligatory until 2025 or 2026. However, that doesn’t mean you shouldn’t start preparing now. When the General Data Protection Regulation (GDPR) came into force in the EU, companies had to make major changes to their operations and the way they handled consumer data, advertising, cookies, and more. This act has the potential to be just as complex and revolutionary in changing the way IoT manufacturers and software providers manage security for their products. What can manufacturers do now to avoid penalties for non-compliance in the future? For starters, there are already technologies available that can help with CRA compliance. The reporting requirements of the EU Cyber Resilience Act are time-sensitive and penalties for non-compliance are high. This means manufacturers have a vested interest in developing efficient ways to communicate discovered vulnerabilities to both consumers and ENISA, as well as to patch those vulnerabilities as quickly as possible. As a result, IoT providers who can utilize a peer-to-peer communication platform that enables remote status reports, updates, and security patches will have a competitive advantage. Additionally, such a platform can allow IoT providers to set up push notifications and security alerts for consumers, enabling the highest level of transparency and communication in case a vulnerability is discovered. It’s also important to keep up with changes to the proposal as laid out by the European Commission, since the CRA as it appears in 2026 may not be the same as the initial draft of the standard. In fact, well-known companies like Microsoft are already making recommendations and critiques of the EU Cyber Resilience Act proposal. Many experts believe the CRA is too broad at the moment and too hard to apply, and that it needs stronger definitions, examples, and action plans to be truly effective. If these critiques are followed, compliance could become a bit less complex and a bit easier to understand in the future, so it would be wise to keep informed of any changes. Final Thoughts While the initial shift to CRA compliance may be challenging, various technologies and cybersecurity tools are already available to help. Integrating these tools and choosing to pursue the highest levels of security in your IoT products today will put you well on your way to fulfilling the requirements of the act before it is even in effect. Good luck. More

IoT Device Management: Streamlining Connectivity in a Connected World

By Manvendra sharma

Types of Edge ML and Enterprise Use Cases

By Margo McCabe

Revolutionizing IoT With Digital Temperature Sensors

By cecilia cheruto

Exploring Edge Computing: Delving Into Amazon and Facebook Use Cases

The rapid growth of the Internet of Things (IoT) and the increasing need for real-time data processing have led to the emergence of a new computing paradigm called edge computing. As more devices connect to the internet and generate vast amounts of data, traditional centralized cloud computing struggles to keep pace with the demand for low-latency, high-bandwidth communication. This article aims to provide a deeper understanding of edge computing, its benefits challenges, and a detailed examination of its application in Amazon and Facebook use cases. Understanding Edge Computing Edge computing is a distributed computing model that moves data processing and storage closer to the source of data generation. Instead of relying solely on centralized cloud data centers, edge computing enables processing to occur at the "edge" of the network, using devices like IoT sensors, local servers, or edge data centers. This approach reduces the amount of data transmitted to and from central data centers, thus easing the burden on network infrastructure and improving overall performance. Edge computing empowers businesses like Amazon and Facebook to revolutionize real-time data processing, delivering seamless experiences, enhanced security, and scalable infrastructure for the ever-growing digital world. Advantages of Edge Computing Reduced latency: Edge computing significantly decreases the time it takes for data to travel between the source and the processing unit by processing data locally. This is essential for applications requiring real-time decision-making. Improved bandwidth utilization: Transmitting large amounts of data to centralized data centers can consume considerable network bandwidth. Edge computing addresses this issue by processing and storing data locally, freeing up bandwidth for other crucial tasks. Enhanced security and privacy: Storing and processing data within local devices or networks helps mitigate privacy and security concerns. Sensitive information remains within the local infrastructure, reducing the risk of unauthorized access or data breaches. Scalability: Edge computing allows organizations to scale their operations more efficiently by distributing workloads across multiple edge devices. This flexibility enables businesses to expand their capabilities without overburdening central data centers. Diving Into Amazon and Facebook Use Cases Amazon: AWS Wavelength for Low-Latency Applications Amazon Web Services (AWS) introduced Wavelength, an edge computing service designed to bring AWS services closer to end-users and devices. This enables developers to build low-latency applications. Wavelength extends AWS infrastructure to the edge of the telecommunications network, allowing developers to deploy their applications in Wavelength Zones — geographically distributed locations connected to the telecom network. One way Amazon uses Wavelength is in deploying real-time gaming and video streaming services. By leveraging Wavelength, Amazon can process and deliver high-quality content with minimal latency, providing a seamless experience for users. For example, game developers can utilize Wavelength to run their game servers at the edge of the network, reducing lag experienced by players and ensuring smooth, responsive gameplay. Additionally, Wavelength can enhance the performance of emerging technologies like augmented reality (AR) and virtual reality (VR), where low latency is crucial for delivering immersive experiences. By processing AR/VR data at the edge, Amazon can provide faster response times and more accurate rendering, resulting in better user experiences. Facebook: Edge Computing for Content Distribution and AI Processing Facebook employs edge computing to optimize content distribution and improve user experiences across its platform. One way Facebook leverages edge computing is through deploying edge data centers, known as Points of Presence (PoPs). These PoPs are strategically located around the world and cache and serve static content, such as images and videos, closer to users, reducing latency and improving content delivery times. Another example of Facebook's use of edge computing is its AI-powered content moderation system. By processing image and video data at the edge, Facebook can quickly analyze and filter out inappropriate or harmful content before it reaches users. This real-time content moderation helps maintain a safe and positive environment on the platform while reducing the workload on central data centers. Moreover, Facebook's edge computing infrastructure can be used to process and analyze user data for targeted advertising. By analyzing user data at the edge, Facebook can deliver personalized ads more quickly and efficiently, improving the overall advertising experience for both users and advertisers. Conclusion Edge computing is transforming the way we process and store data, offering numerous benefits such as reduced latency, improved bandwidth utilization, and enhanced security. Companies like Amazon and Facebook are harnessing the power of edge computing to optimize their services, from low-latency applications like gaming and streaming to content distribution and AI processing. As the demand for real-time data processing continues to grow, edge computing will play an increasingly important role across various industries. However, organizations must address challenges related to infrastructure investment, data management, and security to fully capitalize on the potential of this emerging technology.

By Arun Pandey

Exploring the Evolution and Impact of Computer Networks

In today's interconnected world, computer networks have become the backbone of modern communication and information exchange. They enable the seamless transmission of data, facilitate collaboration, and connect individuals and devices across the globe. This article delves into the evolution, components, and significance of computer networks. We will explore their history, the key networking technologies and protocols, and the impact they have had on various aspects of our lives. Historical Overview The history of computer networks dates back to the 1960s when the concept of connecting computers for communication purposes began to take shape. Let's explore the historical overview of computer networks: ARPANET and Early Networks The U.S. Department of Defense's Advanced Research Projects Agency (ARPA) established the Advanced Research Projects Agency Network (ARPANET) in the late 1960s. The development of contemporary computer networks was sparked by the creation of the first packet-switched network, known as ARPANET. As a result, it enabled resource sharing and collaborative research between computers at various universities and research centers. Ethernet and Local Area Networks (LANs) At Xerox PARC (Palo Alto Research Center), Ethernet was created in the 1970s. Computers within a constrained geographic area can now share resources and communicate thanks to Ethernet's standardization of the LAN protocol. The de facto standard for LAN connectivity, Ethernet, experienced a rapid increase in popularity. TCP/IP and the Birth of the Internet In the late 1970s and early 1980s, the development of the Transmission Control Protocol/Internet Protocol (TCP/IP) by Vinton Cerf and Robert Kahn paved the way for the creation of the Internet. TCP/IP provided a set of protocols for reliable data transmission and ensured compatibility between different networks. The adoption of TCP/IP allowed disparate networks to connect and communicate, forming a global network of networks. Wide Area Networks (WANs) and the Internet's Expansion During the 1980s, the Internet expanded beyond academic and research institutions. Commercial networks and Internet Service Providers (ISPs) emerged, connecting businesses and individuals to the Internet. This expansion led to the creation of wide area networks (WANs) that linked different LANs and extended connectivity over larger geographical areas. World Wide Web and Internet Revolution In 1989, Tim Berners-Lee invented the World Wide Web, introducing hypertext and URLs for easy navigation and information retrieval on the Internet. The World Wide Web revolutionized the way people accessed and shared information, leading to an exponential growth of websites and online services. Internet Technologies and Mobile Connectivity The 1990s witnessed advancements in internet technologies, such as the development of faster networking protocols (e.g., HTTP, SSL/TLS) and the introduction of commercial web browsers. The emergence of mobile devices and wireless networking technologies further expanded internet access, enabling people to connect to the internet on the go. Modern Networking Technologies and Future Trends In recent years, networking technologies have continued to evolve. The deployment of high-speed broadband, the emergence of 5G wireless networks, and the proliferation of Internet of Things (IoT) devices have further transformed the networking landscape. Concepts like software-defined networking (SDN) and network virtualization have gained prominence, offering more flexibility, scalability, and control over network infrastructure. Components of Computer Networks Computer networks are composed of various interconnected components that work together to facilitate communication and data exchange. These components play specific roles in ensuring the smooth operation of the network. Let's explore the key components of computer networks: Network Devices Network devices are physical or virtual devices that enable the transmission and reception of data over the network. Some common network devices include: Routers: Routers connect multiple networks and direct data packets between them based on network addresses. They determine the optimal path for data transmission. Switches: Switches connect devices within a local area network (LAN) and facilitate the exchange of data packets. They direct data to specific devices based on their MAC addresses. Hubs: Hubs are simple devices that connect multiple devices in a LAN. They receive incoming data packets and broadcast them to all connected devices. Modems: Modems enable the transmission of digital data over analog communication lines, such as telephone lines or cable lines. Network Media Network media refers to the physical channels used to transmit data between devices in a network. Different types of network media include: Twisted-Pair Cables: Twisted-pair cables, such as Ethernet cables, are widely used for wired network connections. They consist of pairs of insulated copper wires twisted together to reduce interference. Coaxial Cables: Coaxial cables have a central conductor surrounded by insulating and shielding layers. They are used for cable television connections and some networking applications. Fiber Optic Cables: Fiber optic cables use thin strands of glass or plastic to transmit data as pulses of light. They offer high bandwidth, fast data transmission, and resistance to electromagnetic interference. Wireless Signals: Wireless networks use radio waves, infrared signals, or other wireless technologies to transmit data without the need for physical cables. Wi-Fi, Bluetooth, and cellular networks are examples of wireless networks. Network Protocols Network protocols are a set of rules and procedures that govern data communication between devices in a network. They ensure that data is properly formatted, transmitted, and received. Some common network protocols include: TCP/IP (Transmission Control Protocol/Internet Protocol): TCP/IP is the foundation of the Internet and most computer networks. It provides reliable and secure data transmission across networks. HTTP (Hypertext Transfer Protocol): HTTP is used for communication between web browsers and web servers. It enables the retrieval and display of web pages. FTP (File Transfer Protocol): FTP is used for transferring files between devices in a network. It allows users to upload, download, and manage files on remote servers. DNS (Domain Name System): DNS translates domain names (e.g., www.example.com) into IP addresses that computers can understand. It enables users to access websites using human-readable names. Network Interfaces Network interfaces, also known as network interface cards (NICs), are hardware or software components that enable devices to connect to a network. They provide the necessary physical or virtual interface for data transmission. Network interfaces can be Ethernet ports, Wi-Fi adapters, or virtual interfaces created by network virtualization technologies. Network Services and Applications Network services and applications utilize the network infrastructure to provide specific functionalities. Some examples include: Email Servers: Email servers facilitate the sending, receiving, and storage of emails over a network. Web Servers: Web servers host websites and deliver web pages to clients over the network. DNS Servers: DNS servers provide domain name resolution services, translating domain names into IP addresses. DHCP (Dynamic Host Configuration Protocol) Servers: DHCP servers automatically assign IP addresses and network configurations to devices in a network. Firewalls: Firewalls monitor and control network traffic, ensuring network security by filtering incoming and outgoing data packets. These components work together to establish and maintain a functioning computer network, enabling devices to communicate, share resources, and access services across the network. Networking Technologies Networking technologies encompass a wide range of technologies that facilitate the communication and transmission of data within computer networks. These technologies play a crucial role in enabling reliable, secure, and efficient network operations. Let's explore some of the key networking technologies: Ethernet Ethernet is a widely used networking technology for local area networks (LANs). It defines the standards for wired connections, specifying the physical and data link layer protocols. Ethernet utilizes the Carrier Sense Multiple Access with Collision Detection (CSMA/CD) protocol to manage data collisions and ensure efficient communication. Wi-Fi Wi-Fi (Wireless Fidelity) technology enables wireless connectivity within local networks. It utilizes radio waves to transmit data between devices and access points. Wi-Fi networks provide flexibility, mobility, and convenience, allowing devices to connect without the need for physical cables. Internet Protocol (IP) The Internet Protocol (IP) is a fundamental networking technology that provides a standardized addressing scheme and facilitates the routing of data packets across networks. IP assigns unique IP addresses to devices and enables the delivery of data packets to their destinations. The two prevalent versions of IP are IPv4 and IPv6. Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) TCP and UDP are transport layer protocols that work in conjunction with IP. TCP ensures reliable and ordered delivery of data packets by establishing connections, acknowledging received packets, and retransmitting lost packets if necessary. UDP, on the other hand, provides a connectionless and lightweight protocol for transmitting data packets without the reliability guarantees of TCP. UDP is commonly used for real-time applications and streaming services. Virtual Private Networks (VPNs) VPNs are technologies that create secure and private connections over public networks, such as the Internet. VPNs use encryption and tunneling protocols to ensure the confidentiality and integrity of data transmitted between devices. They provide secure remote access to private networks and enable users to browse the internet privately and anonymously. Wide Area Networks (WANs) Wide Area Networks connect geographically dispersed networks and enable communication over long distances. Various technologies are used to establish WAN connections, including: Leased Lines: Leased lines are dedicated point-to-point connections between two locations, providing high-speed and reliable connectivity. MPLS (Multiprotocol Label Switching): MPLS is a technology used to route network traffic efficiently and securely across WANs. It uses labels to identify and forward data packets along predefined paths. Satellite Connections: Satellite connections utilize satellites in space to transmit data over long distances. They are particularly useful in remote areas where terrestrial infrastructure is limited. Cloud Computing Cloud computing leverages network infrastructure to provide on-demand access to shared computing resources, storage, and applications over the Internet. Cloud services enable businesses and individuals to scale their computing needs, reduce costs, and access resources from anywhere at any time. Software-Defined Networking (SDN) SDN is a networking approach that separates the control plane from the data plane, centralizing network management and control. It allows administrators to programmatically manage and configure network devices, improving flexibility, scalability, and network automation. Network Virtualization Network virtualization technologies enable the creation of virtual network infrastructures on top of physical networks. Virtual networks provide isolation, segmentation, and flexibility, allowing multiple virtual networks to coexist on a shared physical infrastructure. These are just a few examples of the many networking technologies available today. Advancements in networking continue to drive innovation, improving the performance, security, and efficiency of computer networks. Impact of Computer Networks Computer networks have had a profound impact on various aspects of our lives, revolutionizing the way we communicate, access information, conduct business, and interact with the world. Here are some key areas where computer networks have made a significant impact: Communication Computer networks have transformed communication, enabling instant and global connectivity. Email, instant messaging, video conferencing, and social media platforms have revolutionized how we connect with others. Networks have made it possible to communicate with people across the globe in real-time, fostering collaboration, knowledge sharing, and cultural exchange. Information Access and Sharing Computer networks, particularly the internet, have democratized access to information. The internet serves as a vast repository of knowledge, allowing us to search for information, access educational resources, and stay updated on news and events. Online platforms and social media have facilitated the sharing of ideas, creativity, and expertise, empowering individuals to participate in global conversations. Business and E-commerce Networking technologies have transformed the business landscape. E-commerce platforms have opened up new markets, enabling businesses to reach customers globally. Online transactions, digital payments, and secure communication have streamlined business operations. Networks facilitate supply chain management, remote collaboration, and cloud-based services, enhancing productivity, efficiency, and innovation. Education and E-Learning Computer networks have revolutionized education. E-learning platforms and online courses have made education accessible to a wider audience, transcending geographical boundaries. Students can access educational resources, participate in virtual classrooms, and engage in interactive learning experiences. Networking technologies facilitate global educational collaborations, fostering cross-cultural learning and knowledge exchange. Entertainment and Media Networking technologies have transformed the entertainment industry. Streaming services provide on-demand access to movies, TV shows, and music, revolutionizing how we consume media. Social media platforms offer a platform for content creators to share their work, reaching a global audience. Gaming networks enable multiplayer experiences, connecting players worldwide. Healthcare and Telemedicine Computer networks play a crucial role in healthcare. Electronic health records (EHRs) enable a secure and efficient exchange of patient information between healthcare providers, improving care coordination and patient outcomes. Telemedicine leverages networks to deliver remote medical consultations, diagnosis, and monitoring, expanding access to healthcare services, especially in remote or underserved areas. Internet of Things (IoT) The IoT leverages computer networks to connect and control various devices and sensors. It enables smart homes, industrial automation, and intelligent infrastructure. Networks allow devices to communicate, share data, and operate in concert, leading to enhanced convenience, efficiency, and sustainability. Research and Innovation Computer networks have facilitated collaboration and accelerated innovation. Researchers can collaborate across different institutions, sharing data, resources, and expertise. Networked supercomputers and high-performance computing enable complex simulations and scientific breakthroughs. Open-source communities and crowdsourcing platforms leverage networks to foster collaborative problem-solving and innovation. Social and Cultural Impact Computer networks have reshaped social dynamics and cultural interactions. Social media networks have transformed how we form relationships, share experiences, and express ourselves. Online communities have brought together like-minded individuals, fostering new connections and communities beyond geographical boundaries. Networks have given rise to digital activism, enabling individuals to mobilize and advocate for social and political causes. Conclusion Computer networks have advanced significantly from the early ARPANET to the modern, globally connected Internet. As networking technologies have developed, they have sparked societal, economic, and technological changes that have made it possible for people to communicate, share information, and work together easily. As time goes on, developments like 5G, IoT, and emerging technologies will continue to shape the future of computer networks by presenting fresh opportunities for connectivity, innovation, and cross-cultural communication. The way we communicate, work together, and access information has been revolutionized by computer networks. Industry transformation, individual empowerment, and unprecedented global connectivity have all resulted from them. Future networking technology developments should be even more significant as they develop. Computer networks will become even more connected, quick, and capable with the integration of cutting-edge technologies like 5G, edge computing, and artificial intelligence. Accepting these developments is essential, as is tackling the problems they bring with them, like privacy and security issues. We can continue to open up new possibilities and spur innovation across a range of domains by utilizing the potential of computer networks, thereby influencing the future of our interconnected world. Finally, the global connectivity, information exchange, and collaboration that computer networks have made possible have fundamentally changed our society. They have created new opportunities for interaction, learning, commerce, healthcare, and entertainment. We can anticipate even more advancements as networking technologies develop, which will impact how we connect, live, and work in the future.

By Aditya Bhuyan

The State of Data Streaming for Telco

This blog post explores the state of data streaming for the telco industry in 2023. The evolution of telco infrastructure, customer services, and new business models requires real-time end-to-end visibility, fancy mobile apps, and integration with pioneering technologies like 5G for low latency or augmented reality for innovation. Data streaming allows integrating and correlating data in real-time at any scale to improve most telco workloads. I look at trends in the telecommunications sector to explore how data streaming helps as a business enabler, including customer stories from Dish Network, British Telecom, Globe Telecom, Swisscom, and more. A complete slide deck and on-demand video recording are included. General Trends in the Telco Industry The Telco industry is fundamental for growth and innovation across all industries. The global spending on telecom services is expected to reach 1.595 trillion U.S. dollars by 2024 (Source: Statista, Jul 2022). Cloud-native infrastructure and digitalization of business processes are critical enablers. 5G network capabilities and telco marketplaces enable entirely new business models. 5G Enables New Business Models Presentation of Amdocs / Mavenir: A report from McKinsey and Company says, “74 percent of customers have a positive or neutral feeling about their operators offering different speeds to mobile users with different needs”. The potential for increasing the revenue per user (ARPU) with 5G use cases is enormous for telcos: Telco Marketplace Many companies across industries are trying to build a marketplace these days. But the telecom sector might shine here because of its interface between infrastructure, B2B, partners, and end users for sales and marketing. tmforum has a few good arguments for why communication service providers (CSP) should build a marketplace for B2C and B2B2X: Operating the marketplace keeps CSP in control of the relationship with customers A marketplace is a great sales channel for additional revenue Operating the marketplace helps CSPs monetize third-party (over-the-top) content The only other option is to be relegated to connectivity provider Enterprise customers have decided this is their preferred method of engagement CPSs can take a cut of all sales Participating in a marketplace prevents any one company from owning the customer Data Streaming in the Telco Industry Adopting trends like network monitoring, personalized sales, and cybersecurity is only possible if enterprises in the telco industry can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means): Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. "Use Cases for Apache Kafka in Telco" is a good article for starting with an industry-specific point of view on data streaming. "Apache Kafka for Telco-OTT and Media Applications" explores over-the-top B2B scenarios. Data streaming with the Apache Kafka ecosystem and cloud services are used throughout the supply chain of the telco industry. From Telco to TechCo: Next-Generation Architecture Deloitte describes the target architecture for telcos very well: Data streaming provides these characteristics: Open, scalable, reliable, and real-time. This unique combination of capabilities made Apache Kafka so successful and widely adopted. Kafka decouples applications and is the perfect technology for microservices across a telco's enterprise architecture. Deloitte's diagram shows this transition across the entire telecom sector: This is a massive shift for telcos: From purpose-built hardware to generic hardware and elastic scale From monoliths to decoupled, independent services Digitalization with modern concepts helps a lot in designing the future of telcos. Open Data Architecture (ODA) tmforum describes Open Digital Architecture (ODA) as follows: "Open Digital Architecture is a standardized cloud-native enterprise architecture blueprint for all elements of the industry from Communication Service Providers (CSPs) through vendors to system integrators. It accelerates the delivery of next-gen connectivity and beyond — unlocking agility, removing barriers to partnering, and accelerating concept-to-cash. ODA replaces traditional operations and business support systems (OSS/BSS) with a new approach to building software for the telecoms industry, opening a market for standardized, cloud-native software components and enabling communication service providers and suppliers to invest in IT for new and differentiated services instead of maintenance and integration." If you look at the architecture trends and customer stories for data streaming in the next section, you realize that real-time data integration and processing at scale are required to provide the most modern use cases in the telecommunications industry. Architecture Trends for Data Streaming The telco industry applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are: Hybrid architectures with synchronization between edge and cloud in real-time End-to-end network and infrastructure monitoring across multiple layers Proactive service management and context-specific customer interactions Let's look deeper into some enterprise architectures that leverage data streaming for telco use cases. Hybrid 5G Architecture With Data Streaming Most telcos have a cloud-first strategy to set up modern infrastructure for network monitoring, sales and marketing, loyalty, innovative new OTT services, etc. However, edge computing gets more relevant for use cases like pre-processing for cost reduction, innovative location-based 5G services, and other real-time analytics scenarios: Learn about architecture patterns for Apache Kafka that may require multi-cluster solutions and see real-world examples with their specific requirements and trade-offs. That blog explores scenarios such as disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments, and global Kafka. Edge deployments for data streaming are their own challenges. In separate blog posts, I covered use cases for Kafka at the edge and provided an infrastructure checklist for edge data streaming. End-To-End Network and Infrastructure Monitoring Data streaming enables unifying telemetry data from various sources such as Syslog, TCP, files, REST, and other proprietary application interfaces: End-to-end visibility into the telco networks allows massive cost reductions. And, as a bonus, a better customer experience. For instance, proactive service management tells customers about a network outage: Context-Specific Sales and Digital Lifestyle Services Customers expect a great customer experience across devices (like a web browser or mobile app) and human interactions (e.g., in a telco store). Data streaming enables a context-specific omnichannel sales experience by correlating real-time and historical data at the right time in the proper context: "Omnichannel Retail and Customer 360 in Real Time with Apache Kafka" goes into more detail. But one thing is clear: Most innovative use cases require both historical and real-time data. In summary, correlating historical and real-time information is possible with data streaming out-of-the-box because of the underlying append-only commit log and replayability of events. A cloud-native Tiered Storage Kafka infrastructure to separate compute from storage makes such an enterprise architecture more scalable and cost-efficient. The article "Fraud Detection with Apache Kafka, KSQL, and Apache Flink" explores stream processing for real-time analytics in more detail, shows an example with embedded machine learning, and covers several real-world case studies. New Customer Stories for Data Streaming in the Telco Industry So much innovation is happening in the telecom sector. Automation and digitalization change how telcos monitor networks, build customer relationships, and create completely new business models. Most telecommunication service providers use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. Elastic scalability gets even more critical with all the growing networks and 5G workloads. Here are a few customer stories from worldwide telecom companies: Dish Network: Cloud-native 5G Network with Kafka as the central communications hub between all the infrastructure interfaces and IT applications. The standalone 5G infrastructure, in conjunction with data streaming, enables new business models for customers across all industries, like retail, automotive, or energy sector. Verizon: MEC use cases for low-latency 5G stream processing, such as autonomous drone-in-a-box-based monitoring and inspection solutions or vehicle-to-Everything (V2X). Swisscom: Network monitoring and incident management with real-time data at scale to inform customers about outages, root cause analysis, and much more. The solution relies on Apache Kafka and Apache Druid for real-time analytics use cases. British Telecom (BT): Hybrid multi-cloud data streaming architecture for proactive service management. BT extracts more value from its data and prioritizes real-time information and better customer experiences. Globe Telecom: Industrialization of event streaming for various use cases. Two examples: Digital personalized rewards points based on customer purchases. Airtime loans are made easier to operationalize (vs. batch, where top-up cash is already spent again). Resources To Learn More This blog post is just the starting point. Learn more about data streaming in the telco industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases. On-Demand Video Recording The video recording explores the telecom industry's trends and architectures for data streaming. The primary focus is the data streaming case studies. Check out our on-demand recording. Slides If you prefer learning from slides, check out the deck used for the above recording: Slides: The Data of Apache Kafka for Telco in 2023 Case Studies and Lightboard Videos for Data Streaming in Telco The state of data streaming for Telco in 2023 is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, real-time data collection and processing data from network infrastructure and mobile apps, data sharing and B2B partnerships with OTT players for new business models, and many more scenarios. We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Stay tuned; I will update the links in the next few weeks and publish a separate blog post for each story and lightboard video. And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on.

By Kai Wähner CORE

Application Integration for IoT Devices: A Detailed Guide To Unifying Your Ecosystem

As IoT devices pervade every facet of our lives and businesses, the chatter usually revolves around the cool capabilities these devices bring. While that's fascinating, what often goes underappreciated is the role of application integration in amplifying the utility and scalability of these devices. Rather than being a mere enabler, application integration is an equal player in this game, as it not only leverages but also elevates the capabilities of IoT systems. The Real Value of IoT Lies in Integrated Intelligence It's an exhilarating time for IoT aficionados. Smart homes are turning smarter, and businesses are finding new, real-time metrics to measure and improve performance. However, the true potential of these capabilities comes to light only when the devices can collaborate, i.e., when they're properly integrated. Ben Franklin once said, "The whole is greater than the sum of its parts," and nothing could be more accurate when describing IoT and application integration. Without effective application integration, IoT is essentially a collection of isolated capabilities. Tim Berners-Lee, the founder of the World Wide Web, once stressed, "The value of a network grows with the square of its size." This perfectly sums up the situation. When applications and devices work in harmony, you tap into a kind of network effect within your IoT ecosystem. Now, if you're considering this a mere technicality, think again. Connecting IoT devices to applications isn't just about creating a pathway for data to flow. It's about optimizing that pathway for efficiency, scale, and security. API Strategies: Beyond REST While REST APIs have been an industry standard, they aren't always the best fit for IoT, where real-time data synchronization can be crucial. GraphQL and real-time APIs are coming to the forefront, offering more efficient ways to interact with data and services. In complex IoT ecosystems, where a multitude of protocols and languages are at play, middleware can be your best friend. It not only translates but also prepares and secures data for its journey between devices and applications. Essentially, middleware offers a consolidated layer of functionalities, like data normalization and transformation, which are pivotal in maintaining the integrity and usefulness of data. Scalability Through Microservices The number of connected devices is only going to increase, and data volumes will follow suit. Hence, it's crucial to consider scalability right from the get-go. Microservices architecture offers a way to scale different functionalities of your integration platform independently. This is crucial as bottlenecks in any single service could bring down the performance of the entire ecosystem. With greater connectivity comes a more expansive attack surface. The integration layer is not just a pipeline; it's also a potential point of vulnerability. Security cannot be a second thought; it must be integral to your application integration strategy. This means zero-trust models and data encryption at every single stage. The Impact of IPaaS on IoT The influence of Integration Platform as a Service (iPaaS) on IoT can't be overstated. By offering cloud-based integration solutions, iPaaS simplifies and accelerates the integration process. With AI-powered features like automated data mapping and transformation, iPaaS can be a game-changer in managing the complexities of IoT environments. With the emergence of smart cities, connected healthcare, and intelligent manufacturing, we are at a stage where IoT isn't just a value-add; it's becoming essential. This makes the role of application integration even more critical. As these technologies become increasingly entrenched in societal and organizational fabrics, the quality of integration will directly impact the quality of life and operational efficiency. Gearing Up for a Long Haul This isn't a sprint; it's a marathon. As IoT technologies continue to evolve, your integration strategies will need to keep pace. Quick fixes may solve immediate problems, but a robust, scalable, and secure integration strategy will stand the test of time. To delve deeper into this fascinating topic, feel free to explore this comprehensive guide on application integration for IoT. In conclusion, as we stand at the cusp of a new era driven by IoT and digital transformation, let's not overlook the significant role of application integration. Far from being just a background player, application integration is front and center, working in symbiosis with IoT to unlock new avenues of capabilities and efficiencies. It's not just about making devices smarter; it's about making entire ecosystems intelligent, scalable, and secure.

By Pamela Price

Apache Druid: Making 1000+ QPS for Analytics Look Easy

Analytics use cases are evolving with higher volume, low latency queries on the rise. But scaling analytics for high queries per second (QPS) needs some consideration. If your queries are retrieving single-rows in tables with few columns or rows or aggregating a small amount of data, then virtually any database can meet your QPS requirements. But things start getting hard if you have an analytics application (or plan to build one) that executes lots and lots of aggregations and filters across high dimensional and high cardinality data at scale. The kind of application where lots of users should be able to ask any question and get their answers instantly without constraints to the type of queries or shape of the data. Apache Druid is a popular, open-source real-time analytics database that was designed from the ground up for interactive, sub-second queries at tremendous scale and load. Target uses Druid for analytics applications that execute over 4 million queries per day with an average response of 300ms; Confluent powers a customer-facing application requiring 350 QPS on data streaming in at 5 million events per second. 1000s of companies use Druid when there’s a live person or automated decisioning pipeline on the other end of a query. And they can do this under load with far fewer computing resources than with other databases. You might be thinking “those queries must be pre-computed.” But they’re actually not; it’s all raw horsepower driven by Druid’s architecture. So how does it do it? Because open source is awesome, there’s no black box voodoo magic or marketing hype at play here. This post takes you through the key design elements that enable Druid to uniquely achieve 1000+ QPS for analytics without breaking a sweat (or breaking the bank). Optimized Data Format For high QPS, the database has to do 2 things well: process each query lightning-fast and do a lot of them at the same time. Since we’re talking about analytics queries that could involve reading a lot of data, the key to fast processing is to minimize the computing work. Druid’s architecture is very focused on CPU efficiency to process complex queries with potentially trillions of rows in under a second. And this starts with Druid’s data format. It’s common for analytics databases to store data in columns rather than rows. This way aggregate functions will only read the columns in the query. Druid stores in columns too, but it does much more to optimize how data is stored. For starters, Druid is optimized for time-series data. Druid utilizes a timestamp column as a primary dimension for organizing data in files known as segments. This column is indexed in a way that enables efficient filtering and querying based on time intervals or windows and quick locating of the relevant segments. What this allows Druid to do is quickly locate and process only the relevant data when filtering by a time window of interest so you can ask questions like what happened in the last hour, how did that compare to last week, etc. with very fast response times. Additionally, Druid stores dimension columns with dictionary encoding and inverted indexes (aka bitmap indexes) and they are type-aware compressed, which both minimizes how much CPU is needed and reduces the amount of data storage. While the use of indexes isn’t new, Druid’s implementation is out of the box: It’s built into its data format and is much easier to work with than anything else. As data is ingested, Druid automatically manages and updates the indexes. This is in contrast with other databases where indexes require a lot of advance planning and maintenance. Druid doesn’t require anyone to maintain the indexes — they’re all automatic. Scatter/Gather With Pre-Loaded Data The relationship between storage and compute plays a key role in delivering high QPS for analytics: Don’t read the full data set if you can read a smaller index, don’t send data unnecessarily from server to server, and don’t move data from deep storage at query time if you need fast queries. Druid embraces this thinking and at the heart of its interactive query engine is the scatter/gather technique. With scatter/gather, queries are scattered across multiple computing nodes that contain the relevant segments for parallel processing. Each node performs computations on its local subset of the data. Afterward, the results are gathered from those nodes for the final result. But it’s not simply about running queries in parallel. For the fast queries, Druid intelligently pre-loads the segments into the right data nodes so queries are not waiting for data to move from deep storage to where compute is (unlike cloud data warehouses, which separate compute and storage for reports/BI workloads). It’s intelligent because Druid determines the right performance-optimized and time-aware nodes across potentially thousands of nodes and millions of segments to get the right price for performance. Extensive Multi-Threading and Parallelization Many modern databases are designed with multithreading to handle concurrent operations. This allows multiple execution threads to run concurrently, enabling parallel processing of queries. That said, the degree of multithreading in Druid and its specific implementation is a unique technical characteristic of its architecture, which lends itself to be very effective at high QPS for analytics. Druid utilizes multithreading extensively with specialized thread pools, task schedulers, and resource management mechanisms to optimize high performance under load. Here are some key aspects of Apache Druid’s approach to threading: Process-specific parallelization: To say Druid is a distributed architecture would be a bit of an understatement. Druid parallelizes everything through multithreading. From ingestion, to indexing, to segment processing, to query execution are all done in parallel across multiple threads. This approach to parallelization enables efficient utilization of CPU resources and speeds up query execution and data processing. Thread pooling: Druid employs thread pools, which are pools of reusable threads, to manage and control the execution of tasks. Thread pools allow for efficient thread reuse, minimizing the overhead of thread creation and destruction, and enhancing performance by enabling concurrency and workload parallelization. Resource allocation and isolation: Druid’s threading approach involves allocating and isolating specific threads or thread pools for different tasks, such as query processing, data ingestion, and background maintenance. This allows for fine-grained control over resource allocation and prioritization, ensuring optimal performance for different aspects of the system. Dynamic scaling: Druid’s threading model supports dynamic scaling of thread pools based on workload demands. This means that the number of threads can be adjusted dynamically to match the current workload, enabling efficient resource utilization and scalability. If you need to scale for concurrency, then easily borrow from a different pool. These characteristics of Apache Druid’s approach to threading are built to address the requirements of real-time analytics and data exploration use cases where high QPS is critical. The configurability in Druid also enables developers to customize their environments and scale and optimize process-specific resources and thread pools easily. Fine-Grained Data Replication per Segment Earlier we saw that Druid stores data in segments — i.e., files that are primarily partitioned by time and contain a few million rows. Druid replicates data at the segment level vs node-level (which is common in PostgreSQL, MySQL, and Oracle). Segment-level allows for fine-grained control over replication and this plays a key role in high concurrency and high availability too. With segment-level replication, the same segment can be processed concurrently on different nodes. As you can see in the picture below, Segment A is replicated on multiple data nodes. And with scatter/gather, concurrent queries for Segment A can then be executed on all of the nodes that contain that data segment, enabling better parallelization and resource utilization for better query performance and concurrency. While this is a simple view of segment-level replication, Druid scales to support very diverse analytics queries across millions of segments across thousands of nodes without a developer managing the segments or the nodes. At scale, it’s a pretty phenomenal orchestration: The creation of segments, the automatic pre-loading of data into the optimal nodes, the movement of segments for load-balancing, and concurrency to support 1000+ QPS — all done automatically. It’s like this really neat ultra-precision of data distribution meets parallelization meets hands-free developer experience. Druid Is Purpose-Built for High QPS Clearly one of the reasons why 1000s of companies turn to Apache Druid is that it is built for analytics applications with high QPS requirements. And it’s pretty unique in this respect. Recently, Imply completed (and won) a proof-of-concept (POC) at a company building a customer-facing data product. Their core application is used by over 500 million users around the world and they needed to build an application that provides platform usage-insights to customers. As it is a customer-facing application, performance is critical, and the POC required 1300 QPS on complex aggregations and filters on very large data sets. Competing with Clickhouse and Apache Pinot, Apache Druid with Imply far exceeded the expectations, delivering 1400 QPS. So if you’re building or supporting an analytics application requiring high QPS, you’ll be in good hands with Druid.

By David Wang

An Introduction to Subscription Options in MQTT

In Introduction to MQTT Publish-subscribe Pattern, we learned that we need to initiate a subscription with the server to receive corresponding messages from it. The topic filter specified when subscribing determines which topics the server will forward to us, and the subscription options allow us to customize the forwarding behavior of the server further. In this article, we will focus on exploring the available subscription options in MQTT and their usage. Subscription Options A subscription in MQTT consists of a topic filter and corresponding subscription options. So, we can set different subscription options for each subscription. MQTT 5.0 introduces four subscription options: QoS, No Local, Retain As Published, and Retain Handling. On the other hand, MQTT 3.1.1 only provides the QoS subscription option. However, the default behavior of these new subscription options in MQTT 5.0 remains consistent with MQTT 3.1.1. This makes it user-friendly if you plan to upgrade from MQTT 3.1.1 to MQTT 5.0. Now, let's explore the functions of these subscription options together. QoS QoS is the most commonly used subscription option, which represents the maximum QoS level that the server can use when sending messages to the subscriber. A client may specify a QoS level below 2 during subscription if its implementation does not support QoS 1 or 2. Additionally, if the server's maximum supported QoS level is lower than the QoS level requested by the client during the subscription, it becomes apparent that the server cannot meet the client's requirements. In such cases, the server informs the subscriber of the granted maximum QoS level through the subscription response packet (SUBACK). The subscriber can then assess whether to accept the granted QoS level and continue communication. A simple calculation formula: The maximum QoS granted by the server = min ( The maximum QoS supported by the server, The maximum QoS requested by the client ) However, the maximum QoS level requested during subscription does not restrict the QoS level used by the publishing end when sending messages. When the requested maximum QoS level during subscription is lower than the QoS level used for message publishing, the server will not ignore these messages. To maximize message delivery, it will downgrade the QoS level of these messages before forwarding. Similarly, we have a simple calculation formula: QoS in the forwarded message = min ( The original QoS of the message, The maximum QoS granted by the server ) No Local The No Local option has only two possible values, 0 and 1. A value of 1 indicates that the server must not forward the message to the client that published it, while 0 means the opposite. This option is commonly used in bridging scenarios. Bridging is essentially two MQTT Servers establishing an MQTT connection, then they subscribe to some topics from each other. The server forwards client messages to another server, which can continue forwarding them to its clients. Let's consider the most straightforward example where we assume two MQTT servers, Server A and Server B. They have subscribed to the topic # from each other. Now, Server A forwards some messages from the client to Server B, and when Server B looks for a matching subscription, Server A is there too. If Server B forwards the messages to Server A, then Server A will forward them to Server B again after receiving them, thus falling into an endless forwarding storm. However, if both Server A and Server B set the No Local option to 1 while subscribing to the topic #, this problem can be ideally avoided. Retain As Published The Retain As Published option also has two possible values, 0 and 1. Setting it to 1 means the server should preserve the Retain flag unchanged when forwarding application messages to subscribers, and setting it to 0 means that the Retain flag must be cleared. Like the No Local option, Retain As Published primarily applies in bridge scenarios. We know that when the server receives a retained message, in addition to storing it, it will also forward it to existing subscribers like a normal message, and the Retain flag of the message will be cleared when forwarding. This presents a challenge in bridge scenarios. Continuing with the previous setup, when Server A forwards a retained message to Server B, the Retain flag is cleared, causing Server B not to recognize it as a retained message and not store it. This makes retained messages unusable across bridges. In MQTT 5.0, we can let the bridged server set the “Retain” publish option to 1 when subscribing to solve this problem. Retain Handling The Retain Handling subscription option indicates to the server whether to send retained messages when a subscription is established. When a subscription is established, the retained messages matching the subscription in the server will be delivered by default. However, there are cases where a client may not want to receive retained messages. For example, if a client reuses a session during connection but cannot confirm whether the previous connection successfully created the subscription, it may attempt to subscribe again. If the subscription already exists, the retained messages may have been consumed, or the server may have cached some messages that arrived during the client's offline period. In such cases, the client may not want to receive the retained messages the server publishes. Additionally, the client may not want to receive the retained message at any time, even during the initial subscription. For example, we send the state of the switch as a retained message, but for a particular subscriber, the switch event will trigger some operations, so it is helpful not to send the retained message in this case. We can choose among these three different behaviors using Retain Handling: Setting Retain Handling to 0 means that retained messages are sent whenever a subscription is established. Setting Retain Handling to 1 means retained messages are sent only when establishing a new subscription, not a repeated one. Setting Retain Handling to 2 means no retained messages are sent when a subscription is established.

By Weihong Zhang

Is Your Internet of Things Fully Online?

If “green” was the most overused word of the 2010s, for the 2020s, surely it’s “smart.” Smartphones, smartwatches, smart homes, smart clothing, smart appliances, smart shampoo…. We made up that last one, but it wouldn’t be surprising to see it sometime. Collectively, they make up the Internet of Things (IoT) — devices connected via networks that aren’t just a browser or other apps. We created these smart devices because they make life easier. We can sit on the couch and dim the lights with our phone to start a movie, of course, but we can also make sure our doors are still locked when we’re away. These "things" have become essential to our lives, and when they aren’t working, it’s inconvenient at best and dangerous at worst. All these different things have one central feature in common: they are accessed and controlled through mobile devices, mostly our phones. And because they play such a central role in life, mobile operators are facing increased pressure to ensure there are no interruptions or lags in IoT communications. Mobile Visibility Mobile operators are accustomed to making connections like 5G available to devices, but simply having connectivity isn’t enough. Only by testing can they understand how smart devices operate under different conditions like transport mediums, infrastructure variations, and backend services. Occasional spot checks aren’t enough to ensure the user experience is where it needs to be. You need to know how far conditions can be pushed and where bottlenecks are likely to happen using real-world data. There are several key types of testing: Functionality: Each user, device, and operating system is different, so the first step is ensuring that the product works at the most basic level. Reliability: Beyond the user/device level, a test environment needs to be developed that allows you to see it functioning as part of a larger system. Security: With potentially millions of users and large amounts of data created and accessed, data privacy and authentication practices need to be validated. Because connected things produce an enormous amount of telemetry data, the sheer volume of info can hamper efforts to test network functionality. Your IoT testing strategy should include the ability to narrow down the data generated, filtering out useless information. You should also take these needs into account when developing your IoT plan. Universal applicability: Every network is unique, and with the growing complexity inherent in IT today, you need to be able to test traffic on every kind of network and device, including on-premises and cloud-based architecture. It should also integrate with the testing solutions you already have in place. Actionable insights: Information is only useful when it’s something that you can act on, and in particular, you should be able to present findings to non-technical management to inform business strategy. Your IoT testing strategy needs to deliver a simple, customizable dashboard with flexible report options. DevOps support: You should work to ensure the best business outcomes before launching a new app or service by integrating rigorous testing early in the development stage. Realistic testing: Each user represents a unique combination of factors, including geographic location, network configuration, and devices. Your testing should support the full degree of complexity that the real world will present, with highly customizable scripting options. Scalability: Sometimes pre-launch testing shows you have a viable solution, but when it’s scaled to thousands of users or more, unexpected flaws are revealed at the worst time. You should have highly scalable testing options that can script real-world user journeys for a more authentic picture of what to expect upon launch. Automation: Testing that can be done without custom coding frees up your development teams so they can work on more innovative products. You should be able to set up your testing parameters and checks and allow the testing to run with minimal human interaction. It is critical to utilize solutions and tools that help solve the complex digital performance challenges we face today. IoT testing can ensure a seamless user experience, anticipating bottlenecks and performance issues before they occur.

By Jason Haworth

Building a Product Recommendation Engine With Apache Cassandra and Apache Pulsar

The journey to implementing artificial intelligence and machine learning solutions requires solving a lot of common challenges that routinely crop up in digital systems: updating legacy systems, eliminating batch processes, and using innovative technologies that are grounded in AI/ML to improve the customer experience in ways that seemed like science fiction just a few years ago. To illustrate this evolution, let’s follow a hypothetical contractor who was hired to help implement AI/ML solutions at a big-box retailer. This is the first in a series of articles that will detail important aspects of the journey to AI/ML. The Problem It’s the first day at BigBoxCo on the “Infrastructure” team. After working through the obligatory human resources activities, I received my contractor badge and made my way over to my new workspace. After meeting the team, I was told that we have a meeting with the “Recommendations” team this morning. My system access isn’t quite working yet, so hopefully, IT will get that squared away while we’re in the meeting. In the meeting room, it’s just a few of us: my manager and two other engineers from my new team, and one engineer from the Recommendations team. We start off with some introductions and then move on to discuss an issue from the week prior. Evidently, there was some kind of overnight batch failure last week, and they’re still feeling the effects of that. It seems like the current product recommendations are driven by data collected from customer orders. With each order, there’s a new association between the products ordered, which is recorded. When customers view product pages, they can get recommendations based on how many other customers bought the current product alongside different products. The product recommendations are served to users on bigboxco.com via a microservice layer in the cloud. The microservice layer uses a local (cloud) data center deployment of Apache Cassandra to serve up the results. How the results are collected and served, though, is a different story altogether. Essentially, the results of associations between products (purchased together) are compiled during a MapReduce job. This is the batch process that failed last week. While this batch process has never been fast, it has become slower and more brittle over time. In fact, sometimes, the process takes two or even three days to run. Improving the Experience After the meeting, I checked my computer, and it looked like I could finally log in. As I’m looking around, our principal engineer (PE) comes by and introduces himself. I told him about the meeting with the Recommendations team, and he gave me a little more of the history behind the Recommendation service. It sounds like that batch process has been in place for about ten years. The engineer who designed it has moved on; not many people in the organization really understand it, and nobody wants to touch it. The other problem, I begin to explain, is that the dataset driving each recommendation is almost always a couple of days old. While this might not be a big deal in the grand scheme of things, if the recommendation data could be made more up-to-date, it would benefit the short-term promotions that marketing runs. He nods in agreement and says he’s definitely open to suggestions on improving the system. Maybe a Graph Problem? At the onset, this sounds to me like a graph problem. We have customers who log on to the site and buy products. Before that, when they look at a product or add it to the cart, we can show recommendations in the form of “Customers who bought X also bought Y.” The site has this today in that the recommendations service does exactly this: It returns the top four additional products that are frequently purchased together. But we’d have to have some way to “rank” the products because the mapping of one product to every other purchased at the same time by any of our 200 million customers is going to get big, fast. So, we can rank them by the number of times they appear in an order. A product recommendation graph showing the relationship between customers and their purchased products. After modeling this out and running it on our graph database with real volumes of data, I quickly realized that this wasn’t going to work. The traversal from one product to nearby customers to their products and computing the products that appear most takes somewhere in the neighborhood of 10 seconds. Essentially, we’ve “punted” on the two-day batch problem to have each lookup, putting the traversal latency precisely where we don’t want it: in front of the customer. But perhaps that graph model isn’t too far off from what we need to do here. In fact, the approach described above is a machine learning (ML) technique known as “collaborative filtering.” Essentially, collaborative filtering is an approach that examines the similarity of certain data objects based on activity with other users, and it enables us to make predictions based on that data. In our case, we will be implicitly collecting cart/order data from our customer base, and we will use it to make better product recommendations to increase online sales. Implementation First of all, let’s look at data collection. Adding an extra service call to the shopping “place order” function isn’t too big of a deal. In fact, it already exists; it’s just that data gets stored in a database and processed later. Make no mistake: We still want to include the batch processing. But we’ll also want to process that cart data in real-time so we can feed it right back into the online data set and use it immediately afterward. We’ll start out by putting in an event streaming solution like Apache Pulsar. That way, all new cart activity is put on a Pulsar topic, where it is consumed and sent to both the underlying batch database and to help train our real-time ML model. As for the latter, our Pulsar consumer will write to a Cassandra table (shown in Figure 2) designed simply to hold entries for each product in the order. The product then has a row for all of the other products from that and other orders: CREATE TABLE order_products_mapping ( id text, added_product_id text, cart_id uuid, qty int, PRIMARY KEY (id, added_product_id, cart_id) ) WITH CLUSTERING ORDER BY (added_product_id ASC, cart_id ASC); Augmenting an existing batch-fed recommendation system with Apache Pulsar and Apache Cassandra. We can then query this table for a particular product (“DSH915” in this example), like this: SELECT added_product_id, SUM(qty) FROm order_products_mapping WHERE id='DSH915' GROUP BY added_product_id; added_product_id | system.sum(qty) ------------------+----------------- APC30 | 7 ECJ112 | 1 LN355 | 2 LS534 | 4 RCE857 | 3 RSH2112 | 5 TSD925 | 1 (7 rows) We can then take the top four results and put them into the product recommendations table, ready for the recommendation service to query by `product_id`: SELECT * FROM product_recommendations WHERE product_id='DSH915'; product_id | tier | recommended_id | score ------------+------+----------------+------- DSH915 | 1 | APC30 | 7 DSH915 | 2 | RSH2112 | 5 DSH915 | 3 | LS534 | 4 DSH915 | 4 | RCE857 | 3 (4 rows) In this way, the new recommendation data is constantly being kept up to date. Also, all of the infrastructure assets described above are located in the local data center. Therefore, the process of pulling product relationships from an order, sending them through a Pulsar topic and processing them into recommendations stored in Cassandra happens in less than a second. With this simple data model, Cassandra is capable of serving the requested recommendations in single-digit milliseconds. Conclusions and Next Steps We’ll want to be sure to examine how our data is being written to our Cassandra tables in the long term. This way we can get ahead of potential problems related to things like unbound row growth and in-place updates. Some additional heuristic filters may be necessary to add as well, like a “do not recommend” list. This is because there are some products that our customers will buy either once or infrequently, and recommending them will only take space away from other products that they are much more likely to buy on impulse. For example, recommending a purchase of something from our appliance division such as a washing machine is not likely to yield an “impulse buy.” Another future improvement would be to implement a real-time AI/ML platform like Kaskada to handle both the product relationship streaming and to serve the recommendation data to the service directly. Fortunately, we did come up with a way to augment the existing, sluggish batch process using Pulsar to feed the cart-add events to be processed in real time. Once we get a feel for how this system performs in the long run, we should consider shutting down the legacy batch process. The PE acknowledged that we made good progress with the new solution, and, better yet, that we have also begun to lay the groundwork to eliminate some technical debt. In the end, everyone feels good about that. In an upcoming article, we’ll take a look at improving product promotions with vector searching.

By Aaron Ploetz

Natural Interactions in IoT: Combining MQTT and ChatGPT

As IoT technology advances rapidly, it becomes easier to interact with devices and among devices. However, the new challenge in the IoT field is making the interaction more natural, efficient, and smart. Advanced Large Language Models (LLMs) such as ChatGPT, GPT-3.5, and GPT-4, created by OpenAI, have gained much popularity around the world lately. This has created many opportunities for combining General Artificial Intelligence (AGI) with the IoT domain, offering promising avenues for future progress. ChatGPT is an advanced natural language processing application that can easily achieve natural conversations with humans with its excellent natural language processing skills. Message Queuing Telemetry Transport (MQTT) is the main protocol in IoT that enables real-time and efficient data transmission through lightweight and low bandwidth communication and publish/subscribe model. By combining the MQTT protocol with ChatGPT, we can envision a future where intelligent human-machine interaction in the IoT field becomes more seamless and accessible. ChatGPT enables users to control their smart home devices using natural dialogue in the smart home field, enhancing their overall living experience. In the field of industrial automation, ChatGPT aids engineers in efficiently analyzing equipment data, leading to increased productivity and effectiveness. This blog will show you how to combine the MQTT protocol with a natural language processing application like ChatGPT and give you a simple example of using them together for intelligent applications in the IoT field. Basic Concepts Before we start, let's have a quick overview of some fundamental concepts of MQTT and ChatGPT. MQTT As mentioned earlier, the MQTT protocol is a lightweight messaging protocol that uses publish/subscribe model. It is widely applied in various fields such as IoT, mobile Internet, smart hardware, Telematics, smart city, telemedicine, power, oil, and energy. The MQTT broker is the key component for connecting many IoT devices using the MQTT protocol. We will use EMQX, a highly scalable MQTT broker, in our solution to ensure efficient and reliable connection of massive IoT devices and real-time handling and delivery of message and event stream data. We can use an MQTT client to connect to the MQTT broker and communicate with IoT devices. In this blog, we use MQTTX, a cross-platform open-source MQTT client that provides desktop, command line, and web-based applications. It can test the connection with MQTT brokers and help developers quickly develop and debug MQTT services and applications. ChatGPT GPT (Generative Pre-trained Transformer) is a deep learning model that excels at text generation and understanding. ChatGPT can comprehend and produce natural language and have natural and smooth dialogues with users. We need to use the API that OpenAI offers to communicate with the GPT model to achieve ChatGPT's natural language processing skills. ChatGPT Interface Solution Design and Preparation Utilizing the functionalities of the MQTT protocol and ChatGPT, we aim to devise a solution enabling seamless integration and interoperability between the two. We will use the API that OpenAI offers to communicate with the GPT model and write a client script to achieve ChatGPT-like natural language processing functionality. The MQTT client in this script will receive the message and send it to the API, generating the natural language response. The response will be published to a specific MQTT topic to enable the interaction cycle between ChatGPT and MQTT client. We will show the interaction process between ChatGPT and MQTT protocol for message receiving, handling, and delivery through this solution. Please follow the steps below to get ready with the necessary tools and resources. Install EMQX:You can use Docker to install and launch EMQX 5.0 quickly: Python docker run -d --name emqx -p 1883:1883 -p 8083:8083 -p 8883:8883 -p 8084:8084 -p 18083:18083 emqx/emqx:latest You can also install EMQX using RPM or DEB packages besides Docker. Please see EMQX 5.0 Installation Guide for more details. Install the MQTTX desktop application:Go to the MQTTX website and choose the version that matches your OS and CPU architecture. Then, download and install it. Sign up for an OpenAI account and get an API key:Go to OpenAI and sign in or create an account. After that, click on the top right corner and choose View API Keys. Then, under the API keys section, click Create new secret key to make a new API key. Please store this key securely as you will need it for API authentication in later programs. Once you finish these steps, you will have the tools and resources to integrate the MQTT protocol with ChatGPT. For more information and learning materials on how to work with the GPT language model using OpenAI's API, you can check out the OpenAI documentation. Coding After setting up the resources and environment, we will build an MQTT client using the Node.js environment. This client will get messages from an MQTT topic, send data to the OpenAI API, and create a natural language with the GPT model. The natural language created is then published to the specific MQTT topic for integrated interaction. You can also use other programming languages like Python, Golang, etc. based on your needs and familiarity. We will use the API directly to provide a user-friendly illustration, but you can also use the official library, which offers a simpler way to use Node.js and Python. For more information please refer to OpenAI Libraries. Set up the Node.js environment. Make sure Node.js is installed (v14.0 or higher is recommended). Make a new project folder and initialize the project with the npm init command. Then, use this command to install the required dependency packages: Python npm init -y npm install axios mqtt dotenv We use axios to send HTTP requests, mqtt to connect to MQTT servers, and dotenv to load environment variables. Use environment variables. Create a file named .env and put your OpenAI API key in it: Python OPENAI_API_KEY=your_openai_api_key Code the program. Create a new index.js file where you connect to the MQTT broker, subscribe to the specific MQTT topic, and listen for messages. When a message is received, use axios to send an HTTP request to the OpenAI API, create a natural language response, and publish the response to the specific MQTT topic. The following is a list of key codes for each step for your reference: Use the MQTT library to connect to the MQTT broker and subscribe to the chatgpt/request/+ topic by default to get incoming MQTT messages: Python const host = "127.0.0.1"; const port = "1883"; const clientId = `mqtt_${Math.random().toString(16).slice(3)}`; const OPTIONS = { clientId, clean: true, connectTimeout: 4000, username: "emqx", password: "public", reconnectPeriod: 1000, }; const connectUrl = `mqtt://${host}:${port}`; const chatGPTReqTopic = "chatgpt/request/+"; const client = mqtt.connect(connectUrl, OPTIONS); Create a genText function that runs asynchronously and takes the userId parameter. Use axios to make an HTTP client instance and authenticate with the OpenAI API key in the HTTP Headers. Then, make a POST request to the OpenAI API to generate natural language replies. Use the MQTT client to publish the generated replies to a specific topic to which the user is subscribed. Store the historical messages in the Messages array. Python // Add your OpenAI API key to your environment variables in .env const OPENAI_API_KEY = process.env.OPENAI_API_KEY; let messages = []; // Store conversation history const maxMessageCount = 10; const http = axios.create({ baseURL: "https://api.openai.com/v1/chat", headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}`, }, }); const genText = async (userId) => { try { const { data } = await http.post("/completions", { model: "gpt-3.5-turbo", messages: messages[userId], temperature: 0.7, }); if (data.choices && data.choices.length > 0) { const { content } = data.choices[0].message; messages[userId].push({ role: "assistant", content: content }); if (messages[userId].length > maxMessageCount) { messages[userId].shift(); // Remove the oldest message } const replyTopic = `chatgpt/response/${userId}`; client.publish(replyTopic, content, { qos: 0, retain: false }, (error) => { if (error) { console.error(error); } }); } } catch (e) { console.log(e); } }; Finally, save received messages with the topic chatgpt/request/+ in the Messages array and call the genText function to generate and send natural language replies directly to the specific topic to which the user is subscribed. The Messages array can hold up to 10 historical messages. Python client.on("message", (topic, payload) => { // Check if the topic is not the one you're publishing to if (topic.startsWith(chatGPTReqTopicPrefix)) { const userId = topic.replace(chatGPTReqTopicPrefix, ""); messages[userId] = messages[userId] || []; messages[userId].push({ role: "user", content: payload.toString() }); if (messages[userId].length > maxMessageCount) { messages[userId].shift(); // Remove the oldest message } genText(userId); } }); Run the script: Python node index.js We have now finished the fundamental functional aspect of the demo project. Apart from providing the core functionality, the code incorporates a feature that allows users to have access isolation by appending distinct suffixes to specific topics. By preserving the history of previous messages, the GPT model can grasp the context of the conversation and generate responses that are more coherent and contextual, using information from past interactions. The full code is available on GitHub at openai-mqtt-nodejs. Alternative Solution Apart from the above example, another approach to speed up development is to use the EMQX's rule engine and Webhook from the data bridging function. EMQX enables the configuration of rules that initiate a Webhook callback when sending a message to a specific topic. We need to code a simple web service that uses the OpenAI API to work with the GPT model and return the replies created by the GPT model via HTTP. To accomplish the goal of integrated interaction, we have two options: either create a new MQTT client to publish the GPT model's replies to a specific topic, or directly employ the EMQX Publish API. Both approaches allow us to achieve the desired outcome of seamless interaction. This approach can save development costs and quickly build PoC or Demo for users with web services. It does not require an independent MQTT client and uses the EMQX rule engine to simplify the integration process and flexibly handle data. However, it still requires developing and maintaining web services, and Webhook may not be easy and convenient for complex application scenarios. Each of the solutions mentioned above has its benefits, and we can pick a more appropriate solution based on actual business requirements and developer skill level. In any case, EMQX, as the MQTT infrastructure, provides important support for system integration, enabling developers to create project prototypes and advance digital transformation quickly. Demo We can use the MQTTX desktop client to test this demo project after developing the interaction between the MQTT client and the GPT model. The user interface of MQTTX is similar to chat software, making it easier and more suitable for showing interaction with chatbots. First, we need to create a new connection in MQTTX that connects to the same MQTT server as the one used in the previous code examples, that is, 127.0.0.1 . Then, subscribe to the chatgpt/response/demo topic to receive replies and send messages to the chatgpt/request/demo topic. The demo suffix here can be changed to other strings to isolate access between users. We can test this by sending a Hello message: Next, we create some more complex demo environments. When the temperature of a sensor goes beyond a preset threshold, the ChatGPT robot will send an alert message to another MQTT topic, which is connected to a monitoring device, such as a smart watch or smart speaker. The monitoring device can use natural language technology to turn the alert information into speech so that users can receive and understand it more easily. We can also make a smart home environment that includes multiple MQTT topics that match different types of devices (such as lights, air conditioners, sounds, etc.). We will use ChatGPT to generate natural language commands for interacting with these devices in real-time through MQTT clients. Future Prospects By combining ChatGPT and MQTT protocol, you can create an intelligent IoT system with vast potential for smart homes and industrial automation. For example, you can use natural language to control your home devices, such as switches, brightness, color, and other parameters, and enjoy a more comfortable living environment. You can also use ChatGPT and MQTT to manage your industrial devices smartly and improve your manufacturing process. In the future, we can imagine ChatGPT or smarter AGI tools having more of a role in enhancing efficiency and productivity in the IoT field, such as: Message parsing: Analyze the MQTT messages, extract the relevant data, and prepare for further processing and analysis. Semantic understanding: Understand and process the meaning of the MQTT messages and extract more accurate information. Intelligent processing: Use AI technology to process the MQTT messages intelligently and help users find suitable solutions quickly. User feedback: Receive user feedback through MQTT and respond appropriately as an intelligent interaction agent. Virtual assistant: Control smart home devices through language recognition technology as a virtual assistant, providing users with smarter and more efficient services and improving the convenience and comfort of life. Conclusion This blog delves deep into the integration of MQTT and ChatGPT, revealing the exciting possibilities they offer in various applications. By utilizing EMQX, MQTTX, and the OpenAI API, we explore implementing an AI application similar to ChatGPT. Through seamless data reception and forwarding via MQTT, we successfully demonstrate the integration of MQTT and ChatGPT. As AI technology becomes more integrated into products (such as New Bing using GPT models for search engines and GitHub's Copilot), we think that the future trends of AI and IoT technologies will also involve enhancing natural language interactions, making device control smarter, and creating more novel use cases. These technologies are not yet part of the production environment but are on the horizon. In summary, integrating MQTT and ChatGPT shows a promising and exciting field that deserves more attention and research. We hope that these constantly developing innovative technologies will make the world a better place.

By Li Guowei

Building a Real-Time IoT Application With Solace Technology

As a software developer who lives with his aging parents, I was worried about their well-being and wanted to ensure that they were safe and secure at all times. So, I created an IoT application that would help me monitor their activities in real time and alert me in case of any emergency. I started by installing sensors in different parts of the house to monitor temperature, humidity, and sound levels. I then used Solace technology to build an IoT application that would use the data from these sensors to alert me in case of any abnormal activity. I connected the sensors to the Solace messaging platform, which would send messages to my smartphone whenever there was a change in the sensor data. For instance, when the temperature in the house dropped below a certain level, I received an alert on my phone, prompting me to check on my parents’ heating system. Overall, the real-time event-driven architecture (EDA) of my IoT application gave me a sense of security and peace of mind. I was able to keep an eye on my parents’ activities at all times and respond quickly to any changes or emergencies. Solace technology proved to be a reliable and efficient messaging platform that helped me build this robust and secure IoT application. In this article, I’ll provide an overview of how I used Solace technology to build this application. Overview of Event-Driven Architecture EDA is an approach to software architecture that uses events to trigger and communicate between services, applications, and systems. Events can be anything that represents a change in state, such as a sensor reading or a user action. By processing events in real time, organizations can respond quickly to changes and opportunities, enabling faster and more accurate decision-making. Why Choose Solace Technology? When I set out to create my application, I knew that I needed a messaging platform that could handle high volumes of events in real time while providing enterprise-grade security and reliability. After researching various options, I ultimately chose Solace technology for several reasons: Its ability to handle high volumes of events and distribute them to multiple consumers in real time. This was critical for my application, which involved collecting sensor data from multiple devices and processing it in real time. Enterprise-grade security and reliability were important for me because I needed to ensure that the data collected from the sensors was secure and that the messaging platform could handle the demands of a real-world application. Overall, I’m very pleased with my decision to use Solace technology, as it has provided me with the performance, security, and reliability that I needed to build a successful solution. To showcase how I developed this real-time IoT application, I will walk you through a simple application that I created to monitor temperature and humidity. For this, I used a Raspberry Pi and a DHT11 sensor. The application sends events to a Solace message broker that I set up, which then distributes these events to multiple consumers in real time. Setting up the Raspberry Pi and DHT11 Sensor Step 1: Connect the DHT11 Sensor to the Raspberry Pi GPIO Pins When I first started working on my project, I knew I needed a temperature and humidity sensor. After some research, I decided to use the DHT11 sensor because of its digital capabilities and 1-wire protocol. To connect it to my Raspberry Pi, I followed these steps: Shut down my Raspberry Pi to avoid any electrical issues during the connection process. Located the GPIO pins on my Raspberry Pi board. They were easy to find as they were located on the top of the board. Connected the VCC pin of the DHT11 sensor to the 5V pin on my Raspberry Pi, ensuring that the voltage requirements matched. Connected the GND pin of the DHT11 sensor to the GND pin on my Raspberry Pi, ensuring the connection was secure. Connected the DATA pin of the DHT11 sensor to GPIO pin 4, which was the one I had selected to use for the project, but you can use any other GPIO pin you prefer based on your project requirements. Here is the wiring diagram I used for reference: 1 DHT11 Sensor Raspberry Pi 2 VCC -> 5V. 3 GND -> GND 4 DATA -> GPIO 4 Step 2: Install the Adafruit Python DHT Library I found that the Adafruit Python DHT library is a great tool for reading data from the DHT11 sensor using Python. Installing this library was quite easy for me. I just followed these simple steps: Opened the terminal on my Raspberry Pi. Updated my package lists by running the following command: sudo apt-get update Installed the Adafruit Python DHT library by running the following command: sudo pip3 install Adafruit_DHT Note: You may need to install pip3 first if it’s not already installed on your system. You can do this by running the following command: sudo apt-get install python3-pip Step 3: Write a Python Script That Reads Temperature and Humidity Values From the Sensor Once I connected the DHT11 sensor to my Raspberry Pi and installed the Adafruit Python DHT library, I wrote a Python script that reads temperature and humidity values from the DHT11 sensor connected to GPIO pin 4: Python 1 import Adafruit_DHT 2 3 # Set the sensor type and GPIO pin number 4 sensor = Adafruit_DHT.DHT11 5 pin = 4 6 7 # Try to read the temperature and humidity from the sensor 8 humidity, temperature = Adafruit_DHT.read_retry(sensor, pin) 9 10 # Check if the temperature and humidity values were successfully read 11 if humidity is not None and temperature is not None: 12 print('{{'Temperature':{:.1f}'.format(temperature)) 13 print('{{'Humidity':{:.1f}%'.format(humidity)) 14 else: 15 print('Failed to read sensor data.') Explanation of the code: The first line imports the Adafruit_DHT library. The second and third lines set the sensor type (DHT11) and the GPIO pin number (4) that the sensor is connected to. The fourth line tries to read the temperature and humidity from the sensor using the Adafruit_DHT.read_retry() function. This function will attempt to read the sensor data multiple times if it fails the first time. The fifth and sixth lines check if the temperature and humidity values were successfully read. If they were, the values are printed to the console. If not, an error message is printed. Configuring the Solace Message Broker Step 1: Sign Up for a Solace PubSub+ Cloud Account PubSub+ Cloud is a cloud-based messaging service that provides a fully-managed message broker. To use it, you’ll need to sign up for an account. Step 2: Configure the Raspberry Pi Python Script To Connect to the Solace Message Broker Using the Client Username and Password To connect to the Solace message broker from my Raspberry Pi Python script, I used the Solace Python API. Here’s how I configured my Python script to connect to the Solace message broker: Installed the Solace Python API on my Raspberry Pi by running the following command in the terminal: sudo pip3 install solace-semp-config Created a new service by clicking on the “Create Service” button and following the instructions. Obtained the connection details (host, username, password, and port) from the Solace Cloud console. To do this, I clicked on the “Connect” tab and then clicked on my messaging service. Updated my Python script to connect to the Solace message broker using the connection details obtained in the previous step. Here’s an example Python script that connects to the Solace message broker and publishes a message: Python 1 import solace_semp_config 2 import time 3 4 # Set up the connection details 5 solace_host = "<your-solace-host>" 6 solace_username = "<your-solace-username>" 7 solace_password = "<your-solace-password>" 8 solace_port = "<your-solace-port>" 9 solace_vpn = "<your-solace-vpn>" 10 solace_topic = "<your-solace-topic>" 11 12 # Create a new Solace API client 13 client = solace_semp_config.SempClient(solace_host, solace_username, solace_password, solace_vpn, solace_port) 14 15 # Connect to the Solace message broker 16 client.connect() 17 18 # Publish a message to the Solace message broker 19 while True: 20 humidity, temperature = Adafruit_DHT.read_retry(sensor, pin) 21 if humidity is not None and temperature is not None: 22 message = '{{"temperature":{:.1f},"humidity":{:.1f}}'.format(temperature, humidity) 23 client.publish(solace_topic, message) 24 print("Published message: " + message) 25 else: 26 print("Failed to read sensor data") 27 28 # Wait for some time before publishing the next message 29 time.sleep(5) 30 31 # Disconnect from the Solace message broker 32 client.disconnect() Explanation of the code: The first line imports the Solace Python API. The next few lines set up the connection details, including the Solace host, username, password, port, VPN, and topic. The solace_semp_config.SempClient() function creates a new Solace API client using the connection details. The client.connect() function connects to the Solace message broker. The client.publish() function publishes a message to the Solace message broker. The message is a JSON object that contains the temperature and humidity values. The time.sleep() function adds a delay of 5 seconds before publishing the next message. This is done to avoid overwhelming the message broker with too many messages at once. client.disconnect(): This line of code disconnects the Solace API client from the message broker. It’s a good practice to disconnect from the broker when you’re done using it, to ensure that you don’t leave any connections open unnecessarily. Consuming the Events I created a new Python script on my laptop, which has access to the Solace message broker. To build the necessary functionalities, I imported two modules, namely paho.mqtt.client for the MQTT client and json for parsing the incoming JSON message. I defined the callback function that gets triggered every time a message is received. My callback function was designed to parse the incoming message and display it on the console. Next, I created an MQTT client instance, and I set the client ID and username/password as appropriate for my Solace Cloud account. To connect my MQTT client to the Solace message broker, I used the appropriate connection details. I subscribed to the topic that the Raspberry Pi script is publishing to. Finally, I started the MQTT client loop, which listens for incoming messages and calls the callback function whenever a message is received. Here is an example code snippet that demonstrates these steps: Python 1 import paho.mqtt.client as mqtt 2 import json 3 4 # Define the callback function 5 def on_message(client, userdata, message): 6 payload = json.loads(message.payload) 7 print("Temperature: {}°C, Humidity: {}%".format(payload["temperature"], payload["humidity"])) 8 9 # Set up the MQTT client 10 client = mqtt.Client(client_id="my-client-id") 11 client.username_pw_set(username="my-username", password="my-password") 12 13 # Connect to the Solace message broker 14 client.connect("mr-broker.messaging.solace.cloud", port=1883) 15 16 # Subscribe to the topic 17 client.subscribe("my/topic") 18 19 # Start the MQTT client loop 20 client.loop_forever() In this example, the callback function on_message is defined to parse the incoming JSON message and print the temperature and humidity values to the console. The MQTT client is then set up with the appropriate client ID, username, and password and is connected to the Solace message broker. The client subscribes to the topic that the Raspberry Pi script is publishing to, and the client loop is started to listen for incoming messages. When a message is received, the on_message callback function is called. Best Practices and Considerations for Using Solace Technology in Real-World Scenarios When using Solace technology in real-world scenarios, there are several best practices and considerations that should be taken into account. These are important to ensure that your Solace-based real-time event-driven architecture is reliable, scalable, and secure. Let’s dive into each of these best practices and their benefits. Designing for High Availability and Scalability Designing for high availability and scalability is crucial in real-world scenarios as it ensures that your application can handle a large number of events and users without experiencing downtime or performance issues. This involves setting up Solace messaging infrastructure in a clustered, highly available, and fault-tolerant configuration. By doing so, you can ensure that your messaging infrastructure is always available and can handle any load that comes its way. Increased reliability: Your messaging infrastructure will be highly available and fault-tolerant, ensuring that your application can always communicate with Solace. Scalability: The infrastructure can easily handle an increased number of events and users, so you don’t have to worry about any performance issues. Ensuring Data Security and Privacy Data security and privacy are critical in any application, and even more so in real-time event-driven architecture. It’s important to ensure that sensitive data is protected from unauthorized access and that data is transmitted securely. This can be achieved by implementing encryption, access control, and other security measures. Improved security: Protects sensitive data from unauthorized access and ensures that the communication between Solace and other components of the application is secure. Compliance: Ensuring data security and privacy helps meet compliance requirements and protects the reputation of your organization. Monitoring and Managing Performance Monitoring and managing performance are essential to ensure that your application is performing optimally and meeting the desired service level agreements (SLAs). This involves setting up monitoring and alerting mechanisms to proactively identify and address performance issues. Increased uptime: Early detection of performance issues can prevent downtime and ensure that your application is always available. Improved performance: Proactive monitoring and management can help identify bottlenecks and improve the overall performance of your application. Integrating With Other Systems and Technologies Integrating Solace with other systems and technologies is essential to ensure that your application can communicate with other components of your application ecosystem. This involves integrating Solace with other messaging systems, databases, and other components. Improved interoperability: Integrating Solace with other systems and technologies can improve the interoperability of your application ecosystem. Increased functionality: Integration with other systems and technologies can help add new functionalities and features to your application. By following these best practices and considerations, organizations can ensure that their Solace-based real-time event-driven architecture is reliable, scalable, and secure. Designing for high availability and scalability, ensuring data security and privacy, monitoring and managing performance, and integrating with other systems and technologies are crucial for the success of your application. Implementing these best practices will help you achieve your business goals and provide a better user experience. Conclusion Finally, building a real-time IoT application using Solace technology can be an exciting and rewarding experience. Whether you are using a Raspberry Pi, an app, or both, the possibilities are endless for what you can achieve with Solace. I hope that you found this guide helpful and informative, and I encourage you to continue exploring the world of IoT and Solace technology to unlock even more potential in your projects. With the right tools and approach, you can create innovative and impactful solutions that make a real difference in the world. Happy Coding!

By Kingsley Amankwah

IoT

DZone's Featured IoT Resources

Top IoT Experts

The Latest IoT Topics