Building structured data storage strategy for Things data

Building structured data storage strategy for Things data

IOT data packets are generally dependent on the sensors, devices ,gateways, systems that generate them. Also this data can be structured or unstructured.

For the sake of this post lets talk about structured data and challenges to build a mechanism of storing this data into database.

As we all know Java Script Object Notation or JSON (as it is popularly referred to). is the preferred mechanism of data exchange. It is simple and easy to understand and can easily represent complex data structures with ease.
JSON can be nested and multilayered depending on the data. This is where the challenge begins.
Consider a scenario where a gateway is registered and activated on RubiThings. This gateway can be monitoring diverse IOT devices , sensors or even PLCs

The data generated by each device or group of devices or sensors will be diverse. Similarly a typical PLC has 250 tags and any combination of those can be reporting at different frequency.

From a storage perspective RubiThings also needs to be able to handle all these data packets and store the data into a relational database for easy retrieval , grouping , aggregation and reporting.

Thus any payload JSON received from the gateway has to be normalized and made ready to be stored into the data base in such a fashion as to facilitate easy retrieval when invoked from internal / external system for consumption.
What this ensures is fast and easy access

Upside
Using this technique will ensure fast and easy retrieval , ease of sorting and ordering.

Downside
The more complex the JSON the more columns in the table.
In case there are nested lists then lot of repeated rows created with null values

    • Recent Topics

    • I have a doubt

      What are the differences between supervised and unsupervised learning?

    • Building structured data storage strategy for Things data

      IOT data packets are generally dependent on the sensors, devices ,gateways, systems that generate them. Also this data can be structured or unstructured. For the sake of this post lets talk about structured data and challenges to build a mechanism of

    • About the RubiStudio category

      This category consists of all topic discussions related to data preparation & modeling, machine learning, forecasting, textual analytics, and Pro Code possible with RubiStudio.

    • Quick Use of Data Dictionary and Workflow

      Data Dictionary: With the use of pre-existing datasets, a new data dictionary can be produced. It enables you to create new datasets while reusing the existing ones and incorporating new features. By leveraging the preexisting datasets, the Data Dictionary

    • Flow of Information using Sankey Chart in RubiSight

      A Sankey chart, also known as a Sankey diagram or flow diagram, is a type of data visualization that represents the flow of resources, energy, or other quantities between multiple entities or categories. It is often used to visualize the distribution

    • Securing Your Data at Every Level: Introducing Rubisight’s Data Level Security

      In the digital age, data governance and security have emerged as paramount concerns for organizations across industries. With the proliferation of data breaches and regulatory requirements, safeguarding sensitive information and ensuring compliance has

    • Infographics and Specialized Widgets: When to Use, Best Practices, and Limitations

      RubiSight widgets, also known as charts, are the building blocks of visual data storytelling dashboards within the Rubiscape platform. They are essentially visual representations of your data that help you explore, understand, and communicate insights

    • How to include drill-down to details option?

      Question - For example, I have 10 records. 7 of which are ok, 3 are not ok. On dashboard I want to show count and aggregated values for 7 and 3 respectively. And details of 3 or 7 records should be available on user demand. There are two ways to drill

    • Use of HTML for Bookmarking in Rubisight

      What is bookmarking? How do we achieve it using Power BI? Bookmarking is a common feature in various data visualization tools, and it generally refers to the ability to save and revisit a specific state or view of a report/dashboard. This feature allows

    • Rubisight Filters Explained: Finding Insights Faster

      Rubisight filters help you focus on specific data within your dashboard, making it easier to spot trends and gain valuable insights. Here are the three main types: 1. Global Filters: Apply the same filter criteria to all pages on a dashboard. Useful for

    • Banking Analytics: Reimagining the Way Banks Do Business

      Created a Comprehensive dashboard on Banking Analytics. With banking products becoming increasingly commoditized, Analytics can help banks differentiate themselves and gain a competitive edge. This dashboard showcase trends that help management in decision

    • Rubisight Overview - Key Functionalities

      Find the dashboard used in Rubisight Overview Session. This dashboard demonstrated most of the functionalities in Rubisight platform. The funcationalities used in the dashboard are as follows - Data Dictionary used for Rubisight dashboard creation. Calculated

    • LeBron James vs. Michael Jordan: The Greatest Debate in Basketball History

      My new Viz on Rubisight Shapes - This entire dashboard is prepared based on shapes and filters. The debate over who is the greatest basketball player of all time, LeBron James or Michael Jordan, has captivated fans and analysts for years. Both athletes

    • Outlier Detection

      Outlier Detection Description Outlier Detection reveals the extreme values that deviate from the rest of the data in a real-world dataset. Why to use Numerical Analysis – Data Preparation When to use When there are certain values in the data which significantly

    • Hardik vs Rohit - An Inhouse Rivalry!

      The Big Debate - Hardik vs Rohit ---- Mumbai Indians are already out of the IPL and whose fault !!! Is it Hardik ??? The news sources are debating on the two former Indian Player, is it a correct debate ? image723×712 181 KB Find the stats in Rubisight

    • How to convert the data type for a specific data point in Rubiflow?

      In Rubiflow once the data node is processed, you can use an expression from the Data preparaion function (Data integration) and connect to the data source node. In that you will be able to see a functions on the left side - “Convert”, through this functionality

    • Welcome to Community

      The Community hub inside Zoho Desk lets you build a powerful community around your business, while still serving customers via other channels. Your customers can post 4 types of content within your community. These are Ideas, Questions, Problems and Discussions.