reliability studies

Because of the telemetry available from modern equipment, the amount of well-organized data we have available to us is orders of magnitude beyond what we had just a few years back. It is very common now to have access to literally 1000’s of devices, each sending out data payloads with many data elements, at sub-second intervals. There can easily be terabytes of data available for us to analyze and then make decisions with. With so much data available, how best to ingest and use it for making near-real time decisions?