There was a time when the R&D powertrain group at Jaguar Land Rover (JLR) would log up to 500 GB of data per day. How do you sift through such a large number of files? Anjelica Warren, Product Marketing Manager at National Instruments, reviews ways to spend less time analyzing data and creating reports.
A Tech Briefs reader asks:
What are some ways to spend less time analyzing data and creating reports?
Anjelica Warren: Whenever you can, there's always a great benefit in being able to analyze your data inline. That way, once your data is saved to disk, you have captured only the most important information that you need.
But in situations where you cannot do that, there are some ways that you can make your post-processing more efficient. First, make sure you have chosen an analysis tool that includes the built-in functionality that you use most frequently. Having a way to automate or configure a routine of analysis is something that can help save you time in the analysis, instead of doing it manually by hand.
Jaguar Land Rover was using manual analysis before they implemented a solution; they found that, after they created an automated analysis and reporting scheme, they could implement the same post-processing on the order of 20x less time.
Additionally, you want to look for the ability to include custom algorithms. If there is analysis that you've already created, looking for open solutions that give you the ability to take advantage of things you've already developed can also save you time.
What do you think? What are your “Big Data” best practices?
Share your comments and questions below.
This question was part of our Tech Briefs webcast presentation: Is Your Data Acquisition System Prepared for Big Data?