To watch a video demonstration of the Time Series Data Management System, click on the video below and then the full screen icon to see it in full screen mode.
This page contains the video show notes for the demonstration of the Time Series Data Management System (TSDMS). Click on the links below to jump to a topic.
[display home page for time-series-data-management-system.com]
The Time Series Data Managemetn System or TSDMS is a software application that dramatically simplifies the management, analysis, and visualization of time series data. It is optimized to ingest, search, analyze, and visualize big data from nearly any file format and data structure. No application is faster than National Instruments DIAdem, for processing large data acquisition data files. The time series data management system has been designed to be so intuitive, you will be analyzing your data and generating reports without any need for training.
After installing the time series data management system, look for and click on the Windows menu labeled 'TSDMS'. [launch TSDMS] The first time you run the time series data management system, you will need to accept the end user license agreement.
After the application finshes loading, notice how the graphical user interface has been simplified to one panel, four toolbars, and four menus. From those few navigation elements, you may import data, search for imported data, analyze sets of files, and generate reports from multiple files.
No data files have been detected, so it will be suggested that you generate some sample data files for Bulk Import. Click the dialog's 'Yes' button. For video demonstration purposes, only three raw data files will be created. Normally, eighteen data files are written, each with slightly differrent channel content, and metadata in the filenames.
(while the three files are created...) Eighteen ATFX files will be created in one folder, and in another folder, one bus log file will be created. You should always group the files you want to Bulk Import within a folder, with all files of the same file format. The ATFX files contain a mix of channel types from the example files that come with a DIAdem installation.
After the sample raw data files have been created, you will be asked if you want to run Bulk Import to import the files. Click the 'Yes' button.
We are going to import the three raw ATFX files that were created. The folder location for the ATFX files will be the default folder Bulk Import will look for files first. After Bulk Import initializes, the 'Intro' tab provides information about organizing files for import. Click the 'Next' button.
Begin by selecting the folder with the source files to import. It will default to the folder with the raw ATFX files just created. Multiple DataPlugins exist for the .ATFX file extension. Choose the 'ATFX' DataPlugin option and then click the 'Next' button.
The 'Filename' tab allows you to parse metadata from the filenames. Choose the 'Parsing delimiter' of underscore ("_") and see the result. A dialog appears advising you to assign a property name to each column, or delete the metadata. Click 'OK' to close the dialog. Click on the 2nd column header labeled 'ClickToRenameMe2 [Text]' (first row) and a dialog will appear, allowing you to assign a property name, and a property data type. Enter 'Product' for the property name, and choose 'Text' for the data type. Click on the 3rd column header, and assign the name of 'Implement' and data type 'Text'. Click on the 4th column header, and choose 'Delete'. Click on the 5th column header, and assign the name of 'FileSequence' and 'Number'. Click on the 6th column header, and assign the name of 'DaqDevice' and 'Text'. Click the 'Next' dialog to continue to the 'Folder' tab.
The 'Folder' tab allows you to extract metadata from the path to the source folders. The table layout is inverted from the prior 'Filename' tab, but it works the same otherwise. Click on the 'Property Name' cell in row 5 and assign the name 'TestYear' and data type 'Number'. Click on the 'Property Name' cell in row 6 and assign 'TestType' and data type 'Text'. Click on the 'Property Name' cell in rwo 7 and assign 'TestLocation' and the data type 'Text'. Click on the row number for the remaining cells and then click the 'Delete selected row' button. When finished, click the 'Next' button.
You may optionally add a set of custom metadata that will apply to all of the files imported. A default set appears here because a special JSON text file was written when the raw files were created. Any metadata created here is written to a JSON file in the same location as the source data. You may later copy this file to another source folder, edit it, and use it again if you want. This file could contain an extensive set of test conditions and operating conditions. We don't need to add anymore metadata, so click the 'Next' button to continue.
On the 'Time Zone' tab, you define the time zone reference for the data sets being imported. If you don't know the time zone settings, then leave it set to the first option, otherwise make a time zone selection. Click the 'OK' button to begin importing the data.
Each file will be imported by a new instance of DIAdem running in parallel with the current instance of DIAdem. The metadata specified during Bulk Import will be indexed by 'My DataFinder', and then system will be checked to see if any duplicate files exist based on a hash calculated from the raw source files. If duplicates are found, you will be presented with information about the duplicates, and then asked to decide what do do about them. When all import activities are complete, a search is performed, looking for a special hash named 'ImportBatchHash' for the data set imported, causing those files to be returned as the search results. Scrolling the search results from left to right, you can see the metadata added by Bulk Import, and how some items such as 'Product', 'Implement', and 'FileSequence' vary by the content within the filename. The imported files are now categoriezed in a way that allows you to easily differentiate them from other imported data sets, based on this metadata. This concludes the Bulk Import activity.
Exit the time series data management system by choosing the menu 'File' and 'Exit TSDMS'. This will cause the current 'My DataFinder' settings to be saved, and the prior settings to be restored. I am now running DIAdem using its Windows menu icon. Notice that all of my 'My DataFinder' search areas exist as they did previously. Close DIAdem. This concludes this video tutorial.
This tutorial assumes that the prior 'Bulk Import' tutorial activity has been performed on your time series data management system installation. Launch the time series data management system by clicking on the Windows menu icon labeled 'TSDMS'. Whenever you start the application, a search for all imported files will be performed.
We will perform a search based on the metadata added to the imported data. Note that DIAdem calls metadata added to files 'properties'. To begin a search, choose the menu option 'Search' and 'Find by Metadata...', or simply click on the 'Search for data' toolbar button. A dialog named 'Search Files By Metadata' appears. The drop down box 'Property' will be populated with all indexed and optimized file properties. Choose the property named 'ImportBatchHash'. The 'Operator' drop down box is automatically set to the option of '='. The 'Value' drop down box provides a list of values from the index, plus the wildcard asterisk. Choose the hash value. Click the 'Add' button to add this search criteria. Observe that this search criteria has been added to the table in the dialog, and the NAVIGATOR search has been updated with the search criteria and results. At the bottom of the dialog, you can see that 3 files were found for the search conditions.
You may continue to add as many additional search conditions as necessary in order to define the search result desired. You can start all over by clicking the 'Find All' button. Or you may abort and restore any prior search by clicking the 'Cancel' button. Clicking the 'Done' button will close the dialog, leaving the search conditions you built in the NAVIGATOR search window. [click the 'Done' button].
This concludes the search by metadata tutorial.
This tutorial assumes you have previously performed the 'Search By Metadata' tutorial, and the resulting NAVIGATOR search results exist.
Launch the Bulk Analysis dialog by clicking on the toolbar button labeled 'Perform analysis of all files in the search results (Bulk Analysis)...'. Note that the 'Bulk Analysis' dialog will state at the top that '3 files from the Navigator search results will be included in the analysis to be performed'. Additionally, the Data Portal will be visible, and the first file in the search results has been loaded. Click on the channel 'NumericLinear/Speed' and then drag and drop it to the 'Bulk Analysis' dialog 'Y Channel' channel control. Notice that the channel 'Time' will be automatically assigned as the 'X Channel'. Additionally, in the 'Y Stats' tab on the right, the button labeled '0 of 29 statistics...' will be enabled. Click on that button and then check the checkboxes 'Minimum', 'Maximum', and 'Arithmetic Mean'. Click the 'Done' button when finshed. The button label will update to indicate that 3 out of 29 statistics have been selected for analysis.
At the bottom right of the dialog, the button labeled 'Analyze 3 files' is now enabled. Click that button, and the three descriptive statistics will be calculated for each of the three files in the search results. When analyis is complete, a dialog will appear to advise you. The 'Input Channels' in the dialog will be cleared, allowing you to select another Y Channel for analysis.
Expand the channel group '3D_XYZ'. Select the channel 'Y_channel' and then drag and drop it to the Bulk Analysis dialog 'Y Channel' channel control. Note that the X channel, and 20 Z channels will be automatically recognized and assigned to the X and Z channels. This happens because during Bulk Import, that set of channels was recoginized as channels that below to a 3D data structure. Click on the 'Z Status' tab. Check the checkboxes 'Z channel minimum' and 'Z channel maximum'. Click the 'Analyze 3 Files' button. Bulk Analysis will perform 3D statistical analysis on the Z channels, identifying the minimum and maximum values, and where those values occur. When the analysis is complete, the dialog will clear the 'Input Channels' channel controls. Click the 'Close' dialog to quit.
Click on the third file in the search results, and then right click and choose 'Load Data' from the pop up menu. In the Data Portal, click on the channel 'NumericLinear/Speed'. Under it's custom properties, expand the 'Result' property and notice that the minimum, maximum, and arithmetic mean values for the channel have been added as channel properties. Expand the '3D_XYZ' channel group. Click on the first Z channel named 'Z_channel_1', and inspect the custom properties. 'Under 'Result~MaxExtremeValues' will will see the minimum and maximum Z channel values, and the location where those occur.
You can see from this tutorial that you can easily perform statistical analysis of channels across many files with a few dialog inputs. This completes the video tutorial for Bulk Analysis.
This tutorial assumes you have previously performed the 'Search By Metadata' tutorial, and the resulting NAVIGATOR search results exist. It also assumes that the 'Bulk Analysis' tutorial has been performed, calculating descriptive statistics for two channels in every search result (file).
Start 'Bulk Report' by clicking on NAVIGATOR panel button, and then the toolbar labeled 'Generate a report for all files in the search results (Bulk Report).
You may generate two types of Bulk Reports. This dialog allows you to select which of the two Bulk Reports to run. The first type is called 'Single Page Per File', and the second is 'Data Set Summary'. Under those two radio button choices, you can click on buttons to see PDF file examples of a 2D and 3D report for each type. This helps you to quickly visualize what the result will be for each report type.
Select the option 'Single Page Per File', and then click the 'OK' button. the dialog 'Bulk Report - Single Page Per File' will load. The first file in the NAVIGATOR search results will be loaded into the Data Portal. A default PDF file for output has been automatically assigned. You may change this PDF file assignment if desired. The next step is to assign a Report layout to be used. To the right of 'Report layout', click on the '...' button, and a dialog appears, showing you a set of pre-defined report layouts. Choose the template named 'BulkReport_SPPF_portrait_DEMO_3x2Dcht_1x3Dcht.TDR'. This template contains one 2D chart with 3x curves, two 2D charts with 1x curves, and one 3D chart. The REPORT panel behind the dialog will be activated, and the selected report template will be loaded. In the 'Charts' tab, the list box under 'Report Charts' will list the four charts by object name. Click on the first chart named 'Cht2DObj1'. Below the 'Report Charts' list box, a combo box labled '2D chart 'Cht2DObj1' Curve No. ' will be enabled. Choose curve #1 from that combo box. Under 'Channel assignments...' the 'Y Channel' channel control will be enabled. Access the Data Portal and drag and drop the channel 'Speed' into the dialog 'Y Channel' channel control. Under the 'Channel Analysis Results', the button will be enabled, and labeled 0 items of 3 included...'. Click on this button and check the checkbox 'Maximum'. This will cause the 'Maximum' value to be added to the 'CommentsLeft' label in the report when the report is compiled. If you look at the report layout, you will see the 'Speed' channel has been assigned to the curve in the chart.
From the curve drop down box, choose curve # 2. Drag and drop the 'NumericLinear/RPM' channel from the Data Portal to the dialog 'Y Channel' channel control.
From the curve drop down box, choose curve # 3. Drag and drop the 'NumericLinear/Torque' channel from the Data Portal to the dialog 'Y Channel' channel control.
From the curve drop down box, choose curve # 4. Drag and drop the 'NumericLinear/Power' channel from the Data Portal to the dialog 'Y Channel' channel control. At this point, the first chart in the top left of the report layout has four channels assigned from the channel group 'NumericLinear'.
Select the report chart named 'Cht2DObj2'. In the Data Portal, expand the channel group named 'Spikes', and then drag and drop the 'NitrogenOxide' channel to the dialog 'Y Channe' channel control.
Select the report chart named 'Cht2DObj3'. In the Data Portal, expand the channel group named 'Spikes', and then drag and drop the 'NoiseVoltage' channel to the dialog 'Y Channe' channel control.
Select the report chart named 'Cht2DObj4'. In the Data Portal, try dropping the 2D channel 'Spikes/RPM' into the 'Y Channel' channel control. The operation will not be permitted. This is because that chart is for 3D data, and you tried to assign 2D data. In the Data Portal, expand the channel group named '3D_XYZ', and then drag and drop the 'Y_channel' channel to the dialog 'Y Channe' channel control. In the dialog, across from the 'Z Channel', click the corresponding button under 'Channel Analysis Results'. Check the items 'MatMax, 'MatMaxCol', and 'MatMaxRow' to include those results in the 'CommentsRight' label in the report layout.
At this point, all of the charts in the report layout have channel assignments, and the statistical results from two channels will be added to the report. It is now time to define the labels for the report layout. Click the 'Edit Report Labels' button at the bottom left of the dialog. In the row 'Header Left', choose the 'Source file...' option from the drop down box. For 'Footer Middle', assign 'Page # of #'. For 'Footer Right', assign 'Report DD MMM YYYY'. For the 'Sub Title', enter the text '2021 Field Test in CA'. And finally, for 'Title', enter 'NH 25S Tractor'. Note that the 'Comments Left' and 'Comments Right' are disabled, but annotated with 'Analysis~Statistics', indicating they will be populated with channel results from Bulk Analysis. Click the button 'Start Bulk Report Generation'.
Bulk Report will first add instructions defined from the dialog to each file, and then the report will be generated. After the PDF has been written, Bulk Report will attempt to load the file with the default PDF viewer assigned to your Windows system.
Reviewing the report, you can see the header and footer has been populated, and the 'Comments Left' and 'Comments Right' have channel results from the prior Bulk Analysis. This report only contains results for three files, but it could just as easily consist of results from thousands of files.
As you can see, the results from each search result file are assigned to the charts and text objects within a single report page. But sometimes you might only want to calculate the maximum channel value across all of the search results (files). This can be accomplished with the second type of Bulk Report named 'Data Set Summary'.
Close the PDF file.
The online documentation of the time series data management system includes detailed instructions on how you can customize the report layouts, adding your own company logo, changing the color, font, etc. This concludes the video tutorial for the Single Page Per File Bulk Report.
Start 'Bulk Report' by clicking on NAVIGATOR panel button, and then the toolbar labeled 'Generate a report for all files in the search results (Bulk Report). In the dialog 'Bulk Report' that appears, choose the 'Data Set Summary radio button option, and then click the 'OK' button.
The REPORT panel will be activated, and the default report layout will be loaded into a REPORT worksheet. The only options for the report layout is portrait and landscape orientation. A default filename and location is also assigned for the output PDF file.
In the list box labeled 'Channel Statistical Results', you will see a list consisting of channel group / channel names followed by the statistical result that exists for that channel. Click on the 'Channel Statistical Results' item 'Numeric/Speed | Result~Statistics~ExtremeValues~Maximum' and notice that the list box to the right is enabled. In the list box 'Statistic To Calculate Across Results Per File' choose the 'Maximum' option. you can choose any descriptive statistic to calculate on all of the channel 'NumericLinear/Speed' channel maximum results. We want the maximum of the maximum, so choose 'Maximum' from the list box on the right, and then click the button 'Add To Report'.
Look at the Data Portal. The maximum value across all of the files has been calculated and the result added to the Data Portal. When the report is compiled, the results in the Data Portal will be used to generate the report. The report could be generated at this point, but we will add one more calculation.
From the list under 'Channel Statistical Results', click on '3D_XYZ/Z_Channel_1 | Result~MaxExtremeValues~MatMax'. Then from the list on the right labeled 'Statistic To Calculate Across Results Per File' choose 'Maximum'. Click the button 'Add To Report'. Wben that action is complete, click on the button 'Edit Report Labels'.
Look at the possible labels for 'Header Left' The list is extracted from the metadata from all of the search results, and only the items that are the same across all of the files are listed. Choose the option 'Product: NH 25S Sub-Compact WORKMASTER Tractor'. For the 'Footer Middle', choose the drop down list option of 'Page # of #'.
Notice that the 'Title', 'Sub Title' and 'Comments Left' labels are disabled. They will be automatically updated with the channel name and statistical result name. The statistic calculated across all of the search results will appear in the 'Comments Left' text box in the report layout.
Click the button 'Start Bulk Report Generation'. When Bulk Report is finished, the PDF file generated will be automatically opened. The first two pages will each have a chart of the maximum across all files for either the channel 'Speed', or 'Z_channel_1'. Note that the X-Axis for each chart is labeled 'SourceFileHash'. If you review the last page in the report, the 'SourceFileHash' value is mapped out against the full filename from the NAVIGATOR search results. Using the 'SourceFileHash' property keeps the X-Axis label length managable. You can visually see in each chart the maximum value for each file from the search results.
The online documentation of the time series data management system includes detailed instructions on how you can customize the Data Set Summary Report, adding your own company logo, changing the color, font, etc. This concludes the video demonstration of the Data Set Summary Bulk Report.
In this tutoral, you have been introduced to the 'Base Edition' capabilities of the time series data management system application. Details on how to use all of the Base Edition features can be found online in the documentation section of the website.
The Advanced Edition will contain more extensive data analysis and channel manipulation features. See the website at www.time-series-data-management-system.com for a mapping of features by edition.