PQio   The PQDS File Specification

The PQio application provides a quick means to convert PQDIF files to the PQDS.csv format so that event waveform data can be easily reviewed, obfuscated or edited if necessary, and shared with colleagues.

The Power Quality Data Sharing (PQDS) format prescribes the way to structure a CSV file to exchange PQ data -- both waveform and trending data -- in a simple, human readable form. In many GPA applications, like openSEE and TrenDAP, when clicking on the "Export as CSV" button the CSV file produced will comply with the PQDS format standard so that this file can be parsed and used by PQDS compatible systems.

Version 1.2 of this specification is now available in pre-release for review.

  1. Introduction
  2. PQDS Metadata
  3. PQDS Measurement Values
  4. Expected PQDS Parser Behavior
  5. PQDS file naming convention
  6. Example PQDS.csv File
  7. PQDS Log Files
  8. metadata PQDS Codes Descriptions
  9. References

Version 1.1

Version 1.1 - February 2021 -- This version of the PQDS file standard includes changes that enable trending, or interval, data to be included in PQDS files.

1. Introduction

The Power Quality Data Sharing (PQDS) format is a comma-separated file that facilitates the convenient review, editing and sharing of Power Quality (PQ) data -- i.e., disturbance waveforms and trending data values. PQDS is a simple human-to-human format that complements the standard system-to-system data formats of PQDIF and COMTRADE.

The Comma-Separated Value (CSV) file format used by PQDS is supported by almost all spreadsheets and database management systems as well as by standard libraries that are available in most programming languages. In addition, many software applications have the ability to split files based on comma separated values. [1]

The PQDS file format contains two sections -- (1) a metadata section with key-value pair information that describe the measurement values, and (2) a section for measurement values themselves. The measurement values section can contain either time-series voltage and current waveform measurements or time series trending measurements consisting of minimum, maximum and average values over an interval, e.g., 10 minutes. Each of these two sections individually comply with the RFC 4180 CSV format specification with header data to identify the names of the fields that follow. Parsers for PQDS files will expect these files to comply with the seven RFC 4180 rules for CSV formatted files. [2]

The PQDS file format and tools to convert PQDIF and COMTRADE waveform data to PQDS has been developed by the Electric Power Research Institute (EPRI) to support utility disturbance monitoring research. The EPRI PQio program is building a library of waveforms that can be used to evaluate and test automated waveform analytics -- including analytics involving artificial intelligence (AI) techniques.

1.1 Simplicity

All metadata and measurement values included within PQDS files are in standard engineering units without the need for subsequent scaling or adjustment.
Scientific notation for values is not allowed. The time associated with a measurement is in decimal seconds.

If included, the optional PQDS metadata can be used to provide details about a single event in the waveform or general information about the dataset. While complex multi-event cases can be be included in the measurement data itself, no provisions are made for describing multiple events in the PQDS metadata section.

1.2 Extensibility

Users and groups of users can easily extend both the PQDS metadata as well as PQDS time-series waveform data measurement values with low risk of breaking any automation that may have been implemented by other users who have implemented different extensions.

1.3 Shareability

PQDS format is designed to make it easier for the data-owner to improve productivity through collaboration with others -- the business-basis for development of the PQDS format.

  • As a human-readable format with easy-to-use spreadsheet tools for review, there is no ambiguity on the data that is being shared.
  • The ability to selectivity reduce, anatomize or correct the metadata associated with the waveform allows waveform data sharing to align with business confidentiality, security and regulatory requirements on a case-by-case basis.
  • The data is provided to the recipient in the flexible, ubiquitous CSV format so that it can be quickly and easily used without the need for specialized tools.
  • The PQDS format is sufficiently well-structured that it can be used as a format for the automated exchange of waveform data such as can be done through spreadsheet macros at the user-level or through automated programming interfaces. For example, the csv-based format of PQDS enables power quality data to be ported to machine learning tools and other statistical analytic tools that are now commonly available and expect data in this form.

The PQDS format is designed to facilitate time-series measurement value sharing in all its forms -- for example, as waveform data may need to be shared in full detail among work colleagues, or as data may need to be contributed to repositories for power system phenomenon research and analytics improvement. The PQDS supports including (or not including) just the waveform data that is needed for an individual use case. If necessarily, the waveform data itself can also be easily trimmed down to just to portion of the event that is of interest.

2. PQDS Metadata

The PQDS metadata section begins with a header record than contains the following field names:

Column Field Name Requirements
A key Not case sensitive. Note 1
B value The information associated with the key Note 2
C value-units The SI units for numeric values (if any) Note 1
D expected-data-type The expected metadata value type that can be used for validation Note 3
E note Any other descriptive information about the value Note 2
  1. The "key" and "value-units" fields must not exceed 32 characters, must begin with a letter can only contain letters, numerals, the hyphen, underscore and dot characters. Key and value-units fields may be encapsulated in double quotes. The key "waveform-data" is reserved and cannot be used.
  2. "Value" must not exceed 254 characters and "note" must not exceed 4096 characters and both are limited to standard ASCII characters (32 thru 126). Value and note fields must be encapsulated in double-quotes to avoid truncation at the first comma. Value fields that are intended to parsed as numbers should not contain commas -- even if these are wrapped in quotes.
  3. Expected data types are (The data-type field is optional with the data-type of "Text" assumed.):
    • Enumeration (E): zero or positive integer less than 32,768
    • Numeric (N): digits, decimal point and leading sign with a total length less than 16 numerals
    • Alpha-numeric (A): letters, numbers, dot, hyphen and underscore with a total length less than 255 characters
    • Text (T): any ASCII character (32 through 126). Total length limited to 4096 characters.
    • Binary (B): the numeral 0 or 1

Importantly, the entire metadata section of the PQDS.csv file is NOT required and may be omitted entirely without impact to processing the waveform or trending data.

2.1 Reserved PQDS Metadata keys

The following key names are reserved to facilitate use of PQDS to support automated data sharing. Users are welcome to add keys as they may be needed.

2.1.1 metadata about device making the measurements:

metadata Key Description PQDS Data-type
DeviceName Meter or measurement device name T
DeviceAlias Alternate meter or measurement device name T
DeviceLocation Meter or device location name T
DeviceLocationAlias Alternate meter or device location name T
DeviceLatitude Latitude T (35.041058)
DeviceLongitude Longitude T (-85.315074)
AccountName Name of customer or account T
AccountNameAlias Alternate customer or account name T
DeviceDistanceToXFMR The distance (feet) to the upstream transformer N
DeviceConnectionTypeCode The PQDS code for meter connection type E (0-5)
DeviceNotes Description of the device T
DeviceOwner The utility name T

2.1.2 metadata about the electrical asset or load being monitored:

metadata Key Description PQDS Data-type
AssetName Name of the monitored asset T
NominalVoltage-LG Volts - line-to-ground N
NominalFrequency Hertz N (60 default)
UpstreamXFMR-kVA Upstream transformer size in kVA N
LineLength Length of line in units specified (default miles) N

2.1.3 metadata about the nature of the disturbance in the waveform:

Note that these fields are primarily associated with disturbance data and might not apply in the trending-data case.

metadata Key Description PQDS Data-type
EventGUID Globally Unique Event Identifier A (e.g., d571d945-e950-41b5-a3ef-960c410c395e)
EventID A user defined ID for this event A
EventYear Year E (4 digit, e.g., 2019)
EventMonth Month E (1-12)
EventDayofMonth Day E (1-31)
EventHour Hour E (0-23)
EventMinute Minute E (0 to 59)
EventSecond Second E (0 to 59)
EventNanoSeconds Nanosecond N (0 to 999999999)
EventDate Event Date T (e.g., 01/12/2019)
EventTime Event Time T (e.g., 10:09:23.999999999)
EventTimeZone Event Time Zone T (UTC, ET, CT, MT, PT, AT, HT)
EventTypeCode PQDS Event Type Code E (0-5)
EventFaultTypeCode PQDS Fault Type Code E (0 - x)
EventPeakCurrent Amperes N
EventPeakVoltage Volts N
EventMaxVA RMS Maximum A-Phase Volts N
EventMaxVB RMS Maximum B-Phase Volts N
EventMaxVC RMS Maximum C-Phase Volts N
EventMinVA RMS Minimum A-Phase Volts N
EventMinVB RMS Minimum B-Phase Volts N
EventMinVC RMS Maximum C-Phase Volts N
EventMaxIA RMS Maximum A-Phase Amperes N
EventMaxIB RMS Maximum B-Phase Amperes N
EventMaxIC RMS Maximum C-Phase Amperes N
EventFaultI2T Amperes^2 (for fault duration) N
EventPreEventCurrent Amperes (RMS) N
EventPreEventVoltage Volts (RMS) N
EventDuration Decimal seconds N (0-999.999999999)
EventCauseCode PQDS Event Cause Code E (0-12)
EventNotes Description of the event T
EventCauseNotes Description of the event cause T
DistanceToFault Distance from the meter to the fault N

2.1.4 Metadata about the nature of the trending data in the file

Note that these fields are primarily associated with tredning data and might not apply in the disturbance data case.

metadata Key Description PQDS Data-type
StartDate Start Date for trending Data T (in the format "DD-MM-YYYY")

2.1.5 Metadata about the measurements in the time-series data

metadata Key Description PQDS Data-type
WaveformDataTypeFlag 0 = point-on-wave measurements; 1 = RMS measurements B (0 or 1)

2.1.6 Metadata about use restrictions or information classification of the waveform

metadata Key Description PQDS Data-type
WaveformSensitivityCode The PQDS Info Classification Code E (0-3)
WaveformSensitivityNotes Description of waveform sensitivity T

2.2 Adding New Keys -- User Extensions to metadata

Additional keys and values can be added to the metadata section as needed. As these new keys are created the pattern of "domain"."key" should be followed with the "domain" typically being a corporate or organizational abbreviation. This approach avoids key conflicts as metadata is independently extended by multiple parties. e.g., extended metadata key = "EPRI.SequenceID"

3. PQDS Measurement Values

The measurement values section in the PQDS file follows the optional PQDS metadata section. There are two measurement data formats -- one for disturbance waveform values one for trending data (min, max, avg) values, In both cases, the first line after the PQDS metadata section is a header record (or line) which describes the type of data that follows ("waveform" or "trending") as well as the labels for the measurement values.

The following two sections describe each format in more detail.

3.1 Waveform Values

The header record (line) for waveform-data is comprised of the following fields and MUST begin with the string "waveform-data" (not case sensitive).

Column Field Name Requirements
A waveform section flag T (MUST be set to the string "waveform-data") Note 5
B thu J measurement T (MUST one of nine measurement names) Note 6
  1. Column A (the first value) contains time data, or time-stamp, in decimal seconds and should not exceed 999 seconds or 12 decimal places of precision (picoseconds). Optionally, negative time can be shown with time zero representing the trigger point for the disturbance.
  2. Valid values for measurement names to be included in the waveform data header are "va", "vb", "vc", "vn", "ia", "ib", "ic", "in", "f". These names are not case sensitive and can be in any order. Field names and their respective data can be omitted for data that is not available.

3.1.1 Adding New Measurements / User Extensions to Waveform Data

Additional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 6 above.

The header record (line) for trending, or trending, data is comprised of the following fields and MUST begin with the string "trending-data" (not case sensitive).

Column Field Name Requirements
A waveform section flag T (MUST be set to the string "trending-data") Note 7
B thu L measurement T (MUST one of eleven measurement names) Note 8
  1. Column A (the first value) contains time data, or time-stamp, in "HH:MM" format. >>HH<< is the the 2 digit hour in 24H format (00-23) and >>MM<< is the 2 digit minute (00-59). For trending data spanning a single day the date is stored in the "StartDate" field in the metadata section.
  2. Valid values for measurement names to be included in the waveform data header are "va", "vb", "vc", "vn", "ia", "ib", "ic", "in", "f", "vthd", "ithd", "". These names are not case sensitive and can be in any order. Field names and their respective data can be omitted for data that is not available.

The mesurement data following the header record consists of the time-stamp as described in Note 7 followed by a tripplet of values for each measurement. These values are seperated by >>,<< and are always ordered as minimum, average, maximum.

Since trending values are typically recorded over relatively long intervals (e.g., 10 minutes), a common use case for PQDS will be to share trending data for a long periods -- many days. In this case an additional day-header is inserted to define the start of a new day. This header can be skipped for the first day (Day 0)

Column Field Name Requirements
A DayCount N (MUST be set to the day follwing this header) Note 9
B Date T (MUST be in the Date Format described in Note 10)
C Spacing T (MUST be a set of 20 "#")
  1. Column A (the first value) represents the contious count of days starting with Day 0 which is the Date stored in the "StartDate" field in the metadata section.
  2. Column B represents the date of the day following this header. This is optional but if supplied should be in the following format "DD-MM-YYYY", where DD is the 2 digit day of the month, MM is the 2 digit month of the year and YYYY is the 4 digit Year.

3.1.2 Adding New Measurements -- User Extensions to Waveform Data

Additional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 8 above.

4. Expected PQDS Parser Behavior

If the optional PQDS metadata section is included in the PQDS file, it must begin with the three header field names of "key", "value", "value-units" (not case sensitive).

metadata keys do not need to appear in any particular order. Empty lines in the metadata section will be ignored. All fields will be ignored for invalid keys (e.g., one containing special characters or with a length greater than 32 characters). Valid keys will contain 'null' for other fields that are invalid. Other fields may be added to each metadata line (metadata record), but will not be parsed.

PQDS parsers are expected to validate metadata values based on the data-type rules.

4.1 Including Embedded Quotes and Commas

Parsers of PQDS should be expected to behave like Excel when handling strings that include commas and quotes. For example the string: >> "I find that the harder I work, the more luck I seem to have.", Thomas Jefferson << is placed in the CSV as """I find that the harder I work, the more luck I seem to have."", Thomas Jefferson" That is, the overall string is encapsulated in quotes with quotes that are intended represented as double quotes.

5. The PQDS File Naming Convention

There is purposely no standard for PQDS file naming requirements. It is recommended that the PQDS file extension be ".csv"

6. Example PQDS.csv File

6.1 PQDS Containing Waveform Values

DeviceName,"(FL1)",,T,"Meter or measurement device name"
DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter device name"
DeviceLocation,"N/A",,T,"Meter or measurement device location name"
AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
DeviceOwner,"ptco004",,T,"Utility name"
AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
EventGUID,"3a07e27c-0800-98d9-fc11-0747fa0715b6e",,A,"Unique Event Identifier"
EventID,"Event 6",,T,"A user defined Event Name"
WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
waveform-data,va,vb,vc,ia,ib,ic
0,-148.404,0.7707,149.76,-3.51931,-0.0487,-0.082
0.0013,-145.0518,-8.03,152.65,-0.8474,-0.0906,-0.082

DeviceName,"(FL1)",,T,"Meter or measurement device name"
DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter device name"
DeviceLocation,"N/A",,T,"Meter or measurement device location name"
AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
DeviceOwner,"ptco004",,T,"Utility name"
AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
StartDate,"02-01-2019",,T,"Start Date for trending Data"
WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
trending-data,va,vb,vc
23:40,148.404,149.750,156.081,145.203,148.756,150.046,135.214,146.736,151.124
23:50,148.213,148.124,155.234,144.142,149.253,151.770,136.213,145.998,149.235
1,03-01-2019,#####################						
0:00,149.143,149.986,156.876,145.585,148.125,150.124,144.008,146.341,149.432
0:10,148.134,141.231,156.236,145.234,148.235,150.156,140.145,144.444,150.999

7. PQDS Log Files

7.1 Introduction

The Power Quality Data Sharing (PQDS) log file format is a comma-separated value file that enables easy tracking and sharing of the metadata not included in the PQDS file for various reasons, such as anomyizing the electric power disturbance waveforms. It is designed to be a simple and extensible human-to-human data format that complements the PQDS file format.

The PQDS log file format can include multiple sections, each corresponding to a single PQDS file generated by the EPRI PQDS application. The EPRI PQDS application will append a new section to the existing log file for each PQDS file generated.

7.2 Header Section

Each section corresponding to a PQDS file is separated by a single line of 42 >>-<< characters. The first section skips this seperator and imidiately starts with the identifying header.

7.3 Identifying Header

Each section contains a line consisting of two fields separated by a comma. The first field is the date and time at which the corresponding PQDS file is generated. The field is encapsulated in double quotations and formated as d/m/YYYY hh:mm:ss. The date and time are given in UTC to avoid any confusion over the time zone in which the file was generated and formated using a 24 hour format. The second field is a uniquely generated GUID associated with the PQDS file. This GUID corresponds to the EventGUID field in the meta data section of the PQDS file as found in Section 2.1 of this document. Note that the field is also encapsulated by ".

7.4 Meta Data Section

Imidiately following the identifying header the meta data section of the PQDS file is included. Unlike the PQDS file itself all available meta data should be included in the log file. The format of this section is the same as that described in Section 2.1 this document.

7.5 Data Section

The log file does not contain any actual data. However, in order to better describe the content of the PQDS file a header similar to that of the wave form or tredning data section in the PQDS file is included. The entry starts with the word >>waveform-data<< or >>trending-data<< , followed by comma separated list of the channels included in the PQDS file. Suggested values include >>va<<, >>vb<<, >>vc<<, >>vn<<, >>ia<<, >>ib<<, >>ic<<, >>in<<, and >>f<<.

7.6 Example PQDS log File

"1/11/2019 09:34:50","3a07e27c-0800-98d9-fc11-0747fa0715b6e"
DeviceName,"(FL1)",,T,"Meter or measurement device name"
DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter name"
DeviceLocation,"N/A",,T,"Meter or measurement device location name"
AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
DeviceOwner,"ptco004",,T,"Utility name"
AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
EventGUID,"00000000-0000-0000-0000-000000000000",,A,"Unique Event Identifier"
EventID,"Event 6",,T,"A user defined Event Name"
WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
waveform-data,va,vb,vc,ia,ib,ic
---------------------------------------------
"1/12/2019 19:35:04","f7c6281b-0858-22df-ae9a-1191d83f81b4"
DeviceName,"(FL1)",,T,"Meter or measurement device name"
DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter name"
DeviceLocation,"N/A",,T,"Meter or measurement device location name"
AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
DeviceOwner,"ptco004",,T,"Utility name"
AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
EventGUID,"00000000-0000-0000-0000-000000000000",,A,"Unique Event Identifier"
EventID,"Event 6",,T,"A user defined Event Name"
WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
waveform-data,va,vb,vc,ia,ib,ic

8. metadata PQDS Codes Descriptions

8.1 DeviceConnectionTypeCode

The PQDS code for meter connection type with the following codes predefined:

Code Description
0 unknown
1 3 Phase, 4 wires
2 3 Phase, 3 wires
3 1 Phase, 2 wires
4 1 Phase, 3 wires
5 2 Phase, 3 wires

8.2 EventTypeCode

The PQDS code for the event type with the following codes predefined:

Code Description
0 Unknown
1 Fault
2 Voltage sag
3 Voltage swell
4 Interruption
5 Harmonics

8.3 EventFaultTypeCode

The PQDS code for the fault type with the following codes predefined:

Code Description
0 Unknown
1 Single Line to Ground (LG)
2 Line to Line (LL)
3 Double Line to Ground (LLG)
4 3 Phase (LLL)
5 3 Phase to Ground (LLLG)

8.4 EventCauseCode

The PQDS code for the event cause with the following codes predefined:

Code Description
0 Unknown
1 Lightning
2 Animal
3 Vegetation
4 Contamination
5 Smoke
6 Ice or Snow
7 Wind
8 Utility equipment issue
9 Customer equipment issue
10 Vandalism
11 Human error
12 Other

8.5 WaveformSensitivityCode

The PQDS code for the waveform data sensitivity with the following codes predefined:

Code Description
0 Internal use only (i.e., protected, confidential information)
1 Business sensitive (shared under NDA only)
2 Reliability sensitive (shared with other grid entities under NERC data sharing or other reliability or marketing service agreements)
3 Public

9. References

  1. Wikipedia, CSV, 2019, https://en.wikipedia.org/wiki/Comma-separated_values.
  2. Super CSV Package for JAVA, Kasper/GitHub, 2015, http://super-csv.github.io/super-csv/csv_specification.html.
  3. RFC 4180, Format for Comma-Separated Values, 2008 https://tools.ietf.org/html/rfc4180 https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml.