A Scalable Smart Meter Data Generator Using Spark

Nadeem Iftikhar; Xiufeng Liu; Sergiu Danalachi; Finn Ebertsen Nordbjerg; Jens Henrik Vollesen

doi:10.1007/978-3-319-69462-7_2

A Scalable Smart Meter Data Generator Using Spark

Nadeem Iftikhar
, Xiufeng Liu
, Sergiu Danalachi
, Finn Ebertsen Nordbjerg
, Jens Henrik Vollesen

Danmarks Tekniske Universitet
Studerende ved Softwareuddannelsen, UCN

Publikation: Bidrag til bog/antologi/rapport › Konferenceartikel i proceeding › peer review

Abstract

Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter
data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

Originalsprog	Engelsk
Titel	On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences : CoopIS, C and TC, and ODBASE 2017, Proceedings
Redaktører	Herve Panetto, Adrian Paschke, Robert Meersman, Mike Papazoglou, Christophe Debruyne, Walid Gaaloul, Claudio Agostino Ardagna
Antal sider	16
Forlag	Springer
Publikationsdato	2017
Sider	21-36
ISBN (Trykt)	978-3-319-69461-0
ISBN (Elektronisk)	978-3-319-69462-7
DOI	https://doi.org/10.1007/978-3-319-69462-7_2
Status	Udgivet - 2017

Emneord

it

Adgang til dokumentet

10.1007/978-3-319-69462-7_2

Andre filer og links

Link to publication in Scopus

Citationsformater

Iftikhar, N., Liu, X., Danalachi, S., Nordbjerg, F. E., & Vollesen, J. H. (2017). A Scalable Smart Meter Data Generator Using Spark. I H. Panetto, A. Paschke, R. Meersman, M. Papazoglou, C. Debruyne, W. Gaaloul, & C. A. Ardagna (red.), On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences: CoopIS, C and TC, and ODBASE 2017, Proceedings (s. 21-36). Springer. https://doi.org/10.1007/978-3-319-69462-7_2

Iftikhar, Nadeem ; Liu, Xiufeng ; Danalachi, Sergiu et al. / A Scalable Smart Meter Data Generator Using Spark. On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences: CoopIS, C and TC, and ODBASE 2017, Proceedings. red. / Herve Panetto ; Adrian Paschke ; Robert Meersman ; Mike Papazoglou ; Christophe Debruyne ; Walid Gaaloul ; Claudio Agostino Ardagna. Springer, 2017. s. 21-36

@inproceedings{a86ad7ca37ca4140a0b4052378380616,

title = "A Scalable Smart Meter Data Generator Using Spark",

abstract = "Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.",

keywords = "it",

author = "Nadeem Iftikhar and Xiufeng Liu and Sergiu Danalachi and Nordbjerg, \{Finn Ebertsen\} and Vollesen, \{Jens Henrik\}",

note = "Publisher Copyright: {\textcopyright} 2017, Springer International Publishing AG.",

year = "2017",

doi = "10.1007/978-3-319-69462-7\_2",

language = "English",

isbn = "978-3-319-69461-0",

pages = "21--36",

editor = "Herve Panetto and Adrian Paschke and Robert Meersman and Mike Papazoglou and Christophe Debruyne and Walid Gaaloul and Ardagna, \{Claudio Agostino\}",

booktitle = "On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences",

publisher = "Springer",

address = "United States",

}

Iftikhar, N, Liu, X, Danalachi, S, Nordbjerg, FE & Vollesen, JH 2017, A Scalable Smart Meter Data Generator Using Spark. i H Panetto, A Paschke, R Meersman, M Papazoglou, C Debruyne, W Gaaloul & CA Ardagna (red), On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences: CoopIS, C and TC, and ODBASE 2017, Proceedings. Springer, s. 21-36. https://doi.org/10.1007/978-3-319-69462-7_2

A Scalable Smart Meter Data Generator Using Spark. / Iftikhar, Nadeem; Liu, Xiufeng; Danalachi, Sergiu et al.
On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences: CoopIS, C and TC, and ODBASE 2017, Proceedings. red. / Herve Panetto; Adrian Paschke; Robert Meersman; Mike Papazoglou; Christophe Debruyne; Walid Gaaloul; Claudio Agostino Ardagna. Springer, 2017. s. 21-36.

Publikation: Bidrag til bog/antologi/rapport › Konferenceartikel i proceeding › peer review

TY - GEN

T1 - A Scalable Smart Meter Data Generator Using Spark

AU - Iftikhar, Nadeem

AU - Liu, Xiufeng

AU - Danalachi, Sergiu

AU - Nordbjerg, Finn Ebertsen

AU - Vollesen, Jens Henrik

PY - 2017

Y1 - 2017

N2 - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

AB - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

KW - it

UR - https://www.scopus.com/pages/publications/85032688114

U2 - 10.1007/978-3-319-69462-7_2

DO - 10.1007/978-3-319-69462-7_2

M3 - Conference contribution to proceeding

SN - 978-3-319-69461-0

SP - 21

EP - 36

BT - On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences

A2 - Panetto, Herve

A2 - Paschke, Adrian

A2 - Meersman, Robert

A2 - Papazoglou, Mike

A2 - Debruyne, Christophe

A2 - Gaaloul, Walid

A2 - Ardagna, Claudio Agostino

PB - Springer

ER -

Iftikhar N, Liu X, Danalachi S, Nordbjerg FE, Vollesen JH. A Scalable Smart Meter Data Generator Using Spark. I Panetto H, Paschke A, Meersman R, Papazoglou M, Debruyne C, Gaaloul W, Ardagna CA, red., On the Move to Meaningful Internet Systems. OTM 2017 Conferences - Confederated International Conferences: CoopIS, C and TC, and ODBASE 2017, Proceedings. Springer. 2017. s. 21-36 doi: 10.1007/978-3-319-69462-7_2

A Scalable Smart Meter Data Generator Using Spark

Abstract

Emneord

Adgang til dokumentet

Andre filer og links

Fingeraftryk

Citationsformater