Data insertion and archiving in erasure-coding based large-scale storage systems

Given the vast volume of data that needs to be stored reliably, many data-centers and large-scale file systems have started using erasure codes to achieve reliable storage while keeping the storage overhead low. This has invigorated the research on erasure codes tailor made to achieve different des...

Full description

Saved in:
Bibliographic Details
Main Authors: Pamies-Juarez, Lluis, Oggier, Frederique, Datta, Anwitaman
Other Authors: School of Computer Engineering
Format: Conference or Workshop Item
Language:English
Published: 2013
Subjects:
Online Access:https://hdl.handle.net/10356/95760
http://hdl.handle.net/10220/9470
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-95760
record_format dspace
spelling sg-ntu-dr.10356-957602023-02-28T19:17:44Z Data insertion and archiving in erasure-coding based large-scale storage systems Pamies-Juarez, Lluis Oggier, Frederique Datta, Anwitaman School of Computer Engineering International Conference of Distributed Computing and Internet Technology (9th : 2013 : Bhubaneswar, India) DRNTU::Engineering::Computer science and engineering::Data::Data storage representations Given the vast volume of data that needs to be stored reliably, many data-centers and large-scale file systems have started using erasure codes to achieve reliable storage while keeping the storage overhead low. This has invigorated the research on erasure codes tailor made to achieve different desirable storage system properties such as efficient redundancy replenishment mechanisms, resilience against data corruption, degraded reads, to name a few prominent ones. A problem that has mainly been overlooked until recently is that of how the storage system can be efficiently populated with erasure coded data to start with. In this paper, we will look at two distinct but related scenarios: (i) migration to archival -leveraging on existing replicated data to create an erasure encoded archive, and (ii) data insertion - new data being inserted in the system directly in erasure coded format. We will elaborate on coding techniques to achieve better throughput for data insertion and migration, and in doing so, explore the connection of these techniques with recently proposed locally repairable codes such as self-repairing codes. Accepted version 2013-04-09T06:09:51Z 2019-12-06T19:20:57Z 2013-04-09T06:09:51Z 2019-12-06T19:20:57Z 2013 2013 Conference Paper Pamies-Juarez, L., Oggier, F., & Datta, A. (2013). Data Insertion and Archiving in Erasure-Coding Based Large-Scale Storage Systems. Distributed Computing and Internet Technology, 7753, pp.47-68. https://hdl.handle.net/10356/95760 http://hdl.handle.net/10220/9470 10.1007/978-3-642-36071-8_4 167956 en © 2013 Springer. This is the author created version of a work that has been peer reviewed and accepted for publication by Distributed Computing and Internet Technology, Springer. It incorporates referee’s comments but changes resulting from the publishing process, such as copyediting, structural formatting, may not be reflected in this document. The published version is available at: [http://dx.doi.org/10.1007/978-3-642-36071-8_4]. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic DRNTU::Engineering::Computer science and engineering::Data::Data storage representations
spellingShingle DRNTU::Engineering::Computer science and engineering::Data::Data storage representations
Pamies-Juarez, Lluis
Oggier, Frederique
Datta, Anwitaman
Data insertion and archiving in erasure-coding based large-scale storage systems
description Given the vast volume of data that needs to be stored reliably, many data-centers and large-scale file systems have started using erasure codes to achieve reliable storage while keeping the storage overhead low. This has invigorated the research on erasure codes tailor made to achieve different desirable storage system properties such as efficient redundancy replenishment mechanisms, resilience against data corruption, degraded reads, to name a few prominent ones. A problem that has mainly been overlooked until recently is that of how the storage system can be efficiently populated with erasure coded data to start with. In this paper, we will look at two distinct but related scenarios: (i) migration to archival -leveraging on existing replicated data to create an erasure encoded archive, and (ii) data insertion - new data being inserted in the system directly in erasure coded format. We will elaborate on coding techniques to achieve better throughput for data insertion and migration, and in doing so, explore the connection of these techniques with recently proposed locally repairable codes such as self-repairing codes.
author2 School of Computer Engineering
author_facet School of Computer Engineering
Pamies-Juarez, Lluis
Oggier, Frederique
Datta, Anwitaman
format Conference or Workshop Item
author Pamies-Juarez, Lluis
Oggier, Frederique
Datta, Anwitaman
author_sort Pamies-Juarez, Lluis
title Data insertion and archiving in erasure-coding based large-scale storage systems
title_short Data insertion and archiving in erasure-coding based large-scale storage systems
title_full Data insertion and archiving in erasure-coding based large-scale storage systems
title_fullStr Data insertion and archiving in erasure-coding based large-scale storage systems
title_full_unstemmed Data insertion and archiving in erasure-coding based large-scale storage systems
title_sort data insertion and archiving in erasure-coding based large-scale storage systems
publishDate 2013
url https://hdl.handle.net/10356/95760
http://hdl.handle.net/10220/9470
_version_ 1759856327441514496