Storage and access optimization scheme based on correlation probabilities in the internet of vehicles

Following the rapid development of the Internet of vehicles (IoV), many issues and challenges do come up as the storage of large quantities of vehicle network data and improvement of the retrieval efficiency. A great deal of global positioning system (GPS) log data and vehicle monitoring data is gen...

Full description

Saved in:
Bibliographic Details
Main Authors: Bin, Zhou, Yao, Yuhao, Liu, Xiao, Zhu, Rongbo, Sangaiah, Arun Kumar, Ma, Maode
Other Authors: School of Electrical and Electronic Engineering
Format: Article
Language:English
Published: 2021
Subjects:
Online Access:https://hdl.handle.net/10356/150757
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-150757
record_format dspace
spelling sg-ntu-dr.10356-1507572021-08-02T01:42:51Z Storage and access optimization scheme based on correlation probabilities in the internet of vehicles Bin, Zhou Yao, Yuhao Liu, Xiao Zhu, Rongbo Sangaiah, Arun Kumar Ma, Maode School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Internet of Vehicles Small Files Correlation Probability Following the rapid development of the Internet of vehicles (IoV), many issues and challenges do come up as the storage of large quantities of vehicle network data and improvement of the retrieval efficiency. A great deal of global positioning system (GPS) log data and vehicle monitoring data is generated on IoV. When many small files in the conventional Hadoop Distributed File System (HDFS) are accessed, a series of problems arise such as high occupancy rate, low access efficiency and low retrieval efficiency, which lead to degrade the performance of IoV. In an attempt to tackle these bottleneck problems, a small Files Correlation Probability (FCP) model is proposed, which is based on the Text Feature Vector (TFV) presented in this paper. The Small Files Merge Scheme based on FCP (SFMS-FCP) and the Small File Prefetching and Caching Strategies (SFPCS) are proposed to optimize the storage and access performance of HDFS. Finally, experiments show that the proposed optimization solutions achieve better performance in terms of high occupancy of HDFS name nodes and low access efficiency, compared with the native HDFS read-write scheme and HAR-based read-write optimization scheme. This research was supported by the Soft Science Research of Hubei Province (NO.2019ADC071), the Natural Science Foundation of Hubei Province (NO. 2016CFB650), the National Natural Science Foundation of China (NO. 61772562), and the Hubei Provincial Natural Science Foundation of China for Distinguished Young Scholars (NO. 2017CFA043), and Fundamental Research Funds for the Central Universities (CZP19004), and Youth Elite Project of State Ethnic Affairs Commission of China. 2021-08-02T01:42:51Z 2021-08-02T01:42:51Z 2020 Journal Article Bin, Z., Yao, Y., Liu, X., Zhu, R., Sangaiah, A. K. & Ma, M. (2020). Storage and access optimization scheme based on correlation probabilities in the internet of vehicles. Journal of Intelligent Transportation Systems: Technology, Planning, and Operations, 24(3), 221-236. https://dx.doi.org/10.1080/15472450.2019.1612247 1547-2450 https://hdl.handle.net/10356/150757 10.1080/15472450.2019.1612247 2-s2.0-85065998094 3 24 221 236 en Journal of Intelligent Transportation Systems: Technology, Planning, and Operations © 2019 Taylor & Francis Group, LLC. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Electrical and electronic engineering
Internet of Vehicles
Small Files Correlation Probability
spellingShingle Engineering::Electrical and electronic engineering
Internet of Vehicles
Small Files Correlation Probability
Bin, Zhou
Yao, Yuhao
Liu, Xiao
Zhu, Rongbo
Sangaiah, Arun Kumar
Ma, Maode
Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
description Following the rapid development of the Internet of vehicles (IoV), many issues and challenges do come up as the storage of large quantities of vehicle network data and improvement of the retrieval efficiency. A great deal of global positioning system (GPS) log data and vehicle monitoring data is generated on IoV. When many small files in the conventional Hadoop Distributed File System (HDFS) are accessed, a series of problems arise such as high occupancy rate, low access efficiency and low retrieval efficiency, which lead to degrade the performance of IoV. In an attempt to tackle these bottleneck problems, a small Files Correlation Probability (FCP) model is proposed, which is based on the Text Feature Vector (TFV) presented in this paper. The Small Files Merge Scheme based on FCP (SFMS-FCP) and the Small File Prefetching and Caching Strategies (SFPCS) are proposed to optimize the storage and access performance of HDFS. Finally, experiments show that the proposed optimization solutions achieve better performance in terms of high occupancy of HDFS name nodes and low access efficiency, compared with the native HDFS read-write scheme and HAR-based read-write optimization scheme.
author2 School of Electrical and Electronic Engineering
author_facet School of Electrical and Electronic Engineering
Bin, Zhou
Yao, Yuhao
Liu, Xiao
Zhu, Rongbo
Sangaiah, Arun Kumar
Ma, Maode
format Article
author Bin, Zhou
Yao, Yuhao
Liu, Xiao
Zhu, Rongbo
Sangaiah, Arun Kumar
Ma, Maode
author_sort Bin, Zhou
title Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
title_short Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
title_full Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
title_fullStr Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
title_full_unstemmed Storage and access optimization scheme based on correlation probabilities in the internet of vehicles
title_sort storage and access optimization scheme based on correlation probabilities in the internet of vehicles
publishDate 2021
url https://hdl.handle.net/10356/150757
_version_ 1707050388592525312