Optimizing Data Stream Throughput for Real-Time Applications

Kusuma Amilineni; Po Yen Hsu; Ching Hsien Hsu; Hojjat Baghban; Wen Thong Chang; Nithin Melala Eshwarappa

doi:10.1007/978-981-99-2233-8_29

Optimizing Data Stream Throughput for Real-Time Applications

Kusuma Amilineni
, Po Yen Hsu
, Ching Hsien Hsu^*
, Hojjat Baghban
, Wen Thong Chang
, Nithin Melala Eshwarappa

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Many problems, like recommendation services, website log activities, commit logs, and event sourcing services, are often very high volume and high velocity as many activity messages are generated. Various cloud vendors offer services for building message transport and processing pipelines. We present the Apache Kafka optimization process for using Kafka as a messaging system in high throughput and low latency systems. Experiments are conducted with various configurations for observing their effects on performance metrics. The performance, Durability, and reliability trade-offs of Apache Kafka are presented for various data rates, resource utilization, and configurations on how to utilize resources more economically and choose the best Kafka configuration for use cases. This research presents by tuning configuration parameters leveraged throughput by 19%.

Original language	English
Title of host publication	Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings
Editors	Ching-Hsien Hsu, Mengwei Xu, Hung Cao, Hojjat Baghban, A. B. Shawkat Ali
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	410-417
Number of pages	8
ISBN (Print)	9789819922321
DOIs	https://doi.org/10.1007/978-981-99-2233-8_29
State	Published - 2023
Externally published	Yes
Event	International Conference on Big Data Intelligence and Computing, DataCom 2022 - Denarau, Fiji Duration: 08 12 2022 → 10 12 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13864 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	International Conference on Big Data Intelligence and Computing, DataCom 2022
Country/Territory	Fiji
City	Denarau
Period	08/12/22 → 10/12/22

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Keywords

Apache Kafka
latency optimization
real-time application
stream computing
throughput optimization

Access to Document

10.1007/978-981-99-2233-8_29

Cite this

Amilineni, K., Hsu, P. Y., Hsu, C. H., Baghban, H., Chang, W. T., & Eshwarappa, N. M. (2023). Optimizing Data Stream Throughput for Real-Time Applications. In C.-H. Hsu, M. Xu, H. Cao, H. Baghban, & A. B. Shawkat Ali (Eds.), Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings (pp. 410-417). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13864 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-99-2233-8_29

Amilineni, Kusuma ; Hsu, Po Yen ; Hsu, Ching Hsien et al. / Optimizing Data Stream Throughput for Real-Time Applications. Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings. editor / Ching-Hsien Hsu ; Mengwei Xu ; Hung Cao ; Hojjat Baghban ; A. B. Shawkat Ali. Springer Science and Business Media Deutschland GmbH, 2023. pp. 410-417 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{ce2e8bc5e9b1405098be60c0a083f847,

title = "Optimizing Data Stream Throughput for Real-Time Applications",

abstract = "Many problems, like recommendation services, website log activities, commit logs, and event sourcing services, are often very high volume and high velocity as many activity messages are generated. Various cloud vendors offer services for building message transport and processing pipelines. We present the Apache Kafka optimization process for using Kafka as a messaging system in high throughput and low latency systems. Experiments are conducted with various configurations for observing their effects on performance metrics. The performance, Durability, and reliability trade-offs of Apache Kafka are presented for various data rates, resource utilization, and configurations on how to utilize resources more economically and choose the best Kafka configuration for use cases. This research presents by tuning configuration parameters leveraged throughput by 19\%.",

keywords = "Apache Kafka, latency optimization, real-time application, stream computing, throughput optimization",

author = "Kusuma Amilineni and Hsu, \{Po Yen\} and Hsu, \{Ching Hsien\} and Hojjat Baghban and Chang, \{Wen Thong\} and Eshwarappa, \{Nithin Melala\}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; International Conference on Big Data Intelligence and Computing, DataCom 2022 ; Conference date: 08-12-2022 Through 10-12-2022",

year = "2023",

doi = "10.1007/978-981-99-2233-8\_29",

language = "英语",

isbn = "9789819922321",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "410--417",

editor = "Ching-Hsien Hsu and Mengwei Xu and Hung Cao and Hojjat Baghban and \{Shawkat Ali\}, \{A. B.\}",

booktitle = "Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings",

address = "德国",

}

Amilineni, K, Hsu, PY, Hsu, CH, Baghban, H, Chang, WT & Eshwarappa, NM 2023, Optimizing Data Stream Throughput for Real-Time Applications. in C-H Hsu, M Xu, H Cao, H Baghban & AB Shawkat Ali (eds), Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13864 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 410-417, International Conference on Big Data Intelligence and Computing, DataCom 2022, Denarau, Fiji, 08/12/22. https://doi.org/10.1007/978-981-99-2233-8_29

Optimizing Data Stream Throughput for Real-Time Applications. / Amilineni, Kusuma; Hsu, Po Yen; Hsu, Ching Hsien et al.
Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings. ed. / Ching-Hsien Hsu; Mengwei Xu; Hung Cao; Hojjat Baghban; A. B. Shawkat Ali. Springer Science and Business Media Deutschland GmbH, 2023. p. 410-417 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13864 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Optimizing Data Stream Throughput for Real-Time Applications

AU - Amilineni, Kusuma

AU - Hsu, Po Yen

AU - Hsu, Ching Hsien

AU - Baghban, Hojjat

AU - Chang, Wen Thong

AU - Eshwarappa, Nithin Melala

PY - 2023

Y1 - 2023

N2 - Many problems, like recommendation services, website log activities, commit logs, and event sourcing services, are often very high volume and high velocity as many activity messages are generated. Various cloud vendors offer services for building message transport and processing pipelines. We present the Apache Kafka optimization process for using Kafka as a messaging system in high throughput and low latency systems. Experiments are conducted with various configurations for observing their effects on performance metrics. The performance, Durability, and reliability trade-offs of Apache Kafka are presented for various data rates, resource utilization, and configurations on how to utilize resources more economically and choose the best Kafka configuration for use cases. This research presents by tuning configuration parameters leveraged throughput by 19%.

AB - Many problems, like recommendation services, website log activities, commit logs, and event sourcing services, are often very high volume and high velocity as many activity messages are generated. Various cloud vendors offer services for building message transport and processing pipelines. We present the Apache Kafka optimization process for using Kafka as a messaging system in high throughput and low latency systems. Experiments are conducted with various configurations for observing their effects on performance metrics. The performance, Durability, and reliability trade-offs of Apache Kafka are presented for various data rates, resource utilization, and configurations on how to utilize resources more economically and choose the best Kafka configuration for use cases. This research presents by tuning configuration parameters leveraged throughput by 19%.

KW - Apache Kafka

KW - latency optimization

KW - real-time application

KW - stream computing

KW - throughput optimization

UR - https://www.scopus.com/pages/publications/85161417598

U2 - 10.1007/978-981-99-2233-8_29

DO - 10.1007/978-981-99-2233-8_29

M3 - 会议稿件

AN - SCOPUS:85161417598

SN - 9789819922321

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 410

EP - 417

BT - Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings

A2 - Hsu, Ching-Hsien

A2 - Xu, Mengwei

A2 - Cao, Hung

A2 - Baghban, Hojjat

A2 - Shawkat Ali, A. B.

PB - Springer Science and Business Media Deutschland GmbH

T2 - International Conference on Big Data Intelligence and Computing, DataCom 2022

Y2 - 8 December 2022 through 10 December 2022

ER -

Amilineni K, Hsu PY, Hsu CH, Baghban H, Chang WT, Eshwarappa NM. Optimizing Data Stream Throughput for Real-Time Applications. In Hsu CH, Xu M, Cao H, Baghban H, Shawkat Ali AB, editors, Big Data Intelligence and Computing - International Conference, DataCom 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 410-417. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-981-99-2233-8_29

Optimizing Data Stream Throughput for Real-Time Applications

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this