Cardinality Based Rate Limiting System for Time-Series Data

  • Sudeep KumarEmail author
  • Deepak Kumar Vasthimal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12403)


Massive monitoring systems that require high availability and performance for both ingestion and retrieval of data are often encountered with rogue streams of data having a high cardinality. The management of such high cardinality data sets for time-series data and a performance sensitive system is challenging. The challenges primarily arise as the time-series data sets, typically needs to be loaded onto a limited memory space before results can be returned to the client. This affects the number of incoming queries that can be supported simultaneously. Too many time-series can potentially degraded read performance and thereby affect user experience. Our proposed rate-limiting system described herein seeks to address a key availability issue on a high-volume, time-series system by using a dynamic cardinality computation in combination with a central assessment service to detect and block high cardinality data streams. As a result of this technical improvement, anomalous logging behavior is detected quickly, affected tenants are notified, and hardware resources are used optimally.


Cloud computing Hyperlog Cardinality Time-series Micro-services Rate-limiting Metrics 



This work has been performed with support of eBay Inc. The views expressed are those of the authors and do not necessarily represent eBay Inc. eBay Inc. or any of its subsidiaries is not liable for any use that may be made of any of the information contained herein.


  1. 1.
    Korenberg, M.J., Paarmann, L.D.: Orthogonal approaches to time-series analysis and system identification. IEEE Signal Process. Mag. 8(3), 29–43 (1991)CrossRefGoogle Scholar
  2. 2.
    Zheng, Y., Li, M.: ZOE: fast cardinality estimation for large-scale RFID systems. In: 2013 Proceedings IEEE INFOCOM, Turin (2013)Google Scholar
  3. 3.
    Vasthimal, D.K., Kumar, S., Somani, M.: Near real-time tracking at scale. In: IEEE 7th International Symposium on Cloud and Service Computing (SC2). Kanazawa 2017, pp. 241–244 (2017)Google Scholar
  4. 4.
    Chabchoub, Y., Hebrail, G.: Sliding hyperloglog: estimating cardinality in a data stream over a sliding window. In: 2010 IEEE International Conference on Data Mining Workshops, NSW, Sydney (2010)Google Scholar
  5. 5.
  6. 6.
    Sellisa, T.K.: Intelligent caching and indexing techniques for relational database systems. Information Systems, Elsevier (1988)Google Scholar
  7. 7.
    Bennett, J.M., Bauer, M.A., Kinchlea, D.: Characteristics of files in NFS environments. In: Proceedings of the ACM SIGSMALL/PC Symposium on Small Systems, Toronto, Ontario, Candada (1991)Google Scholar
  8. 8.
    V, D.K., Shah, R.R., Philip, A.: Centralized log management for pepper. In: 2011 IEEE Third International Conference on Cloud Computing Technology and Science, Athens, 2011, pp. 1–3 (2011)Google Scholar
  9. 9.
    Han, H., Giles, C.L., Manavoglu, E., Zha, H., Zhang, Z., Fox, E.A.: Automatic document metadata extraction using support vector machines. In: 2003 Joint Conference on Digital Libraries, 2003. Proceedings., Houston, TX, USA (2003)Google Scholar
  10. 10.
    Vasthimal, D.: Robust and Resilient migration of data processing systems to public hadoop grid. In: 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion), Zurich, 2018, pp. 21–23 (2018)Google Scholar
  11. 11.
    Gupta, D., et al.: Difference engine: harnessing memory redundancy in virtual machines. Commun. ACM 53(10), 85–93 (2010)CrossRefGoogle Scholar
  12. 12.
    Raghavan, B., Vishwanath, K., Ramabhadran, S., Yocum, K., Snoeren, A.: Cloud Control with Distributed Rate Limiting. In: SIGCOMM (2007)Google Scholar
  13. 13.
    Hightower, K., Burns, B., Beda, J.: Kubernetes: Up and Running: Dive Into the Future of Infrastructure, 2nd edn. O’Reilly Media, Sebastopol (2019)Google Scholar
  14. 14.
    Ngan, H.Y.T., Pang, G.K.H.: Novel method for patterned fabric inspection using Bollinger bands. Opt. Eng. 45(8), 087202 (2006)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.eBay Inc.San JoseUSA

Personalised recommendations