Grand Challenge: High Performance Stream Queries in Scala

Dantong Song, Kaiwen Zhang, Tilmann Rabl, Prashanth Menon, and Hans-Arno Jacobsen.

In DEBS, 2015.

Abstract

Traffic monitoring is an important stream processing application which is highly dynamic and requires aggregation of spatially colocated data. Inspired by this, the DEBS 2015 Grand Challenge uses publicly available taxi transportation information to compute online the most frequent routes and most profitable areas. In this paper, we describe our solution to the DEBS 2015 Grand Challenge written in Scala. Our large-scale solution employs Apache Spark, while our challenge implementation is highly specialized and can process events at a 10 ms latency and at a throughput of 114,000 events per second.

Download



Tags: grand challenge, spark, scala, taxi monitoring


Readers who enjoyed the above work, may also like the following:


  • Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing.
    Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, and Hans-Arno Jacobsen.
    In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, 2012.
    Best DEBS Challenge Award - Public Voting.
    Tags: event processing, grand challenge
  • PSBench: A Benchmark for Content- and Topic-based Publish/Subscribe Systems.
    Kaiwen Zhang, Tilmann Rabl, Yi Ping Sun, Rushab Kumar, Nayeem Zen, and Hans-Arno Jacobsen.
    In Middleware Demos, 2014.
    Tags: pub/sub, pub/sub applications, publish/subscribe, benchmarking
  • MADES - A Multi-Layered, Adaptive, Distributed Event Store.
    Tilmann Rabl, Mohammad Sadoghi, Kaiwen Zhang, and Hans-Arno Jacobsen.
    In Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems, 2013.