From Big Data to Fast Data: Efficient Stream Data Management

This manuscript provides a synthetic overview of research on data management in support of stream processing. It address all stages of the stream processing pipeline: data collection and in-transit processing at the edge, transfer towards the cloud processing sites, ingestion and persistent storage. First, the general context of stream data management is presented in light of the recent transition from Big to Fast Data. After highlighting the challenges at the data level associated with batch and real-time analytics, we introduce a subjective overview of proposals to address them. They bring solutions to the problems of in-transit stream storage and processing, fast data transfers, distributed metadata management, dynamic ingestion and transactional storage. The integration of these solutions into functional prototypes and the results of the large-scale experimental evaluations on clusters, clouds and supercomputers demonstrate their effectiveness for several real-life applications ranging from neuro-science to LHC nuclear physics. Finally, these contributions are put into the perspective of the High Performance Computing - Big Data convergence.