← #78 MsBIP møde nr. 78, 30. august 2023

1001 sources - ingestion & scraping of data

How do you ingest data from thousands of sources, 24/7/365?

What role does caching and deduplication play in a real-time-always-on setup ?

In this session we cover the data ingestion setup of InCommodities.

Which methods and technologies are employed and how do they each play a role in ensuring that data moves as fast as possible from market signal and in to the data platform.