← #78 MsBIP møde nr. 78, 30. august 2023
1001 sources - ingestion & scraping of data
How do you ingest data from thousands of sources, 24/7/365?
What role does caching and deduplication play in a real-time-always-on setup ?
In this session we cover the data ingestion setup of InCommodities.
Which methods and technologies are employed and how do they each play a role in ensuring that data moves as fast as possible from market signal and in to the data platform.