Scheduler 3.0

During the intended follow-up project Owlin will develop the next generation scheduler system (‘Scheduler 3.0’) replacing the old linear computational pipeline. In the new system the computation flow consists of a user selectable number of logic units and data flow is governed by input/output types which includes the possibility of recursion. Data records will be stored in an AP (Available and Partition tolerant) data storage and will allow full text search on all the generated data records.....This has a number of advantages over existing infrastructure. The code editing will be “notebook” style instead of classical editor style. In combination with the smaller units of computation this will mean that writing production configuration code will become even more accessible, even to people with limited coding experience. In the old system caching is on the level of datasets (usually there is a one to one mapping between client projects and datasets). This means amongst other things that if new companies get added the dataset needs to be generated from scratch (“backfilling”), which can take a number of days. In the new system, caching is much more granular, and this faster and less resource intensive.....This project connects with the top sector ‘HTSM/ICT’ and links up with the themes ‘Big Data’. Having large amounts of data means that they have to be stored somewhere. Moreover, this data has to be structured in a way that it adds value to the user. Data management systems that have been developed accordingly in the recent decades have become increasingly more complex, often building on top of legacy systems, adding layer after layer of infrastructure and computing steps.....This has had a number of effects that have to be taken into accounting building a business. On boarding of new development staff takes a lot of time and investments (and developers are already very expensive), added tasks to the computational pipeline take much time to execute and calculations take up more and more server space raking up data-handling costs to significant portions of business expenses. Therefore, new strategies are needed to unlock the potential of Big Data and manage related resources...