Skip to main content

Everything everywhere, all at once

img

The world is changing, and so is the way we do science. The amount of data we are generating is increasing exponentially, and the amount of data we can process is increasing at a linear rate. We are generating more data than we can process, and we are generating more data than we can store. Often this leads to weeks of waiting for the data to be processed, and then weeks of waiting for the results to be analysed.

With Arkitekt we wanted to change this. We wanted to enable users to process their data in real-time, and to be able to analyse the results in real-time, and we wanted the developer to not have to worry about this too much.

Real-time workflows​

Arkitekt is built to enable users to design analysis workflows that can as easily run post-hoc* on already generated data, as they can run in real-time on data while it is being generated. This requires a bit of a different approach to designing and thinking about bioimage workflows: Instead of focussing on "batch analysis", Arkitekt fully embraces "streaming analysis" to analyses data as it is being generated, no only cutting down on the time it takes to get results, but because of immediate feedback enabling paradigms like "smart microscopy".

Facing bottlenecks​

Real-time analysis is a very challenging task, and often unavoidable bottlenecks will arise, rendering your real-time analysis very much week-long. These bottlenecks can be caused by the hardware ( an old computer), the software (a long running deep learning algorithm), or the network (very slow network connection / latency). When using Arkitekt, we wanted to enable the user to identify and circumvent these bottlenecks and to fix them as they arise.

Arkitekt is therefore designed to provide simple interfaces to debug, monitor and fix these bottlenecks, while the analysis is still running. T his enables the user to be able to fix the bottlenecks as they arise, and to be able to continue the analysis in real-time. Even though Arkitekt is a server-client application, that in its default configuration enables the user to run the analysis on multiple computers, you can decide to run/schedule workflows latency on one application, enabling even workflows that are sensitive to sub-millisecond latency.