Arkitekt Next

Developing the Arkitekt Platform is an ongoing process. While we are happy with the released version of Arkitekt, we are always looking for ways to improve the platform and make it more user-friendly, stable, and reliable. After having adapted the platform to a set of applications, we have now learned a lot more about the needs of users and are working on a lessons-learned-version of Arkitekt that will address the most dire pitfalls of the current version.

From here be dragons

We are working tirelessly to bring the next version of Arkitekt to you. Currently we are stepping into a pre-alpha phase and are looking for early adopters that want to help us test the new version of Arkitekt. If you are interested in testing the new version of Arkitekt, please go ahead and check out the installation instructions for the next version of Arkitekt. Rest assured that the "paper version", will remain available and rest frozen in time to ensure reproducability of the paper.

The shape of things to come

While we could dicusss hours about the new features of the next version of Arkitekt, we have decided to focus on the most important ones here and give you already a sneak peak of what is to come, this is by no means a comprehensive list of all the features that we are working on, but it should give you a good idea of what to expect.

Improved User Experience

Users, Users, Users! While we want Arkitekt to be a powerful tool for bioimage analysis, we also want it to be easy to use. Let's face it, if you followed the tutorial and got your first workflow running you saw all of the hurdles. This is why we are working hardcore on improving the user experience of Arkitekt. starting with Orkestrator our desktop control app. Here we are focusing on the following areas:

Improved Design and Layout

The design and layout of "Orkestrator" boasts a lot of features and functionality, but it is not very user-friendly. With the new interface we are working on, we are aiming to make the platform more user-friendly and intuitive, streamlining the UI and UX.

No more Reservations

While Johannes was desperately trying to make reservations work, we realized that the concept of "reserving" a node was not very intuitive and was causing a lot of confusion (additionally it was a source of many bugs). So we got rid of it ! The new version of Arkitekt will allow you to assign tasks to nodes directly without having to reserve them first. B

Look how easy it is to start a task:

Improved Workflow Design

Designin visual workflows should be a breeze and honestly in Orkestrator is currently a bit complex and not very intuitive. We have reworked and are continuing to iterate on the workflow design and making it more intuitive. The new prototype alreadly includes features such as auto node suggestion, auto-repair of broken workflows, and more and uses design patterns that users are already familiar with from other workflow design tools like Blender or Unreal Engine. Here is a sneak peak of the new design:

Improved Data management

Data management is a major pain point in the bioimage community and we are hoping to address some outstanding issues with next version of Arkitekt. Here we are focusing on the following areas:

Introducing the Knowledge Graph and Ontologies

A vision of Arkitekt was to capture the reality of biological systems in a database that allows users to create arbitraty measurements and interactively link them to each other. But the current version of Arkitekt does not fully allow to capture the intricacies of biological interactions intuitevely. The next version of Arkitekt will introduce the Knowledge Graph a new feature that will allow users to assign meaning to their analyzed data following a more natural language based approach (e.g. this ROI delineates an Axon) and link them to other data (e.g. this Axon is connected to this Dendrite). For this Arkitekt will now feature a Graph Database (based on the amazing Apache AGE project) that will allow users to interactively create and explore the relationships between their data using their own vocabulary.

This short video illustrates a very manual approach of labeling and assigning meaning to ROIS in an image. Of course the process shoud be ideally automated, but maybe it can give you an idea of what we are aiming for.

On Ontologies

Ontologies and Knowledge Graphs are a huge topic and are sadly not coverable in this short introduction. We will of course provide more information on how to create and use ontologies in the next version of Arkitekt in the documentation, and explain you how they might just help to make your data and your knowledge more sharable.

What we are aiming for it to interlink your Knowledge Graph with your data and metadata, and to allow you to interactively explore the relationships between your data and metadata. We want Programmers and Database savy people to use standard graph query languages to interact with the Knowledge Graph, and non-programmers to use a more intuitive interface to explore the relationships between their data and metadata.

note

This is a prelimnary version of the Knowledge Graph and we are working on improving the user experience and the features of the Knowledge Graph. We would like to tightly integrate the Knowledge Graph with the Views and Protocols features of the next version of Arkitekt.

Insert video of the Knowledge Graph here

Introducing Views

Metadata management is hard. To often does the metadata get lost or is not properly documented, and often because the metadata system is to rigid to accomodate the needs of the users. The next version of Arkitekt will introduce the concept of "Views". Views are ways to describe the metadata of your data in a flexible way. Your first channel of the image is supposed to be displayed in a certain color? Just add a RGBView to that channel. You want to describe the acquisition parameters of your image? Just add a AcquisitionView to the image. You want to indicate that this images pixel values represent labels (in a segmentation Map), just add a PixelView to the image, and map the labels to your Knowledge Entities. While your image-data might never change, the way you describe it can be as flexible as you need it to be.

Introduction of Protocols

Protocols are a way to describe the steps of sample preparation, image acquisition, and image analysis in a structured way, and to link specific protocol-steps to your image data. Imaging you want to filter all your images based on which a specific antibody was used in the primary staining? If you added the antibody to the protocol, you can now filter your images (attached via ProtocolViews) based on that information.

Stability

Stability is a major focus of the development on Arkitekt Next. As the platform grew organically out of our research needs, some parts of the platform are not as stable as we would like them to be. We are working on improving the stability of the platform and making sure that it is robust and reliable. Here we are focusing on the following areas:

Adressing Failure Hell

One of the arguabily biggest features of the Arkitekt platform is its ability to have applications run code that depend on other applications. May it be that you schedule a workflow on one app, that calls a plugin on another app. Or when when you have written a plugin that internally calls another plugin.

This is a powerful feature, but our current implementation is not as stable as we would like it to be. Especially when apps fail, and we have to cascade the failure to other apps that depend on the failed app. Currently propagating this change to the next app needs is handled on the client side, and is therefore not as reliable as we would like it to be. The next version of Arkitekt will adress this feature and make sure that the platform is more robust by introducing an automatic way to handle the cascading failure of tasks on the server side.

Adressing Apps without Servers

Arkitekt is buld around the concept of apps that want to connect to the platform. But this platform is by design modular as we wanted to make it easy for sys admins to only choose the modules they need. Do not care about mikro? Just don't install it. Kluster is exactly what you want because you have a dask-cluster? Just add it in your docker-compose file.

So how do we make sure that the apps can find services that they need to connect to? The Arkitekt answer to this is "Fakts", a small protocol that we developed for the service discovery inside Arkitekt. And it does work great for its intended purpose. However while transitioning from the Paper version to the Next version we realized that apps do not really have a way of declaring specific version requirements and communicate them before they are installed. This is a major issue as it can lead to the installation of apps that are not compatible with your underlying services. Fakts next will adress this issue and make sure that the platform is more robust and reliable by introductin dependencies to the Fakts protocol, which are declared in the app manifest and can stop you from installing an app that is not compatible with your services.

The old manifest

name: "My App"
version: "1.0.0"

The new way

name: "My App"
version: "1.0.0"
dependencies:
    - kluster: "live.arkitekt.kluster@paper"
    - mikro: "live.arkitekt.mikro@paper"
    - rekuest: "live.arkitekt.rekuest@next"

Breaking changes

This is probably the biggest breaking change in arkitekt and will hinder old apps from connecting to the platform. We will update the existing apps to the new manifest format, but you might have to update your apps to the new manifest format as wll.

Improved Documentation

We are working on improving the documentation of Arkitekt. While the current documentation is comprehensive, it can be overwhelming for new users. While we will keep the expert section we will also add a new section for beginners and intermediate users, that will guide them through the platform step by step. We are also working on improving the search functionality of the documentation, so that users can easily find the information they are looking for. All of which has not yet happend.

Improved Backend Stack

We are working on improving the backend stack of Arkitekt. Currently Arkitekt is built on top of a stack of Graphene, Django, and Postgres. While We are happy with the two latter, Graphene has some limitations and we are going to replace Strawberry which is a new GraphQL library that provides a more type safe and intuitive API. This stack is is already powering everything that you are seing here, and we are working on making it more stable and reliable.

Performance Updates

A major focus of the development on Rekuest will be on improving the performance of the platform, when assigning tasks. While the current version of Arkitekt is not slow by any means, we are working of scqueezing out every bit of performance from the platform, when it comes to distributed task assignment. One such tasks is going to be the introduction of "ephemeral" tasks that will not have to be stored in the database and can be assigned directly to the worker. This feature does not yet exist, but we are working on it.

Improved API Design

We are working on improving the API design of Arkitekt. While the current API is powerful, it does not follow all the best practices. THe next version of Arkitekt will have a more consistent and intuitive API based on the GraphQL best practices.

Improved Security

We are working on improving the security of Arkitekt. While currently Arkitekt is secure by design, most security features are not enforced. The next version of Arkitekt will have more security features that are enforced by default.