Improve scheduler actor by TheHeroBrine422 · Pull Request #63 · FyraStack/odorobo

TheHeroBrine422 · 2026-07-10T06:38:40Z

intended changes for this PR

improve actor cache
- currently the actor cache is setup in a way that doesn't really support the scheduler updating it when messages happen. this makes the cache very difficult to use because it can't be relied on for up to date information which means we can barely use it as a cache. a specific issue i hit with this was trying to implement affinity/placement groups. if we rely on the cache data for calculating affinity, we could break the affinity rules due to race conditions.
- i am replacing the generic version I made with one that is just integrated into the scheduler directly, because otherwise I don't really feel comfortable doing direct modifications to the cache in other segments of the scheduler
- on top of this, the vm cache arguably needs slightly different code because we basically need the ability to query by either actor id or vmid, and so it needed some sort of modification to support this either way.
- todo: have messages sent to scheduler directly update the cache
Placement groups #28
improve scheduling algorithm
- i realized that currently the scheduling algorithm doesnt consider the resource requirements of the new vm. so for example if an agent had 15/16 vcpus scheduled, and then a new VM was to be scheduled that needed another 16 vcpus, it could theoretically schedule it to that agent. which would be bad

All that is left is testing and review by another maintainer.

this was done for two reasons: 1) the vm and agent actor caches have slightly different requirements, specifically around keys (actorid vs vmid) 2) the scheduler needs direct control of the cache so the cache can be updated in response to messages it forwards. being possibly up to 1 second out of date in this data could cause problems. In theory the scheduler could update the generic cache, but I don't think using a generic cache if we need to modify it directly anyway really makes sense. Signed-off-by: Caleb Jones <caleb@calebgj.io>

Signed-off-by: Caleb Jones <caleb@calebgj.io>

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

…k/odorobo into caleb/fix-scheduler-actor-cache

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security

clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

…ctor-cache

halfcyan · 2026-07-16T23:39:26Z

i will look at this when i get back home. have you run it through clippy?

halfcyan · 2026-07-17T01:58:46Z

-    ) -> Result<RemoteActorRef<AgentActor>, Report> {
-        let mut best_agent = None;
-        let mut best_agent_score = 0u32;
+    // someone should likely give caleb a firm talking to about code duplication due to this section, but things are just different enough that trying to make them one function requires usage of a lot of generics which feels even worse. so i dont know what to do. cappy please fix. i hate this.


i don't think that it's that big of a deal here tbh. it's functional and readable in a way that you might lose if you were to try to use generics.

halfcyan · 2026-07-17T02:00:41Z

+// todo: i dont like the way this cache is setup. I think we may need to change it later, but it is hard to figure out what the optimal solution is without doing it at least once.
+//  especially when we haven't fully made decisions about some other things.
+//
+// todo: we should improve the cache to not have agents and vms send the full data on every update.


i'm sure there are off the shelf delta libraries. hell, git's is pretty decent. i can look into this a bit probably? not necessary for this pr but worth putting in outline or something. or opening a new pr/issue.

halfcyan · 2026-07-17T02:01:08Z

+//  I looked at kameo streams to make this better, but they aren't really intended for this kind of long term update use case.
+//  They use rust futures::stream which seems to be more intended for you have an iterator for example that will create data, but not like full on sending messages.
+//  This could likely be done pretty easily by having two get data messages.
+//   Option 1: One that creates a session and sends the full data and then only sends diffs after that.


this sounds more robust to me tbh

halfcyan · 2026-07-17T02:01:53Z

+    //  idk. will figure it out later.
+    //
+    //  new related problem: i just realized vmid, actorid pairs dont have to be unique.
+    //  if a vm is migrating from one actor to another, there might be two actors with the same vmid.


why was my first thought here to use a linked list

halfcyan · 2026-07-17T02:02:39Z

+    //  if a vm is migrating from one actor to another, there might be two actors with the same vmid.
+    //
+    //  additional context (05/05/2026): we almost may need a way to store them without a ulid, due to how CH migration works.
+    //  the question becomes if we want to abstract CH migration away entirely from the scheduler.


what would the scope be like on this? i don't think it needs to be through the scheduler, but i think the scheduler needs to integrate with it. we need to rebalance nodes occasionally.

halfcyan · 2026-07-17T02:04:34Z

+    //  the best solution for at least some of this is almost certainly having an external reliable DB (such as etcd) to store some of these things permanently.
+    //  we will need that specifically for what VMs are supposed to be running, because if a large percentage of the cluster goes down, including the manager, we need a way to recover.
+    //  and i dont think leaving that on dashboard which could have high latency is a good idea.
+    //  alternatively we could have the other manager nodes try to keep track of that data, but i think we are going to run into issues with keeping the state consistent between all nodes.


yeah honestly i think it's worth having a separate distributed db for this tbh. we don't want it to be integrated with odorobo in any way beyond like... a docker compose stack kinda thing. if odorobo fails, the database must retain data as it was when odorobo went down. i wonder if we could host a db with anyone who we peer with like cloudflare?

halfcyan · 2026-07-17T02:06:27Z

+
+                parent_actor_ref.link_remote(&vm_actor).await?;
+
+                let vm_actor_id = vm_actor.id();


is there a reason you can't just pass vm_actor.id() for this since it only gets used in the keepalive_tasks.insert() function call?

meant to leave this on line 151

halfcyan · 2026-07-17T02:07:46Z

+            if let Ok(data) = actor_ref.ask(&GetVMInfo { vmid: None }).await {
+                let vmid = data.vmid;
+
+                vm_actorid_ulid_map.insert(actor_ref.id(), vmid); // should we be doing this on every loop? idk. but we at least need to do it on the first iteration given we don't know the mapping before that


i feel like that's a good way to eat a couple cycles if it's not necessary for each loop

halfcyan · 2026-07-17T02:14:07Z

-    async fn on_update(
+    // this function intentionally only checks against the cache. this has some positives and negatives:
+    // positive: it will never trigger any network requests so its very fast, and having to do network requests for scoring whenever we want to schedule a vm is likely a bad idea
+    // negative: it technically has a delayed view of the cluster, meaning that some things that happened in the future, may not exist yet. so we need to be careful about how this is done so affinity rules are not accidentally broken. mostly this means, if we do anything that could affect the outcome of an affinity rule (ex: network request to an agent), we need to update the cache, before we do the action.


assuming anything later in the cache is referencing anything that changed as a result of this, i think it's fine. so like if you have multiple VM creations get scheduled, as long as each subsequent one references the one before it.

TheHeroBrine422 added 13 commits May 1, 2026 12:04

improve scheduling algorithm to use estimated future values.

09c41f0

Signed-off-by: Caleb Jones <caleb@calebgj.io>

committing before lunch so i dont have code just sitting on my machine

d0dd375

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor caches and placement groups

4449502

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor (still not finished)

7a63518

Signed-off-by: Caleb Jones <caleb@calebgj.io>

more work on scheduler affinity stuff. still not finished.

11186cd

Signed-off-by: Caleb Jones <caleb@calebgj.io>

theoretically write scheduler affinity scoring. need to test this.

2202c17

Signed-off-by: Caleb Jones <caleb@calebgj.io>

remove ipam actor

fedb58b

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

e26b7d7

fix bug with getting all agent and vm actors in actor finders

29c2a84

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

99e313f

commiting stuff before cypress rebase. was in the middle of testing

2fde501

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'caleb/fix-scheduler-actor-cache' of github.com:FyraStac…

da311a3

…k/odorobo into caleb/fix-scheduler-actor-cache

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:14

Merge branch 'main' into caleb/fix-scheduler-actor-cache

5274b03

Signed-off-by: Caleb Jones <caleb@calebgj.io>

cargo fmt

674c5a9

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:22

fix bugs created by merge commit

06c7885

cargo fmt

2f1ad4e

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 4 commits July 11, 2026 03:13

random debugging stuff

1561884

Merge branch 'main' into caleb/fix-scheduler-actor-cache

a575032

fix: remove unused AffinityDirection enum

188a2b9

feat: remove VMs from actors if unreachacble by updaters.

43b6f56

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 2 commits July 16, 2026 05:37

chore: cargo fmt

a784bab

Merge remote-tracking branch 'origin/main' into caleb/fix-scheduler-a…

e42ee50

…ctor-cache

TheHeroBrine422 changed the title ~~Caleb/fix scheduler actor cache~~ Improve scheduler actor Jul 16, 2026

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

fix: better extended_vm_set removal and setup

81d5b9c

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

chore: clippy and fmt

ecbc2e9

TheHeroBrine422 marked this pull request as ready for review July 16, 2026 20:23

TheHeroBrine422 requested review from halfcyan and korewaChino July 16, 2026 20:23

halfcyan approved these changes Jul 17, 2026

View reviewed changes


		parent_actor_ref.link_remote(&vm_actor).await?;

		let vm_actor_id = vm_actor.id();

Uh oh!

Conversation

TheHeroBrine422 commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-advanced-security AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halfcyan commented Jul 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants