Skip to content

Fix Swoole Cache store races and improve hot-path performance#414

Open
binaryfire wants to merge 19 commits into
0.4from
feat/swoole-store-concurrency-intervals
Open

Fix Swoole Cache store races and improve hot-path performance#414
binaryfire wants to merge 19 commits into
0.4from
feat/swoole-store-concurrency-intervals

Conversation

@binaryfire

@binaryfire binaryfire commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes several correctness issues in the Swoole table cache store under cross-worker access. The most important one is add(): it previously checked exists() and then wrote the value as two separate operations, so two workers could both observe a missing key and both return true. It also treated physically present but expired rows as existing. That breaks the normal cache add() contract, which is used as a single-winner primitive for dedupe, replay protection, and similar workflows. A concrete example is Fortify's two-factor TOTP replay protection: it writes a replay cache key with the cache repository's add() method so the same code cannot be accepted twice during the replay window. With the old Swoole store, two workers racing on the same replay key could both win.

The same shared-row problem affected other cache operations. increment(), decrement(), and touch() could lose concurrent updates, cleanup and eviction could delete a row that another worker had just recreated, and the old read path wrote whole rows back while updating hit metadata, which could clobber a concurrent value or expiration change. This PR adds short shared row locks around the small critical sections that need atomicity, keeps normal live reads lock-free, changes hit metadata writes to narrow metadata updates, and rechecks cleanup/eviction candidates before deleting them.

Performance is part of the fix. The old store wrote a full row back on every cache hit, even when the active eviction policy did not use hit metadata. It also ran stale cleanup after every successful put() and add() miss, which scanned the table and made write cost grow with table size. The new store skips hit-metadata writes for TTL/no-eviction reads, updates only the needed metadata for LRU/LFU reads, replaces per-write table scans with an O(1) memory-pressure check, avoids building a Carbon object for production timestamp reads, and removes a Collection allocation from the Swoole table write guard. Normal reads still do not take row locks.

This PR also fixes interval cache refresh for Swoole. Interval metadata was stored in the shared table, but the list of registered interval keys was local PHP memory, so the manager process could not discover intervals registered by workers. The manager timer also only ran eviction. Interval registrations are now stored in shared internal metadata rows, a small sharded index lets the manager discover them, and a dedicated interval refresh timer refreshes due intervals without adding work to normal cache misses or hot reads.

The Swoole store now also implements the framework cache lock API. This is a natural extension of the same shared-memory machinery needed for the cache fixes above: row locks already had to exist for add() and the other single-key mutation paths, so wiring them into a real LockProvider does not add overhead to normal reads or writes just to support locks.

The design keeps the driver in the same role: a very fast in-memory cache backed by Swoole\Table. It avoids global locks for ordinary single-key writes, avoids running user callbacks while a row lock is held, keeps interval index work on the manager timer instead of request paths, and maps logical keys through seeded xxh128 before touching Swoole\Table so long user keys cannot be truncated into the same physical row. Writes to different keys only contend when they land on the same lock stripe; same-key writes serialize because that is the required behavior for a shared key.

Microbenchmarks

Median ops/sec over five CLI benchmark rounds on the same machine, PHP binary, and Swoole extension. These are raw SwooleStore method calls, not cache repository or facade calls. Write benchmarks are shown at several live-row fill levels because 0.4 scanned the table on every successful write, while this PR uses an O(1) memory-pressure check and only scans under pressure. N/A rows are after-only because cache locks did not exist in 0.4.

Operation Before ops/sec PR ops/sec Change Notes
get() hit, TTL policy 62,962 310,558 4.9x faster Hot reads no longer write hit metadata.
get() hit, no-eviction policy 62,540 314,251 5.0x faster Hot reads no longer write hit metadata.
get() hit, LRU policy 62,858 207,015 3.3x faster Partial metadata update instead of full-row write.
get() hit, LFU policy 63,555 231,686 3.6x faster Uses Swoole table incr() for hit count.
put(), empty table 1,315 192,421 146.4x faster O(1) memory-pressure check instead of full stale scan.
put(), 1k live rows 434 194,406 447.6x faster Write cost stays roughly flat as table grows.
put(), 10k live rows 61 188,608 3,093.5x faster Old path scanned the table on every write.
put(), 20k live rows 31 188,876 6,028.3x faster Old path scaled with table size.
add() miss, empty table 515 157,348 305.5x faster Single-winner write plus O(1) pressure check.
add() miss, 1k live rows 284 156,697 551.4x faster Write cost stays roughly flat as table grows.
add() miss, 10k live rows 59 160,069 2,734.0x faster Old path scanned the table on every successful add.
add() miss, 20k live rows 31 156,471 5,004.8x faster Old path scaled with table size.
add() existing live key 1,112,505 237,032 78.7% slower Now checks logical expiration under the row lock.
increment(), same key 46,499 159,597 3.4x faster Correct read-modify-write under a row lock.
increment(), 1,024 rotating keys 45,738 157,116 3.4x faster Unrelated-key throughput remains high.
touch() existing key 44,229 175,888 4.0x faster Partial expiration update under a row lock.
lock()->acquire()/release() N/A 54,841 New New supported operation.

The one slower benchmark is add() for an already-live key. The old path was only a physical exists() check; the new path has to check logical expiration under the row lock to enforce the add() contract. It still runs at about 237k ops/sec in this CLI benchmark.

Interval refresh runs on the manager timer, off request paths. In the same benchmark, one refresh tick costs about 66 microseconds with no intervals, 78 microseconds with one interval, and 917 microseconds with 100 intervals.

Row Concurrency and Locks

  • Added SwooleTableState so each Swoole table has shared pre-fork row-lock stripes and a shared hash seed.
  • Changed the store to use bounded seeded physical table keys instead of passing raw user keys to Swoole\Table, avoiding Swoole key truncation and separating user rows from internal control rows.
  • Fixed add() so it is atomic for a single key and treats expired rows as logically absent.
  • Fixed put(), increment(), decrement(), touch(), and forget() by protecting their single-key mutations with row locks.
  • Kept live get() lock-free and cheaper: TTL/no-eviction reads only read and unserialize, LRU reads update only last_used_at, and LFU reads use Swoole's numeric incr() for the hit counter.
  • Changed production timestamp reads to use microtime(true) while preserving Carbon::setTestNow() behavior for tests.
  • Changed the Swoole table value-size guard from a Collection/closure path to a plain loop, preserving the same exception while avoiding per-write allocation.
  • Changed stale cleanup and eviction to scan without locks, then lock and recheck each delete candidate before deleting it.
  • Removed per-write stale cleanup scans. Normal writes now pay an O(1) memory-pressure check, and stale cleanup runs on the timer path or under memory pressure.
  • Changed flush() to delete user cache rows while preserving internal control rows for locks and interval metadata.
  • Added SwooleLock and made SwooleStore implement LockProvider and CanFlushLocks, including owner-aware release, force release, restore, refresh, remaining lifetime, and flushLocks().
  • Added sequential and process-level regression tests for atomic add(), concurrent increments, lock acquisition, long key hashing, eviction/flush behavior, LRU/LFU metadata updates, and lock API behavior.

For more details, see: docs/plans/2026-07-03-swoole-store-row-concurrency-and-locks.md

Interval Cache Refresh

  • Moved interval metadata into seeded internal i: control rows and registered those metadata keys in sharded x: index rows.
  • Changed refreshIntervalCaches() to discover intervals through the shared index, so the manager process can refresh intervals registered by workers.
  • Replaced the eviction-only timer listener with CreateSwooleTimers, which registers both the existing eviction timer and a dedicated interval refresh timer.
  • Added interval_refresh_interval config for the manager refresh cadence.
  • Preserved same-instance fallback: the worker that registered an interval can still resolve it before the first manager tick.
  • Kept normal misses cheap: get() only uses interval fallback for keys registered on the current store instance and never scans the shared interval index.
  • Kept interval discovery off request paths: the manager timer reads at most 64 small index rows per Swoole store, then checks only registered intervals.
  • Added refresh claims with refreshingAt so timer refreshes and local fallback do not run the same resolver at the same time.
  • Added stale-claim recovery so a worker crash after claiming an interval does not freeze refreshes until restart.
  • Runs interval resolvers outside row locks, completes metadata only after the public cache value is written, and clears/reports failures without marking failed refreshes as successful.
  • Preserves interval metadata and index rows through normal flush(), stale cleanup, and eviction while allowing the public cached value to be flushed and recreated by the next refresh.
  • Added regression tests for manager discovery, multi-interval index shards, same-instance fallback, first-tick behavior, due checks, stale claims, resolver exceptions, failed writes, null-returning resolvers, and timer wiring.

For more details, see: docs/plans/2026-07-03-swoole-store-interval-cache-refresh.md

Validation

  • composer fix
    • PHP CS Fixer: clean
    • PHPStan: no errors
    • Parallel tests: passing

Summary by CodeRabbit

Release Notes

  • New Features

    • Added atomic lock support for the Swoole cache driver, including lock acquire/release/refresh and force-release.
    • Introduced separate Swoole timers for eviction and interval cache refresh, configurable via interval_refresh_interval.
  • Bug Fixes / Improvements

    • Reworked Swoole cache concurrency and interval refresh coordination across workers to prevent clobbering and stale refresh issues.
    • Improved putMany() result semantics across cache stores and tightened eviction/refresh correctness.
  • Documentation

    • Updated cache/queues/session/events/Octane docs to include Swoole lock and interval-refresh behavior.
  • Tests

    • Added extensive Swoole concurrency, interval refresh, timer wiring, and heap/putMany coverage.

binaryfire added 14 commits July 3, 2026 15:55
Add the signed-off implementation plans for the Swoole table cache work.

The row-concurrency plan records the agreed design for shared pre-fork table state, striped row locks, hashed physical table keys, lock-free reads, row-safe mutators, cleanup and eviction behavior, and Swoole cache lock support.

The interval-cache plan is committed alongside it so the follow-up work has its reviewed design preserved before implementation starts.
Introduce SwooleTableState as the cache driver's shared pre-fork state object. It owns the Swoole table, the seeded physical-key hash seed, and the shared striped Swoole\Atomic row locks used by row lifecycle operations.

Refactor SwooleTableManager and CacheManager so Swoole cache stores receive that shared state instead of a raw table. This keeps the table and its synchronization primitives together and ensures workers inherit the same shared-memory objects.

Rework SwooleStore around short seeded physical table keys for user rows, interval metadata, and lock rows. This avoids Swoole table key truncation, separates control rows from user cache entries, and lets flush operations preserve internal state intentionally.

Keep live cache reads lock-free while replacing the old full-row get metadata write with policy-specific metadata updates. LRU uses a partial timestamp write, LFU uses Swoole's numeric incr, and ttl/noeviction reads do not write metadata.

Protect single-key mutators with the shared row stripes. add now performs a logical expired-aware check-and-set under one row lock, increment/decrement/touch preserve current expiration without lost updates, and forget coordinates with concurrent writers.

Make cleanup and eviction race-safe by scanning without locks, then rechecking and deleting each candidate under its row lock. Eviction is gated behind memory pressure on write paths, skips control rows, and stops when no progress can be made.

Move the existing local interval metadata onto hashed control keys so this implementation stands alone before the follow-up interval refresh redesign. Normal cache flushes delete user rows while preserving interval metadata.

Add SwooleLock and wire SwooleStore into LockProvider and CanFlushLocks. Lock acquire, release, refresh, owner lookup, force release, and lock flushing all use the same shared row-lock machinery.
Update the Swoole store test suite to work through the new hashed physical table keys instead of raw logical Swoole table keys.

Add regressions for expired-aware add semantics, live-row preservation, policy-specific hit metadata updates, expired-read cleanup, increment/decrement expiration preservation, touch behavior, control-row preservation, memory-pressure-gated eviction, exact-expiration stale cleanup, long-key collision avoidance, and seeded hash behavior.

Cover SwooleStore's new lock-provider surface in sequential tests, including lock acquisition, owner-checked release, force release, restoreLock, refresh, remaining lifetime, flushLocks, and funnel support.
Add process-level SwooleStore tests that share one pre-created SwooleTableState across forked Swoole processes, matching the production pre-fork table and atomic-lock model.

Cover the races that motivated the fix: concurrent add on a missing key, concurrent add over a logically expired physical row, concurrent increment updates, and concurrent lock acquisition. Each regression asserts that only the correct winner count is observed and that the final stored value remains consistent.

The test opts out of coroutine wrapping and terminates child processes after writing results so inherited PHPUnit/Testbench shutdown handlers do not interfere with the parent runtime.
Remove SwooleStore from the negative funnel support assertions now that the store implements LockProvider through a real shared Swoole table lock implementation.

Keep the remaining unsupported-store coverage for stores that still do not provide cache locks, while positive Swoole funnel behavior is covered in the Swoole store test suite.
Update the current Boost documentation to reflect that the Swoole cache driver now supports atomic cache locks.

Add Swoole to the lock-capable driver lists used by cache lock flushing, unique listeners, unique jobs, WithoutOverlapping jobs, and session blocking.

Only src/boost/docs is changed; the stale v0.3 docs tree remains untouched.
Store interval registrations as shared control-plane metadata under seeded internal table keys and register those metadata rows in sharded interval index rows. This lets the manager process discover interval caches without relying on worker-local PHP arrays.

Refreshes now claim metadata with a microsecond token before resolver execution, run resolvers outside row locks, complete only after the public cache value is written, and clear or reclaim stale claims safely. Same-instance fallback uses the same claim path while ordinary cache misses still avoid interval-index work.

This fixes manager-discovery gaps, overlapping refresh races, stale claim freezes, resolver-failure timestamp poisoning, and interval control-row namespace leakage while keeping request hot paths fast.
Replace the eviction-only Swoole cache timer listener with a listener that registers both eviction and interval refresh timers for every configured Swoole cache store. The manager process now periodically calls refreshIntervalCaches(), allowing shared interval metadata to seed and refresh public cache values across workers.

Add a small SwooleTimer wrapper around Swoole\Timer::tick so timer registration can be tested cleanly without patching static extension calls. Wire the renamed listener through CacheServiceProvider and add the interval_refresh_interval configuration default in milliseconds.

This keeps interval refresh work out of request hot paths while giving interval caches a dedicated short refresh cadence independent of stale-record eviction.
Add dedicated coverage for the shared interval metadata and index design. The tests verify seeded internal metadata keys, sharded index registration, idempotent registration, same-instance fallback before the first timer tick, cross-instance manager refresh, due checks, and successful metadata completion.

Cover concurrency and failure edges around refresh claims: fresh claims block overlapping resolver work, stale claims are reclaimable, claim completion and clearing do not overwrite newer claims, resolver exceptions are reported or rethrown appropriately, and failed public value writes clear claims without marking refresh success.

Add regressions for control-row preservation through flush and eviction, cheap generic misses that do not consult the interval index, index-row expiration renewal, multi-interval shared-index discovery including same-shard accumulation, and null-returning resolvers preserving live public cache rows.
Add focused tests for the Swoole cache timer listener. The suite verifies that each configured Swoole cache store receives both an eviction timer and an interval refresh timer, while non-Swoole stores are ignored.

The tests cover configured intervals and defaults, and exercise the registered callbacks to ensure they resolve the configured cache store and call evictRecords() and refreshIntervalCaches() respectively.

This protects the manager-process wiring that makes shared interval cache refreshes automatic without requiring a live Swoole timer in the test process.
Document the interval_refresh_interval option in the Swoole table cache configuration example and explain how it differs from eviction_interval. Eviction remains responsible for stale-record pruning and memory-pressure cleanup, while interval_refresh_interval controls manager-driven interval cache refresh checks.

Update the Octane cache interval docs to refer to Hypervel's cache system and describe the first-tick behavior introduced by shared manager refreshes: shared values are seeded by the manager refresh timer, while the registering worker can still resolve through local fallback immediately.

Only the current Boost docs are updated; the old docs/docs tree remains untouched.
Document the final performance decisions added after review of the Swoole cache row-concurrency work.

The plan now records why production timestamp reads use microtime while preserving Carbon::setTestNow for tests, and why the Swoole table write guard uses a plain loop instead of allocating a Collection and closure on every write.

This keeps the design record aligned with the final implementation and explains the performance/correctness tradeoffs for future maintenance.
Use microtime(true) for production Swoole cache timestamps while keeping the existing Carbon path when test time is frozen.

getCurrentTimestamp() is used by reads, writes, expiration checks, locks, and interval claims. Avoiding Carbon object construction removes a hot-path allocation without changing the stored timestamp shape or breaking Carbon::setTestNow-based tests.

Focused Swoole cache tests and the full composer fix pipeline passed after this change.
Replace the Collection/closure path in SwooleTable::set() with a plain foreach loop while preserving the string column size guard and ValueTooLargeForColumnException behavior.

Every Swoole cache write goes through this wrapper, including put, add, increment, touch, lock writes, interval writes, and LRU metadata updates. Removing the per-write Collection and closure allocations keeps the guard intact while reducing write-path overhead.

Add a focused regression test that writes a nine-character value into an eight-byte string column and verifies the same exception path still fires.
@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8e9d0529-0f3d-491c-a2f5-ec660033b506

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR redesigns the Swoole cache store around shared table state, striped row locks, hashed internal keys, lock-provider support, and manager-driven interval refresh timers. It also updates docs and expands test coverage for concurrency, intervals, locks, eviction, and timer wiring.

Changes

Swoole cache store redesign

Layer / File(s) Summary
Design plans
docs/plans/2026-07-03-swoole-store-row-concurrency-and-locks.md, docs/plans/2026-07-03-swoole-store-interval-cache-refresh.md, docs/plans/2026-07-04-swoole-store-review-follow-up-hardening.md
Two planning docs describe Swoole row locking, hashed keys, lock-provider support, interval refresh, timer ownership, and follow-up hardening items.
Table state and manager wiring
src/cache/src/CacheManager.php, src/cache/src/SwooleTableState.php, src/cache/src/SwooleTableManager.php, src/cache/src/SwooleTable.php
SwooleTableState adds striped atomic locks and backoff; SwooleTableManager returns table states; SwooleTable::set() validates string column sizes inline; CacheManager passes the table state into SwooleStore.
Store, locks, and timer wiring
src/cache/src/SwooleStore.php, src/cache/src/SwooleLock.php, src/cache/src/SwooleTimer.php, src/cache/src/Listeners/CreateSwooleTimers.php, src/cache/src/CacheServiceProvider.php, src/foundation/config/cache.php, src/boost/docs/cache.md, src/boost/docs/events.md, src/boost/docs/octane.md, src/boost/docs/queues.md, src/boost/docs/session.md
SwooleStore adds hashed keys, row-locked cache mutations, interval metadata/index handling, eviction counting, and lock-provider methods; the new lock, timer, provider, config, and docs updates wire and describe that behavior.
Core cache behavior tests
tests/Cache/CacheSwooleStoreTest.php, tests/Cache/FunnelUnsupportedStoresTest.php, tests/Cache/CacheDatabaseStoreTest.php, tests/Cache/CacheMemoizedStoreTest.php, tests/Cache/CacheRetrievesMultipleKeysTest.php, tests/Cache/CacheStackStoreTest.php, tests/Cache/CacheTaggedCacheTest.php, tests/Cache/LimitedMaxHeapTest.php, tests/Integration/Cache/FailoverStoreTest.php
CacheSwooleStoreTest expands coverage for reads, writes, eviction, flush, and locks, while the other cache tests update putMany semantics and heap validation.
Concurrency, interval, and timer tests
tests/Cache/CacheSwooleStoreConcurrencyTest.php, tests/Cache/CacheSwooleStoreIntervalTest.php, tests/Cache/CreateSwooleTimersTest.php
New suites cover forked-process cache and lock concurrency, interval registration and refresh claims, and Swoole timer scheduling and callbacks.

Estimated code review effort: 5 (Critical) | ~120 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the PR’s main focus on fixing Swoole cache store races and improving hot-path performance.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/swoole-store-concurrency-intervals

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the Swoole cache store for multi-worker correctness and improves hot-path performance by introducing shared row-lock striping, seeded hashed table keys to avoid Swoole key truncation collisions, cheaper hit-metadata updates, and maintenance work moved off per-write paths. It also adds first-class support for the framework cache lock API and implements cross-worker interval cache refresh driven by manager-process timers.

Changes:

  • Add shared SwooleTableState (row-lock stripes + hash seed) and refactor SwooleTableManager / CacheManager wiring to pass state into SwooleStore.
  • Make SwooleStore atomic for add() and other single-key mutations, keep live reads lock-free, and add lock + interval-refresh support (including new manager timers).
  • Expand the test suite and docs to cover new concurrency/lock/interval semantics and configuration.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/Cache/FunnelUnsupportedStoresTest.php Removes the “Swoole doesn’t support locks” assertion; keeps stack/session as unsupported for funnels.
tests/Cache/CreateSwooleTimersTest.php Adds coverage for manager timers registering eviction + interval-refresh ticks and dispatching callbacks to the correct store.
tests/Cache/CacheSwooleStoreTest.php Updates/expands store-level tests for atomic add(), LRU/LFU metadata behavior, flush/eviction behavior, key hashing, and lock API behavior.
tests/Cache/CacheSwooleStoreIntervalTest.php Adds dedicated interval-refresh tests (shared index discovery, claims, failure handling, flush/eviction invariants).
tests/Cache/CacheSwooleStoreConcurrencyTest.php Adds multi-process regression tests for add(), increment(), and lock acquisition under contention.
src/foundation/config/cache.php Adds interval_refresh_interval config for Swoole store manager refresh cadence.
src/cache/src/SwooleTimer.php Introduces an injectable wrapper around Swoole\\Timer::tick() to simplify timer testing.
src/cache/src/SwooleTableState.php Adds shared state for a table: striped Atomic locks and a shared hash seed.
src/cache/src/SwooleTableManager.php Refactors manager to resolve/cache SwooleTableState instead of raw tables; adds createState().
src/cache/src/SwooleTable.php Removes Collection allocation from the column-size guard and uses a simple loop.
src/cache/src/SwooleStore.php Implements row-locking for key mutations, seeded hashed keys, safe eviction/stale cleanup behavior, interval shared index + refresh claims, and lock-provider support.
src/cache/src/SwooleLock.php Adds a Swoole-backed lock implementation supporting acquire/release/forceRelease/refresh/remaining-lifetime.
src/cache/src/Listeners/CreateTimer.php Removes the old eviction-only manager timer listener.
src/cache/src/Listeners/CreateSwooleTimers.php Adds manager timers for both eviction and interval refresh per configured Swoole store.
src/cache/src/CacheServiceProvider.php Wires the new CreateSwooleTimers listener on manager start.
src/cache/src/CacheManager.php Updates Swoole driver creation to use SwooleTableState (not a raw table).
src/boost/docs/session.md Documents swoole as supporting atomic locks for session blocking.
src/boost/docs/queues.md Documents swoole as supporting atomic locks for unique jobs and WithoutOverlapping.
src/boost/docs/octane.md Updates wording to Hypervel and documents first-tick interval refresh behavior + same-instance fallback.
src/boost/docs/events.md Documents swoole as supporting atomic locks for unique listeners.
src/boost/docs/cache.md Documents new Swoole interval refresh config and adds swoole to flushLocks support list.
docs/plans/2026-07-03-swoole-store-row-concurrency-and-locks.md Adds detailed design/plan doc for row-locking, hashed keys, eviction changes, and lock API implementation.
docs/plans/2026-07-03-swoole-store-interval-cache-refresh.md Adds detailed design/plan doc for shared interval index and manager-driven refresh claims.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/Cache/CacheSwooleStoreConcurrencyTest.php
Comment thread src/cache/src/SwooleStore.php Outdated
@greptile-apps

greptile-apps Bot commented Jul 3, 2026

Copy link
Copy Markdown

Greptile Summary

This PR delivers a thorough correctness and performance overhaul of the Swoole cache store. The most critical fix is making add() atomic: it now executes the existence check and write as a single row-locked unit, closing the race that let two concurrent workers both see a missing key and both return true (directly affecting primitives like Fortify\u2019s TOTP replay protection). Per-write stale-table scans are replaced with an O(1) memory-pressure check, eliminating the O(n) write cost that scaled with table size.

  • New SwooleTableState \u2014 provides pre-fork striped Atomic row locks and a seeded xxh128 key hasher; SwooleStore now implements LockProvider + CanFlushLocks, adding a full cache-lock API backed by the same machinery.
  • Interval refresh \u2014 metadata and a sharded index written to shared Swoole table rows let the timer discover intervals registered by any worker; stale-claim recovery prevents a crashed refresher from freezing future refreshes.
  • Ancillary fixes \u2014 LimitedMaxHeap::insert no longer inserts items worse than the current top when full; DatabaseStore::putMany stops returning false on zero-affected-rows upserts; RetrievesMultipleKeys, StackStore, and TaggedCache putMany all attempt every key before returning a combined result.

Confidence Score: 5/5

Safe to merge. The core concurrency fixes are well-reasoned, critical sections are small and non-yielding, and the design correctly shares Atomic objects via pre-fork shared memory.

The atomic add() fix, lock-free reads, row-locked mutations, and LimitedMaxHeap correction are all correct and well-tested. The two style notes (timer in worker 0 instead of the manager process, and the hasSeparateLockStore documentation gap) are non-blocking observations. No correctness regressions were found.

CreateSwooleTimers.php — the switch from the Swoole manager process to worker 0 is worth understanding if large-table eviction latency in worker 0 becomes observable in production.

Important Files Changed

Filename Overview
src/cache/src/SwooleStore.php Major rewrite fixing atomic add(), lock-free hot reads, row-lock-protected mutations, interval refresh, and adding a full cache lock API. Logic is well-reasoned and consistent.
src/cache/src/SwooleTableState.php New file providing striped row-lock Atomics and a seeded hash. Acquisition ordering in withAllRowLocks prevents deadlock. Crash recovery limitation acknowledged in comments.
src/cache/src/SwooleLock.php New Lock implementation delegating to SwooleStore. Implements acquire, release, forceRelease, refresh, and remaining lifetime correctly.
src/cache/src/Listeners/CreateSwooleTimers.php Replaces manager-process CreateTimer with worker-0 timers. Both eviction and interval refresh now run in worker 0’s event loop rather than the dedicated manager process.
src/cache/src/LimitedMaxHeap.php Fixes a real bug: old insert() always called parent::insert() even when the heap was full and the candidate was worse than the top, causing the heap to exceed its limit.
src/cache/src/DatabaseStore.php putMany now correctly handles empty arrays and no longer returns false when upsert reports zero affected rows (identical-value case).
src/cache/src/RetrievesMultipleKeys.php putMany now attempts all keys even after an earlier failure, initializes result to true (correct for empty arrays), and removes the manyResult ?: false quirk.
src/cache/src/SwooleTableManager.php Now returns SwooleTableState instead of SwooleTable, wiring in the row-lock Atomics and hash seed. Logic is unchanged except for state wrapping.

Reviews (4): Last reviewed commit: "fix: normalize cache putMany results" | Re-trigger Greptile

Comment thread src/cache/src/SwooleTableState.php
Comment thread src/cache/src/SwooleStore.php
Comment thread src/cache/src/SwooleStore.php
Comment thread src/cache/src/SwooleStore.php
Comment thread src/cache/src/SwooleStore.php

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/Cache/CacheSwooleStoreIntervalTest.php`:
- Around line 346-354: The tests around
testThrowingTimerResolverIsReportedAndCanRetry are leaving a mocked global
Container instance in place, which can leak the ExceptionHandler into later
tests. Update the affected test methods to wrap each
Container::setInstance($container) usage in try/finally, and in the finally
block restore or clear the global container state after the test finishes. Use
the existing Container::setInstance, ExceptionHandler, and
createState/createStore test setup points to locate and apply the fix.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cada0b1a-8d63-4973-b50d-420ec0be5929

📥 Commits

Reviewing files that changed from the base of the PR and between 916ff5b and 2eb651d.

📒 Files selected for processing (23)
  • docs/plans/2026-07-03-swoole-store-interval-cache-refresh.md
  • docs/plans/2026-07-03-swoole-store-row-concurrency-and-locks.md
  • src/boost/docs/cache.md
  • src/boost/docs/events.md
  • src/boost/docs/octane.md
  • src/boost/docs/queues.md
  • src/boost/docs/session.md
  • src/cache/src/CacheManager.php
  • src/cache/src/CacheServiceProvider.php
  • src/cache/src/Listeners/CreateSwooleTimers.php
  • src/cache/src/Listeners/CreateTimer.php
  • src/cache/src/SwooleLock.php
  • src/cache/src/SwooleStore.php
  • src/cache/src/SwooleTable.php
  • src/cache/src/SwooleTableManager.php
  • src/cache/src/SwooleTableState.php
  • src/cache/src/SwooleTimer.php
  • src/foundation/config/cache.php
  • tests/Cache/CacheSwooleStoreConcurrencyTest.php
  • tests/Cache/CacheSwooleStoreIntervalTest.php
  • tests/Cache/CacheSwooleStoreTest.php
  • tests/Cache/CreateSwooleTimersTest.php
  • tests/Cache/FunnelUnsupportedStoresTest.php
💤 Files with no reviewable changes (1)
  • src/cache/src/Listeners/CreateTimer.php

Comment thread tests/Cache/CacheSwooleStoreIntervalTest.php
Tighten the Swoole cache follow-up fixes from the bot review pass.

Store interval resolvers as serialized bytes so metadata serialization inside row-lock sections only handles scalar data. Resolver serialization now happens before the metadata lock is acquired, and resolver unserialization/invocation happens after the refresh claim lock is released.

Preserve existing interval refresh metadata when an interval is re-registered. This keeps lastRefreshedAt and fresh refreshingAt claims intact while still allowing the resolver and refresh interval to be updated for future refreshes.

Harden row-lock critical sections by using allowed_classes=false for increment payload unserialization, preventing object wakeups while the stripe is held and preserving the existing numeric failure behavior for non-numeric payloads.

Make control-row detection explicit for interval metadata, interval indexes, and lock rows. Unknown raw rows now behave like cache data and can be removed by flush or eviction instead of being protected forever.

Document the Swoole Table primitive limitation in SwooleTableState, including the requested @todo marker for future native CAS / set-if-absent / delete-if-current support.

Update the implementation plans to match the final resolver-bytes and explicit-control-key design.

Tests add regressions for object wakeups during increment, interval metadata serialization boundaries, interval re-registration state preservation, and raw legacy row cleanup.

Validation: vendor/bin/phpunit tests/Cache/CacheSwooleStoreTest.php; vendor/bin/phpunit tests/Cache/CacheSwooleStoreIntervalTest.php; composer fix.
Comment thread src/cache/src/SwooleTableState.php
Comment thread src/cache/src/SwooleStore.php
@binaryfire

Copy link
Copy Markdown
Collaborator Author

@coderabbitai @greptile-apps @copilot thanks for the reviews. I went through the inline comments and pushed the valid follow-up fixes in fe4da129.

Fixed in that commit:

  • hardened increment() so object payloads cannot run __wakeup() while a row stripe is held;
  • changed interval metadata to store resolver bytes, with resolver serialization before the metadata lock and resolver unserialization/invocation after the refresh claim lock is released;
  • preserved lastRefreshedAt and refreshingAt when an interval is re-registered;
  • changed isControlKey() to an explicit i: / x: / l: allowlist so legacy/raw rows are cleaned normally;
  • added a clarifying comment at the null-returning interval resolver fallthrough;
  • updated the plan docs to match the final design.

A few comments were intentionally left as no-code-change items and are answered inline:

  • the all-stripe flush() lock is the intended writer barrier for a maintenance operation;
  • spinlock crash recovery is limited by Swoole\Atomic not having owner-death detection, and timeout force-reset would be unsafe, so the limitation is documented with a @TODO for future native Swoole Table primitives;
  • the extra callback argument in the concurrency helper is harmless in PHP;
  • the test container is already reset by the globally registered AfterEachTestSubscriber.

Validation after the follow-up fixes: focused Swoole cache tests passed, composer fix passed, and the benchmark table in the PR description has been refreshed.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

@binaryfire, thanks for the detailed summary. Here's my take on the items you flagged as no-code-change:

  • flush() all-stripe lock: Agreed, using the writer barrier for a maintenance-only operation is a reasonable tradeoff.
  • Spinlock crash recovery: Makes sense — since Swoole\Atomic has no owner-death detection, a timeout-based force-reset could release a lock while the original holder is still mid-mutation, causing corruption. Documenting with a @TODO for future native primitives is the right call for now.
  • Extra callback argument in the concurrency helper: Agreed, harmless in PHP given loose arity checking for closures.
  • Test container reset via AfterEachTestSubscriber: Good to know, thanks for confirming that's already handled globally.

I'll take a fresh look at the fixes in fe4da129, including the increment() hardening against __wakeup(), the resolver-bytes change for interval metadata, the lastRefreshedAt/refreshingAt preservation on re-registration, and the explicit isControlKey() allowlist.

(=^・ω・^=) glad to see these tightened up


✏️ Learnings added
Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-03T20:37:33.955Z
Learning: In Hypervel's `SwooleStore` (src/cache/src/SwooleStore.php) and `SwooleTableState` (src/cache/src/SwooleTableState.php), row-lock crash recovery via a stale-timeout force-reset is intentionally not implemented because `Swoole\Atomic` has no owner-death detection; forcing a reset on timeout could release a lock while the original holder is still mid-mutation, causing data corruption. This limitation is documented with a `TODO` for future native Swoole Table primitives.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-03T20:37:33.955Z
Learning: In Hypervel's `SwooleStore`, `isControlKey()` uses an explicit allowlist of prefixes (`i:` for interval metadata, `x:` for the interval index, `l:` for lock rows) to identify internal control rows, so legacy/raw rows without these prefixes are cleaned up normally by eviction/stale-cleanup instead of being treated as protected control rows.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-03T20:37:33.955Z
Learning: In Hypervel's test suite, the container used in cache tests (e.g., tests under `tests/Cache/`) is already reset between tests by a globally registered `AfterEachTestSubscriber`, so individual tests don't need to manually reset the container.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@greptile-apps

greptile-apps Bot commented Jul 3, 2026

Copy link
Copy Markdown

All five items check out in fe4da129:

increment() wakeup guardunserialize($record['value'], ['allowed_classes' => false]) is in place. No object methods run while the stripe is held, and the (int)(result + $value) cast preserves the existing non-numeric fallthrough behavior.

Resolver serialized before the lockinterval() calls serialize(new SerializableClosure($resolver)) to produce the bytes before entering withRowLock, so the metadata row lock only wraps scalar reads/writes. Unserialization and invocation happen after withRowLock returns the claim in refreshIntervalCache.

Re-registration preserves lastRefreshedAt / refreshingAt — the existing !== null branch in interval() copies both fields from the existing row into $metadata before calling putIntervalMetadataByInternalKey.

isControlKey() is an explicit allowlist — the method now checks str_starts_with for i:, x:, and l: only. Unknown raw rows fall through and are treated as evictable user data.

@TODO in SwooleTableState — the class-level docblock and the @TODO for native CAS/set-if-absent/delete-if-current are present.

The explanations for the intentional no-code-change items (all-stripe flush lock as a writer barrier, spinlock crash recovery limitation documented rather than unsafe timeout-reset, extra callback argument harmlessness, and test container reset via the global subscriber) are all reasonable.

Everything looks good. The PR is in good shape.

@binaryfire binaryfire requested a review from albertcht July 3, 2026 20:50
Record the signed-off follow-up plan for PR #414 after the bot review and maintainer review loop.

The plan captures the concrete bugs being hardened: manager-process timer execution, over-eviction from the bounded heap, stale eviction candidates, interval stale-overwriter races, hot-key spin behavior, lock-free hit metadata trade-offs, and inconsistent multi-key write results.

It also documents the rejected alternatives and the final testing strategy so the implementation can be audited against the decisions that were reviewed before coding.
Move Swoole cache timer registration from OnManagerStart to AfterWorkerStart and elect non-task worker 0 as the single timer owner.

Interval refresh resolvers can use coroutine-backed framework services, so running them from the manager process is unsafe. The stale cleanup and eviction timer moves with the interval timer so Swoole cache maintenance has one worker-owned execution model.

Keep the eviction callback pointed at evictRecords() directly so flushStaleRecords() still runs periodically even when the table is below the memory threshold.

Update the Boost cache docs to describe the elected worker behavior and extend CreateSwooleTimers tests to cover worker 0, other workers, task workers, configured intervals, defaults, and callbacks.
Fix the bounded eviction heap so it rejects invalid limits and discards non-better candidates once full. This makes single-pass policy eviction honor eviction_proportion instead of allowing the candidate heap to grow beyond the intended batch size.

Replace full-row eviction candidates with compact fingerprints that include a non-cryptographic value hash and every mutable metadata column. Candidate deletion now re-reads under the row lock and deletes only if the row still matches the scanned fingerprint, preserving stale-delete protection without copying serialized payloads into PHP memory under pressure.

Guard interval refresh public-value commits by rechecking ownership after the resolver returns. The commit prep restamps the claim, lost claims return without writing or clearing, and the public write stays outside the metadata lock to avoid nested metadata-to-user row locking and eviction work inside the metadata critical section.

Add a small failed-CAS backoff to striped Swoole table locks after a short spin burst. The uncontended path remains one successful CAS, while hot-key contention no longer burns CPU indefinitely.

Keep cache hit metadata lock-free on the read hot path and document the accepted expired shell-row behavior. Add regression coverage for heap limits, single-pass LRU/LFU/TTL eviction, stale eviction candidates, lock-free hit shell rows, interval stale-overwriters, lost claims, commit-time cadence, stale-claim timeout selection, stderr exception fallback, and SwooleStore putMany failure aggregation.
Make fallback and native multi-key cache writes return true for empty batches, attempt every key, and return false if any attempted write fails.

RetrievesMultipleKeys and StackStore now aggregate with put() first so later writes are not skipped after an earlier failure. Tests pin the ordering because reversing it would silently short-circuit the batch.

DatabaseStore now treats non-empty upsert completion as success even when the affected-row count is zero, which is valid for identical-value upserts; real database failures surface as exceptions. Empty database batches now return true without issuing an upsert.

TaggedCache had the same empty-batch aggregation bug on the finite-TTL path, so it now matches Repository::putManyForever([]) and the rest of the cache API.

Add regression coverage for the shared trait, database upsert no-op writes, stack failure aggregation, memoized/failover delegation, and tagged empty batches.
@binaryfire

Copy link
Copy Markdown
Collaborator Author

@greptileai @coderabbitai Thanks for the review feedback. I pushed a follow-up hardening pass that addresses the SwooleStore issues and the adjacent cache write-result issues we found while tracing the same paths.

What changed:

  • Swoole cache timers now run from elected non-task worker 0 via AfterWorkerStart, not from the manager process. Interval resolvers can use coroutine-backed framework services, and stale cleanup / eviction still runs through evictRecords() so expired cache and lock rows are pruned even when the table is below the memory threshold.
  • Policy eviction now honors eviction_proportion in a single pass. LimitedMaxHeap no longer grows beyond its limit, and eviction candidates store compact fingerprints instead of full serialized rows. Before deleting, the store re-reads the row under the row lock and only deletes if the fingerprint still matches, so stale scan candidates cannot remove a row that was updated after the scan.
  • Interval refresh commits are now guarded after the resolver returns. If a refresher loses or has its stale claim reclaimed, it returns without writing or clearing the newer owner's claim. Successful refreshes restamp ownership before writing the public value, then complete with the commit timestamp.
  • Row-lock acquisition now adds a tiny usleep(1) backoff only after a burst of failed CAS attempts. The uncontended path remains the same single successful CAS.
  • recordHit() remains lock-free for the read hot path, with the accepted expired-shell-row behavior documented and covered by tests.
  • putMany() result semantics are now consistent: empty batches are successful no-ops, every key is attempted, and the result is false if any attempted write fails. This covers Swoole, the shared fallback trait, Stack, Database, and Tagged cache paths, with Memoized and Failover delegation pinned by tests. Database no-op upserts are treated as success because affected-row count is not a reliable success signal for identical-value writes.

I also added regression coverage for the timer election, bounded heap behavior, single-pass LRU/LFU/TTL eviction, stale eviction candidate rechecks, interval stale-overwriter/lost-claim cases, lock-free hit shell rows, and the multi-key write result contract.

Verification: composer fix is green: CS Fixer clean, PHPStan clean, and the full parallel test suite passed.

@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown

@binaryfire Thanks for the thorough writeup — this addresses a solid list of concurrency and correctness edges. A few notes before I dig into the diff:

  • Moving timers to an elected non-task worker 0 via AfterWorkerStart makes sense for letting interval resolvers use coroutine-backed services; I'll check the election logic for split-brain risk if worker 0 restarts.
  • The fingerprint-recheck-before-delete pattern for eviction candidates is a good way to avoid the stale-scan-removes-fresh-row race — I'll verify the fingerprint comparison covers all mutation paths (put/increment/decrement/touch) so no update can slip through undetected.
  • The refresh commit guard (bail without writing/clearing if claim was reclaimed) closes the "lost claim overwrites the winner's completion" gap — I'll look at how the commit path re-verifies ownership right before writing lastRefreshedAt.
  • The usleep(1) backoff only after a burst of failed CAS attempts sounds like a sensible tradeoff between uncontended latency and pathological spin under contention.
  • The putMany() result contract (empty batch = success, every key attempted, false if any attempted write fails) is a good consistency fix across drivers — I'll check the Database no-op-upsert-as-success reasoning against actual driver behavior (e.g., MySQL affected-rows quirks for ON DUPLICATE KEY UPDATE).

I'll do a fresh pass over the new commit now.

(=^・ω・^=) nice to see the hardening pass land


✏️ Learnings added
Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-04T05:30:55.363Z
Learning: In Hypervel's `SwooleStore` cache driver, eviction/timer maintenance work was moved to run from an elected non-task worker (worker 0) via the `AfterWorkerStart` event, instead of the Swoole manager process, so interval resolvers can use coroutine-backed framework services. `evictRecords()` still prunes expired cache and lock rows even when the table is below the memory pressure threshold.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-04T05:30:55.363Z
Learning: In Hypervel's `SwooleStore`, policy-based eviction (LRU/LFU/TTL) honors `eviction_proportion` in a single pass using a bounded `LimitedMaxHeap` that doesn't grow past its limit. Eviction candidates store compact fingerprints (not full serialized rows); before deleting, the store re-reads the row under the row lock and only deletes if the fingerprint still matches, preventing stale scan candidates from removing rows updated after the scan.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-04T05:30:55.363Z
Learning: In Hypervel's `SwooleStore` interval refresh flow, refresh commits are guarded after the resolver returns: if a refresher loses its claim or the claim was reclaimed as stale by another worker, it returns without writing the public value or clearing the newer owner's claim. Successful refreshes restamp ownership before writing the public value, then complete by setting the commit timestamp.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-04T05:30:55.363Z
Learning: In Hypervel's `SwooleTableState` row-lock acquisition, a tiny `usleep(1)` backoff is added only after a burst of failed CAS attempts; the uncontended path remains a single successful CAS with no sleep.

Learnt from: binaryfire
URL: https://github.com/hypervel/components/pull/414

Timestamp: 2026-07-04T05:30:55.363Z
Learning: In Hypervel's cache layer, `putMany()` result semantics were standardized across `SwooleStore`, the shared fallback trait, `StackStore`, `DatabaseStore`, and `TaggedCache`: empty batches are successful no-ops, every key is attempted, and the overall result is `false` if any attempted write fails. `Memoized` and `Failover` cache decorators delegate this contract. For `DatabaseStore`, no-op upserts (identical value writes) are treated as success because affected-row count is not a reliable success signal in that case.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@greptile-apps

greptile-apps Bot commented Jul 4, 2026

Copy link
Copy Markdown

The hardening pass looks solid overall. A few observations:

Timer election — The AfterWorkerStart / worker 0 approach is clean. If worker 0 dies and is restarted by Swoole, the new process fires AfterWorkerStart again and re-registers both timers automatically. No special recovery path needed.

LimitedMaxHeap eviction — The bounded max-heap logic is correct. When full, it only admits a new entry if that entry is smaller than the current top, then extracts the top (the current maximum). This means the heap retains exactly the $quantity entries with the smallest metric values (lowest last_used_at, lowest used_count, or soonest expiration) — i.e., the most eviction-worthy candidates. Correct for all three policies.

One remaining LRU edge caserawPutSerialized writes only value and expiration. For a brand-new key, Swoole Table initializes unspecified float fields to 0.0. Under LRU policy, a freshly written entry that has never been read has last_used_at = 0.0, making it the oldest candidate in the eviction scan. If evictRecordsIfNeeded() runs on the same put() call that created the entry, the fingerprint will match and the entry can be deleted immediately — so put() returns true but a subsequent get() returns null. This is rare (requires being exactly at the memory threshold) but is a silent correctness gap under LRU. Setting last_used_at = now in rawPutSerialized for user rows would close it, though it adds a field to every write.

flushLocks() guardhasSeparateLockStore() always returns true, so the RuntimeException is dead code. Minor, but worth noting if the method is ever expected to work on a shared store in the future.

putMany short-circuit$result = $this->put(...) && $result is correct: the put is always called (leftmost operand), and the assignment is false if any put fails. All keys are attempted regardless of earlier failures.

Interval two-phase commit — The restamp-on-commit approach (refreshingAt = commitNow) is the right guard against a stale overwriter clobbering a newer owner's claim during a slow public write. Looks correct.

The LRU new-entry issue is the only thing I'd flag as a functional concern. Everything else in the hardening pass is sound.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants