Today, during the project, we attempted to add a permission-related feature to the system. While implementing a new authentication method, we introduced a breaking change that prevented web users from logging in.

This modification should have been caught by our automated checks. However, the check was accidentally bypassed, allowing the bug to leak into the production environment.

Subsequently, we revoked the bypass permission capability to prevent similar incidents from occurring.

We are continuing to work on a fix for this incident. A hotfix is being prepared for deployment to the live environment.

We are continuing to work on a fix for this incident.

The problem stems from a recent update where we modified the update rules for the OAuth connector. Our goal was to implement a more proactive refresh mechanism to ensure a valid token is available whenever the system runs.

However, this new refresh rule introduced a bug specifically related to the Google OAuth refresh process. Because this issue was difficult to replicate within our automated checking procedures, it was unfortunately deployed to the live environment.

To prevent this from happening again, we are currently working on implementing automated validation for the OAuth connector workflow to ensure these types of issues are caught before release.

Guest Download Failure Due to Parallel Race Condition on Overlapping Mount Paths

What Happened

On 2026-04-10, agent jobs with multiple storages failed during VM initialization. The guest-download binary, which runs inside the VM to download and extract storage archives, crashed with canonicalize ENOENT errors when processing skills storages.

Root Cause

guest-download extracts storage archives in parallel (up to 4 concurrent threads). A recent change added a remove_dir_all(target_path) call at the start of each thread to clean stale files on VM reuse (keep-alive).

The storage mount paths have a guaranteed parent-child overlap:

- Instructions mount at /home/user/.claude

- Skills mount at /home/user/.claude/skills/{name}

When threads run concurrently, the parent path's remove_dir_all deletes child directories already created by sibling threads, causing those threads to fail with ENOENT.

Impact

• Scope: All jobs

• Duration: ~11 hours (2026-04-09 16:38 UTC — 2026-04-10 03:21 UTC)

Timeline (UTC)

Fix

• Removed remove_dir_all from download_and_extract() — threads now only do create_dir_all + streaming tar extraction

• Disabled --keep-alive in CI and production to avoid VM reuse until proper stale file cleanup is implemented (#8757)

Our most recent deployment script encountered several issues during the latest release, which prevented the database migration from executing as planned.

However, the application service proceeded with the deployment under the assumption that the database execution was successful.

This has resulted in an inconsistency between the live application and the database versions, making it impossible to create a new event activity.

We are currently conducting an investigation to determine why the database upgrade scripts failed to execute correctly.

The issue originated with the most recent security update.

When opening the login page, a blank screen is displayed. Since the username and password fields are not visible, users are unable to complete the login process.

We are continuing to work on a fix for this incident.