Today, during the project, we attempted to add a permission-related feature to the system. While implementing a new authentication method, we introduced a breaking change that prevented web users from logging in.

This modification should have been caught by our automated checks. However, the check was accidentally bypassed, allowing the bug to leak into the production environment.

Subsequently, we revoked the bypass permission capability to prevent similar incidents from occurring.

We are continuing to work on a fix for this incident. A hotfix is being prepared for deployment to the live environment.

We are continuing to work on a fix for this incident.

The problem stems from a recent update where we modified the update rules for the OAuth connector. Our goal was to implement a more proactive refresh mechanism to ensure a valid token is available whenever the system runs.

However, this new refresh rule introduced a bug specifically related to the Google OAuth refresh process. Because this issue was difficult to replicate within our automated checking procedures, it was unfortunately deployed to the live environment.

To prevent this from happening again, we are currently working on implementing automated validation for the OAuth connector workflow to ensure these types of issues are caught before release.

Postmortem: Agent Task Execution Failure Due to Model ID Format Mismatch

What Happened

On 2026-04-08, all vm0 agent task runs failed to start. Users were unable to execute any agent tasks across all built-in model provider types.

Root Cause

The Claude Code CLI and Anthropic API require model IDs in hyphenated format (e.g., claude-opus-4-6). However, the model ID constants in our codebase and the values stored in the database used dot notation (e.g., claude-opus-4.6). When vm0 attempted to launch an agent, the CLI rejected the model ID at startup, blocking all task execution.

Impact

• Scope: 100% of agent task runs across vm0, anthropic-api-key, and claude-code-oauth-token provider types

• Duration: ~30 minutes (09:47 UTC — 10:17 UTC)

• Not affected: OpenRouter and Vercel AI Gateway providers (use a separate anthropic/claude-* naming convention)

Timeline (UTC)

Fix

1. Renamed all model ID constants in MODEL_PROVIDER_TYPES and VM0_MODEL_TO_PROVIDER from dot to hyphen format

2. Ran DB migration 0230 to backfill affected rows in model_providers.selected_model, vm0_api_keys.model, and credit_pricing.model

3. Updated dev seed scripts and all related test suites

Our most recent deployment script encountered several issues during the latest release, which prevented the database migration from executing as planned.

However, the application service proceeded with the deployment under the assumption that the database execution was successful.

This has resulted in an inconsistency between the live application and the database versions, making it impossible to create a new event activity.

We are currently conducting an investigation to determine why the database upgrade scripts failed to execute correctly.

The issue originated with the most recent security update.

When opening the login page, a blank screen is displayed. Since the username and password fields are not visible, users are unable to complete the login process.

We are continuing to work on a fix for this incident.