VM0 Status - Incident history https://status.vm0.ai VM0 Fri, 24 Apr 2026 03:06:08 +0000 Due to a performance degradation with the upstream provider, our Sonnet 4.6 model is currently affected. Type: Incident Duration: 45 minutes Affected Components: API Service Apr 24, 03:06:08 GMT+0 - Investigating - We suggest temporarily switching to other models Apr 24, 03:50:56 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 45 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 24</var>, <var data-var='time'>03:06:08</var> GMT+0</small><br><strong>Investigating</strong> - We suggest temporarily switching to other models.</p> <p><small>Apr <var data-var='date'> 24</var>, <var data-var='time'>03:50:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 24 Apr 2026 03:06:08 +0000 https://status.vm0.ai/incident/cmocbwfxz003ie7yg902jt5xx https://status.vm0.ai/incident/cmocbwfxz003ie7yg902jt5xx Currently unable to log in to the website. Type: Incident Duration: 57 minutes Affected Components: Web Page Apr 21, 10:40:51 GMT+0 - Investigating - We are currently investigating this incident. Apr 21, 11:02:54 GMT+0 - Identified - We are continuing to work on a fix for this incident. A hotfix is being prepared for deployment to the live environment. Apr 21, 11:33:59 GMT+0 - Monitoring - We are currently releasing a hotfix to resolve this issue. Apr 21, 11:37:38 GMT+0 - Resolved - This incident has been resolved. Apr 21, 11:40:46 GMT+0 - Postmortem - Today, during the project, we attempted to add a permission-related feature to the system. While implementing a new authentication method, we introduced a breaking change that prevented web users from logging in. This modification should have been caught by our automated checks. However, the check was accidentally bypassed, allowing the bug to leak into the production environment. Subsequently, we revoked the bypass permission capability to prevent similar incidents from occurring. Type: Incident

Duration: 57 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>10:40:51</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>11:02:54</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. A hotfix is being prepared for deployment to the live environment..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>11:33:59</var> GMT+0</small><br><strong>Monitoring</strong> - We are currently releasing a hotfix to resolve this issue..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>11:37:38</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>11:40:46</var> GMT+0</small><br><strong>Postmortem</strong> - Today, during the project, we attempted to add a permission-related feature to the system. While implementing a new authentication method, we introduced a breaking change that prevented web users from logging in. This modification should have been caught by our automated checks. However, the check was accidentally bypassed, allowing the bug to leak into the production environment. Subsequently, we revoked the bypass permission capability to prevent similar incidents from occurring..</p> ]]> Tue, 21 Apr 2026 10:40:51 +0000 https://status.vm0.ai/incident/cmo8htmpf0053d4cajuvbcelh https://status.vm0.ai/incident/cmo8htmpf0053d4cajuvbcelh The system is unable to run any tasks. Type: Incident Duration: 4 minutes Affected Components: Runner Apr 21, 05:49:09 GMT+0 - Investigating - We are currently investigating this incident. Apr 21, 05:50:58 GMT+0 - Identified - We are continuing to work on a fix for this incident. Apr 21, 05:52:41 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 4 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>05:49:09</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>05:50:58</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>05:52:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 21 Apr 2026 05:49:09 +0000 https://status.vm0.ai/incident/cmo87eiuo00113ul7sh57uf2p https://status.vm0.ai/incident/cmo87eiuo00113ul7sh57uf2p The Google Workspace connector is currently not working properly. Type: Incident Duration: 1 hour and 18 minutes Affected Components: Connector, Web Page, Runner, API Service, Storage Apr 17, 11:48:01 GMT+0 - Investigating - We are currently investigating this incident. Apr 17, 12:07:14 GMT+0 - Identified - We are continuing to work on a fix for this incident. Apr 17, 12:40:02 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 17, 13:05:59 GMT+0 - Resolved - This incident has been resolved. Apr 17, 13:23:06 GMT+0 - Postmortem - The problem stems from a recent update where we modified the update rules for the OAuth connector. Our goal was to implement a more proactive refresh mechanism to ensure a valid token is available whenever the system runs. However, this new refresh rule introduced a bug specifically related to the Google OAuth refresh process. Because this issue was difficult to replicate within our automated checking procedures, it was unfortunately deployed to the live environment. To prevent this from happening again, we are currently working on implementing automated validation for the OAuth connector workflow to ensure these types of issues are caught before release. Type: Incident

Duration: 1 hour and 18 minutes

Affected Components: , , , ,

<p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>11:48:01</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>12:07:14</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>12:40:02</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>13:05:59</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>13:23:06</var> GMT+0</small><br><strong>Postmortem</strong> - The problem stems from a recent update where we modified the update rules for the OAuth connector. Our goal was to implement a more proactive refresh mechanism to ensure a valid token is available whenever the system runs. However, this new refresh rule introduced a bug specifically related to the Google OAuth refresh process. Because this issue was difficult to replicate within our automated checking procedures, it was unfortunately deployed to the live environment. To prevent this from happening again, we are currently working on implementing automated validation for the OAuth connector workflow to ensure these types of issues are caught before release..</p> ]]> Fri, 17 Apr 2026 11:48:01 +0000 https://status.vm0.ai/incident/cmo2ugmg50g0gz3zj3tc7npdk https://status.vm0.ai/incident/cmo2ugmg50g0gz3zj3tc7npdk We are experiencing a failure with the Agent task runner, which is currently preventing our Agents from executing tasks. Type: Incident Duration: 1 hour and 6 minutes Affected Components: Runner Apr 14, 09:50:09 GMT+0 - Resolved - This incident has been resolved. Apr 14, 08:43:55 GMT+0 - Investigating - We are currently investigating this incident. Apr 14, 09:35:13 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 14, 09:50:08 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 6 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>09:50:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>08:43:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>09:35:13</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>09:50:08</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 14 Apr 2026 08:43:55 +0000 https://status.vm0.ai/incident/cmnydkazx02elz659ubr0pxta https://status.vm0.ai/incident/cmnydkazx02elz659ubr0pxta storage download failed Type: Incident Duration: 1 hour and 37 minutes Affected Components: Runner Apr 10, 02:14:36 GMT+0 - Investigating - We are currently investigating this incident. Apr 10, 03:00:13 GMT+0 - Identified - We are continuing to work on a fix for this incident. Apr 10, 03:51:11 GMT+0 - Resolved - This incident has been resolved. Apr 10, 04:35:41 GMT+0 - Postmortem - # Guest Download Failure Due to Parallel Race Condition on Overlapping Mount Paths ### What Happened On 2026-04-10, agent jobs with multiple storages failed during VM initialization. The `guest-download` binary, which runs inside the VM to download and extract storage archives, crashed with `canonicalize` ENOENT errors when processing skills storages. ### Root Cause `guest-download` extracts storage archives in parallel (up to 4 concurrent threads). A recent change added a `remove_dir_all(target_path)` call at the start of each thread to clean stale files on VM reuse (keep-alive). The storage mount paths have a guaranteed parent-child overlap: \- Instructions mount at `/home/user/.claude` \- Skills mount at `/home/user/.claude/skills/{name}` When threads run concurrently, the parent path's `remove_dir_all` deletes child directories already created by sibling threads, causing those threads to fail with ENOENT. ### Impact * **Scope:** All jobs * **Duration:** \~11 hours (2026-04-09 16:38 UTC — 2026-04-10 03:21 UTC) ### Timeline (UTC) | Time | Event | | ------------------ | ------------------------------------------------------------------------------------ | | 2026-04-09 16:38 | Code change merged — added remove\_dir\_all pre-cleanup in parallel download threads | | 2026-04-10 \~02:00 | Job failures reported on prod-3 | | 2026-04-10 \~02:45 | Root cause identified via prod SSH log analysis | | 2026-04-10 03:21 | Fix merged and deployed | ### Fix * Removed `remove_dir_all` from `download_and_extract()` — threads now only do `create_dir_all` \+ streaming tar extraction * Disabled `--keep-alive` in CI and production to avoid VM reuse until proper stale file cleanup is implemented (#8757) Type: Incident

Duration: 1 hour and 37 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 10</var>, <var data-var='time'>02:14:36</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 10</var>, <var data-var='time'>03:00:13</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Apr <var data-var='date'> 10</var>, <var data-var='time'>03:51:11</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 10</var>, <var data-var='time'>04:35:41</var> GMT+0</small><br><strong>Postmortem</strong> - # Guest Download Failure Due to Parallel Race Condition on Overlapping Mount Paths ### What Happened On 2026-04-10, agent jobs with multiple storages failed during VM initialization. The `guest-download` binary, which runs inside the VM to download and extract storage archives, crashed with `canonicalize` ENOENT errors when processing skills storages. ### Root Cause `guest-download` extracts storage archives in parallel (up to 4 concurrent threads). A recent change added a `remove_dir_all(target_path)` call at the start of each thread to clean stale files on VM reuse (keep-alive). The storage mount paths have a guaranteed parent-child overlap: \- Instructions mount at `/home/user/.claude` \- Skills mount at `/home/user/.claude/skills/{name}` When threads run concurrently, the parent path's `remove_dir_all` deletes child directories already created by sibling threads, causing those threads to fail with ENOENT. ### Impact * **Scope:** All jobs * **Duration:** \~11 hours (2026-04-09 16:38 UTC — 2026-04-10 03:21 UTC) ### Timeline (UTC) | Time | Event | | ------------------ | ------------------------------------------------------------------------------------ | | 2026-04-09 16:38 | Code change merged — added remove\_dir\_all pre-cleanup in parallel download threads | | 2026-04-10 \~02:00 | Job failures reported on prod-3 | | 2026-04-10 \~02:45 | Root cause identified via prod SSH log analysis | | 2026-04-10 03:21 | Fix merged and deployed | ### Fix * Removed `remove_dir_all` from `download_and_extract()` — threads now only do `create_dir_all` \+ streaming tar extraction * Disabled `--keep-alive` in CI and production to avoid VM reuse until proper stale file cleanup is implemented (#8757).</p> ]]> Fri, 10 Apr 2026 02:14:36 +0000 https://status.vm0.ai/incident/cmns9w8cj06a5zia892px46qr https://status.vm0.ai/incident/cmns9w8cj06a5zia892px46qr We are currently experiencing an outage. Users of the V0 Managed Token service may be affected. Type: Incident Duration: 1 hour and 14 minutes Affected Components: Runner Apr 8, 09:45:18 GMT+0 - Investigating - We are currently investigating this incident. Apr 8, 09:53:24 GMT+0 - Investigating - This incident is related to our most recent release. We are currently investigating the situation and working to resolve an issue that occurred during the deployment process. Apr 8, 10:22:56 GMT+0 - Monitoring - We have submitted the fix code and are observing whether the problem has been resolved. Apr 8, 10:46:12 GMT+0 - Resolved - fixed Apr 8, 10:57:19 GMT+0 - Postmortem - # Postmortem: Agent Task Execution Failure Due to Model ID Format Mismatch ### What Happened On 2026-04-08, all vm0 agent task runs failed to start. Users were unable to execute any agent tasks across all built-in model provider types. ### Root Cause The Claude Code CLI and Anthropic API require model IDs in hyphenated format (e.g., `claude-opus-4-6`). However, the model ID constants in our codebase and the values stored in the database used dot notation (e.g., `claude-opus-4.6`). When vm0 attempted to launch an agent, the CLI rejected the model ID at startup, blocking all task execution. ### Impact * **Scope:** 100% of agent task runs across `vm0`, `anthropic-api-key`, and `claude-code-oauth-token` provider types * **Duration:** \~30 minutes (09:47 UTC — 10:17 UTC) * **Not affected:** OpenRouter and Vercel AI Gateway providers (use a separate `anthropic/claude-*` naming convention) ### Timeline (UTC) | Time | Event | | ----- | ----------------------- | | 09:47 | Fix PR opened (#8511) | | 10:17 | PR merged and deployed | | 10:17 | Task execution restored | ### Fix 1. Renamed all model ID constants in `MODEL_PROVIDER_TYPES` and `VM0_MODEL_TO_PROVIDER` from dot to hyphen format 2. Ran DB migration `0230` to backfill affected rows in `model_providers.selected_model`, `vm0_api_keys.model`, and `credit_pricing.model` 3. Updated dev seed scripts and all related test suites Apr 8, 10:58:59 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 14 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>09:45:18</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>09:53:24</var> GMT+0</small><br><strong>Investigating</strong> - This incident is related to our most recent release. We are currently investigating the situation and working to resolve an issue that occurred during the deployment process..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>10:22:56</var> GMT+0</small><br><strong>Monitoring</strong> - We have submitted the fix code and are observing whether the problem has been resolved..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>10:46:12</var> GMT+0</small><br><strong>Resolved</strong> - fixed.</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>10:57:19</var> GMT+0</small><br><strong>Postmortem</strong> - # Postmortem: Agent Task Execution Failure Due to Model ID Format Mismatch ### What Happened On 2026-04-08, all vm0 agent task runs failed to start. Users were unable to execute any agent tasks across all built-in model provider types. ### Root Cause The Claude Code CLI and Anthropic API require model IDs in hyphenated format (e.g., `claude-opus-4-6`). However, the model ID constants in our codebase and the values stored in the database used dot notation (e.g., `claude-opus-4.6`). When vm0 attempted to launch an agent, the CLI rejected the model ID at startup, blocking all task execution. ### Impact * **Scope:** 100% of agent task runs across `vm0`, `anthropic-api-key`, and `claude-code-oauth-token` provider types * **Duration:** \~30 minutes (09:47 UTC — 10:17 UTC) * **Not affected:** OpenRouter and Vercel AI Gateway providers (use a separate `anthropic/claude-*` naming convention) ### Timeline (UTC) | Time | Event | | ----- | ----------------------- | | 09:47 | Fix PR opened (#8511) | | 10:17 | PR merged and deployed | | 10:17 | Task execution restored | ### Fix 1. Renamed all model ID constants in `MODEL_PROVIDER_TYPES` and `VM0_MODEL_TO_PROVIDER` from dot to hyphen format 2. Ran DB migration `0230` to backfill affected rows in `model_providers.selected_model`, `vm0_api_keys.model`, and `credit_pricing.model` 3. Updated dev seed scripts and all related test suites.</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>10:58:59</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 8 Apr 2026 09:45:18 +0000 https://status.vm0.ai/incident/cmnpv44m100l0xnamhsd9whby https://status.vm0.ai/incident/cmnpv44m100l0xnamhsd9whby We are currently experiencing some service outages online. Type: Incident Duration: 3 minutes Affected Components: Web Page, API Service Mar 26, 08:23:50 GMT+0 - Investigating - The agents are temporarily unable to perform conversations or other tasks. We are currently investigating this incident. Mar 26, 08:27:03 GMT+0 - Resolved - This incident has been resolved. Mar 26, 08:28:16 GMT+0 - Postmortem - Our most recent deployment script encountered several issues during the latest release, which prevented the database migration from executing as planned. However, the application service proceeded with the deployment under the assumption that the database execution was successful. This has resulted in an inconsistency between the live application and the database versions, making it impossible to create a new event activity. We are currently conducting an investigation to determine why the database upgrade scripts failed to execute correctly. Type: Incident

Duration: 3 minutes

Affected Components: ,

<p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>08:23:50</var> GMT+0</small><br><strong>Investigating</strong> - The agents are temporarily unable to perform conversations or other tasks. We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>08:27:03</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>08:28:16</var> GMT+0</small><br><strong>Postmortem</strong> - Our most recent deployment script encountered several issues during the latest release, which prevented the database migration from executing as planned. However, the application service proceeded with the deployment under the assumption that the database execution was successful. This has resulted in an inconsistency between the live application and the database versions, making it impossible to create a new event activity. We are currently conducting an investigation to determine why the database upgrade scripts failed to execute correctly..</p> ]]> Thu, 26 Mar 2026 08:23:50 +0000 https://status.vm0.ai/incident/cmn77hak4000cqot3pmimzkuh https://status.vm0.ai/incident/cmn77hak4000cqot3pmimzkuh Service temporarily unavailable Type: Incident Duration: 30 minutes Affected Components: Web Page, API Service Mar 18, 04:19:36 GMT+0 - Investigating - We are currently investigating this incident. Mar 18, 04:49:34 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 30 minutes

Affected Components: ,

<p><small>Mar <var data-var='date'> 18</var>, <var data-var='time'>04:19:36</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 18</var>, <var data-var='time'>04:49:34</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 18 Mar 2026 04:19:36 +0000 https://status.vm0.ai/incident/cmmvj8e1j05gjezxdenn7qed4 https://status.vm0.ai/incident/cmmvj8e1j05gjezxdenn7qed4 The website is not allowing users to log in normally. Type: Incident Duration: 38 minutes Affected Components: Web Page Mar 12, 05:52:18 GMT+0 - Investigating - We are currently investigating this incident. Mar 12, 05:52:49 GMT+0 - Investigating - When opening the login page, a blank screen is displayed. Since the username and password fields are not visible, users are unable to complete the login process. Mar 12, 05:54:25 GMT+0 - Identified - The issue originated with the most recent security update. Mar 12, 06:22:48 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 12, 06:30:07 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 38 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>05:52:18</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>05:52:49</var> GMT+0</small><br><strong>Investigating</strong> - When opening the login page, a blank screen is displayed. Since the username and password fields are not visible, users are unable to complete the login process..</p> <p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>05:54:25</var> GMT+0</small><br><strong>Identified</strong> - The issue originated with the most recent security update..</p> <p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>06:22:48</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>06:30:07</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 12 Mar 2026 05:52:18 +0000 https://status.vm0.ai/incident/cmmn1whyg006ddyuxih5nnm4z https://status.vm0.ai/incident/cmmn1whyg006ddyuxih5nnm4z www.vm0.ai cannot directly redirect to platform.vm0.ai. Type: Incident Duration: 33 minutes Affected Components: Web Page Feb 11, 13:04:13 GMT+0 - Investigating - We are currently investigating this incident. Feb 11, 13:14:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Feb 11, 13:36:50 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 33 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>13:04:13</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>13:14:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>13:36:50</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 11 Feb 2026 13:04:13 +0000 https://status.vm0.ai/incident/cmli1k8ru0042hh84qomwdnme https://status.vm0.ai/incident/cmli1k8ru0042hh84qomwdnme The platform.vm0.ai cannot be opened. Type: Incident Duration: 16 minutes Feb 6, 13:28:00 GMT+0 - Investigating - We are currently investigating this incident. Feb 6, 14:18:09 GMT+0 - Identified - We determined that the issue originated from a recent deployment of the platform frontend code. Feb 6, 14:29:55 GMT+0 - Resolved - This incident has been resolved. Feb 6, 14:38:03 GMT+0 - Postmortem - This incident was caused by a code refactoring that consolidated references to CLERK\_PUBLISHABLE\_KEY across several web sites. However, due to an oversight in omitting the variable name in the deployment script, the platform failed to locate the legacy CLERK\_PUBLISHABLE\_KEY in the production environment, resulting in page failures and inability to use [platform.vm0.ai](http://platform.vm0.ai). The related API services and container services were not affected. The follow-up remediation plan primarily includes attempting to validate required environment variables during the build phase to prevent problematic code from being deployed. Additionally, introducing e2e testing for [platform.vm0.ai](http://platform.vm0.ai) to ensure the happy path workflow functions normally. Type: Incident

Duration: 16 minutes

<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:28:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:18:09</var> GMT+0</small><br><strong>Identified</strong> - We determined that the issue originated from a recent deployment of the platform frontend code..</p> <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:29:55</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:38:03</var> GMT+0</small><br><strong>Postmortem</strong> - This incident was caused by a code refactoring that consolidated references to CLERK\_PUBLISHABLE\_KEY across several web sites. However, due to an oversight in omitting the variable name in the deployment script, the platform failed to locate the legacy CLERK\_PUBLISHABLE\_KEY in the production environment, resulting in page failures and inability to use [platform.vm0.ai](http://platform.vm0.ai). The related API services and container services were not affected. The follow-up remediation plan primarily includes attempting to validate required environment variables during the build phase to prevent problematic code from being deployed. Additionally, introducing e2e testing for [platform.vm0.ai](http://platform.vm0.ai) to ensure the happy path workflow functions normally..</p> ]]> Fri, 6 Feb 2026 13:28:00 +0000 https://status.vm0.ai/incident/cmlayuga407c3a1jxpcjvfqui https://status.vm0.ai/incident/cmlayuga407c3a1jxpcjvfqui Agent can't run Type: Incident Duration: 4 hours and 58 minutes Affected Components: Runner Jan 19, 02:40:00 GMT+0 - Investigating - We are currently investigating this incident. Jan 19, 02:45:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Jan 19, 02:50:00 GMT+0 - Identified - This glitch comes from a recent runner deployment, and the team is trying to fix the issue Jan 19, 05:50:18 GMT+0 - Identified - Locating the problem comes from a recent database change, the team is trying to fix the data that caused the problem Jan 19, 06:21:45 GMT+0 - Identified - We have now restored normal operation for both the database and task dispatcher, and are currently working on getting the Claude code in the sandbox back online. Jan 19, 07:03:49 GMT+0 - Identified - We have confirmed that the issue lies in the way VM0 calls the claude code. The minimal fix has been finalized and is currently being redeployed. Jan 19, 07:38:24 GMT+0 - Resolved - This incident has been resolved. Jan 19, 08:23:59 GMT+0 - Postmortem - Postmortem: Claude Code Hanging in Sandbox Date: 2026-01-19 Severity: P0 Duration: \~4 hours ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SUMMARY Production agent runs were failing silently. Claude Code started but never produced output, timing out after 15+ minutes. Root Cause: stdin was configured as "pipe" but never closed, causing Claude Code to hang waiting for EOF. Why CI missed it: CI uses mock-claude which doesn't check stdin state. Real Claude Code does. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ THE BUG Before (hangs - stdin pipe never closed): spawn(cmd, args, { stdio: \["pipe", "pipe", "pipe"\] }) After (works - stdin is /dev/null, immediate EOF): spawn(cmd, args, { stdio: \["ignore", "pipe", "pipe"\] }) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ HOW WE FOUND IT 1. SSH into sandbox 2. cat /tmp/vm0-agent-\*.log → empty (no Claude output) 3. ps aux | grep claude → process alive, using 23% memory 4. ps -p 510 -o wchan → ep\_pol (waiting on I/O) 5. ls -la /proc/510/fd/0 → stdin connected to pipe 6. Manual "claude --print hello" → works (TTY mode) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ WHY NO ROLLBACK Release included database migration. Forward-fix was safer. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ INVESTIGATION NOISE Runner npm publish failure (@vm0/core not built) was unrelated but consumed investigation time. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ACTION ITEMS ✅ Fix stdin → "ignore" ([#1316](https://github.com/vm0-ai/vm0/pull/1316)) ✅ Add spawn unit tests ([#1319](https://github.com/vm0-ai/vm0/pull/1319)) ✅ Fix CI publish jobs ([#1306](https://github.com/vm0-ai/vm0/pull/1306), [#1318](https://github.com/vm0-ai/vm0/pull/1318)) 🔲 Add real Claude test in CI (TODO) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ KEY LESSON Mock ≠ Real: CI must include at least one test with real Claude Code to catch behavior differences like stdin handling. Type: Incident

Duration: 4 hours and 58 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>02:40:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>02:45:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>02:50:00</var> GMT+0</small><br><strong>Identified</strong> - This glitch comes from a recent runner deployment, and the team is trying to fix the issue.</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>05:50:18</var> GMT+0</small><br><strong>Identified</strong> - Locating the problem comes from a recent database change, the team is trying to fix the data that caused the problem.</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>06:21:45</var> GMT+0</small><br><strong>Identified</strong> - We have now restored normal operation for both the database and task dispatcher, and are currently working on getting the Claude code in the sandbox back online..</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>07:03:49</var> GMT+0</small><br><strong>Identified</strong> - We have confirmed that the issue lies in the way VM0 calls the claude code. The minimal fix has been finalized and is currently being redeployed..</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>07:38:24</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 19</var>, <var data-var='time'>08:23:59</var> GMT+0</small><br><strong>Postmortem</strong> - Postmortem: Claude Code Hanging in Sandbox Date: 2026-01-19 Severity: P0 Duration: \~4 hours ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SUMMARY Production agent runs were failing silently. Claude Code started but never produced output, timing out after 15+ minutes. Root Cause: stdin was configured as "pipe" but never closed, causing Claude Code to hang waiting for EOF. Why CI missed it: CI uses mock-claude which doesn't check stdin state. Real Claude Code does. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ THE BUG Before (hangs - stdin pipe never closed): spawn(cmd, args, { stdio: \["pipe", "pipe", "pipe"\] }) After (works - stdin is /dev/null, immediate EOF): spawn(cmd, args, { stdio: \["ignore", "pipe", "pipe"\] }) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ HOW WE FOUND IT 1. SSH into sandbox 2. cat /tmp/vm0-agent-\*.log → empty (no Claude output) 3. ps aux | grep claude → process alive, using 23% memory 4. ps -p 510 -o wchan → ep\_pol (waiting on I/O) 5. ls -la /proc/510/fd/0 → stdin connected to pipe 6. Manual "claude --print hello" → works (TTY mode) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ WHY NO ROLLBACK Release included database migration. Forward-fix was safer. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ INVESTIGATION NOISE Runner npm publish failure (@vm0/core not built) was unrelated but consumed investigation time. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ACTION ITEMS ✅ Fix stdin → "ignore" ([#1316](https://github.com/vm0-ai/vm0/pull/1316)) ✅ Add spawn unit tests ([#1319](https://github.com/vm0-ai/vm0/pull/1319)) ✅ Fix CI publish jobs ([#1306](https://github.com/vm0-ai/vm0/pull/1306), [#1318](https://github.com/vm0-ai/vm0/pull/1318)) 🔲 Add real Claude test in CI (TODO) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ KEY LESSON Mock ≠ Real: CI must include at least one test with real Claude Code to catch behavior differences like stdin handling..</p> ]]> Mon, 19 Jan 2026 02:40:00 +0000 https://status.vm0.ai/incident/cmkkqswpo002w10pflokikw48 https://status.vm0.ai/incident/cmkkqswpo002w10pflokikw48