pnpmInstall: Throws strange exceptions: Solved with "--no-parallel"

[jschneider]

I get these exceptions when calling gradle pnpmInstall from my project root (with several sub projects configured for pnpm):

> Task :internal:closed:webapps:meistercharts.com:pnpmInstall FAILED
Caching disabled for task ':internal:closed:webapps:meistercharts.com:pnpmInstall' because:
  Caching has not been enabled for the task
Task ':internal:closed:webapps:meistercharts.com:pnpmInstall' is not up-to-date because:
  Output property 'nodeModulesDirectory' file /home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/node_modules/@astrojs/check has been removed.
  Output property 'nodeModulesDirectory' file /home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/node_modules/@astrojs/check/LICENSE has been removed.
  Output property 'nodeModulesDirectory' file /home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/node_modules/@astrojs/check/dist has been removed.
Starting process 'command '/home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/build/node/pnpm/pnpm-v8.15.1/bin/pnpm''. Working directory: /home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com Command: /home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/build/node/pnpm/pnpm-v8.15.1/bin/pnpm install
Successfully started process 'command '/home/johannes/projects/com.cedarsoft.monorepo/internal/closed/webapps/meistercharts.com/build/node/pnpm/pnpm-v8.15.1/bin/pnpm''
Scope: all 12 workspace projects
../../../..                              | Progress: resolved 1, reused 0, downloaded 0, added 0
../../../..                              |    +2358 ++++++++++++++++++++++++++++
../../../..                              | Progress: resolved 2358, reused 288, downloaded 0, added 0
../../../..                              | Progress: resolved 2358, reused 1799, downloaded 0, added 0
../../../..                              | Progress: resolved 2358, reused 2352, downloaded 0, added 0
../../../..                              | Progress: resolved 2358, reused 2352, downloaded 0, added 0, done
 ENOENT  ENOENT: no such file or directory, unlink '/home/johannes/projects/com.cedarsoft.monorepo/node_modules/.pnpm/node_modules/tunnel-agent'

The exact error message changes every time:

 ERR_PNPM_LINKING_FAILED  Error: ENOTEMPTY: directory not empty, rmdir '/home/johannes/projects/com.cedarsoft.monorepo/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/langs'

 ERR_PNPM_LINKING_FAILED  Error: ENOTEMPTY: directory not empty, rename '/home/johannes/projects/com.cedarsoft.monorepo/node_modules/.pnpm/[email protected][email protected]/node_modules/postcss-calc_tmp_15783' -> '/home/johannes/projects/com.cedarsoft.monorepo/node_modules/.pnpm/[email protected][email protected]/node_modules/postcss-calc'

The exception does not happen if adding "--no-parallel".

[deepy]

Since caching is not enabled we can rule out Gradle writing to the directory
And unless your subprojects are installing to the same node_modules we can rule out the plugin as well

Which then sadly would leave pnpm, you can work around the issue by using a shared build service as a semaphore, limiting the number of installs that can run in parallel

But before going down that route, just to confirm, all the pnpm installs aren't writing to /home/johannes/projects/com.cedarsoft.monorepo/node_modules/ right?

[jschneider]

The pnpm install tasks are writing all to the same node_modules dir in the project root.
We have a pnpm "monorepo" with a pnpm-workspace.yaml

[bryanstern]

I'm running into this as well with a similar project set up. Here's my workaround that I added in my root build.gradle.kts.

// Ensures that only one PnpmInstallTask runs at a time. The build service acts as a semaphore.
 class EmptyBuildService : BuildService<BuildServiceParameters.None> {
     override fun getParameters(): BuildServiceParameters.None = TODO("Not yet implemented")
 }
 val service = gradle.sharedServices
     .registerIfAbsent("pnpmInstallService", EmptyBuildService::class) {
         maxParallelUsages = 1
     }
 subprojects {
     tasks.withType<PnpmInstallTask>().configureEach {
         usesService(service)
     }
 }