Unable to use NAT-networking on Windows

[nikhofmann]

I saw that the latest release of nerdctl (1.3.0) has improved support (mainly due to this PR: #1953) for networking on Windows and I tried to make it work.

Unfortunately I could not find any configuration of the networking / NAT CNI plugins that work with nerdctl on Windows.
(At least not with either the 'out-of-the-box' experience or without showing errors on container startup).
See my efforts / investigation below.

My questions are:

• Is there a CNI plugin for NAT networks on Windows that works with the current release of nerdctl? (In the sense of sing the out-of-the-box behavior & using the default configuration nerdctl creates).
• If not, how should the CNI plugin configuration be structured so that nerdctl can be used to run containers with a network attached without showing errors?
• (As more of an understanding kind of question: Is it meant for nerdctl to manage the HNS networks on Windows by itself, or do we have to create these manually?)

Following is my investigation into the issue:

My current setup is:

• Windows Server 2019
• containerd version v1.6.19
• nerdctl version v1.3.0
• CNI plugin for Windows in version v0.3.0 (installed from here https://github.com/microsoft/windows-container-networking)

I verified that the underlying setup works with the following steps:

• Create a new HNS network manually:

  New-HnsNetwork -Name testNet -Type NAT -AddressPrefix "10.5.0.0/24"

• Configure the nat plugin:

  {
     "cniVersion": "0.3.0",
     "name": "testNet",
     "type": "nat",
     "master": "Ethernet",
     "ipam": {
        "routes": [
          {
            "gateway": "10.5.0.1"
          }
        ],
        "subnet": "10.5.0.0/24",
        "type": ""
      },
     "capabilities": {
         "portMappings": true,
         "dns": true
     }
   }

• Use ctr to run a base image and verify network interface & connectivity:

  ipconfig

    PS C:\> ctr run --cni --rm mcr.microsoft.com/windows/nanoserver:ltsc2019 test-network ipconfig
    
    Windows IP Configuration
    
    
    Ethernet adapter vEthernet (default-test-network_testNet):
    
       Connection-specific DNS Suffix  . :
       Link-local IPv6 Address . . . . . : fe80::7bfc:17be:b053:1016%18
       IPv4 Address. . . . . . . . . . . : 10.5.0.235
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 10.5.0.1
  

  ping

    PS C:\> ctr run --cni --rm mcr.microsoft.com/windows/nanoserver:ltsc2019 test-network ping www.github.com
  
    Pinging github.com [140.82.121.4] with 32 bytes of data:
    Reply from 140.82.121.4: bytes=32 time=26ms TTL=53
    Reply from 140.82.121.4: bytes=32 time=26ms TTL=53
    Reply from 140.82.121.4: bytes=32 time=26ms TTL=53
    Reply from 140.82.121.4: bytes=32 time=26ms TTL=53
    
    Ping statistics for 140.82.121.4:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 26ms, Maximum = 26ms, Average = 26ms
    PS C:\>

I then installed the latest version of nerdctl and removed the manually created HNS network as well as the CNI plugin configuration (to start from a clean state).

When I try to run the same image & command as above I get the follwoing output:

PS C:\> nerdctl run --rm mcr.microsoft.com/windows/nanoserver:ltsc2019 ipconfig
time="2023-04-13T02:19:22-07:00" level=warning msg="networking setup error has occurred" error="plugin type=\"nat\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.2.0\" \"0.3.0\"]"
time="2023-04-13T02:19:22-07:00" level=warning msg="failed to revert container \"0b196dd44dcff6dbb23a378aa296416974b92bab3eb511d94c6f3bbc52a55a42\" networking settings" error="plugin type=\"nat\" failed (delete): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.2.0\" \"0.3.0\"]"
time="2023-04-13T02:19:22-07:00" level=fatal msg="plugin type=\"nat\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.2.0\" \"0.3.0\"]"
PS C:\>

This also created the following CNI plugin configuration:

{
  "cniVersion": "1.0.0",
  "name": "nat",
  "nerdctlID": "d919a100ce6b45524d415d52d088d5817587c6dd8c3691b03b8063c44d043523",
  "nerdctlLabels": {
    "nerdctl/default-network": "true"
  },
  "plugins": [
    {
      "type": "nat",
      "master": "Ethernet",
      "ipam": {
        "routes": [
          {
            "gateway": "10.4.0.1"
          }
        ],
        "subnet": "10.4.0.0/24",
        "type": ""
      }
    }
  ]
}

I could not find any NAT CNI plugins for windows that support the CNI versions 1.0.0.

I also tried to use the same CNI plugin configuration that already worked with the ctr command above with no luck:

PS C:\> nerdctl run --rm --net testNet mcr.microsoft.com/windows/nanoserver:ltsc2019 ipconfig
time="2023-04-13T03:02:38-07:00" level=warning msg="networking setup error has occurred" error="plugin type=\"nat\" name=\"testNet\" failed (add): hcnCreateNetwork failed in Win32: Invalid JSON document string. (0x803b001b) {\"Success\":false,\"Error\":\"Invalid JSON document string. {{Type,UnknownEnumValue}}\",\"ErrorCode\":2151350299}"
time="2023-04-13T03:02:38-07:00" level=fatal msg="plugin type=\"nat\" name=\"testNet\" failed (add): hcnCreateNetwork failed in Win32: Invalid JSON document string. (0x803b001b) {\"Success\":false,\"Error\":\"Invalid JSON document string. {{Type,UnknownEnumValue}}\",\"ErrorCode\":2151350299}"
PS C:\>

If I again create the correct HNS network manually networking seems to first indicate an error, but then the container starts and has an IP / can reach other hosts:

• Using the plugin configurations:

  nerdctl-nat.conf

  {
    "cniVersion": "1.0.0",
    "name": "testNet",
    "nerdctlID": "d919a100ce6b45524d415d52d088d5817587c6dd8c3691b03b8063c44d043523",
    "nerdctlLabels": {
      "nerdctl/default-network": "true"
    },
    "plugins": [
      {
        "type": "nat",
        "master": "Ethernet",
        "ipam": {
          "routes": [
            {
              "gateway": "10.4.0.1"
            }
          ],
          "subnet": "10.4.0.0/24",
          "type": ""
        }
      }
    ]
  }

  testNet.conf

    {
       "cniVersion": "0.3.0",
       "name": "testNet",
       "type": "nat",
       "master": "Ethernet",
       "ipam": {
          "routes": [
            {
              "gateway": "10.5.0.1"
            }
          ],
          "subnet": "10.5.0.0/24"
        }
     }

• A manually created HNS network:

  New-HnsNetwork -Name testNet -Type NAT -AddressPrefix "10.5.0.0/24"

• The following commands work, but indicate errors:

     PS C:\> nerdctl run --rm --net testNet mcr.microsoft.com/windows/nanoserver:ltsc2019 ping www.google.ch
     time="2023-04-13T03:32:54-07:00" level=warning msg="networking setup error has occurred" error="<nil>"
     
     Pinging www.google.ch [142.251.209.35] with 32 bytes of data:
     Reply from 142.251.209.35: bytes=32 time=18ms TTL=117
     Reply from 142.251.209.35: bytes=32 time=18ms TTL=117
     Reply from 142.251.209.35: bytes=32 time=18ms TTL=117
     Reply from 142.251.209.35: bytes=32 time=20ms TTL=117
     
     Ping statistics for 142.251.209.35:
         Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
     Approximate round trip times in milli-seconds:
         Minimum = 18ms, Maximum = 20ms, Average = 18ms
     PS C:\>

[jeikabu]

Just chiming in I have the same issue.
Tried changing the cniVersion to "0.3.0", but that fails with:

time="2023-05-05T14:22:46-07:00" level=warning msg="networking setup error has occurred" error="<nil>"

[TBBle]

I just looked into this, and that's actually not an error, it's a bug in nerdctl. The nil means the call got no error; nerdctl still outputs the warning message though.

That said, I didn't check if this trick works, as I did the exact same thing, and then forked the Windows CNI plugin to support CNI 1.1.0, before I noticed that nerdctl bug. I'm still testing the changes, but will hopefully put my changes up for review this weekend.

Edit: Just noticed the original poster tried this trick and it worked.

[TBBle]

• The following commands work, but indicate errors:

The error here is a bug in nerdctl. The error="<nil>" actually means success, and nerdctl should not output the warning when that happens.

I've put up a PR for WinCNI to support CNI 1.0.0 configs like current nerdctl generates: microsoft/windows-container-networking#96, and even with this change, the above faulty warning appears, but the container still works. I put up a PR that includes a fix for that faulty warning: #2587

[realflash]

I'm trying to tell - I think based on commit and release dates that this PR isn't in the 0.3.0 release of CNI at that repository. Is that so? Am I to conclude that I can't have networking in a container on Windows until a new release is made? That doesn't seem to align with all the documentation implying it is possible.

[TBBle]

That's correct, that PR, or rather its successor, microsoft/windows-container-networking#101, was merged, but no new release has been made since. So for now, you'll need to build your own copy of nat.exe (or whichever CNI plugin you want to use).

I'm not sure myself if there's a reason no new release has been made.