WDS and DNS

Recently I fired up a laptop that needed to be refreshed on a PXE boot to the WDS server. It sat there after contacting DHCP successfully and eventually output the error: TFTP Timeout.
Upon doing some research, I found that this was actually related to me having a DNS server role and a WDS server role on the same box.

I guess what happens is the DNS overtakes extra ports that roll into the WDS server. I believe this is actually common.. although having a DNS and WDS server together aren’t as common.


It turns out that the well-known security patch for DNS last summer breaks WDS, but only if DNS and WDS are running on the same box. This is also valid for Windows Server 2008 and Windows Server 2008 R2 which has the patch built-in.
In brief, the issue with DNS and WDS on the same machine is that DNS will bind to 2500 ports in the ephemeral port range (49152-65535). Since the 2500 ports are randomly chosen, there’s a chance that those 2500 ports will completely cover WDS’s port range (which is 6400 to 65000 by default). Since the DNS ports are randomly chosen each time the DNS service starts, it’s possible that changing the WDS port range will fix the problem now, but next time the machine restarts they will conflict again.
For this reason, the proper fix is to make the WDS port range larger than the DNS port range. There’s also an option in Windows Server 2008 R2 to have WinSock dynamically allocate ports to WDS that aren’t in use, which is controlled by a registry key as indicated below. However, this fix may not work if you are using a third party firewall that requires a rule for WDS since the ports chosen will be random.

Workaround #1: Don’t install DNS on the same server as WDS  🙂
Workaround #2: If you are running Windows Server 2003 or Windows Server 2008, on the WDS Server properties, in the network settings tab, change the range from say 50000 to 65000 (assuming you don’t have any other apps using these ports). You can also set this from the command line using WDSUTIL:

wdsutil /set-Server /Transport /StartPort:50000 /EndPort:65000
If you are running Windows Server 2008 R2 you can also configure WDS to dynamically query WinSock for available ports instead of using a port range. This is done by setting the UdpPortPolicy value in the registry to 0. You will find it in the HKLM\System\CurrentControlSet\Services\WDSServer\Parameters key. You need to restart WDS after changing the value.
For more info, see: http://support.microsoft.com/kb/977512