Server 2008 and Windows Firewall

I stumbled across something very interesting today regarding these two components that is worth sharing, especially for anyone else managing a Server 2008 environment. Many of us got used to disabling the firewall in Server 2003 unless we needed to solve very specific security requirements with limited resources. The Firewall in Vista/Server 2008 has been completely revamped and is now a truly stateful firewall complete with profiles, profile-specific rules, and the ability to separately manage inbound/ outbound traffic. Host-based firewalls certainly add complexity to any environment and one has to be very cognizant of hosted services as well as consuming applications and users. Troubleshooting can be frustrating if you forget that “oh yeah that server has its firewall turned on”.

In my current dev environment I disabled the firewall on all servers, 2003 and 2008, via GPO. This is how you do it in XP and 2003 so the same applies to 2008, right? While this does in fact disable the firewall in 2008, there appears to be an underlying system dependency on the firewall service in Server 2008. I installed the April Windows updates on both a Server 2008 STD (x86) VM and a Server 2008 STD (x64) physical host. After rebooting, both servers came back up fine but were completely inaccessible from the network. No PING, no RDP, no CIFS. From the servers themselves, outbound traffic flowed without issue, to the internet, ping other hosts, browsing, etc. This was baffling especially since the firewall was disabled. Nothing should be blocking traffic! I followed the usual troubleshooting methods of digging through logs, checking driver revs, I even added a new vNIC and reinstalled vmTools in my VM thinking that maybe the virtual device had become corrupted somehow. Nothing worked!!

So I decided to do a packet capture to dig a bit deeper hoping that maybe the answer would become clear in the annals of the IP stack. Ok I see ICMP echo-requests coming into the server but it’s not replying, hmm. Let’s try a telnet to TCP 3389, I see SYN messages but the server isn’t sending a SYN-ACK back, what is going on here?? It seems that the server is flat choosing to not respond to inbound traffic. The firewall is definitely disabled.

Thinking this had something to do with Windows update, I opened a sev3 case with Microsoft. In the meantime I continued to troubleshoot and finally developed a work-around before they called me back. There is no stated dependency or published best practice (that I’ve seen) that recommends leaving the firewall enabled on Server 2008 lest you risk network problems. I think most people assume that it’s a security feature that can be safely disabled without adversely affected normal server performance or function [as it should be]. To fix the problem I enabled and started the firewall service, then in the firewall management MMC set all 3 profiles to allow/allow for inbound/outbound access. Both problem servers are now accessible and functioning perfectly on the network.

This experience changes the way I will setup Server 2008 environments as it appears the firewall is a mandatory service that the network service depends upon for normal operation. Microsoft doesn’t have a KB or any information yet about this issue so the support people are kicking my case up to the product teams for review. I also submitted a change request on the Connect site so hopefully someone will take notice and at the very least write a KB stating that the firewall service is a required component of Server 2008. Ideally this dependency will be dropped altogether but we’ll see. Here is my submission:

https://connect.microsoft.com/WindowsServerFeedback/feedback/ViewFeedback.aspx?FeedbackID=464554

To sum: leave your firewalls turned on in Server 2008 and if you wish to make them transparent then do so by allowing all traffic inbound/outbound for all profiles. Don’t disable the service!!

7 comments:

  1. Hi Weestro. Did you ever hear back from Microsoft on this issue? I see there are no comments on your Microsoft Connect submission. I'm seeing the exact same issue on a couple new Windows 2008 SP2 with IE8, fully patched servers. Disable the firewall, can't ping, RDP, connect in any method. Enable firewall, ping, RDP, everything else functions fine given the pertinent rules are in place.

    ReplyDelete
  2. No I have received no feedback from MS yet. I guess this issue doesn't have enough exposure yet or most people run the firewall in production? Feel free to add a comment on Connect, as more people come forward they'll hopefully take a closer look at this.

    ReplyDelete
  3. Hey again Weestro, following up on my previous post. I took another look at this and was thinking it may be IP6-related. I disabled IP6, rebooted, and with the Firewall off, I could remotely connect whereas I couldn't before. I removed the registry key that disables IP6, rebooted and I could still connect remotely with the Firewall off. Odd. Obviously not related to the registry key disabling IP6, more likely the reboot. So I manually re-enabled the Firewall and rebooted. Server booted up, logged in, Firewall still enabled. Hmmm, makes me question how computer policy is applied in Windows 2008 as my GPO should have disabled the Firewall on bootup, make note of this point for later. Ran GPUPDATE /FORCE to re-apply GPO, Firewall now disabled, I can't ping, RDP, remotely connect again. Reboot the server, connections are back.

    Fresh server now, my second server. I verified that I couldn't connect remotely in any shape or form with the Firewall off via GPO. Server has not been rebooted since the GPO was first applied. I rebooted the server, making no changes whatsoever to IP6 or anything else. After reboot, Firewall still off, I can now remotely connect.

    In summary, it seems that if the Firewall is disabled via GPO, something else, as you say, a dependent service is not getting shut off. I am assuming that the reboot then does not restart this dependency given the Firewall never started on bootup so therefore the dependency doesn't either.

    Now here is the kicker. This only happens when the Firewall is disabled via GPO. I enable the Firewall and reboot. I can remotely connect to the server. Manually disable the Firewall in Control Panel and I can still remotely connect. Re-enable the Firewall, can still connect. Run GPUPDATE /FORCE, Firewall gets disabled via GPO, can no longer connect. I can recreate this problem at will now.

    Last thought. I previously mentioned that there appears to be issues with applying GPOs in general. Last night before I went home, I enabled the Windows Firewall. When I came in this morning, Windows Firewall was still enabled. Sixteen hours later and the GPO did not re-apply and disable the Firewall? Appears group policy is not automatically refreshing. When I reboot, group policy is not getting applied either. Seems I cannot get group policy to apply outside of GPUPDATE /FORCE and then the Firewall problem occurs. I'm running a Windows 2003 domain/forest, no Windows 2008 domain controllers. Maybe the 2003 GPO that disables the Firewall is not shutting down something properly in Windows 2008?

    ReplyDelete
  4. Interesting... So your GPO doesn't stop the firewall service (it shouldn't) but does the service actually not get marked as disabled? And the policies are being successfully applied?

    My environment is 100% 2008 and I had the same issues disabling the service via GPO. So whatever hidden dependency is missing here too. The weird thing is that my servers worked fine for months with no problems at all which makes me wonder if a recent update changed something in the IP stack. I should also mention that I haven't loaded SP2 on anything but my DCs yet...

    ReplyDelete
  5. I haven't fleshed out the exact circumstances surrounding the GPO issues I am seeing. All I know so far is that rebooting the server which should re-apply group policy or its normal refresh of group policy does not stop the Windows Firewall service if it is running. However GPUPDATE /FORCE does stop the Windows Firewall service as well as mark it as Disabled. I'll probably look into that a little more to figure out the exact circumstances once I have a little more time.

    I did comment on and validate your post over at Microsoft Connect though. I submitted reproducible steps to recreate the problem in the comment as I can recreate it 100% of the time now. I just tried it on another server that someone else installed and was able to reproduce it there too. Most certainly looks like a bug to me.

    ReplyDelete
  6. I noticed this issue using Windows NLB on 2008R2, with the firewall disabled the NICs would not converge, as soon as I turned it on they converged. Now, maybe I'm thinking to simply here, but shouldn't off mean off. I don't feel comfortable leaving Windows Firewall on, but now it seems I must get over it and use their firewall. I have done as you suggested and told it to allow all traffic on all profiles. Thank you for the post. It was good to know someone else was seeing the same thing I am.

    ReplyDelete
  7. I had a similar issue where a server came back up with the firewall disabled. had to go into c:\windows\system32\config\tvx and then delete the *.blf and *.regtrans-ms files from this directory. I then rebooted the server and it came up fine with no issues with the firewall being damaged.

    The relivancy is that I was having the exact same symptoms as you were.

    ReplyDelete

Powered by Blogger.