Filed Under (Geekspeak, Work) by Justin on 2008-02-12

Firebox X750eI’ve previously written about a slight problem caused by our Watchguard X750e firewall at the office. I still think it’s a great piece of hardware, but I occasionally run into little snafus like this one with Google Maps. I couldn’t find the info I needed on Google to fix this, so I had to figure it out on my own.

I’ve always known Google Maps to be a little quirky from the two computers I use while at work, but never thought to track down what was causing it. By quirky, I mean that more often than not, I would be left map-less and would have to do a hard refresh multiple times before it would draw the map or load the satellite imagery. After popping open the console today to see what Watchguard has to say about the problem, this is the log entry I see:

2008-02-12 09:21:21 Deny dns/udp 1059 53 1-Trusted 0-External ProxyDeny: DNS Invalid response (DNS-00) src_ip_nat=”66.20.xxx.xxx” src_port_nat=”14340″ proxy_act=”DNS-Outgoing”

In the above, represents my internal DNS server, is a DNS server apparently owned by Google, and 66.20.xxx.xxx is the outward facing interface on our firewall. What really struck me as odd about this, was that my internal DNS should only be querying the DNS servers at AT&T – not at Google. After a little trial and error, I found that I just needed to modify one setting in the default Watchguard DNS proxy to make this work. By default, this policy is aptly named “DNS” in your Fireware Policy Manager.

  1. Open the DNS policy and go to the “Properties” tab
  2. Next to the “Proxy action:” drop-down dialog, click the “View/Edit Proxy” button
  3. In the “General” section, under “Protocol Anomaly Detection Rules” change the drop-down next to “Not of class Internet” to “Allow” and click OK
  4. When prompted, give your newly-modified proxy settings a new name and save them
  5. Click OK to close the “Edit Policy Properties” dialog
  6. Save your new policy changes to your Firebox

I hope someone else finds this useful.

Note: At this point, I have no real idea exactly what the setting means, but it has fixed my problem and I haven’t seen any further side-effects. If anyone “in-the-know” thinks that allowing those queries is a problem, please let me know. I take no responsibility for any issues this may cause on your network.

(2) Comments   
Filed Under (Geekspeak) by Justin on 2007-01-05

Since I installed our new Watchguard Firebox X750e, I’ve been unable to use Subversion from work. I always received this kind of response:

jmoore@www:/usr/web/wordpress> svn switch http://svn.automattic.com/wordpress/branches/2.0
svn: PROPFIND request failed on ‘/wordpress/tags/2.0.4’
svn: PROPFIND of ‘/wordpress/tags/2.0.4’: 400 Bad request: request method not supported (http://svn.automattic.com)

I suspected the new Firebox was to blame, but up until a few weeks ago, I never really cared enough to get in to it. With the release of WordPress 2.0.6 today, I needed to upgrade our intranet and set out to figure out what was up with the HTTP Proxy on the Firebox. These are the errors logged in the

ProxyDeny: HTTP Request method unsupported (HTTP – Outbound-01) src_ip_nat=”66.20.x.x” src_port_nat=”55839″ proxy_act=”HTTP-Client-ESI” method=”PROPFIND”

I almost immediately found this once I started Googling around, which didn’t leave me with a lot of hope because I didn’t want the HTTP Proxy turned off for ALL the hosts on the inside of my network – I liked the protection it was offering except the ones I needed to use SVN from.

My solution ended up being rather simple. Create a new filter policy instead of a proxy and add only the IP addresses of the machines which I use Subversion on in the “From” box. A few clicks will get you up and going. For those of you who like the step-by-step type instructions, “fire” up your Policy Manager and follow along.

  • Click Edit->Add Policy
    Watchguard SVN Fix 01
  • Expand the Packet Filters find and click on HTTP (or HTTPS if the SVN server is running on SSL) and then click the Add button
    Watchguard SVN Fix 02
  • For Name: type in ‘SVN via HTTP‘ or whatever description you want
  • Click the Add button underneth the From: area and use the Add Other… button to add the host IP addresses of the machines you want to use SVN on. Also, be sure to remove the ‘Any-Trusted‘ from that list, or all hosts on the Trusted portion of your network will begin using this new filter instead of the normal HTTP Proxy. When finished, click OK to close the dialog box
    Watchguard SVN Fix 03
    Watchguard SVN Fix 04
  • Click OK to save the new packet filter and then click the Closebutton.
  • Save your new configuration file to your Firebox and watch SVN come alive for those hosts you defined.

Extra Credit: I found a doc on Watchguard’s website explaning why you can’t just add PROPFIND to the allowed HTTP Request types in the HTTP Proxy, and I can sort of see things from their point of view. They only let you add allowed types that are defined in the HTTP 1.1 RFC 2616 spec. However, given the popularity of Subversion and WebDAV along with the age of the spec, they really need to get with the program and patch-up Fireware to make this easier on network admins.

(0) Comments   
Filed Under (Geekspeak, Work) by Justin on 2008-06-03

Since posting last week about my troubles treading in to SonicWALL water, I think my issues have all been resolved and things are really humming right along. Truth be told, the problem was really a combination of user-error and user-ignorance. If you follow me at all on Twitter, you already know that I had major, show-stopping issues over a couple of days last week. WAN->LAN traffic would flow perfectly fine for a while, and then at some completely arbitrary point, it would stop passing traffic to certain internal hosts. Like I said, this was show stopping.

You might like some background on our implementation. When ESI received our first IP assignment from BellSouth many years ago on our fractional T-1, it was a short and simple /29 network which yielded six usable addresses, minus one that had to be assigned to the ISP provided Cisco router, leaving us with five addresses we could use. A couple years ago (three to be exact), we outgrew that, and in an attempt to keep a 1:1 ratio on NAT rules, we acquired an additional subnet – but this time a significantly larger /28 was assigned. For whatever reason, we never bothered moving all the services to the new subnet. With Watchguard, it was never an issue and was easy enough to run them all simultaneously. The first subnet was configured on the WAN interface and the additional addresses were added as “virtual addresses” on that interface and they were then available to create rules/policies with.

Enough with the history lesson – let’s move on to the present. Personally, I’ll take 99% of the blame for the user-error part. Upon receiving the NSA 3500, I was anxious to get started, so I unboxed it and started exploring the web admin interface. It was confusing and quite different for someone like me, who prior to this, had only had experience with Watchguard gear. After questioning many SonicWALL users whom I respect, I started firing off the Public Server Wizard and creating all my NAT Policies and Firewall Rules. This was apparently mistake number one. While it does indeed work, I later found out that Mark doesn’t suggest that method. Creating them manually gives you a bit more granualar control, leaves a little less cruft in the auto-generated NAT Policies, and the real kicker – you end up gaining a much better understanding of how the Firewall Rules and NAT Policies work together.

Moreno and I made plans to work on this last Thursday morning. I went in to the office early and decided I’d put the SonicWALL NSA back in production while we finished up. Around 7:30 AM, I made the swap and rebooted the Cisco from AT&T to flush any routing related issues out. A few services came online with the NSA and Mark continued cleaning up the rest of the NAT rules and chasing rabbits. By 8:15 or so, we had everything working except one Linux server which was non-mission critical, so we decided to hold-off on that one for a while and see how things went. Things went extremely well throughout the day, but as soon as we finished watching the LOST season finale on Thursday night around 11:15 PM, my Treo started buzzing with down alerts. I started checking things, and finally at this point, it dawned on me that all the services that were failing were on the newer /28 subnet. Surely that must bare some significance. I rebooted the NSA to no avail. After about an hour, traffic magically started flowing to all hosts again so I called it a night, texted Mark to let him know about the issue, and set the alarm for 6:00 AM.

I crawled out of bed around 6:30 Friday morning and got to the office around 7:15. At this point, I had resolved to myself that it was either do or die. I was not going to revert back to Watchguard a third time. If we couldn’t fix the SonicWALL by noon, I was going to wash my hands of it and let Moreno have his gear back. I chatted with Mark around 8:00 and he was bumfuzzled and en route to meet with another client but promised to look at my logs if I’d send them over and also open a ticket with SonicWALL to get the issue resolved ASAP.

I kept hammering away. Sitting idly and waiting on Mark and SonicWALL was not part of my playbook and for some reason, neither was calling support directly. I’m just not the kind of guy most of the time. For some reason, I’d rather spend several  hours resolving an issue on my own than just calling and asking someone. Maybe it’s pride? Regardless, around 9:00 I turned to Google and the SonicWALL website and within 15 minutes, I discovered SonicWALL KBID 3726 titled “SonicOS: Configuring Multiple Subnets Using Static ARP with SonicOS Enhanced” which outlines these simple instructions:

Follow these instructions to create a second subnet on an interface:

  • Create a static ARP assignment. Enable the “publish entry” check box.
    1. Login to the SonicWALL’s Management page.
    2. Select Network > ARP.
    3. Click the ADD button under Static ARP Entries.
    4. IP Address – Specify the IP address to which the SonicWALL should be assigned on the additional subnet.
    5. Interface – Specify the interface (LAN / WAN / OPT / WLAN) where the additional subnet resides.
    6. Publish Entry – Enabling this option causes the SonicWALL to respond to ARP queries for the specified IP address with the SonicWALL’s MAC address. This box must be checked when creating additional subnets.
    7. Click OK.
  • Select Network > Routing.
  • Select Add. Create the following new route policy:
    • Source: ANY
    • Destination: Create new address object
      • Name the object for your secondary subnet
      • Zone Assignment of your secondary subnet
      • Type: Network
      • Network: Enter the Network address of the secondary subnet
      • Netmask: Enter the Subnet mask of the secondary subnet
      • Click OK
    • Service: ANY
    • Gateway:
    • Interface: Select the interface the secondary subnet resides on
    • Metric: 20
    • Comment: Label policy so it can be identified at a later date
    • Click OK
  • A ssecondary subnet on the LAN interface will use the default NAT Policy & Access Rules. Access rules & NAT policies may be added.

As soon as I created the ARP assignment, traffic started flowing but I went ahead and created the static route as well for good measure.

I’ve got a couple more SonicWALL posts to come. One about how AWESOME the hardware-based VPN is and another about exactly how to configure the Firewall Rules and NAT Policies without using the Wizard. Tomorrow (Tuesday) morning, I’m heading to Charlotte for the SonicWALL Roadshow and then in the afternoon, I’m deploying my first TZ 150 endpoint to a remote office, along with another special piece of hardware. I’ll try to get some pictures of all that and the VPN post up up by the weekend.

Overall, I’m quite happy with the NSA 3500 and my SonicWALL VPN solution. For now, I’ll withhold my formal review and recommendation on Moreno until I see his final invoice – if he cuts me some slack, I’ll cut him some on the blog.

(6) Comments   
Filed Under (Geekspeak, Work) by Justin on 2008-05-28

As previously mentioned, we’re switching away from a one-year old Watchguard Core x750e to SonicWALL NSA 3500 at my place of employment in order to deploy a nice, widespread (geographically speaking), and expensive VPN.

I received the first half of my gear from Mark Moreno two weeks ago and immediately unboxed the NSA. It’s quite a purdy device! It’s sleek, silver, and has a very bright blue LED on the front. I powered it up and upon logging in to the web management interface, I was equally impressed by how shiny and web 2.0 the web UI was. Sadly, that’s where my enthusiasm ends for SonicWALL right now. I started digging around and was just overwhelmed at the options and difference in terminology between the NSA and the Watchguard. After talking it up in the CITRT IRC channel, I was informed that the “public server wizard” was the way to go with configuring NAT policies since SonicWALL  actually needs THREE rules to create one NAT rule. Not only the the NAT policies have to be defined, but then there is the firewall policy. Best I can tell, to NAT one port to one service would require the following steps without the wizard:

  1. Create “Address Objects”
  2. Create “Service” or “Service Group” if not predefined
  3. Create Firewall rule
  4. Create the three NAT policies

While four steps seems simple, it’s a lot of clicking and a lot of digging around, and so far, I’m not a fan. The wizard did a good enough job for some of my rules, but others don’t work right (will work for a few hours and then stop) and others don’t even work at all. At this point, the firewall is doing WAY too good of a job at blocking services from the outside world!

I’m sure it’s a PEBKAC or maybe even an ID ten T error, because so many people just love their SonicWALL stuff. A few minutes ago, I said this in the IRC channel, and I think it’s fairly accurate at a certain level:

<wantmoore> i’d almost go out on a limb and say “windows is to linux as watchguard is to sonicwall”
<DavidSzp>    wantmoore: That’s an interesting analogy
<wantmoore>    watchguard: much easier to do stuff and make it work. sonicwall: a lot more flexibility, but not nearly as straightforward
<stephensflc>    I would totally agree with that statement at this point
<wantmoore>    the analogy doesnt stick where cost is concerned though 😉
<wantmoore>    in that regard, watchguard is a WHOLE lot cheaper. sonicwall will nickel and dime you to death

And I’ll stand by those statements for now. I’m sure that Moreno will help me get my issues resolved and I’ll join the Happy SonicWALL Club soon enough. Until then, I really miss my Watchguard and I’ll be hanging out in the corner with my friend Ed talking about our plans to startup and anti-SonicWALL user group.

(6) Comments   
Filed Under (Geekspeak, Work) by Justin on 2008-04-19

The blogging has been pretty sparse here lately, and I’m quite aware of that fact. Most of the action for me has started happening over on Twitter because its so quick and easy.

I kicked off a major project or three at work this week and they’re all dependant on one central project: a massive VPN rollout. For those who don’t know, I work for smallish company with a rather large footprint. Counting our corporate office where I work, we have 25 locations. Each branch office relies on several services housed at corporate and connect to these sources over the wide-open internet. A VPN has always been on my radar, but never really been considered fiscally until about two week ago when we finally decided to bite the bullet and do it.

My initial plan was to use OpenVPN running on top of Linksys WRT54GL’s at the branches and grab a new Dell R200 to be the hub of the VPN. After about three days of flashing different firmwares on my Linksys at home (OpenWRT, dd-wrt, Sveasoft) and trying to make it work, I threw in the towel and went back to the drawing board. I’ve mentioned before that we have a Watchguard Firebox Core X750e at corporate and I’ve been mostly happy with it. A quick look showed the Firebox Edge X10e to be the cheapest endpoint available to this with Watchguard hardware. However, at around $300 each, this would really quickly get very expensive. Not only that, but I couldn’t come up with anyone I knew who had a Watchguard deployment of this size to ask their advice and opinions.

As I was seeking advice from my pals in the Church IT RoundTable IRC channel, by some act of providence, Mark Moreno decided to grace us with his presence in the channel. I definitely need to insert a disclaimer and an apology at this point. Mark is a guy who is really knowledgeable and passionate about the product he sells and he makes no excuses for being a salesmen either. As a result, I’ve always given Moreno a really hard time about SonicWALL gear, mostly just for kicks. He knows to expect this kind of trash-talk whenever I’m around and always takes the ribbing in good fun. As Mark came in, I made the joke to everyone that I could probably make him drool with the details of the VPN project I was working on. Sure enough, he took the bait and started putting together some quotes. His initial number was somewhere in the range of $15,000 and I just laughed. We talked back and forth over the course of a few days and settled on a new  SonicWALL NSA 3500 to replace the Firebox at corporate and 22 SonicWALL TZ 150 endpoints to go to the branch offices and stripped them down to firmware only – no UTM or support options. The best part of all this is that Mark is preconfiguring all the VPN tunnels before shipping the hardware to me. I really give major props to Mark and SonicWALL for working hard to match an absolutely INSANE price that I found on NewEgg for the TZ 150. The final pricetag with hardware and his consulting time was just a little more than half of his original quote – quite a substantial savings!

Once the VPN is in place, I finally be able to rollout the IP phones I’ve been sitting on for a year to the branch offices, implement a web-based time clock for our staff employees, and join the remote PC’s to our Active Directory domain, which opens a TON of doors for management of these remote computers (software deployment and security patches via WSUS to name a couple).

So, if you don’t hear from me in the next few weeks, just check my Twitter feed (also conveniently located in the sidebar of this blog) or drop me an email or leave a comment. I’ve got a lot of work to do! Thanks again to Mark Moreno for making this a reality from a budget standpoint. I’ll update here as we progress with the implentation and you can bet I’ll let you know if I hit any snafus specifically related to SonicWALL.

(6) Comments   
Filed Under (Geekspeak, Work) by Justin on 2006-09-26

It’s 7:45 PM as I start to write this and I’m still in the office. I just finished upgrading our old trusty firewall, a Watchguard Firebox III 700, to a much faster (and sexier!) Firebox X750e. It’s thinner (by one-half rack unit) and much faster than it’s predecessor. So far, I’d say that the Fireware platform which the new box runs on is a huge step ahead from the previous WFS stuff that the older generation of Fireboxen ran on. I’ve been monitoring it’s traffic for the past 1h 25m 29s now and all seems to be well. Next on the agenda is probably going to me moving our SMTP gateway virus scanning to this puppy and off the old Symantec product atop Win2k Server which has become a huge thorn in my side lately.

All I know is this – I have to make sure everything here is ROCK SOLID before we leave for Russia. T-minus 9 days and counting…

(1) Comment   
Filed Under (Geekspeak) by Justin on 2006-08-03

I’m long overdue for a format/install on my Windows XP machine at work. Courtesy of AIDA32:

Operating System Properties
OS Name: Microsoft Windows XP Professional
OS Code Name: Whistler
OS Language: English (United States)
OS Kernel Type: Uniprocessor Free
OS Version: 5.1.2600 (WinXP Retail)
OS Service Pack: Service Pack 2
OS Installation Date: 1/18/2005

I knew it had been a while, but I didn’t realize it was over 18 months ago. The realization that it needed to be done occured yesterday when I downloaded the 30-day trial of Adobe Acrobat Professional and it wouldn’t launch after I installed it. So, I set out this morning to gather everything I would need to perform this feat, and abruptly changed my plans for the day after visiting the Add/Remove Programs dialog and making a list of all the applications I would need to reinstall…
Read the rest of this entry »

(2) Comments    Read More