Tag Archives: Net

GoDaddy Outage: RFC for Dummies

September 11, 2012 07:58 | 10 Comments |

MoïseYesterday was a black day for GoDaddy.com. During a few hours all they hosting services were interrupted. Mail, websites but, worse, all the DNS services were unavailable. The outage was caused by a member of Anonymous as said on Twitter but it’s not yet clear. Personally, who’s behind the attack, I don’t care! Results were the same: millions of websites remained unreachable during hours. Other people started to blame GoDaddy and to exort customers to move to another provider. Do you really think other companies would resist to a massive DDoS attack? I don’t!

Let’s leave this apart and focus on the consequences. Lot of websites were simple not reachable because the hostnames could not be resolved. Wait? When  I connected for the first time to the Internet (and – trust me – I’m here for a while!), everybody told me that this super-network was derived from a military project. The goal was to build an super-strong meshed network being able to resist to almost any attack from the “enemies”. Today, we are in 2012 and millions of sites are affected by a “simple” attack! Is there a problem somewhere?

Are people entitled to complain against GoDaddy for not providing the services they subscribed to? Is moving quickly to an alternate provider the best choice? I don’t think so. My idea is that Internet became today a real media like any other one and people tend to forget the complexity that exists behind nice websites with beautiful interfaces and plenty of features. Internet (read: “the set of all protocols used to build the Internet“) relies on RFC’s (“Request For Comments“). Those documents are memorandum published by the IETF (“Internet Engineering Task Force“) and describe how to build a working Internet. As a developer, manufacturer or designer, those RFC’s must be seen as golden rules for you!

Back to the GoDaddy story! There is a very interesting RFC2182 with the title: “Selection and Operation of Secondary DNS Servers“. If you read it (please do!), you will find best practices to define secondary DNS servers for your domain(s). How many do you require? How to deploy them? Let’s take a simple example: digitalz.org. This domain is hosted by GoDaddy:

$ dig digitalz.org ns

; <<>> DiG 9.8.1-P1 <<>> digitalz.org ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20319
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;digitalz.org. IN NS

;; ANSWER SECTION:
digitalz.org. 3600 IN NS ns13.domaincontrol.com.
digitalz.org. 3600 IN NS ns14.domaincontrol.com.

;; Query time: 68 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Sep 11 08:30:25 2012
;; MSG SIZE rcvd: 85

But if you look at the two registered nameservers (ns13 & ns14):

$ host ns13.domaincontrol.com.
ns13.domaincontrol.com has address 216.69.185.7
ns13.domaincontrol.com has IPv6 address 2607:f208:206::7
$ host ns14.domaincontrol.com.
ns14.domaincontrol.com has address 208.109.255.7
ns14.domaincontrol.com has IPv6 address 2607:f208:302::7

Both are part of the same backbone belonging to GoDaddy:

NetRange: 216.69.128.0 - 216.69.191.255
CIDR: 216.69.128.0/18
OriginAS:
NetName: GO-DADDY-COM-LLC
NetHandle: NET-216-69-128-0-1
Parent: NET-216-0-0-0-0
NetType: Direct Allocation
RegDate: 2004-05-24
Updated: 2012-02-24
Ref: http://whois.arin.net/rest/net/NET-216-69-128-0-1
NetRange: 208.109.0.0 - 208.109.255.255
CIDR: 208.109.0.0/16
OriginAS:
NetName: GO-DADDY-COM-LLC
NetHandle: NET-208-109-0-0-1
Parent: NET-208-0-0-0-0
NetType: Direct Allocation
RegDate: 2006-04-12
Updated: 2012-02-24
Ref: http://whois.arin.net/rest/net/NET-208-109-0-0-1

Finally, have a look at the BGP routes to access those IP ranges: They are announced via the same path (AS-26496)

BGP routing table entry for 208.109.255.0/24, version 111874851
Paths: (5 available, best #2, table Default-IP-Routing-Table)
Multipath: eBGP iBGP
  Advertised to update-groups:
     3         
  26496
    195.69.144.26 (metric 20) from 195.26.4.255 (195.26.4.255)
      Origin IGP, metric 1000, localpref 100, valid, internal
      Community: 5577:2000 5577:2100 5577:2103 5577:5000 5577:5002
      Originator: 195.26.4.133, Cluster list: 0.0.0.2
  26496
    195.69.144.26 (metric 20) from 195.26.4.254 (195.26.4.254)
      Origin IGP, metric 1000, localpref 100, valid, internal, best
      Community: 5577:2000 5577:2100 5577:2103 5577:5000 5577:5002
      Originator: 195.26.4.133, Cluster list: 0.0.0.1
  46786 26496
    199.59.206.17 from 199.59.206.17 (204.26.60.249)
      Origin IGP, localpref 100, valid, external
      Community: 5577:2000 5577:2100 5577:2150 5577:2199 5577:5000 5577:5001
  46786 26496
    199.59.206.29 from 199.59.206.29 (204.26.60.249)
      Origin IGP, localpref 100, valid, external
      Community: 5577:2000 5577:2100 5577:2150 5577:2199 5577:5000 5577:5001
  3549 26496
    208.178.63.97 from 208.178.63.97 (67.17.80.136)
      Origin IGP, metric 100, localpref 49, valid, external
      Community: 3549:4698 3549:31528 5577:1000 5577:1001 5577:5000 5577:5001

As you can imagine, any issue with this BGP autonomous system would have huge impacts on the services (being multi-homed would not solve all the problems). There are plenty of nightmare stories about BGP issues. In this case, best practices are to use multiple DNS servers spread geographically (ex: one on each continent) and connected to multiple backbones totally independant. In other words: Don’t put all your eggs in the same basket! Always keep in mind that RFC’s are your best friends. Follow and implement them to increase the availability of your online services.

Outages like the one of GoDaddy are always good opportunities to remind best practices. We learn by doing mistakes!

Posted in: Net | Tagged: , , , ,

Why is the World IPv6 Day Important?

April 6, 2011 08:15 | Leave a Comment |

IPv6 ClockThe World IPv6 Day is scheduled on 8th of June. During 24 hours, big players on the Internet will make their content available via IPv6. Popular websites, like Google, Facebook and many more, will be available via IPv4 or IPv6 depending on your network configuration. This blog is already available via IPv6 for a few months and I would like to invite all website owners and network administrators to take part of this worldwide live test. Why? Not because it will be funny or interesting from a technical point of view. Because, while introducing IPv6, mis- or default- configurations will also introduce more issues on your infrastructure and it could have a negative impact on your business. Here follows a good example that I faced yesterday.

I’m running my own mail server based on a Postfix setup. My SMTP relay has a dual IP stack but is not (yet) connected to the world using IPv6. It means that applications IPv6 ready will detect the presence of an IPv6 stack. For an unknown reason (for sure, a mistake), I added the following directive in my Postfix config:

  $ sudo postconf -e "inet_protocols = all"

It enables both IPv4 and IPv6. It also means that Postfix will perform AAAA DNS lookups. And here come the problems… I sent some e-mails to an organization fully IPv6 ready (yes, they are!) and those were rejected and blocked in the outgoing mail queue. Let’s have a look at this organization MX records:

  $ dig company.com mx
  [...]
  ;; QUESTION SECTION:
  ;company.com.            IN    MX

  ;; ANSWER SECTION:
  company.com.        900    IN    MX    10 mx3.company.com.
  company.com.        900    IN    MX    10 mx2.company.com.
  company.com.        900    IN    MX    10 mx1.company.com.
  company.com.        900    IN    MX    10 mx4.company.com.
  $ host mx1.company.com
  mx1.company.com has address 78.x.x.x
  mx1.company.com has IPv6 address 2a00:x::x:40
  mx1.company.com has IPv6 address 2a00:x::x:38
  mx1.company.com has IPv6 address 2a00:x::x:44
  mx1.company.com has IPv6 address 2a00:x::x:30
  [...]

The four MX records have both IPv4 and IPv6 addresses. This is why Postfix failed to deliver the emails. As described in the IPv6 Postfix support page, the order of IPv6/IPv4 outgoing connection attempts is not (yet?) configurable: IPv6 is always probed first.

As my Postfix detected a dual IP stack, it tried to deliver the e-mails over IPv6. This failed due to the lack of IPv6 connectivity! The problem was solved by forcing IPv4 only:

  $ sudo postconf -e "inet_protocols = ipv4"

If you have your own SMTP relay and offer IPv6 connectivity, a good idea could be to define backup MX records with IPv4 supports. In case of problems related to IPv6, e-mails will be relayed via your backup servers! This example shows how implementing IPv6 could affect regular operations.  That’s the purpose of the World IPv6 day. It’s not too late to prepare yourself…

Posted in: IPv6, Net | Tagged: ,

Belgium Would Like to Track Your Emails for Two Years?

August 12, 2009 20:48 | 1 Comment |

Crime Scene
Mr Stefaan De Clerck , the Belgian Justice Minister announced a new plan to fight (cyber-)criminals. Belgian ISP’s could be asked forced to keep a trace of all communications via e-mail for two years. Read the article on the RTBF website in French (or via Google Translate). The main reason invoked by the Minister is to help judges to track criminals and send official requests to the Internet Service Providers. My opinion is that the problem is taken the wrong side: Is two years a relevant delay to treat a file? A wise choice would be to give more resources to the Belgian justice and reduce this amount of time!

A few remarks now… By “communications via e-mail”, the Justice Minister means everything required to “track” who sent an e-mail to who, when and via which IP address. Luckily, the content of the e-mails is not concerned (right now?).

On a radio station, I heard that only e-mail addresses with .be domain names will be concerned by this measure. Big free e-mail providers like Google or Yahoo! don’t have to be afraid. Keep cool guys! A major question is: where Internet Providers will collect their data?

  • At MTA (Mail Transfer Agent) level? They will simply keep SMTP servers log files for a longer period.
  • At IP (layer 4 is the ISO model) level? They will sniff all the traffic passing thru port TCP/25?

I suppose it should be the first case: Almost all providers already filter traffic on port 25 and force their users to use their SMTP relays (to reduce spam). Webmail interfaces are also very common today and use HTTP(S) traffic.

Other reflexions on the flight… Are Belgian authorities certain that criminals use skynet.be or telenet.be e-mail addresses? Do they even still use e-mail services? The ex-top-one application is more and more replaced by real-time communications like Instant Messenging or micro-blogging like Twitter. Is it relevant to keep for two years e-mail sent between a student at the university and his parents?

Second point, Am I an ISP? I own my own domains (some of them in the .be top-level domain), my own servers and I manage my own e-mail addresses. Do I need to keep a trace of all my e-mail communications? A lot of .be e-mail addresses are hosted in foreign countries (MX records pointing to hosts outside Belgium). What about them?

The Belgian Internet Provider Association (ISPA) already gave a negative feedback about this project of law. The biggest issue for them will be the huge costs (in term of storage capacity) to keep a log of all customers activity. Even worse, Internet Providers could be forced to adapt their prices to cover the required investments. From my point of view, I clearly do NOT agree to pay an extra fee to my ISP to keep logs of a service that I do NOT use! I never used and I will never use the e-mail address linked to my access.

Once again, like for the Great Firewall of Belgium project, only “the average Joe” user will be tracked by this law, real-criminals already have the tools to keep their communications safe.

Posted in: Belgium, ISP, Net | Tagged: ,

Easy Geolocalization of IP Addresses

August 10, 2009 20:00 | 5 Comments |

Google Map Marker

If there was only one annoying operation for me, it will be to search contact information about IP addresses!

Often, network administrators and security guys have to found out to who is assigned a given IP address (example: when analysis logs or doing forensics searches). Useful information linked to IP addresses are contact information (technical, abuse), country and routing information (autonomous systems).

The whole IPv4 addresses space (2^32 addresses) is split in blocks and almost all of them are assigned by an organization called IANA (Internet Assigned Numbers Authority). This address space is divided in blocks of /8 (255^3 addresses) and assigned. But IANA cannot handle all the requests for IP addresses. That’s why there is some kind of a delegation system. The Internet is divided into several geographical zones where a “sub” authority, called “Regional Internet Registry“, handles local requests and follows the good usage of its own assigned IP blocks. Finally, each ISP redistributes its assigned addresses by splitting the blocks into smaller pieces (using bigger subnet masks). Finally, we have this kind of chain of delegation:

IANA -> Regional Internet Registry -> Internet Providers -> Customers

Source: http://www.iana.org/numbers/

Source: http://www.iana.org/numbers/

An Internet Service Provider located in Belgium will ask to its local IANA representative for IP addresses. For Belgium, it’s the RIPE (“Réseaux IP Européens“). Once IP addresses have been assigned to this ISP, all relevant information are stored in a “whois” database and the ISP can start to distribute them to its customers. Its responsibility will be to add customers information into the same whois database. Such database is used to store information about IP addresses, domain names or autonomous systems. To query a whoid DB, you need a client which is called… “whois” on UNIX (clients are available for all operating systems). To query a database just search for information. As an example, let’s search for more information about the IP address behing www.twitter.com:

$ host www.twitter.com
www.twitter.com is an alias for twitter.com.
twitter.com has address 168.143.162.116
$ whois -h whois.ripe.net 168.143.162.116
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '168.0.0.0 - 168.255.255.255'

inetnum:      168.0.0.0 - 168.255.255.255
netname:      EU-ZZ-168
descr:        Various Registries
country:      EU # Country is really world wide
remarks:      These addresses were issued by
              The IANA before the formation of
              Regional Internet Registries.
              
org:          ORG-NCC1-RIPE
admin-c:      iana1-RIPE
tech-c:       iana1-RIPE
status:       ALLOCATED UNSPECIFIED
mnt-by:       RIPE-NCC-HM-MNT
mnt-lower:    RIPE-NCC-HM-MNT
mnt-routes:   RIPE-NCC-RPSL-MNT
source:       RIPE # Filtered

organisation:   ORG-NCC1-RIPE
org-name:       RIPE NCC
org-type:       RIR
address:        RIPE Network Coordination Centre
address:        P.O. Box 10096
address:        1001 EB Amsterdam
address:        The Netherlands
phone:          +31 20 535 4444
fax-no:         +31 20 535 4445
e-mail:         hostmaster@ripe.net
admin-c:        CREW-RIPE
tech-c:         CREW-RIPE
mnt-ref:        RIPE-NCC-RIS-MNT
mnt-ref:        RIPE-NCC-HM-MNT
mnt-by:         RIPE-NCC-HM-MNT
source:         RIPE # Filtered

role:         Internet Assigned Numbers Authority
address:      see http://www.iana.org.
e-mail:       bitbucket@ripe.net
admin-c:      IANA1-RIPE
tech-c:       IANA1-RIPE
nic-hdl:      IANA1-RIPE
remarks:      For more information on IANA services
remarks:      go to IANA web site at http://www.iana.org.
mnt-by:       RIPE-NCC-MNT
source:       RIPE # Filtered

In the example above, we sent a whois query to whois.ripe.net and the database reported that the object has been assigned by IANA. No more information is provided. Now, you know why searching for IP addresses is so boring! Often, you don’t know which whois database to query to get the relevant information! Worst, the IANA whois database only contains domain names for which IANA is authoritative! Except for some well known blocks directly related to your business, you can’t remember for all of them which whois server to use. Let’s try another DB: whois.arin.net. Bingo! It found something:

$ whois -h whois.arin.net 168.143.162.116

OrgName:    NTT America, Inc. 
OrgID:      NTTAM-1
Address:    8005 South Chester Street
Address:    Suite 200
City:       Centennial
StateProv:  CO
PostalCode: 80112
Country:    US
<Remaining stuff deleted>

If you’re not lucky, you can now imagine the nightware to find out the right whois server! You’ll have to perform two or three queries before a successful search.

Recently, I found a nice website developed by a German company and called utrace. It allows you to enter a domain name, a host name or an IP address and will search the right whois database for you. It will also show you the IP address location on a Google map:

Click to enlarge

Click to enlarge

Once located on the map, the IP address can be re-used to perform a query against the right whois server and extra information will be displayed. As an extra, a small widget is available for your web pages as well as a PHP API. A very useful online tool! (I’m sure that other services like this one exists, feel free to share)

Posted in: Net, Websites | Tagged: ,

The Great Belgian Firewall is Back?

July 24, 2009 23:17 | Leave a Comment |

The Great Belgian Firewall

During the month of April, the Belgian authorities decided to prevent a very controversial web-site to be accessed by Belgian citizens (read my previous post). According to ZDNet.be, the idea of a firewall is back on the table.

The article states that host or domain names could be blocked via an official procedure (primary targets are pedophilia and phishing websites). Immediately some questions popped out of my mind… Of course, I’m against any form of censorship on the Internet. The “network of networks” is just a new media with one major issue: it can spread illegal material all over the world in a few seconds or give to bad guys a huge base of targets but it never created new illegal activities. They just took a new form! For ever, people tried to steal things or to break laws. That’s part of the human behavior! To make a ugly comparison with the road traffic: Closing our highways will not prevent trucks to deliver goods in our Belgian Ardennes, it will just take more time using alternative roads.

Back to the Internet. Will this blacklist really prevent pedophiles to continue exchanging illegal material? Clearly not! Protecting our children from visiting such sites is one thing. Anyway, I’m surfing the web for years and I never found pedophilia sites via Google or any other search engines. Fighting against the sources is much more important.

And what about the work load? New illegal sites are opened and closed so quickly everyday. Who will review them and ask the Internet providers to block them? Based on which criteria, websites will be blacklisted or not? Will the list be publicly available? What will be the procedure to remove a “clean” domain name from the blacklist? On a business point of view, to be able to quickly block new websites, the ISP infrastructure could have a non negligible cost. What about the smaller ISP? Will they have some delays to implement the solution?

On a technical point of view, I suppose the blacklist will be implemented at DNS level. Where will the blacklist be hosted? By federal authorities? All Internet providers will have to sync a local copy? What if a hacker successfully injects safe domain names into the list? Bypassing your ISP DNS is so easy! The opendns.com future will be assured! And big companies who maintain their own DNS? Will they be forced to implement the blacklist? And multi-national companies with DNS servers hosted in foreign states? Personally, I’m not using my ISP DNS for years!

To conclude, if the Great Belgian Firewall is deployed, a proper communication must be done on this topic! The goal of authorities is to protect “the average joe”. But Joe has rights and duties. He must respect the laws but he must also receive information to be aware of these laws. Once again, global censorship is not the right solution. I’m an adult, I can distinguish good from bad things by myself, I don’t need someone to decide for me. Of course, if I choose the “bad”, I know the risks…

Source: http://www.zdnet.be/news/105509/zwarte-lijst-voor-belgische-surfers-omstreden/ (Google translation).

Posted in: Belgium, Net | Tagged: ,

DShield Web Honeypot – Alpha Preview Release

February 17, 2009 22:26 | Leave a Comment |

SANS ISC

isc.sans.org announced today the Alpha availability of the DShield Web Honeypot:

“The goal of the Web honeypot project is inline with the original DShield project, the data collected through the sensors feed the Dshield web database where human volunteers as well as machines pour through the data looking for abnormal trends and behavior. In addition, we would like to use the honeypot data to measure web attack prevelance and find objective metrics to recommend protective measures. The data collected will also be shared with the research community upon request later this year and be made available in aggregated form via the DShield website.”

Like the classic DShield service, you’ll be able to submit your logs to SANS for further processing and correlation. How does it work? You install the honeypot on a machine (preferably a dedicated machine or a virtual one) which will receive all the garbage on port HTTP(80). Requests will be logged and sent to ISC.

I read the FAQ and found an interesting question: “Is it legal? Their answer is quite fuzzy. I would say that honeypots are legal until you don’t attract the bad guys. Example: if you announce “DiVX for free” on a homepage and catch/log them it’s illegal. On the other hand, if they come by themselves, it’s legal.

Check out the website for more information.

Posted in: Net, Security, Software | Tagged: ,

Do We Need a New Internet?

February 16, 2009 17:36 | Leave a Comment |

New!

An article has been published this weekend in the NY Times about a hot topic: “Do we need a new Internet?“.

To resume, the journalist, John Markoff, explained that the current Internet is not able to survive to attacks from malicious codes (viruses, malwares and other nice things). Unfortunately, deploying a brand new Internet (based on IPv6 like suggested John) will never solve the problem. Why?

Whatever the protocols or applications used, business will grow and a high-revenue e-business means a growing interest from criminals! They evolved from destructive attacks (remember the DDoS golden years) to more malicious activities to stealth valuable data.

First, if you consider a chain, its strength will be equal to the one of the weakest link. The same model applies to security (and Internet): in this case, the weakest link is… the user! This is by nature: Humans tend to trust too easily. And IPv6 or any new bullet-proof technology will never change their behavior. However, new tools can warn the user and help them to take the right choice (basically, “to click on this link or not?” or “to open this attached file or not?“). User education stays a key topic (regular awareness trainings are required).

On the other side, do we really need a new Internet to face the growing demand in bandwidth and IP addresses? For years now, organizations like IANA or RIPE warn about a very-soon lack of IPv4 addresses (just as reminder: 32 bits == 4294967296 addresses). A one-time switch from IPv4 to IPv6 is impossible. Cohabitation will continue for years with mixed environments (Examples: backbones running on IPv4 and private networks on IPv6). IMHO, the highest demand in IP addresses will come from new electronic devices which will require network connectivity but which will not be publicly available in the Internet (think about sensors of all kinds).

Finally, take care with IPv6. It’s not yet widely deployed and almost unknown to the end-users. At this moment, it means, except the standard security features introduced in the new protocol version, a global lack of security around IPv6. If not properly secured, it will be for sure an important vector of penetration in networks or systems. Just my 0.02€!

Posted in: Net | Tagged: ,

France: IP Addresses are no Longer Considered as Private?

February 10, 2009 21:24 | 1 Comment |

Privacy

My last post was a little bit funny but the quote came from a French newspaper article about a judgment which can have an effect similar to an earthquake in France! It’s about the “privacy” aspect of IP addresses. A previous justice decision made by Court of Appel in Rennes was broken by another Court of Cassation.

Previously, an IP address was considered as private and was protected by a French organization: the CNIL (“Commission nationale de l’informatique et des libertés“). Nobody was authorized to track IP addresses without the knowledge of the Internet users. Now, according to the judgment, anybody may collect IP addresses and contact the authorities (Police) to request the address details (who used the address and when).

This is a major step forward for anti-piracy organizations which track users on P2P networks. They can safely let sniffers run on P2P networks and collect data. But… Technically an IP address can be easily spoofed. What will happen if your IP address is found in a “bad users” list and your details communicated to anti-piracy organizations? How to prove your innocence?

The article is available here (in French).

Posted in: Net | Tagged:

Google != Internet

January 31, 2009 17:52 | 1 Comment |

Google Monster

As said in a previous post, during a few minutes this afternoon, Google decided to tag all the websites as “potentially dangerous”.

This was quickly fixed but immediately reported by several sites:

(I’m sure it will be a buzz in the coming hours!)

Google is a major player on the Internet: For free, it will take care or your e-mails, your documents, your websites statistics, your calendar and the list is long! When Google suffers of a bug or a technical problem, the visibility is immediate.

But, Google is NOT the Internet!( some new comers or low-level Internet users can imagine this, Google is everywhere). Google is in trouble? Use another service! So simply!

Posted in: Net, Websites | Tagged:

Google Suddenly Decided that the Whole Internet is Dangerous?

January 31, 2009 17:14 | Leave a Comment |

Google Logo

What happened to Google this afternoon (16:00 GMT+1)? Suddenly, all websites are reported as “suspicious”:

Click to enlarge

Click to enlarge
Posted in: Net, Websites | Tagged: