HackDig : Dig high-quality web security articles for hackers

A Tale of CenturyLink Backdoors, PCI Compliance, and Pain. Lots of Pain.

2014-08-11 20:35

I have a client with an ActionTec M1000 modem running firmware QA02.5-3.60.3.0.8.6-M1000. It’s on a business CenturyLink DSL line and routes for five public IP addresses. For ease of writing, I’m going to use 1.1.1.1 through 1.1.1.5 to refer to the IP addresses. The modem itself is assigned the fifth IP address, (.5). The office firewall is assigned the first (.1), the office hybrid VOIP PBX has the second (.2), the website has the third (.3) and the fourth IP address (.4) is unassigned. Simple so far, right?

The small business has been slowly but surely edging towards PCI compliance with the help of their acquiring banks and payment processors. One of the check boxes they have to fill is passing an automated PCI compliance scan. One of the scans was against the office’s external IP address. That particular scan continually failed on two ports that had an unusual listening service. Apparently 1.1.1.1:4567 and 1.1.1.1:51080 were open and responding to HTTP and HTTPS requests respectively (remember, .1 is the office’s public IP address). Port 4567 is always seen as failing the scan, but port 51080 is only occasionally seen to be failing. When looking into this trouble, I did not find any documentation within the organization that made reference to those ports as being opened or forwarded to an internal device. No one knew where those open ports had come from.

Here is an example of the remarks on a failing scan of port 4567:

Description: Web Server Uses Basic Authentication Without HTTPS
Synopsis: The remote web server seems to transmit credentials in clear text.
Impact: The remote web server contains web pages that are protected by ‘Basic’ authentication over plain text.
An attacker eavesdropping the traffic might obtain logins and passwords of valid users.
Data Received: The following pages are protected. /:/ realm=”WebAdmin” /html/:/ realm=”WebAdmin”
Resolution: Make sure that HTTP authentication is transmitted over HTTPS.
Risk Factor: Medium/ CVSS2 Base Score: 4.0
AV:N/AC:H/Au:N/C:P/I:N/A:N

When I was first made aware of that failure report, I thought “That’s odd.” The office IP address is assigned to a SonicWall Firewall that is pretty well locked down. I know each of the firewall rules almost by heart and you can count what’s allowed through the firewall using the fingers on one hand. Port 4567 is most certainly not allowed! Only email, an https user portal, and a VPN end point have rules allowing traffic. This is… bizarre. To say the least.

I nmap’d the office IP address and sure enough, port 4567 was listening and responding to HTTP requests with an HTTPAUTH login prompt. THIS IS MADNESS! There is nothing on the SonicWall that is allowing 4567 and there is a bog standard default deny rule for all things that aren’t explicitly allowed. “Was the SonicWall itself rooted?!” I began to fret.

When telnetting to the socket, I would get the following error:

Escape character is '^]'.
GET / HTTP/1.0
 
HTTP/1.0 401 Unauthorized
Server: 
Content-Type: text/html
Date: Thu, 06 Feb 2014 01:22:07 GMT
Last-Modified: Thu, 06 Feb 2014 01:22:07 GMT
Accept-Ranges: bytes
Connection: close
WWW-Authenticate: Basic realm="WebAdmin"
 
<HTML>
<HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY BGCOLOR="#cc9999" TEXT="#000000" LINK="#2020ff" VLINK="#4040cc">
<H2>401 Unauthorized</H2>
Authorization required for the URL '/'.
<HR>
<ADDRESS><A HREF=""></A></ADDRESS>
</BODY>
</HTML>
Connection closed by foreign host.

Basic realm=”WebAdmin”? What on earth?! But wait… my spidey sense tingled. Or more accurately, the dim recesses of a distant corner of my memory began to glow. The last time I saw an HTTP authentication login box associated with an external address for this organization was their ActionTec modem. But that requires HTTPS, is sitting on the standard port of 443, and most importantly is on a completely different IP address! So that couldn’t be it. And yet…

I typed the modem’s external address into my web browser (secured WAN administration access was allowed on the modem at that time). The URL https://1.1.1.5/ caused a self-signed certificate error. I accepted the error, knowing that it was the proper certificate for the little modem, and then saw a login box:

Okay so that’s all norm… WAT. WebAdmin?!

I scrambled back to http://1.1.1.1:4567 (the external IP address for the firewall!) and saw a plain HTTP login prompt. This was not over HTTPS. I did not have to accept a self signed SSL certificate. With much fear and trembling, I typed in the username and password for the administrative user of the modem. Waves of confusion and anger swept over me as I stared at the information home page for the modem’s web administration interface. I had successfully logged in to the modem. Using the firewall’s IP address. Over HTTP.

In disbelief, I opened another tab and typed in the modem’s external address https://1.1.1.5/. I had to accept the self signed certificate again. I then logged in with the exact same username and password. Once again, I was staring at the modem’s web administration page.

I tried http://1.1.1.5 and got no response. Of course! There’s no administration web server running on port 80. I disallowed unsecure remote administration in the modem’s options. There’s not even a redirect to the https site that’s on port 443. I then tried https://1.1.1.5:4567 (notice the ‘s’). Keep in mind that’s the modem’s IP address but with that strange port I used on the firewall’s IP address that was sending me to the modem’s administration page. I got no response. I then tried http://1.1.1.5:4567. I got the login box and was able to log in with the modem’s administrative credentials.

I then tried http://1.1.1.2:4567. The public IP address of the PBX. I got the login box for the modem. I next tried http://1.1.1.3:4567 (the website). Yet again, I got the modem’s login box. Finally, I triedhttp://1.1.1.4:4567, the IP address that has never had anything assigned to it in the seven years this client has leased the /29 block. I got the modem’s login box.

What is this I don’t even.

malwat

A Brief History of Port 4567

Let’s take a step back and look at a history of port 4567 as relates to both this client’s past and the history of the Internet at large.

If I nmap -Pn the entire netblock, every IP address, regardless of if there’s a device assigned to it or not,even the network and broadcast addresses, will show a response on port 4567. Nmap likes to say that it’s the TRAM service purely based on the IANA’s Service Name and Transport Protocol Port Number Registry document.

Port 4567 TCP and UDP were apparently assigned to a service named TRAM that was designed in the late 1990s. Developed by an engineer at Sun, it appears to be something similar to multicast. It doesn’t seem to have ever caught on or been developed into a production set of tools.

Nevertheless, it’s still in the IANA’s Port Number Registry so if anything is running on port 4567, a quick look at the official list of ports will suggest that it’s TRAM. Let me be the first to say that TRAM is not what’s running on port 4567 of this CenturyLink modem.

So let me get this straight: I can turn on the remote management page for the modem, which does a few things for me:

  • I can only access the administration page on one single WAN IP address. The one that’s directly assigned to the modem.
  • I have to go across HTTPS and the standard port of 443.
  • I can turn the remote admin page on and off at will and lock things down by only allowing administrative access from the LAN.

However, I also have the non-option of using the “secret” port 4567 ingress point that offers me the following:

  • Hijacks all traffic to every IP address that it routes to as long as the destination port is 4567.
  • Does not accept HTTPS traffic at all, but only HTTP traffic.
  • Cannot be turned off.

Even if I turn off the “official” remote administration option within the modem, port 4567 is still open, still accepting only HTTP traffic, and I can still log in with full administrator privileges with the exact same account that the official administration page requires.

Oh But Wait, U Kan B Sekure 2!

Hark! What is the other port that the PCI compliance scans would occasionally flag as having vulnerable services running? That would be port 51080. You don’t suppose…

I tried https://1.1.1.1:51080 and received a familiar certificate error. After accepting the certificate, I was taken to the modem’s login page. This time, magnificently secured with SSL. Well, secure as long as we ignore the poor implementation of SSL that’s running with known vulnerabilities and lighting the PCI compliance scan up like Broadway.

At this point, I began to wonder just how long this had been going on.

A History of Failure

After briefly considering a new career as an alcoholic, I decided to delve into this client’s past and check the company that performs the PCI compliance scans for a detailed history. I was in luck! My client’s dashboard of information at the PCI scanning company came complete with a detailed history of each scan.

As I read the history, my shock was only paralleled by my anger. There was a time when my client had passed the scan on their office’s WAN IP address. However, oddities eventually began showing up on various quarterly scans. Keep in mind that this scan directly targets the IP address associated with the office’s WAN firewall, and that IP address alone. The modem, PBX, and other IP addresses on the /29 are not targeted.

The first failing scan took place all the way back in March of 2012. The culprit of the failure? Port 51080 had suddenly shown as being open with the notation: “Description: SSL certificate is signed with weak hash function: MD5 Severity: Potential Problem”. No documented service or business need has ever required that port to be opened and there’s no history of it being opened on the firewall. This happened entirely without my client’s action or knowledge. What this meant was that sometime in March 2012, Qwest / CenturyLink pushed out an update to the modem that in essence implanted a backdoor into the ActionTec m1000 modem, but it was a backdoor that affected every IP address that it routed for. Furthermore, the modem that was purchased (not leased) by my client.

SIDE NOTE: Why didn’t this get caught sooner? The client has been working through a lot of PCI compliance requirements and moved their 30 year old non-profit quite a ways into the 21st century through no small efforts over the course of a few years. Some of the finer points, such as monitoring the automated scan reports, did get missed in the shuffle of discarded knuckle-busters and transitioning to the wonderful world of paperless transactions. They literally changed every single part of how they handled money transactions, sometimes multiple times, in the quest to overhaul their financial workflows. They’re way better now, so good for them. Moving on…

As I moved from the quarterly scan taken in March of 2012 to the next scan taken in June, I saw that the mysterious port 51080 was gone by the next scan, replaced instead by a firehose blast of problems reported with a service running on port 4567. There were eight vulnerability warnings in total, six associated with what was running on port 4567, one for port 80, and another for port 443.

Once again, let me continue to beat this dead horse: This was ostensibly the WAN IP address for the SonicWall firewall at the office. However, nothing has ever run on or been forwarded to ports 4567 or 80. Port 443 was associated with the user portal for Microsoft Small Business Server. Only one out of those eight failing scan results was for something allowed through the firewall itself, in spite of port 4567 and 80 being blocked with no NAT rules on the firewall.

Furthermore, the vulnerabilities that were listed are completely alien to anything that has ever been in the office. Here are the scan results for port 4567 in June 2012:

Result #1

Title: Web server vulnerability

Impact: /modules.php?letter=%22%3E%3Cimg%20src=javascript:alert(document.cookie) ;%3E&op=modload&name=Members_List&file=index: Post Nuke 0.7.2.3-Phoenix is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Result #2

Title: Web server vulnerability

Impact: /myphpnuke/links.php?op=MostPopular&ratenum=[script]alert(document.c ookie);[/script]&ratetype=percent: myphpnuke is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Result #3

Title: Web server vulnerability

Impact: /myphpnuke/links.php?op=search&query=[script]alert(‘Vulnerable); [/script]?query=: myphpnuke is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Result #4

Title: Web server vulnerability

Impact: /phpimageview.php?pic=javascript:alert(8754): PHP Image View 1.0 is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Result #5

Title: Web server vulnerability

Impact: /forum_members.asp?find=%22;}alert(9823);function%20x(){v%20=%22: Web Wiz Forums ver. 7.01 and below is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Result #6

Title: Web server vulnerability

Impact: /members.asp?SF=%22;}alert(‘Vulnerable’);function%20x(){v%20=%22 : Web Wiz Forums ver. 7.01 and below is vulnerable to Cross Site Scripting (XSS). CA-2000-02.

Finally, here’s the vulnerability result for port 80:

Title: Web server vulnerability

Impact: Default account found for ‘NTLM’ at / (ID ”, PW ‘_Cisco’). Cisco device.

There is only one Cisco device in the company, and that’s a LinkSys e4200 WAP for office wireless access. The account to log into the e4200 is not the default. However, that’s a moot point because this is a scan against the IP address for the office’s firewall, a completely unrelated device. The e4200 does not have a WAN presence! I searched around on the web for what Cisco devices carry a blank default username and a password of “_Cisco”. The only device that came back was a Cisco Aeronet WAP. No device branded as Cisco Aeronet has ever existed in this organization at any time, nor has any WAP of any brand had that publicly routed IP address.

The next quarterly scan, this time in September 2012, showed more anomalies! Port 51080 was back, this time with three warnings:

Port 51080 Warning #1

Title: TLS Protocol Session Renegotiation Security Vulnerability

Impact: The vulnerability allows man-in-the-middle attack.

Port 51080 Warning #2

Title: SSL certificate is signed with weak hash function: MD5

Impact: The SSL/TLS certificate is signed with a weak hash function. An attacker may be able to forge a SSL/TLS certificate that would appear to be valid for the website. This may allow an attacker to perform a man-in-the- middle attack against the SSL-secured website.

Port 51080 Warning #3

Title: SSL server accepts weak ciphers

Impact: A remote attacker with the ability to sniff network traffic could decrypt an encrypted session.

And 4567 was still there, but this time with only two warnings:

Port 4567 Warning #1

Description: Web Server Uses Basic Authentication Without HTTPS

Synopsis: The remote web server seems to transmit credentials in clear text.

Port 4567 Warning #2

Title: web server uses cleartext HTTP Basic authentication (/)

Impact: Poor authentication practices may leave the web application vulnerable to authentication attacks.

The next quarterly scan in December showed only one odd failure, and that was for port 4567:

Description: Web Server Uses Basic Authentication Without HTTPS

Synopsis: The remote web server seems to transmit credentials in clear text.

Concerning port 4567, the rest of the quarterly scans until the day that I looked deeply into this trouble in early 2014 continued with only the above two warnings. Port 51080 seemed to disappear in the security warnings for about a year, until early 2014 when, you guessed it, it was back, this time with a single error in the scan report:

Title: TLS Protocol Session Renegotiation Security Vulnerability

Impact: The vulnerability allows man-in-the-middle attack.

Resolution: For OpenSSL, [http://www.openssl.org/source/] upgrade to 0.9.8l or higher.

For Microsoft IIS web servers, install the appropriate patch available through [http://technet.microsoft.com/en- us/security/bulletin/MS10-049] Microsoft Security Bulletin 10-049.

For other types of products, consult the product documentation.

Risk Factor: Medium/ CVSS2 Base Score: 5.8

Inspecting the Modem Itself

I decided to inspect the modem to see if there was any indication within the firmware that this is a feature. Maybe I can turn it off, or perhaps mess with the routing tables or firewall rules. Anything to get this horrible behavior to stop. I looked at the firewall settings within the modem (Modem? Router? Firewall? Yes, this is one of those annoying consumer-grade-but-we’ll-sell-it-on-business-lines-as-well style of units that tries to be all things to all people).

The firewall feature of the modem was turned off. There were no NAT rules at all. No routing rules except for the public subnet information. Remote management was turned off (no really).

Also, of interest is that the firmware version is listed as QA02.5-3.60.3.0.8.6-M1000. At the time of this problem, that was documented on ActionTec’s website as being the latest firmware for the M1000 device. I can find reference to that version of firmware going back to 2008. An official CenturyLink document for the firmware is stamped with the year 2010.

Here’s the twist: the stated firmware version pre-dates the version of the observed vulnerabilities that were implanted into the device. The vulnerabilities detected on the open ports seem to indicate a series of modifications that took place after the stated timeframe of the firmware version, thus completely outside of the firmware update or documentation cycles.

One of my first concerns when this ordeal started was that the modem had perhaps been auto-updating without anyone’s knowledge, however that wasn’t the case. There is no auto-update facility for the modem’s firmware. By all external appearances, the only way to update the firmware is for someone to log into the web interface and manually upload and update a firmware image. At least, that’s the story that’s made publicly available. (Certainly something could be easily scripted with curl, wget, etc. but I wouldn’t expect an ISP to do something as hacky as that.)

To summarize, there is no visible means of determining that the modem is intercepting all traffic destined to ports 4567 and 51080 across every IP address that it routes for. There is no indication of a continuous stream of updates modifying the firmware of the modem. There is no way to stop this behavior or disable the exposed ports and vulnerable services.

Further Toying

I wanted to get a deeper grasp of how this traffic tampering was taking place. From the office’s LAN, I attempted to access port 4567 on all of the public IP addresses, but was not greeted with the modem’s login page. I’m going to assume that’s because the traffic is not passing through the WAN interface of the modem to get to the /29 side. For example, accessing .2 (the PBX) from the office LAN goes out through .1, but is entirely contained on the /29 side of the modem.

Thinking further, I wondered what would happen if a legitimate service was running on one of the /29′s hosts at port 4567. Perhaps there was enough intelligence in the modem to see if the intended host’s IP address was really offering up a service on 4567 and then pass it through instead of intercept it. Crazy, right?

I set up a NAT rule on the firewall to port forward 4567 to the office’s Ricoh multifunction printer admin page. When accessing the public IP address for the firewall over port 4567 from a workstation on the office’s LAN I was presented with the Ricoh’s admin page. When accessing the firewall’s public IP address over port 4567 from a remote host, I got the modem’s administration page.

Nice.

ISP Engineers to the… Rescue?

I didn’t have anything else to do but contact the ISP. I knew this was going to be a circus. Fortunately, the organization uses a mediator company that interfaces with the ISP on their behalf. The arrangement is nice because the mediator company has authority to speak on the organization’s behalf, knows how to play the ISP game, and also has no skin in the game and is just as happy to move us to a different ISP if we’re not happy with the current service provider. All of this for free because the mediator company works off a commission of the total monthly bill, regardless of who the ISP is.

With the help of the mediator company, I fashioned an email briefly describing the trouble and sent it to three “engineers” at CenturyLink. The response? “Uhh… we’ll send this on to someone who can help.” To cut a long story short, it took roughly five weeks and multiple emails, redirections, and befuddled engineers before we were finally told to call DSL support.

That’s right. We were told to just call regular tier 1 DSL support and work our way up the ladder. I may or may not have excused myself to punch the sofa cushions.

Mediator company to the rescue once more! My impassioned contact at the company called DSL support and demanded her way through the offshored support system right into a Sr Engineer’s desk phone located somewhere in Idaho. I explained my predicament to the engineer who immediately and plainly stated “Yeah you can’t change that on those modems. You need the PK5000. Anything else I can help you with?”

Bam! Simple as that.

Problem: Solved. At Least, Until the Next Forced Firmware Update

And that’s exactly what happened. We ordered a different modem that was said to not have those ports open. I won’t drone on with the details that involved one DOA modem, one modem of a completely different model number, a destroyed firewall, and a CenturyLink employee unexpectedly transferring to a different office in the middle of it all. In the end, the organization received the intended modem and it actually worked like we were told it would! No longer was the modem hijacking traffic. No longer was the PCI compliance scan failing.

There was no official explanation by CenturyLink, nor did I expect one. They’re a huge ISP that has to manage millions of CPE devices, and they apparently do so with a horrible, hacky, insecure method of hijacking all traffic and forcing ports open across entire subnets.

The organization I was consulting for had a “business class DSL” line, which is really not “business class” at all. Nothing DSL can be considered business class, and certainly not with the same modem that a home user would receive. I explained to the business that this was to be expected, in some way, because of the class of internet connection they had. They understood.

In the end, while trying to make sense of the scenario, the only explanation I could find that seemed to fit the symptoms was something called TR-069. TR-O69 is a CPE WAN Management Protocol (CWMP) and, to use Wikipedia’s words: “As a bidirectional SOAP/HTTP-based protocol, it provides the communication between customer-premises equipment (CPE) and Auto Configuration Servers (ACS). [...] CWMP is a text based protocol. Orders sent between the device (CPE) and auto configuration server (ACS) are transported over HTTP (or more frequently HTTPS)”

Well there you go. That sounds about right to me, based on the information I found and was given. I just didn’t expect it to be ramrodded into the equipment in such an inelegant and insecure way, and for no one at CenturyLink to apparently know about it until I found one John Wayne-esque engineer that took two seconds to give me an answer. This also sounded very much like the “Easter egg: DSL router patch merely hides backdoor instead of closing it” problem that was first discovered on a Linksys WAG200G DSL modem/router.

In the end, my client’s problem was solved, and I have at least a few more reasons to dislike CenturyLink. Got a similar story? Anything to add? Hit me up in the comments.


Source: -fo-stol-niap-dna-ecnailpmoc-icp-sroodkcab-knilyrutnec-fo-elat-a/82/40/4102/moc.nimdaybbuneht

Read:6236 | Comments:0 | Tags:No Tag

“A Tale of CenturyLink Backdoors, PCI Compliance, and Pain. Lots of Pain.”0 Comments

Submit A Comment

Name:

Email:

Blog :

Verification Code:

Tools

Tag Cloud