Wednesday, August 24, 2005
Two years and dozens of worm variants after the W32.Blaster.worm worm infected millions of machines using an easy to block TFTP callback mechanism, the latest variant of the Zotob family is using the same technique. The W32.Zotob.K worm may spread on some networks more successfully than previous variants, all of which attempt to exploit the MS05-039 buffer overflow defect in Windows systems. Using this technique, a worm author trades complexity in one area of the worm design (the overall transport logic) for simplicity in another (the code which exploits the buffer overflow). Previous variants have connected to the victim computer on port 139 or port 445, where it hopes to find an unpatched software agent listening. Then, a packet is sent containing some things that the victim expects to receive, and some things it does not -- all must be arranged very precisely. This package includes the message which trips the buffer overflow, and the code the attacker seeks to run on the remote system immediately thereafter -- which includes a copy of the worm. It turns out that most variants of the worms that exploit MS05-039 directly have been limited in their ability to spread, even on networks of systems entirely vulnerable and unpatched. Their slow spread appears to be due to a quirk -- the attempt to execute the complicated instructions and upload the entire worm to the victim will sometimes fail, causing the target system to reboot, without having first been infected. The TFTP callback allows a simpler package to be delivered through the buffer overflow, and probably makes it more reliable as a result. Instead of a big payload with lots of instructions, a small payload can be delivered. Basically, the worm says, "Hey, call me back." The attacking, worm-infested computer first sets up a listener on port 69, which is able to respond to TFTP requests. It's a small bit of code and it has become standard fare in the "off the shelf" worm building toolkits. The instructions sent through the buffer overflow ask the victim computer to fetch a file from the attacker, using a TFTP client software utility built into Windows, and then execute the resulting file. Organizations which have continued exposure of large numbers of Windows systems (unpatched for MS05-039, and with NULL sessions enabled) should consider blocking the TFTP port 69/udp on internal routers before these new variants hit your network. If this TFTP callback on port 69/udp is so easy to block, why do so many organizations still have it open on their networks? It turns out that many network devices including routers and switches occasionally use TFTP to communicate with network management consoles. This is another good reason why the port should be blocked -- just remember to leave it open to and from a small number of network management consoles or subnets, not throughout the entire network. You can easily block these TFTP callback worms without interfering with your ability to manage routers and switches. Will this become an arms race with new variants opening the TFTP callback trojan on a different port each time? Perhaps. Some worms exploiting the MS05-039 vulnerability apparently open their own FTP server on a high numbered port, using that for a callback transport rather than TFTP. However, the TFTP callback remains a popular exploit, and it's easy enough to block it. The TFTP program on Windows seems to be hard-wired to call to port 69, which explains the continued popularity of this particular port. Permanently blocking this port deprives the worm of a propagation technique with a long and successful history. Worm authors might possibly switch to a different protocol. The other obvious choices, FTP and HTTP, would seem to place a greater burden on the instructions that need to be sent through the buffer overflow exploit, sending the worm author back to square one -- a worm that doesn't propagate very well because the buffer overflow exploit is too fragile. In any case, you won't likely be chasing TFTP all over the port map. It has stayed right there on port 69 for years, and partitioning your internal network on this port remains an effective strategy for mitigating the spread of many worm variants.
A number of systems administrators have asked me for some nice authoritative PDF and official looking references to support them in discussions with management regarding recovery strategies (see IRC Botnets: The Needle and The Damage Done). Senior managers are not particularly impressed by blogs, you know. Here are a few that I had handy. Note that this guidance is not universal. AntiVirus vendors, in particular, commonly claim that cleanup tools are sufficient recovery in most cases, although lately even a few of them have become more cautious in their claims for their cleanup tools. If you find any other nicely authoritative references on this topic, let me know and I'll add them to this page. The first reference actually surprised me. I was working on a large team in a gargantuan organization, and a client asked a Microsoft Security Consultant if Microsoft recommended re-imaging from worm attacks, and if they in fact practiced this type of recovery for internal problems. The gentleman responded with a clear and concise explanation that it was one of the 10 Immutable Laws of Security. I was so stunned I forgot to ask about the other 9. It turns out that it's actually number 1 and 2 on their list. (They also have a reasonable overview of Responding to IT Security Incidents . The NIST documentation is more detailed, but this overview is reasonably concise and might be helpful for managers.) 10 Immutable Laws of Security
Computer Security Incident Handling Guide
Law #2: If a bad guy can alter the operating system on your computer, it's not your computer anymore In the end, an operating system is just a series of ones and zeroes that, when interpreted by the processor, cause the computer to do certain things. Change the ones and zeroes, and it will do something different. Where are the ones and zeroes stored? Why, on the computer, right along with everything else! They're just files, and if other people who use the computer are permitted to change those files, it's "game over". To understand why, consider that operating system files are among the most trusted ones on the computer, and they generally run with system-level privileges. That is, they can do absolutely anything. Among other things, they're trusted to manage user accounts, handle password changes, and enforce the rules governing who can do what on the computer. If a bad guy can change them, the now-untrustworthy files will do his bidding, and there's no limit to what he can do. He can steal passwords, make himself an administrator on the computer, or add entirely new functions to the operating system.Steps for Recovering from a UNIX or NT System Compromise
Keep in mind that if a machine is compromised, anything on that system could have been modified, including the kernel, binaries, datafiles, running processes, and memory. In general, the only way to trust that a machine is free from backdoors and intruder modifications is to reinstall the operating system from the distribution media and install all of the security patches before connecting back to the network. Merely determining and fixing the vulnerability that was used to initially compromise this machine may not be enough. We encourage you to restore your system using known clean binaries. In order to put the machine into a known state, you should re-install the operating system using the original distribution media.The CERT® Guide to System and Network Security Practices
An intruder may have altered user data and application program areas. Examples where this may occur includeNIST Special Publication 800-61
Use the latest trusted backup to restore user data. For files that have not been compromised, you can consider using the backup that was made closest in time to when an intrusion was detected to avoid user rework. This should be done with caution and is based on having a high level of confidence that restored user files were not compromised. Regardless, you need to encourage users to check for any unexpected changes to their files and warn them about the risk of compromise.
- installing back doors to provide future access. For example, an intruder installs a program in a local user directory that is called each time the user logs in, providing an unprotected login shell that can be accessed by anyone via the Internet.
- compromising user data to sabotage the user's work. For example, an intruder makes small changes to spreadsheets that go unnoticed. Depending on how the spreadsheets are used, this can cause minor to major damage.
Computer Security Incident Handling Guide
5.4.3 Eradication and Recovery (Malicious Code Incident Handling) Antivirus software effectively identifies and removes malicious code infections; however, some infected files cannot be disinfected. (Files can be deleted and replaced with clean backup copies; in the case of an application, the affected application can be reinstalled.) If the malicious code provided attackers with root-level access, it may not be possible to determine what other actions the attackers may have performed.(91) In such cases, the system should either be restored from a previous, uninfected backup or be rebuilt from scratch. The system should then be secured so that it will not be susceptible to another infection from the same malicious code. 6.4.3 Eradication and Recovery (Unauthorized Access Incident Handling) Successful attackers frequently install rootkits, which modify or replace dozens or hundreds of files, including system binaries. Rootkits hide much of what they do, making it tricky to identify what was changed.(94) Therefore, if an attacker appears to have gained root access to a system, handlers cannot trust the OS. Typically, the best solution is to restore the system from a known good backup or reinstall the operating system and applications from scratch, and then secure the system properly. Changing all passwords on the system, and possibly on all systems that have trust relationships with the victim system, is also highly recommended.Checking Microsoft Windows® Systems for Signs of Compromise
If a rootkit is installed on your system, it will be extremely hard to detect. At present, there are only two tools that we aware of that can aid the discovery of a rootkit, and the associated procedures are extremely difficult to follow. It is for precisely this reason we would recommend simply reinstalling the operating system ; it will take far less effort and time. Indeed, it could be argued that these procedures should only be used for either academic curiosity and forensics of an attack, or if the system is of extreme importance. Regardless of your findings, it is still highly likely that a compromised machine will always remain compromised, and thus cannot be trusted.The following document from the US CERT is less rigorous than the others cited here, as well as a bit ambivalent. Note that it's also internally inconsistent. The document advises trying antivirus cleanup, but then states that reinstallation is "the only way to ensure" a secure recovery of a system. The US CERT should revise this document to be more clear, and to be more clearly in alignment with the overwhelming weight of sound advice, industry best practices, and with consideration of the dramatic increase in sophistication of automated botnet attacks in the last few years. Recovering from a Trojan Horse or Virus
If the previous step failed to clean your computer, the only available option is to reinstall the operating system. Although this corrective action will also result in the loss of all your programs and files, it is the only way to ensure your computer is free from backdoors and intruder modifications.
Sunday, August 21, 2005
Since August 15th, many organizations have been struggling to recover from the onslaught of the various worms exploiting the MS05-039 Universal Plug and Play (UPnP) buffer overflow exploit. Those unfortunate enough to see large numbers of systems hit by one or more worm variants face the usual challenge of recovering the systems. Microsoft and the AntiVirus Vendors are eager to help you recover your systems, with several offering their own custom cleanup tool to eradicate the worms. Victims of this crop of botnetswould do well to heed the long standing recommendation of information security experts. Recover your contaminated systems by re-imaging them from pristine media, particularly if they were able to contact the outside world even for a few minutes using an IRC control channel. It's often difficult for non-technical management to weigh the risks involved with any given malware outbreak. This week I've heard this sentiment expressed almost exactly the same way from managers in several different organizations:
"It's just a virus, right? I have those on my home PC all the time and nothing bad has ever happened."Well, I'm sorry to be the bearer of the bad news, but that's not the way it is, certainly not any longer. The largest identity theft to date, in which up to 40 million credit card numbers were recently stolen was reported to be due to a "computer virus". That was undoubtedly a pretty bad event for quite a few people. It can take many months, even years, for an innocent individual to recover from problems deriving from the theft of their identity. Far more people than you might think are affected by identity theft, as described in the Federal Trade Commission – Identity Theft Survey Report from two years ago. Experts acknowledge that the problem is getting worse, as large scale automated attacks by worms and botnets are employed to harvest identity data. Worms and bots execute arbitrary code on the zombied systems hosting them. They typically run with Administrator rights and can do anything the computer can do -- and they start doing it within seconds after the systems is exploited. These things are not just hypothetical. Here are a few of the things that zombied PC systems have been observed to perform, at the request of remote attackers, controlling zombied systems from outside the corporate firewall. By the way, these are not alarmist proclamations, rather, they are mundane work-a-day activities of the typical botnet, observed and documented by many independent security consultants.
- contact an IRC channel at a remote location, and receive arbitrary instructions
- update the bot software, install new bot modules
- scan penetrated networks for other vulnerabilities
- probe the vulnerable systems and spread the bots
- perform denial of service attacks on other networks
- harvest (find and upload to remote servers) private, sensitive, secret or classified documents from hard drives
- harvest passwords, user names, and other login information (from the Windows Registry, the Internet Explorer cache, cookies, and text or document files on the system)
- harvest email addresses, contact information
- sniff network traffic to capture passwords and other information
- install rootkits, trojans, keystroke loggers and other malicious software
- use the system to send spam
Thursday, August 18, 2005
Microsoft and the AntiVirus Vendors (perhaps a decent name for a band) tend to think of "threat" in terms of the number of machines infected, how many are vulnerable, and certain other primitive measures of damage done by a worm, such as "does it delete data files". By those measures, this worm appears benign. In fact this current crop of worms is far more harmful than some of the most famous worms from a couple years ago. Rather than hitting many millions of machines, these worms hit only a few hundred thousand or a few million perhaps (infestations inside large corporate and government networks are hard to count from the outside, hiding many infected systems.) When the worms are released, they do the most damage in the first few hours. They immediately search the hard drives for interesting files and upload them to remote servers. This damage is done, to the tune of thousands of files and hundreds of MB of data, before you learn which port to block at your firewall. They steal user identity information, documents, and files that store encrypted passwords so they can be cracked at the convenience of the attacker. They often leave very little in the way of evidence about what they have done. If you get lucky and capture an IRC session used to control these things, you'll understand the true nature of the threat. Many infected systems this week were being actively controlled from outside the corporate firewall by hostile forces. I've recently seen a captured IRC session which includes automated traffic from the zombied bots, as well as conversation traffic between members of a team of human attackers who immediately noticed (and thought it was funny) when the client blocked the IRC port published by the antivirus vendors. We have very little forensic evidence on this, but what we do have indicates that the bots appear to have automatically switched to another port/server combination and nary a beat was skipped. Managers at all levels of corporations and government need to understand that these worms are a very serious threat today. Even though the number of systems infected might be smaller than in previous outbreaks, these worms and bots are dramatically more sophisticated. The security industry needs to come up with better measures of the threat level, which include the risk of data theft, identity theft, and execution of arbitrary command and code on internal systems.
Tuesday, August 16, 2005
Many organizations are struggling with containment of worms and botnet invasions this week, as a result of the MS05-039 vulnerability and myriad variant worms exploiting it (Zotob, Spybot, Esbot, Rxbot, bobax, et. al.) Most of these organizations have patch management, firewalls, IDS and AntiVirus systems in place as part of a layered defense. They may suffer dozens or hundreds of compromised systems regardless of these efforts at prevention. The current crop of worms are nearly all bots -- remote controlled software agents that call out of your network to a remote server, looking for instructions from an attacker. Containing the outbreaks can dramatically reduce the cost of the later cleanup. If you have a large network, with a large population of vulnerable machines, and you don't have a containment strategy in place, consider the following tactics.
Network PartitioningConsider partitioning your internal network on the ports used by the worm to spread. This worm seems to favor port 445, but some variants also employ port 139. Block these inbound at VPN and dial-up access points. Consider creating zones within your enterprise that are partitioned on these ports, at least until you get all your systems patched. If an outbreak occurs, the damage can be contained within a zone.
Egress FilteringDon't wait for the AntiVirus vendors to capture and analyze all the variants to determine what ports to block. Start by blocking all of the standard IRC ports if you don't have a critical business need for IRC (most organizations don't). The standard IRC ports are used surprisingly often for botnet control, because they can sometimes be set up on existing IRC servers with relative ease. Although there is a small chance that someone in your organization might be using IRC for legitimate purposes, consider directing them to use a more modern Instant Message protocol, like AIM, Yahoo IM, MSN IM, Jabber/XMPP, etc. Standard IRC ports should be blocked indefinitely. There are other ports associated with IRC, registered with the IANA , but they don't seem to be in use for botnet control at this time. We might need to expand this list at a later time. Note also that several of these ports are identified as commonly used by IRC servers, but not registered with the IANA, they currently show as "unassigned"). 6660/tcp 6661/tcp 6662/tcp 6663/tcp 6664/tcp 6665/tcp 6666/tcp 6667/tcp 6668/tcp 6669/tcp 7000/tcp Also, in your perimeter routers, block and log the IRC ports used by the known variants, as documented by the various AntiVirus Vendors. During an outbreak, don't wait for a variant to hit your network. Block and log the IRC ports as soon as the variant is documented. Have someone one your team assigned to review the emergent documentation from three or four major AntiVirus vendor web sites, and update your perimeter egress filtering rules at least twice a day during an outbreak.
Disable NULL sessionsIf you have a software distribution system in place, and if you haven't done it already, consider disabling NULL sessions on the Windows systems which haven't been patched yet. This can be accomplished with a tiny package and distributed much more quickly than large system patches. It appears that this will prevent an infection of an unpatched Windows 2000 system, allowing you more time to patch. Microsoft declined to confirm or deny that the UPnP vulnerability could be exploited with NULL sessions disabled, but apparently the current crop of worms and bots all rely on NULL session to perform the exploit.
Smart BombsPractice adroit and vigilant application of cleanup tools as part of your containment strategy, but not recovery (apologies to George F. Kennan). Cleanup tools can be deployed to contaminated systems to kill and delete the probing worm process which is spreading through buffer overflow exploits. If you can test and deploy it quickly enough, such tools can be part of a layered defense -- even if they are the "last line" of that defense. Focus testing on system compatibility -- will it accidentally wreck something on the system, making recovery harder? Probably not, but it's a good idea to check it out in your test lab. Don't squander precious time during the early phases of an outbreak by trying to validate that a cleanup tool kills every variant on your network. Deploy it only to systems that are contaminated and probing other systems. This is an attempt to slow the spread of the worm -- remember, you're engaged in containment, not recovery. If you have good reason to believe it will kill some of the variants on your network, send it out to contaminated systems after basic compatibility testing has been performed against your system image baseline. As follow-up, you can focus your limited forensics resources on the systems that continue to spew worm traffic, despite the cleanup tool, and return with an improved version to taunt the silly worm a second time.
Intrusion SuppressionOur FireBreak AntiWorm can help you identify those infected systems within moments of the start of an outbreak, and with an extremely low false positive rate (no false positives at all on a typical network). FireBreak AntiWorm can also significantly impede the progress of zero day worms, allowing you more time to respond. The system is appliance based and dramatically simpler than traditional IDS systems. Our AntiWorm solution can be deployed very quickly, so if you're having trouble putting the lid on the worm this week, don't wait -- call us today.
Monday, August 15, 2005
Zotob reared its slimy head this morning. It exploits a defect in Windows systems (UPnP MS05-039) for which a patch has been available less than a week. Zotob is undoubtedly the first of what will be many Week Zero Worms exploiting this defect -- not quite a Zero Day worm, but close enough to wreck havoc. Every time this happens, the internet discussion forums are flooded with snide comments from smug systems administrators, along these paraphrased lines:
"I patched all 652 of my systems this week before the worm hit. Any organization being hit by this worm is incompetent."Well, probably not. These well-run one-man shops do impress with their ability to deploy patches quickly and offer some hope for the rest of the universe. However, the prima donna types that make it happen generally don't really understand the magnitude of the problem in a large corporation with, say, 50,000 TCP/IP devices, mostly running Windows. It's not just a matter of patching 77 times as many systems in that same week. The unstable tower of complex software architectures built up on top of the typical network of Windows systems in a large enterprise makes it quite a bit more difficult to plan and execute a system upgrade or a configuration change or even an operating system patch in a larger environment. Explaining this to management in large organizations isn't very hard. Getting them to agree to do something to fix the underlying problems, however, is almost impossible. The people in charge of keeping the engines running are not the same people in charge of all the complicated attachments that get connected to them. All of these arbitrary "business drivers" may be carefully considered by IT people, who conclude that they need to meet the needs of the "customer" (e.g. another business unit, which is often a profit center carrying clout with Senior Management) and concede to the complicated attachments. These other business units are often engaged, sometimes knowingly, in a game of externalized cost. They may buy a software system that must be deployed to every desktop, rather than one that users can access from a web server. Worse yet, they may build one, without divining the best practices which help prevent high-maintenance software architectures. An increasing burden builds up on the IT staff over time. Most of this stuff is extraordinarily difficult to measure. But these costs don't go away. They come back to bite. Other times support issues arise within the IT organization itself, and a clever solution is devised. Often entirely too clever. Unfortunately, this "can do" attitude of most IT shops is sometimes their undoing. Clever solutions interwoven through the layers of the distributed systems and the various creaky but mandated optional components combine to make an overall system architecture which is relatively brittle. Then a worm hits. In a panic, patches are applied, things break, and the mess is cleaned up later. A post-mortem is performed. In the post-crisis exhaustion, the IT organization struggles to put the pieces back together and move forward on the latest set of tasks from the latest set of business drivers. In the standard ongoing chaos, the recommendations are ignored. A few weeks later, another defect, another worm, another crisis which possibly could have been averted in a better world. It's a nasty vicious cycle, but it's definitely related to the sheer size of an organization and its network. So please, all you smug fully patched systems administrators, don't be so hard on your collegues who didn't get 50,000 PCs patched in the same week that you patched 700. This worm gave you several days to patch them, and it took you more than a day. The next worm could hit before the patch is available, and it could be you turning to the forums for advice on how to impede the spread of the worm on your network, contain the damage, and recover your systems. When your number comes up, these folks will have unfortunate experience that you might be able to draw upon.