Posted by & filed under Security, Windows / Server.

February’s Patch Tuesday was eventful to say the least. Many have noticed that Patch #977165 (Security Bulletin MS10-015) causes a blue screen on some systems (Stop Error: PAGE_FAULT_IN_NONPAGED_AREA). While the initial outrage was directed at Microsoft for a shoddy patch, eventually it was found that the Alureon Rootkit was the cause of the blue screen after KB977165 was installed.

But don’t worry! – the makers of the Alureon Rootkit have actually updated it and patched the flaw! Hurray!

For everyone else there are two options:

  1. Use a LiveCD to scan your hard drive for the rootkit and remove it. This will resolve the issue. Try Knoppix STD (http://www.knoppix-std.org/) or BartPE (http://www.nu2.nu/pebuilder/).
  2. Remove MS10-015 (977165) from your system.

How to remove Security Bulletin MS10-015 (977165) from your system

  1. Insert the Windows XP disc into the CDROM. You may need to adjust your BIOS settings to boot the CDROM first.
  2. When the “Welcome to Setup” screen appears, press “R.”
  3. Select the installation you wish to access (there should be only one option for most systems).
  4. Enter the administrator password when asked.
  5. Once at the Recovery Prompt, press ENTER after typing the following command: CHDIR $NtUninstallKB977165$\spuninst
  6. Press ENTER after typing the following:  BATCH spuninst.txt
  7. Press ENTER after typing the following:  systemroot
  8. Press ENTER after typing the following:  exit
  9. Remove the Windows XP CD and restart.

Posted by & filed under General, Hardware, Windows / Server.

Backups are a necessary evil in System Administration, and although most of us dislike the process, it is by far the most critical element under the IT umbrella. I like to think of the whole process as a Recovery Plan instead of a backup plan, because in the end all I care about is that the data is recovered properly and quickly. One of the biggest pitfalls new System Administrators or System Administrators new to a particular company do, is that they do not test their own backups. Not being able to recover information from a system you designed or recommended is the quickest and surest way to get fired. 

 

1. Risk Assessment

Although this sounds simple, a true risk assessment is rarely done and far out of the reach of the average business. Although hiring actuaries and combing through insurance statistics is ideal, this is far from what companies are willing to do for a data recovery plan. Many System Administrators find companies that have not experienced data loss are less willing to be thorough in their analysis and budgeting.

One of the first things in a recovery plan is to write down the possible external risks, some examples:

  1. Fire
  2. Flood
  3. Earthquake
  4. Tornado
  5. Physical Break-in / Theft
  6. Virtual Break-in (Cracker)

Ask your Insurance Company what some likely issues will be, they will be happy to tell you every possible disaster scenario.

Next, think of internal risks such as:

  1. Viruses/Malware
  2. Data Corruption
  3. Hardware Failure (Electrical or Mechanical)
  4. User Error (accidental deletion)
  5. User Malice (non-accidental deletion)

To supplement risks, look through your company’s history for any previous data loss and the reasons for it. Enlist the help of your industry colleagues for any scenarios not on the list. You’ll be amazed at some of the “once-in-a-lifetime” stories you’ll hear – some may be applicable to you.

 

2. Impact Rating

An impact analysis on each scenario listed above should be created. This involves the hardware, software and data (all systems) that are affected and how. Does the impact involve a full or partial outage? If hardware is likely damaged, how quickly can it be replaced? Etc. Suppose there is a fire in a server room and the servers are damaged. You have the LTO tapes as backups, but no server, and no LTO drive to restore it with. How many days will it take to get an LTO drive and from where? During this phase of planning a vendor and consultant list should be compiled.

 

3. Risk Rating

This can be included in the budgetary section, or done beforehand. Combine the risk assessment items and impact ratings and sort them. This is important. You should implement a recovery plan that encompasses as many items as possible. By sorting, it makes the budgetary step easier when you need to cut coverage because of costs. 

 

4. Methods

There are many methods to ensure data continuity in certain scenarios, but be careful as there is rarely a one-size-fits-all approach to backups.

Mirroring: Is designed to mitigate single-point hardware failures. If your database server fails, having a mirrored server will ensure your data is available. Mirroring at another location may also solve router, switch and connection issues for external clients. Mirroring does not protect against corruption, viruses, user error or malice. On-site mirroring does not protect against theft, fire, flood, etc.

Removable Backup Media: Backups to media protect against issues such as corruption, viruses, user error or malice. If you leave these items on-site they will not protect against theft, fire and flood. If they are taken off-site, it will take longer to retrieve your data in case of loss. With backup tapes, hard drives and cds, the backup data itself is typically a day or older. If this method of backing up is your only method, be sure your business can survive with older data. Be sure to have multiple days of backups, or a weekly backup with incremental backups per day. Often times users will not report data loss until days after the event, by which time relevant backups have already been overwritten with newer, useless backups.

Non-removable Backup Media: Items such as NAS (Network Attached Storage), DAS (Direct Attached Storage) or SAN (Storage Area Network) can be used to backup servers, virtual machines and data. The issue with these is that they are not removable. This will not protect against theft, fire, flood, etc.

Be careful of proprietary systems used to backup your data. Be sure to audit your recovery scenario regularly to ensure your backups can be recovered. Companies go out of business, and items are discontinued. Do you have any backups on jazz drives? How difficult would it be to recover if you had to find a new jazz drive? Don’t know what a jazz drive is? Exactly!

 

5. Budgetary Concerns

With your sorted list in hand, you can now plan for the items you need to mitigate any disasters. Protecting against many scenarios may prove to be prohibitive in cost. If you do not make the budgetary decisions, be sure that your list is as comprehensive as possible. It is up to IT to determine the impact of all scenarios, and it is up to the budgetary members to determine how much they want to spend. If they say no, you have at least outlined all the possibilities.

In your cost analysis, include replacement or redundancy items such as:

  1. Backup Storage (Tapes & Tape Drives, Hard Drives, CDs)
  2. Backup Servers, NAS, SAN, DAS
  3. Mirrored Servers
  4. Redundant Connections (Internet and Cabling)
  5. Backup Routers, Switches, etc

Part of your impact analysis should include what is damaged or lost. If you have the tapes, but no tape drive, you will need to replace it in order to retrieve the data. Make sure you have the ability to read your backups when you need. If it takes 3 days to ship a new tape drive, but the cost is minimal, consider having a backup tape drive in stock.

While they are numbers out there for determining how much to spend on a backup and recovery system, you should make your decisions based on the impact and risk. If your data is your business’ main asset, you should spend a larger chunk of your budget to protect it. If time is critical in retrieving your data, the solution may include keeping extra hard drives, servers, router and switches in stock. If time is not an issue and an outage can be handled for days, you can order items at the time of recovery.

Determining budget can be a mix of preventative costs and the cost of downtime to the business (lost sales, lost productivity). Ideally disaster scenarios should have a cost to the business attached to them. If a server failure results in $0 productivity for the day, the overall impact can be many thousands of dollars per day – that fact alone may convince management to have a redundant server available.

 

6. Deployment and Testing

Don’t forget this step! Backups are useless unless they can be recovered. Take a weekend to simulate a likely recovery scenario. You may be surprised at all the “gotchas” when recovering data. Common stumbling blocks include not backing up database logs that are critical to recovery (ex. Exchange Server), or recovering to dissimilar hardware (Ex. RAID5 on a different controller).

Posted by & filed under General, MySQL, PHP.

Many content management systems and forums were originally created using the latin1 character set and the latin1_swedish_ci collation in MySQL. The problem many of these systems are facing today is the growing demand for multi-language content using special characters that cannot be accurately represented in the latin1 character set. This is where utf8 comes in.

The problem with simply converting a database from latin1 to utf8 is in the data itself. When you convert a database or table to a different character set or collation, it does not convert the content held within the tables. What happens is users end up with ‘strange’ characters in their data.

strange_a_e latin1_convert_to_utf8

You can prevent this by taking steps to protect the data before you convert the database or tables.

An overview of the steps:

  1. Backup the database. No, really, do it.
  2. Alter the string field types to their binary equivalents in all tables. This will protect the data during conversion.
    • VARCHAR to VARBINARY
    • CHAR to BINARY
    • LONGTEXT to LONGBLOB
    • MEDIUMTEXT to MEDIUMBLOB
    • TEXT to BLOB
    • TINYTEXT to TINYBLOB
  3. Convert the database and tables from latin1 to utf8 character set and latin1_swedish_ci to utf8_general_ci collation.
  4. Convert the binary field types back to their original string field types.
  5. Backup the resulting database.
  6. Finished

Ex. Generic convert column to BLOB from TEXT

ALTER TABLE tbl_name MODIFY column_name BLOB

Ex. Generic convert table to new character set

ALTER TABLE tbl_name CHARACTER SET charset_name COLLATE collation_name

Ex. Generic convert database to new character set utf8

ALTER DATABASE database_name CHARACTER SET utf8;

Posted by & filed under General, Security.

Some time ago a co-worker had mentioned someone had been rummaging through her files on her computer. She had expressed some concern over the situation as the files in question were pertaining to a terminated employee. I nodded in sympathy and asked her the following questions:

  • When did this happen?
    Her response: After she had left work for the evening, but before she came in the next day.
  • Did you log-off of your computer in the evening?
    Her response: No.
  • Did you lock your office in the evening?
    Her response: No.
  • Is there anyone currently employed that would have an interest in those files?
    Her response: Yes.
  • Did you tell the Boss?
    Her response: No.

Because the office didn’t have any form of employee tracking, we could not find out who was in the building, let alone who accessed the files. While management was trying find out who did it, I was focusing more on the measures that could have been taken to prevent it. The worker had not logged off her machine or locked her office in the evening. As with all things in IT, the typical process response is always reactive instead of proactive. If the managers had taken my concerns seriously with regards to physical and virtual security months before, the situation would not have happened. This example was the catalyst I needed to affect a change in policy regarding passwords, automatic log-off, and certain aspects of physical security.

balancing-it-security-venn

People, Process, Technology

Everyone has seen the People, Process, Technology Venn diagrams prevalent in business literature. I believe the most effective security practices involve a balance of all three categories to succeed – the process has to be sound, the technology relevant, and the people informed. Relying on any one of these categories too much will surely result in failure. No matter how locked-down a server is, if someone writes their password on a post-it note on the monitor, it is no longer secure. If there is no process in place to direct the people or the technology on the correct actions to take to be secure, it will fail.

Social Engineering

It seems the most vulnerable aspect of security lies in people’s tendency to succumb to social engineering tricks. Medium-sized companies are especially vulnerable as they may not have the means to implement physical security and also lack the close-knit employee base to detect outsiders easily. How easy do you think it would be to walk into a medium-sized company with a Canon shirt and convince them you are there to fix the copier?

I’d like to hear your own experiences with security, including what you think are the most important factors in creating a successful security policy.

Posted by & filed under Security, Windows / Server.

Last night Malwarebyte’s AntiMalware program detected a false positive of the Atapi.sys driver and associated registry keys. As you may know, Atapi.sys is required by the storage system in Windows, and as such deleting it will render the system unbootable.

If you have the reboot on error checked in your system properties, your system will continuously reboot itself without giving an error. If you have a stop on reboot option checked, you will see the STOP error 0x0000007B.

I have included a zip file with the following registry keys and atapi.sys (5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)) taken from a fresh install of Windows XP SP2. Apparently it only affects SP2 installations.

Registry Keys Infected:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\atapi
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\atapi
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\atapi

Files Infected:
C:\WINDOWS\system32\drivers\atapi.sys

A few ways to fix this:

Use another machine to load Atapi.Sys and Registry Keys

  1. Install the hard drive into another Windows computer and copy the computer’s good Atapi.sys driver (Windows\System32\Drivers) to your hard drive. Or download the XP SP2 one here.
  2. Put the hard drive back into the original computer and select “Last known good configuration” to boot – this will restore the registry keys.
  3. If the “Last known good configuration” doesn’t work, you may try editing the registry hive of your installation via another computer. Put the hard drive into a second machine and load the hive within that machine’s registry editor. If you are uncomfortable doing this, create a LiveCD of Windows (below).

Make a BartPE or LiveCD of Windows

  1. Go to http://www.ubcd4win.com/howto.htm and follow the instructions to make a LiveCD of Windows. You can also use BartPE, but the following instructions may be different (http://www.nu2.nu/pebuilder/).
  2. From a known good machine, export the above 3 registry keys to a USB drive.
  3. From a known good machine, copy the atapi.sys driver from Windows\System32\Drivers to a USB drive. Or download the XP SP2 keys and atapi.sys here (zip).
  4. When the LiveCD loads (this will take a while), attach the USB key to the machine. Copy the atapi.sys to your machine’s Windows\System32\Drivers directory.
  5. In the LiveCD’s Windows environment, go to: Start>Program Files>Registry Editors>Regedit (remote).
  6. You will be prompted to select a user from your machine to edit. Most likely it is “Administrator.”
  7. Go to File and Import in the registry editor.
  8. Import each of the 3 .reg keys you exported from a known good machine.
  9. Restart your computer, taking out the LiveCD.
  10. Everything should work.

Use Windows Repair

I don’t like this option because it does not always work. You may also need to reinstall or fix some programs after this procedure.

  1. Put the Windows XP disc into the machine.
  2. When the machine boots into the Setup environment, it will give you the following options:
    To setup Windows XP now, press ENTER.
    To repair a Windows XP installation using Recovery Console, press R.
  3. Press ENTER, not R.
  4. On the next screen, it will detect a previous installation and ask if you want to repair it. Choose to do so.
  5. Windows will go through the setup by reinstalling all default options and drivers. You will need your Windows XP key.