Facebook Scalability Story

Here's a synopsis of a presentation given as part of the CNS lecture series, discussing Facebook's architecture.

Even though it is just an overview, it is pretty detailed, and gives interesting little factoids such as:

• 600k photos/sec
• php for the front end
• 30K servers, in two datacenters
• ”Work fast and don’t be afraid to break things.”

There's lots of neat stuff in there.

Listing User Last Logon with Powershell

This script assumes that you have enabled auditing on successful logins (by default, it doesn't).

The general process it follows is:

• retrieve the Security event log
• pulls login information up to the last reboot
• gets unique usernames and the time they logged in
• writes it all out to a text file

It's still a little raw, but it works. It runs very slow over the network; I'll work up one that uses the PSJob facilities...

function getLastBoot( $computername )
{
 $wmi = Get-WmiObject -Class Win32_OperatingSystem
 return $wmi.ConvertToDateTime( $wmi.LastBootUpTime )
}

function getTopDates()
{
 $logins = @()
 $input | foreach {
  $rec = $_
  
  # this is ugly...I'm going thru the list twice
  $hasit = ($logins | where {($_.UserName -eq $rec.UserName) -and ($_.MachineName -eq $rec.MachineName)})
  
  if( $hasit )
  {
   for( $x = 0; $x -lt $logins.Count; $x++ )
   {
    if(($rec.UserName -eq $logins[ $x ].UserName ) -and
    ($rec.MachineName -eq $logins[ $x ].MachineName ))
    {
     $logins[ $x ] = $rec
    }
   }
  }
  else
  {
   $logins = $logins + @(,$rec)
  }
 }
 return $logins
}

$start_time = (Get-Date)
Write-Host "starting all $start_time"

$target_computers = @( "dal1mspwb16",
      "dal1mspwb19",
      "dal1mspwb36",
      "dal1mspwb37",
      "dal1mspwb12",
      "dal1mspwb35")

# $target_computers = @( "dal1msdwb34" )
      
$target_computers | foreach {
 $target = $_
 $lastboot = getLastBoot( $target ) 

 Remove-Item "iis_logins_$target.txt" -ErrorAction SilentlyContinue


 Write-Host "processing $target :" (get-date)
 Get-EventLog -LogName "Security" -ComputerName $target -After $lastboot | 
 select -Property UserName, MachineName, TimeGenerated -Unique |
 sort -Property TimeGenerated |
 getTopDates |
 Out-File -Append -FilePath "iis_logins_$target.txt"
 
 Write-Host "completed $target :" (get-date)
} 

$end_time = (Get-Date)

Write-Host "complete"
Write-Host "Started: $start_time"
Write-Host "Finished: $end_time"
Write-Host ($end_time - $start_time)

Creating System Restore Points with Powershell

Getting ready to install a new video driver? What about that "interesting" piece of software you found?

Aren't you worried that it is going to screw up your computer?

Well, if so, Windows XP/Vista/7 have a facility known as "System Restore Points". Basically, these are snapshots of your filesystem. There is a good chance they are already enabled, and being used by Windows Update.

What about those other times, though?

If you have Powershell 2.0 installed (whaddya mean, you don't? Get on it!), then you have a couple of commands to help you out:

Checkpoint-computer creates a system restore point.

Restore-computer reverts to the specified restore point

So, before you install that problematic driver/update/app, this is a quick and dirty way to cover your butt.

Be warned, though, this will restore everything - including any changes to any files you may have made.

You Don't Know Jack About Software Maintenance

Communications of the ACM has this article up on their site.

Software maintenance involves moving an item away from its original state. It encompasses all activities associated with the process of changing software. That includes everything associated with "bug fixes," functional and performance enhancements, providing backward compatibility, updating its algorithm, covering up hardware errors, creating user-interface access methods, and other cosmetic changes.

In software, adding a six-lane automobile expressway to a railroad bridge is considered maintenance—and it would be particularly valuable if you could do it without stopping the train traffic.

The article asserts that this can be managed, because it has been managed in the past. However, it is pretty weak on the "how" - only that it can be done.

Powershell v1.0, IIS6, and remote machines

I found this snippet on the web:

$computer="server" 
$co = new-object System.Management.ConnectionOptions 
#$co.Username="domain\username" 
#$co.Password="password" 
$co.Authentication=[System.Management.AuthenticationLevel]::PacketPrivacy 
#$co.EnablePrivileges=$true; 
$wmi = New-Object System.Management.ManagementObjectSearcher 
$wmi.Query="Select * From IIsApplicationPool" 
$wmi.Scope.Path="\\$computer\root\MicrosoftIISv2" 
$wmi.Scope.Options=$co 
$wmi.Get() | foreach { $_.name } 
In PowerShell v2.0 there is a new parameter, -Authentication,  to specify 
the authentication level (one line): 
gwmi -class IIsApplicationPool -namespace "root\MicrosoftIISv2" -computer 
$computer -authentication PacketPrivacy |  foreach { $_.name} 

Live failover for Xen images

Remus promises to bring to the Xen hypervisor live failovers. In other words, if a host system crashes, another machine picks up the load without any interruption to services.

From the Remus website:

Remus provides transparent, comprehensive high availability to ordinary virtual machines running on the Xen virtual machine monitor. It does this by maintaining a completely up-to-date copy of a running VM on a backup server, which automatically activates if the primary server fails. Key features:

• The backup VM is an exact copy of the primary VM. When failure happens, it continues running on the backup host as if failure had never occurred.


• The backup is completely up-to-date. Even active TCP sessions are maintained without interruption.


• Protection is transparent. Existing guests can be protected without modifying them in any way.

This is neat because the only thing that I know of that provided this functionality is VMWare, which costs big bucks (and still holds the tools advantage). Xen is free, and available for Linux and OpenSolaris.

Using RUNAS for SQL Management Studio

Ran across this today, and just don't want to lose it.

The short version: if you need to connect to a Windows SQL server in a different domain, the runas command has a /netonly switch.

runas /netonly /user:domain\username “C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\Ssms.exe”

Neat, huh?

Pash - PowerShell for Unix

Regular readers of this blog - both of you (Hi Mom, and that guy in Australia who subscribed to the RSS) - know that I loves me some MS PowerShell. I've called it a "game changer", because it greatly simplifies Windows administration. It is good enough that I've abandoned cygwin on my Windows systems; PowerShell is better than bash.

Well, somebody has released a version of PowerShell that runs on *nix systems: Pash. It builds on Mono, so there is a huge library of objects for it to work on. I don't know how well it works, yet, but I'm certainly going to be trying it.

IT, Users, and Communication

I was going to let this article slide, and not get all meta-bloggy about it, but the rebuttal really tweaked me. It's all about what users get to install on their work machines, and IT's reaction.

They both miss the point, I think.

Mr. Manjoo related a story about Firefox, and the crowd cheered. If there was really that much demand for it, then it was a failure on the IT department's part to know that it was wanted, and if they knew that, not at least acknowledging it clearly. There's plenty of good reasons not to upgrade.

What Mr. Manjoo missed is that there are tradeoffs to the freedom to install whatever you want, most of them related to support. A lot of IT policy is driven by how much they have to provide that support. Less money means coarser support - heavily locked down machines, aggressive re-imaging, or similar. Things that don't require a lot of people time.

The confirmation bias that both articles triggered in me, though, was that it clearly showed that in neither case is the IT department and the users communicating.

Good IT is hard, not just because of the technology involved, but because you have to make long term decisions which will permit you to react to users ever-changing needs and wants.

Remember, we're here for them, not the other way around. When I walk into a shop that doesn't live that attitude, I know I'll find a lot of problems.

Svnserve, and Solaris 10

I had to go through the trouble of getting svnserve to run as an SMF-managed service on Solaris 10, so there's no reason you should, too.

Create the method script.

This script uses rc-like syntax. The xml manifest (coming up!) uses this.

vi /lib/svc/method/svc-svnserve

The contents:

#!/sbin/sh

case $1 in
 start)
     svnserve -r /var/svnroot -d ;;
 stop)
     /usr/bin/pkill -x -u 0 svnserve  ;;
 *)
     echo Usage is $0 { start | stop }
     exit 1 ;;
esac

exit 0

Fix the permissions:

chmod 555 /lib/svc/method/svc-svnserve
chown root:bin /lib/svc/method/svc-svnserve

Test it with:

sh /lib/svc/method/svc-svnserve start

Try to connect, list, etc., make sure it works the way you want it to.

Create the SMF manifest

vi /var/svc/manifest/site/svnserve.xml

The manifest, itself

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='SUNWsvn:svnserve'>
       <service
               name='site/svnserve'
               type='service'
               version='1'>
               <single_instance/>
               <dependency
                       name='loopback'
                       grouping='require_all'
                       restart_on='error'
                       type='service'>
                       <service_fmri value='svc:/network/loopback:default'/>
               </dependency>

               <exec_method
                       type='method'
                       name='start'
                       exec='/lib/svc/method/svc-svnserve start'
                       timeout_seconds='30' />
               <exec_method
                       type='method'
                       name='stop'
                       exec='/lib/svc/method/svc-svnserve stop'
                       timeout_seconds='30' />
               <property_group name='startd' type='framework'>
                       <propval name='duration' type='astring' value='contract'/>
               </property_group>
               <instance name='default' enabled='true' />
               <stability value='Unstable' />
               <template>
               <common_name>
               <loctext xml:lang='C'>
               New service
               </loctext>
               </common_name>
               </template>
       </service>
</service_bundle>

Check your work

Check the xml with:

xmllint --valid /var/svc/manifest/site/svnserve.xml

Then let's see if the smf stuff likes it:

svccfg validate /var/svc/manifest/site/svnserve.xml

If everything looks good so far...

Importing the manifest

svccfg import /var/svc/manifest/site/svnserve.xml

It should show up under svcs in maintenance. Let's fix that:

svcadm enable svnserve:default

If it doesn't start, check /var/svc/log/site-svnserve:default.log

You should be all nicely integrated now.

Redmine on Glassfish

Okay, this took me a couple of days to piece together, so here it is for posterity.

Redmine is a software-development project management system, written in Ruby on Rails. The demo looks good, I tried it out on a small scale, liked it, and am ready to really use it.

I like my RoR running under Java where I can keep an eye on it, and normally this isn't a problem. Just warble it up, and deploy. Nothing to it.

However, this time around was a little more interesting.

Get everything working per these instructions, running Webrick, etc., under jruby instead of ruby.

The .jar that comes with the latest stable JRuby (as of this writing) has a bug. There's a newer one here which works. It should be replace whatever jruby-completes are in jruby-1.3.0\lib\ruby\gems\1.8\gems\warbler-0.9.13\lib. Thanks to this little post for that fix.

Next, run warble config, and modify the newly-created config/warble.rb. Make whatever other changes you might want (like including your jdbc driver in config.gems), and add lang to config.dirs.

Then warble it up, and away you go.

Also, it really helps if you don't have some weird file corruption issue to help you misdiagnose things, spin your wheels, and get needlessly frustrated for a day and a half. Really, try to avoid that part. Just try it on one of the other ten zillion machines you've got laying around a little sooner, dumbass.

Update

If you happen to be using redmine not only in this configuration, but behind nginx as a proxy, here's a trick to get around the non-relative paths you find all over the place:

        
server {
   listen 80;
   server_name whatever.example.com;
   location / {
      proxy_pass http://server/redmine;
   }
   location /redmine {
      proxy_pass http://server/redmine;
   }

Hope this helps.

Powershell, and log4net

EZ money:

"--- Loading log4net ---"

[System.Reflection.Assembly]::LoadFrom("$pslib\log4net.dll") | out-null;
"Loaded log4net.dll"

[System.IO.FileInfo]$fi = new-object System.IO.FileInfo "$pslib\log4net.xml"
[log4net.Config.XmlConfigurator]::Configure( $fi )
"log4net configured with $pslib\log4net.xml"

$log = [log4net.LogManager]::getLogger("default")
"`$log = (new logger `"default`")"

""

'Cause sometimes we wanna be fancy...

ZFS and Upgrading Drives

Yea, okay, so this is common. In fact, I've done it with VMs before. I still think it is neat that I was able to upgrade an array without any downtime.

Before:

root@bluelight:~# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data    888G   511G   377G    57%  DEGRADED  -
rpool   298G  6.28G   292G     2%  ONLINE  -

After:

root@bluelight:~# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data   2.73T   511G  2.23T    18%  ONLINE  -
rpool   298G  6.27G   292G     2%  ONLINE  -

I ran into one snag. When I issued zpool replace data c4d0s0, it gave me a message about there being no such device. The solution was zpool replace data c4d0. I suspect that "s0" at the end of the original disk was a slice, and probably an artifact of how I set the system up to begin with.

The larger impact is that I've got a lot of room at very little cost. There aren't as many advantages to the high-end gear that there used to be. Everything I used is commodity class, or free. It doesn't suit all situations, but there isn't a reason for small businesses to be paying a lot of money.

Powershell $profile - Is This Great Filler, or What?

Some new java stuff, a couple of nice little helper functions: qfind, for searching the filesystem and title, to change the console title.

$projects = $env:USERPROFILE + "\Documents\Visual Studio 2008\Projects"
$sysdir = $env:USERPROFILE + "\sys"

# misc stuff
new-alias -name npp -value "C:\Program Files\Notepad++\notepad++.exe"
function qfind( $start, $like ) { get-childitem $start -name $like -recurse }

function title( $msg ) { $host.ui.RawUI.WindowTitle = $msg }

# git stuff

$gitexe = $sysdir + "\Git\bin\git.exe"
new-alias -name vi -value $sysdir\Vim\vim72\gvim.exe
$env:EDITOR = "npp"
new-alias -Name git -Value $gitexe 

# java stuff

$jdk = "jdk1.6.0_12"
$java_home = "C:\Program Files\Java\$jdk"
$env:JAVA_HOME = $java_home

new-alias -name jruby -value $sysdir\jruby\bin\jruby.bat

new-alias -name jar -value "$java_home\bin\jar.exe"
new-alias -name ant -value "c:\sys\ant\bin\ant.bat"
new-alias -name javac -value "$java_home\bin\javac.exe"
function jetty() { java -jar start.jar etc/jetty.xml }
$env:CATALINA_HOME = "c:\sys\tomcat6"
new-alias -name tomcat -value "c:\sys\tomcat6\bin\startup.bat"

# MS Stuff

new-alias -name nant -value "$sysdir\nant\bin\nant.exe"
new-alias -name msbuild -value C:\Windows\Microsoft.NET\Framework\v2.0.50727\msbuild.exe

title( "General" )

Hope this helps. At least it is harder to lose :)

Update: Once I got to looking at it here, I decided it could be better laid out. So, it is a little prettier, now.

The State of Hypervisors: Meh

I've been dinking around with the various virtualization technologies: VMWare ESX, Solaris xvm, MS Hyper-V, and Ubuntu/Linux kvm (I still have VirtualBox on the todo list).

kvm and xvm, despite different underlying technologies, are pretty similar. They get the job done, but they aren't much fun to manage. The various crowds are making it better, everyday (eucalyptus, which uses Xen on Linux, is especially promising, but personally untested).

VMWare wins the management crown, hands down. Highly complex configurations, an abundance of tools, simple administration, and well-known in the industry.

They need to be doing a lot more, though. MS' Hyper-V 2.0 promises to have live migrations (a big selling point for VMWare), and is nearly as easy to manage. It won't take long for MS to catch up, and it is almost free with Windows.

Here's something that kills me for all of them, though. It can be difficult to move VMs from one version of the software to another version, which kills flexibility. If I want to run the VM on my laptop on a big fat server for awhile, moving it from the workstation version to the server was not simple.

It should be simple. I realize that if I wanted to do something similar on a large scale, I could do it by implementing some conventions and standards, with a bunch of scripts to hold it together. I'm lazy. That's the point here.

My Hypervisor Grail is this: a seamless transition from any hardware, to any hardware. At the moment, I'm whining about moving from my laptop to a server, but I'd like to be able to move it out to a cloud of some flavor also (right now, I'm using AWS EC2, and it is sufficient).

It would be super-cool if I could choose among images at boot, and have it backed up at shutdown, without the compromise of hard drive space and/or performance.

We're getting there, at least:

Xen 3.3 also contains a wealth of new features contributed by vendors collaborating in the new Xen Client Initiative (XCI), a Xen.org community effort to accelerate and coordinate the development of fast, free, compatible embedded Xen hypervisors for laptops, PCs and PDAs. The XCI is targeting three use cases: using Xen to run "embedded IT" VMs that allow remote support, security and service of PCs through embedded IT applications without any impact on the user's primary desktop OS;
"instant on" applications that can be immediately available as separate VMs from the user's primary desktop OS; and "application compatibility" VMs, which allow legacy PC applications to run as VMs, alongside the user's primary desktop OS. XCI member companies are already shipping Xen client hypervisors embedded in chipsets, PCs and laptops.

I think, for my next build-out, I'm going to try Eucalyptus on a server (sacrificing the Win2008 machine - good OS, but not as good as Solaris, which stays and gets big fat hard drives). I like the idea of being able to push VMs from inside the datacenter (okay, in this case, my closet) to the cloud.

MSBuild, and Cancelling Long Running Processes

I really like the syntax and organization associated with the various build tools for getting stuff done. I'd hoped to leverage those qualities to handle some system tasks - and I will, anyway.

I ran into a snag, though. If a task involves a long-running external process - say, robocopy'ing a large directory structure, killing the msbuild process doesn't kill the child process. Robocopy keeps on running, leading to a lot of cursing about how slow the network is until you notice it running two dozen times.

BTW, here's a powershell snippet to kill all robocopy processes:

get-process -Name robocopy | foreach-object { $_.Kill() }

Under normal circumstances, the runaway processes shouldn't be a problem. Under other circumstances, this could be a big deal.

MSBuild exposes its object model. It should be possible to either write a wrapper, or a better exec task. One way or the other, it isn't as cut and dried as I'd hoped.

SocketShifter - Redirect Sockets on Windows

A trick I've often used to get around firewalls is ssh port forwarding. Basically, after SSH connects to a server, you can get it to forward local ports through the tunnel to arbitrary ports on the other end.

So, if I connect from work to my ssh server listening on my home router, I can forward my local port 1000 (which isn't being used) to port 80 on one of my internal machines, and connect to that webserver via http://localhost:1000.

Windows servers don't have many good options for an ssh server. The only one I've used was the version that comes with Cygwin, and it wasn't really all that good (hangs, crashes, non-killable).

Introducing SocketShifter, an app that performs a similar magic, only on Windows.

But wait, there's more!

It uses something called the .NET Service Bus, an offering from Microsoft's Azure. This promises to allow two machines to connect, regardless of intermediate firewalls (and how do I prevent this?). It also means you have to sign up for an account in order to use it.

I'll be sticking with ssh, for now, but this service bus thing looks kind of interesting.

msbuildtasks - A collection of MSBuild Tasks (duh)

I'm starting to use MSBuild for some system-level stuff. I've been torn between that, NAnt, and Powershell for commonly executed admin tasks (backups, deploys, things like that).

The most tempting choice was PowerShell. It can do just about anything I'd need it to. That flexibility creates a problem, too: I don't want to have to maintain these things. If I use a full-featured language, then follow up maintainers will fall into two camps: the ones who add more complexity until it becomes too hard to maintain, and the ones who won't understand any of it at all.

I was a little jealous of NAnt, because it has matured very nicely. That has to be installed - never mind how easy - and needs to be justified.

Then I ran across msbuildtasks, from the same people who brought us Subversion. Lots of shiny stuff (I love it!) including tasks for IIS sites (I don't know if it does IIS7), a slew of build and deploy related tasks, SQL tasks, Registry tasks, and more.

Then I found the MSBuild Extension Pack, which is chock-full of shiny promise, too.

So, MSBuild is our winner (found a good tutorial, and yet another). It comes with .Net 2.0, which is installed anyway (okay, you have to copy those extensions, but that could be the first task :) ), it has the features I'm looking for, thanks to Tigris, and it is proven in the field. I'm sure it will suck, too, but at least I'll know how much and in what ways it sucks. If it doesn't work out, I'll look for something else.

Update: Despite the title of this article, I'm not even using the msbuildtasks package, yet. The MSBuild Extension Pack has a lot more goodies, and everything that I've needed so far.

Unattended, A Windows Deployment System

Unattended. From their site:

Features include:

• Automated install of operating system, hotfixes and applications.
• Full documentation and source code.
• Support for floppy, CD-ROM, and "nothing but net" installs.
• True unattended installation, not disk imaging.
• No Windows servers required; use your Unix servers instead.
• No Unix servers required; use your Windows servers after all.
• Completely free.

When you are finished setting up Unattended, you will be able to boot any PC from a floppy, from a CD-ROM, or directly from the network, answer a few questions, and come back an hour or two later to a fully-installed Windows workstation.

Update: I gave it a look-see, and it looks like it would be useful for Windows 2003, and that's about it. Considering deployments of that are dropping off of a cliff, it doesn't look useful.

Eucalyptus - AWS Workalike

I keep rediscovering Eucalyptus, an project aiming to be a clone of Amazon Web Services. I need to give them a real test run...