Operations is as Important as Development

A couple of recent events have inspired me to write this post. The second was Reddit's needing help. The first was an event on my current contract.

My takeaway from Reddit's problem is brief: they finally figured out that just adding new features isn't enough. They have to get their house in order, and they have to take steps to keep it that way. To be fair, I don't know that the source of their problem was poor operational planning, in fact, it sounds like they devote a fair amount of time and effort to it.

The event at work was something of which I've seen way too much. A business wants to roll out a pretty sophisticated reporting application, high availability, super fast - you know, what everyone wants. My part of the whole thing was to determine what hardware and software resources to provide, and then build them out (it's a big enterprise shop, so there's a lot more "fun" associated with it).

It is easy to say "I want multiple web servers", it is another thing to build for multiple web servers. There's a few ways to do it, and what direction is chosen determines what to build.

This particular business not only hadn't really thought about it, they didn't want to think about it. Okay, fine, I can build for that, too - but it is gonna cost them. To the tune of about a half million dollars, just to start. With only a little bit of planning and forethought, by utilizing resources already in place they could get everything they wanted for a few thousand. That's a business decision - if the business can justify that much money, it's their budget, and who am I to complain?

While I may not complain, I can't help but notice that this doesn't really seem like smart business. In that particular case, I really don't know enough about what is going on from the business side to really make that kind of judgement.

However, I have been in enough situations where I did know enough about what is going on to be able to make that judgement. I've seen it enough that I don't want to be in a position where those decisions are made, or diminished. In fact, it is in my best interest to let these shops ignore this importance - because I charge top dollar for coming in and fixing it after the fact. If a shop wants to make a poor business decision which is good for my business...well, that's a good business decision on my part, isn't it?

If you're not bothered by the idea of getting yourself in such a tight bind that you have to shell out thousands of dollars for someone like me (and yes, thousands of dollars is accurate. Sometimes tens of thousands), then save yourself some time and stop reading.

System administration shouldn't be considered after the fact. It shouldn't be considered a necessary evil. It deserves as much prominence and dedication as any other part of the development process. An application isn't just the application that is installed or written - what it is running on is just as important.

System administration has it's own demands,it's own skillsets, it's own schedule, and it's own rhythm. This should be recognized up-front, and it should be given first-class priority. I'm probably being repetitious at this point, but it seems like it really isn't appreciated.

So, I'll say it again: too many businesses, large and small, waste money and time because they don't recognize the importance of their operations, and treat them as afterthoughts. Too many businesses limit themselves, their options, their very potential, because they don't recognize this.

I started to write this with an eye towards specific steps, specific considerations, and a lot of ideas to think about for implementing this. As I wrote, though, it only became clearer that the first step was pointing out what doesn't seem obvious: operations are important. Learn that, know that, believe that, and live that, and you won't have to worry about paying someone like me a ridiculous amount of money when it is least convenient to you.

Ignore this, and that's fine with me. My contact info is to the right, and I sincerely hope to hear from you.

Exporting MongoDb with Powershell

I'm just going to leave this right here. This fixes up the output from mongoexport to create an array, suitable for XMLSpy...

function Export-Wmi
{
 Param (
  [Parameter(mandatory=$true)]
  [string]
  $DbName,
  
  [Parameter(mandatory=$true)]
  [array]
  $CollectionNames
 )
 
 Process {
  $CollectionNames |
  foreach {
   $output = ""
   $cmd = ""
   $docs = C:\Path\To\Mongo\bin\mongoexport.exe `
    --db $DbName `
    --collection $_ `
    2>$null
   
   $output += "["
   $bfirst = $true
   foreach( $doc in $docs )
   { 
    if( -not $bfirst )
    {
     $output += ","
    }
    
    $output += $doc
    
    $bfirst = $false
   }
   
   $output += "]"
   
   $output | Out-File -FilePath "C:\Path\To\Data\$_.json"  
  }
 }
}

Export-Wmi -DbName "reports" -CollectionNames @("coll1", "coll2" )

Powershell, MongoDB, and WMI

I had reason to write a short cmdlet which converts WMI objects to MongoDB docs. It ain't much, but it works so far:

 [Cmdlet( VerbsData.ConvertTo, "MongoDoc" )]
 public class ConvertToMongoDoc : Cmdlet
 {
  [Parameter( Mandatory = true, ValueFromPipeline = true )]
  public PSObject InputObject { set; get; }

  protected override void ProcessRecord()
  {
   Document converted = (Document)WmiConvert( InputObject.BaseObject );
   this.WriteObject( converted );
  }

  protected object WmiConvert( object obj )
  {
   if( null == obj )
    return null;

   object newObj = obj;


   Type objType = obj.GetType();

   TypeCode objTypeCode = Type.GetTypeCode( objType );
   String objTypeName = objType.FullName;
   
   if( objType.IsGenericType )
    return obj;

   switch ( objTypeCode )
   {
    case TypeCode.String:
     return obj;
   }

   switch ( objTypeName  )
   {
    case "System.TimeSpan":
     return ( (TimeSpan)obj ).Ticks;

    case "System.Int16":
    case "System.UInt16":
     return System.Convert.ToInt32( obj );

    case "System.UInt64":
    case "System.UInt32":
     return System.Convert.ToInt64( obj );

    case "System.Byte":
     return null;

    case "MongoDB.Driver.Document":
     return obj;
   }

   if( objType.IsArray )
   {
    ArrayList aTmp = new ArrayList();
    foreach ( var s in (object[])obj )
    {
     aTmp.Add( WmiConvert( s ) );
    }
    return aTmp;
   }

   if( 0 == objTypeName.IndexOf( "System.Management." ) )
   {
    Document tmpdoc = null;

    switch( objTypeName )
    {
     case "System.Management.ManagementObject":
     case "System.Management.ManagementBaseObject":
      tmpdoc = new Document();
      ManagementBaseObject mbo = (ManagementBaseObject) obj;
      tmpdoc.Add( "WmiPath", mbo.SystemProperties["__PATH"].Value );
      tmpdoc.Add( "WmiServer", mbo.SystemProperties["__SERVER"].Value );
      AddPropsToDoc( tmpdoc, mbo.Properties );

      newObj = tmpdoc;
      break;

     default:
      break;
    }

    return newObj;
   }
   return newObj;
  }

  protected void AddPropsToDoc( Document doc, object obj )
  {
   foreach ( var propData in (PropertyDataCollection)obj )
      {
       doc.Add( propData.Name, WmiConvert( propData.Value )  );
      }
  }
 }

Hope this helps.

Enumerating Powershell NoteProperties

I recently was asked to stuff a custom PSObject into a bunch of key value pairs, so I thought I'd put it here. Is there a better way?

$props = ($File | get-member -MemberType NoteProperty)
foreach( $p in $props )
{
     # write-verbose (invoke-expression "`$File.$($p.Name)")
     $doc.Add( $p.Name, (invoke-expression "`$File.$($p.Name)") )
}

I Want an Online-persona Manager

There's been a lot of hubbub about Facebook and its changing privacy settings. The short version is that they are changing them to the advantage of people who want to make a buck out of you and your friends. It didn't start out that way, and the change has people in a bit of an uproar.

Thing is, Facebook does serve a valuable function - otherwise there would be nothing to exploit. An easy way to keep a business account, and a personal account. Many people use LinkedIn (for professional contacts) and Facebook (for personal) in this manner, including myself.

What the Facebook uproar is making clear is that people value the partitions they build into their lives, and want those partitions maintained online. Another thing that is clear is that as users, we can be sure that policies will change, and we won't like some of those changes.

I don't want to quit Facebook. I like Facebook, for what it is. I don't begrudge Zuckerberg a few bucks off of me, as long as he provides a service I want to use.

What I want is an easier way to manage my profiles, so that I not only have control over what's published, and where, and I want it to be easy.

I want to maintain my own profiles in a centralized location - my hard drive, my backup.

I want to write something once, and then choose, with high granularity, where it will be published.

I want it to be universal, and work with damn near everything.

I want it to be simple, with one interface, like Pidgin does for IM.

I want it to be scriptable, because by now I'm just being greedy :)

I don't want project Diaspora. I'm happy for the kids, and I wish them well, but all they promise is that I'll either have to fire up a server for myself, or I'll have to trust someone else. Again.

I trust my laptop. I trust my mobile phone (sort of). I trust my thumb drive. I trust TrueCrypt. It is my data, and I should control it.

There's a reason businesses try to find out so much about us: it has value. Google has the luxury to walk away from the extra revenue; they're getting enough.

Interestingly enough, Microsoft has an initiative that addresses some of the problem. Called U-Prove, it is BSD-licensed to help drive adoption. It primarily deals with managing both identity and authorization in financial transactions - I don't know if it could be adapted to social sites. This Ars article goes into greater depth.

I dinked around with building something like this on my own, but honestly, it looks like a bit more work than I can tackle right now. So, if there's somebody out there that thinks is a good idea, and has the wherewithal to pull it off, please steal this idea - I'll be one of your early adopters, I promise.

Update: I came up with another way to describe it while discussing this privately (Hi, Tracey!):

I've got six accounts in Outlook right now, all pointing to different places. My personal domains don't know about my google addresses, and heck, my google addresses don't even "know" about one another. (Reading the Spam folders for each one is an education in targeted advertising.) I want the same thing, only for the various social networks.

Powershell grep'ing

I've been asked about searching and replacing in files a few times, so I thought I'd put an example up here. This is a way to do it - and you'll almost certainly want to change it around to better suit your scenario.

cd C:\Users\michael.DOMAIN\Workspace\Powershell
function pgrep
{
 Param (
 
  [string]
  [parameter(mandatory=$true)]
  $Like,
 
  [bool]
  $Recurse = $false,
  
  [string]
  $BasePath
 )
 
 Process {
  if( -not $BasePath )
  {
   $BasePath = $PWD
  }
  
  # there's gotta be a slicker way
  $recurse_string = ""
  if( $Recurse )
  {
   $recurse_string = "-Recurse"
  }
  
  Invoke-Expression "Get-ChildItem -Path $BasePath $recurse_string" | 
  where {$_.PSIsContainer -eq $false } |
  foreach {
 
   $file = $_
   # $file.FullName
   # Get-Content -Path $file.FullName | where { $_ -eq "MachineName" }
   if( Get-Content -Path $file.FullName | where { $_ -like $Like } )
   {
    $file
   }
  } 
 }
}

function preplace
{
 Param (
  [string]
  $Path,
  
  [string]
  $From,
  
  [string]
  $To
 )
 
 Process {
  $fi = New-Object -TypeName System.IO.FileInfo -ArgumentList $Path
  $tmpfile = "$($fi.DirectoryName)\$($fi.Name).tmp"
  Remove-Item -Path $tmpfile -ErrorAction SilentlyContinue
 
  Get-Content $Path | 
   foreach {
    $line = $_
    $line.Replace( $From, $To )
   } |
   Add-Content -Path $tmpfile
  
  Remove-Item -Path $Path -ErrorAction SilentlyContinue
  Rename-Item -Path $tmpfile $Path
 }
}


pgrep -Like "*MachineName*" -BasePath C:\Users\michael.DOMAIN\Workspace\Powershell |

foreach {
 $fi = $_
 Write-Host "updating $($fi.FullName)"
 preplace -Path $fi.FullName -From "MachineName" -To "MachineFoo"
}

Getting Command Line Parameters Using Powershell

I'm just going to leave this here...

(gwmi win32_process -filter "name='powershell.exe'").commandLine

Open Source Installation Packages for Windows

Well, at least MS is acknowledging the problem: installing (not to mention developing) OSS applications on Windows is a complete pain in the ass. They have announced some vaporware to deal with the situation.

Now, if they would only take the same approach with everybody's app. Windows would really benefit from a Debian/RedHat package management system: A single place to look for applications and installers. It works for Apple's iPhone, doesn't it?

Emacs, Clojure, and Windows 7

Okay, I had to wrestle with this. Maybe that makes me dumb, but if you're reading this, then you ain't doin' much better. I've had an on-and-off again, love/hate relationship with emacs over the years, and I'm swinging back to "on/love" while dinking around with Clojure. Emacs really is the best Lisp-type environment, even on Windows.

To give credit where due, I started with this good post on freegeek.in, but some bits were outdated.

Pre-requisites

First, get emacs installed. Find the latest version in this folder, and extract it someplace - no installation is really necessary.

You also need git. Let's assume you get through all of that.

Last, but not least, you should have a JDK installed.

Add the path to git and java.exe to your PATH, and set JAVA_HOME to your JDK. If you don't want to do that, then start a command prompt, set them manually, and do anything emacs-y from there.

set PATH=%PATH%;c:\sys\git\bin;c:\sys\emacs-23.1\bin
set JAVA_HOME=c:\program files\Java\jdk1.6.0_19
runemacs

Emacs Starter Kit

Next, install the Emacs Starter Kit. This is a huge time-saver. Many of my previous complaints about emacs tended to be having to learn so much just to modify the environment. This takes care of a majority of that. One of the nice, and really important bits is ELPA - a package management system for emacs. In one fell swoop, 80% of my complaints have been silenced.

Anyway, download the ESK from github

git clone git://github.com/eschulte/emacs-starter-kit.git C:\Users\<userdirectory>\AppData\Roaming\.emacs.d

If you have a C:\Users\<userdirectory>\AppData\Roaming\.emacs file, rename it. You can copy it to custom.el, and it will be used as normal. Most of the action occurs in init.el, so that's a good place to look if you really don't like something.

Nope, we're not quite there yet. Before you can use it, you need to install org-mode.

cd C:\Users\<userdirectory>\AppData\Roaming\.emacs.d
git submodule init
git submodule update

Fire up emacs, and M-x package-list-packages. Scroll right on by that tempting "clojure-mode", and put your cursor on "swank-clojure", then press "i" (to select it for installation), then "x" (to start the installation).

---

Okay, here's where I ran into a problem: when trying to use ELPA, I got the following error: "Local variables entry is missing the suffix". The nice thing about obscure error messages is that they are usually pretty easy to search on. (There's a patch here, only the file name is "package.el"

You'll be adding a function, and modifying package-unpack-singe to use it.

(defun package-write-file-no-coding (file-name excl) 
 (setq buffer-file-coding-system 'no-conversion) 
 (write-region (point-min) (point-max) file-name nil nil nil excl))

The ESK instructions say to build org-mode after that, but launching emacs seemed to take care of that.

---

Using it

Woooo! Installed! Isn't this easier than NetBeans? (well, actually "yes" for me, since it was some frustration with Enclojure that drove me back to emacs)

Launch emacs, and type M-x slime. If you have everything set up right, you should be prompted to install clojure. This will not only install clojure, but also get you the swank-clojure.jar you'll be needing for projects.

Projects

Assuming you got your SLIME prompt in the step before, creating a project is easy, but not as easy as it could be.

Create the following directory structure wherever it is you want your project to go:

root
  +--src
  +--lib
  +--classes

In the lib directory, put your clojure, clojure-contrib, and swank-clojure jars. You can find them in C:\Users\<username>\AppData\Roaming\.swank-clojure Then you should be able to start a new project with:

M-x swank-clojure-project

That should do it - hope it helps

Cleaning up OpenNMS Assets

I've had OpenNMS running on the network for awhile, and here's a problem I ran into: my network is too dynamic. Specifically, it doesn't play nice with DHCP clients. Every time the IP address changes, OpenNMS considers it a new node.

What's screwed up is if you delete the old nodes, the asset info sticks around, even if you didn't add any actual asset information.

I got things straightened up enough that OpenNMS and I are getting along fairly well. There was still all this old asset information gumming up the works. I didn't see anything in the UI or help to get rid of them, so I did it the hard way.

Log into the postgres box hosting the database, and don't forget to use it. Then run the following:

delete from assets 
using node
where node.nodeid = assets.nodeid
and node.nodetype = 'D'

That should clear it up, and it doesn't appear to break anything. If you have any trouble after running this, please let me know in the comments, and I'll look into it (since it will probably be a problem for me, too).

Hope this helps.

Untangle, the DIY Network Gateway Software

I like building my own home routers. The ones you can pick up from NewEgg are nice, and they do their job admirably. Sometime back, though, I developed a taste for building my own.

Until just recently (we'll get to that), I was running pf on FreeBSD. Love it. Best damn firewall software out there, and if you need something really custom, but don't want to spend a lot of money, that's the way to go.

My old firewall was a little Epia mini-itx (that's small and low-powered), but it handled three network segments just fine, as well as a couple of ancillary tasks. It's been dying for awhile, though. Since it's my house, and that's the only place I get to really play fast-and-loose, I kept it going.

Well, it died, as these things do, last weekend.

I found an old box - another Epia, as it happens - with only two network ports (no wireless segment, for now), and set up Untangle, the "Open Source Network Gateway". There wasn't much to it, put in the CD, answer some questions, do a lot of waiting (the temporary machine is really slow).

It can be set up as a router or a bridge, the latter permitting easy integration into an existing network. There's no need to set up anything on the clients; everything happens transparently. Some of the niftier free modules include an ad blocker, virus blocker (http, smtp, pop3, imap - again, with no configuration), web filters, and so on.

It also provides the typical soho services: DHCP, DNS, port forwarding, etc.

All of that is open source, and free to download. They also have additional modules, on a subscription basis.

So far, it works well for the two segments it is handling. Performance is fine, as long as I don't get crazy with the various modules. There's also a bunch of administration functions I haven't delved into, so I may not be giving it the credit it deserves.

The new hardware should be in this week - it's looking like four network ports, maybe a wifi card, and a nice little Atom processor. It should be faster, in general, but it is going to have a lot more to do. I'll update when I've got a stronger opinion about it.