From StackOverflow

def stripped(f):
    for l in f:
        line = l.rstrip()
        if line and not line.startswith("#"):
            yield line

with(open "foo.txt", "r") as f:
    for line in stripped(f):
        print line

class Color(object):
  """Wrapper around termcolor to allow colors to be disabled."""

  @classmethod
  def Setup(cls, enabled=True):
    if enabled:
      cls.me = cls.WithColor
    else:
      cls.me = cls.WithoutColor

  @classmethod
  def WithColor(cls, msg, color=None, on_color=None, attrs=None):
    return termcolor.colored(msg, color, on_color, attrs)

  @classmethod
  def WithoutColor(cls, msg, color=None, on_color=None, attrs=None):
    (color, on_color, attrs) = (color, on_color, attrs)  # gpylint
    return msg

MacOS 10.7: Monitor your Wi-Fi with Wi-Fi Diagnostics

Lion ships with an app called Wi-Fi diagnostics. It allows you to monitor Wi-Fi networks your computer is connected to and collect various kinds of information. This utility could be very useful for finding problems or doing research on your Wi-Fi networks.

Enabling OS X Screen Sharing via the CLI

If you’re already at your Mac’s desktop, you can simply turn on Screen Sharing. But if you (1) didn’t plan ahead or (2) worry about security, here’s how to turn it on.

Converting from a string to boolean in Python

>>> import ast
>>> ast.literal_eval("True")

John Carmack Slashdot Archive

All of Carmack’s comments on Slashdot, from 1999 to 2008. See also his .plan file updates from 1996-2007 here.

Deploying node.js on Amazon EC2

Creating an AWS “micro” instance running Ubuntu 10.04LTS and setting up a Node.js server with deployment managed by Capistrano and process management by Bluepill.

If I Launched a Startup

Collection of linked advice from Ryan Roberts, aka @StartupLawyer.

Clever Algorithms

“This book is concerned with ‘clever algorithms’, which are algorithms drawn from many sub-fields of artificial intelligence not limited to the scruffy fields of biologically inspired computation, computational intelligence and metaheuristics. The term ‘clever algorithms’ is intended to unify a collection of interesting and useful computational tools under a consistent and accessible banner. ”

Deft

Emacs mode for quickly browsing, filtering, and editing directories of plain text notes, inspired by Notational Velocity. It was designed for increased productivity when writing and taking notes by making it fast and simple to find the right file at the right time and by automating many of the usual tasks such as creating new files and saving files.

Warehouse-Scale Computing: Entering the Teenage Decade

ACM keynote from FCRC 2011 by Luis Barroso Alvarez. Covers both Google-specific stuff and general datacenter industry trends.

Keymaster

A simple micro-library for defining and dispatching keyboard shortcuts.

The Problem

Dropbox recently made news by updating their Terms of Service to explicitly state that (if asked by law enforcement) they can and will decrypt your files and hand them over.

As set forth in our privacy policy, and in compliance with United States law, Dropbox cooperates with United States law enforcement when it receives valid legal process, which may require Dropbox to provide the contents of your private Dropbox. In these cases, Dropbox will remove Dropbox’s encryption from the files before providing them to law enforcement.

This seemingly contradicts some language in their marketing materials that would suggest that user files are stored in an encrypted format on Dropbox’s backend and are completely inaccessible even to Dropbox employees.  This is similar to what competing tools like Jungle Disk do explicitly:  the files are encrypted before they ever leave your computer and the decryption key is only ever stored locally.

Unfortunately, it now seems possible that Dropbox is not encrypting files on the backend, given the comments from their CTO at the end of the TUAW article linked above.  Given this, it seems prudent to look for a solution that allows you to keep using Dropbox, but encrypt sensitive data before it’s ever uploaded to the cloud.

Enter EncFS

EncFS is an encrypted pass-through filesystem that’s implemented in userspace.  What’s that mean?  Pass-through means (basically) that the encryption is done on a file-by-file basis, as opposed to an encrypted block device, such as TrueCrypt or Apple’s DMG image format, where you have one monolithic encrypted “bucket” into which you put your files.

In a nutshell, an EncFS volume looks (mostly) like an ordinary folder on your hard drive. Anything you save into that folder gets transparently encrypted and stored in a separate folder elsewhere on your hard drive (the “backing store”). As far as any applications know, they’re working with plain, ordinary, unencrypted files but these files are never actually read or written on disk. Only the encrypted versions are actually accessed behind the scenes.

It’s this backing store functionality that makes EncFS a perfect companion to Dropbox. You read and write your files on your hard drive as normal (inside the virtual folder), and EncFS transparently stores the encrypted versions in your Dropbox folder, where they are synced in real-time to the cloud. Since each file is encrypted individually, Dropbox can sync changes incrementally instead of having to sync the whole disk image. To be fair, they do differential syncing, so only the actual changed bits of the image get transferred over the wire, but to my mind syncing the individual files is still better. File-based encryption also makes it much smoother to access your files from multiple machines at the same time, since Dropbox can resolve conflicts at the individual file level.

“Implemented in userspace” means that EncFS runs as an ordinary process (a daemon), like a web or FTP server, as opposed to a kernel module.  This is done through use of the FUSE (Filesystem in Userspace) project, which is a kernel module and provides the hooks into the file handling subsystem that something like EncFS, sshfs, or GMailFS requires to work their magic.

MacFUSE

FUSE was originally (and still is) a Linux project, using a loadable module for the Linux kernel to provide the necessary hooks to userspace filesystems like EncFS.  Fortunately, Amit Singh from Google (author of the terrific book “MacOS X Internals“) developed a MacOS X port, MacFUSE, which provides the same userspace API, but hooks into the Darwin kernel instead.  Thus, we can use EncFS, sshfs, and most if not all of the other FUSE filesystems on the Mac.

Installation

• download MacFUSE from Google Code. Don’t worry about the latest version being from 2008. It still works fine with (32-bit) OS X 10.6. Run the installer and click through, accepting the defaults.
• download EncFS pre-built for OS X from Google Code.  You want the file named something like “EncFS-10.5-2.0.0.zip” even though it says it’s for Leopard.  Run the installer and again accept all of the defaults.  This will install the EncFS FUSE module and the related CLI tools (in /usr/local/bin)

Creating Your First Secure Volume

On my machines, I keep the backing store (the actual files on disk) in my Dropbox, which is ~/Dropbox/Secure. The virtual folder where I can access these files is in ~/Documents/Secure.

$ encfs ~/Dropbox/Secure ~/Documents/Secure
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> p

Paranoia configuration selected.

Configuration finished.  The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 2:1:1
Filename encoding: "nameio/block", version 3:0:1
Key Size: 256 bits
Block Size: 512 bytes, including 8 byte MAC header
Each file contains 8 byte header with unique IV data.
Filenames encoded using IV chaining mode.
File data IV is chained to filename IV.

-------------------------- WARNING --------------------------
The external initialization-vector chaining option has been
enabled.  This option disables the use of hard links on the
filesystem. Without hard links, some programs may not work.
The programs 'mutt' and 'procmail' are known to fail.  For
more information, please see the encfs mailing list.
If you would like to choose another configuration setting,
please press CTRL-C now to abort and start over.

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism.  However, the password can be changed
later using encfsctl.

New Encfs Password: dontuseme
Verify Encfs Password: dontuseme

The Results

Creating and manipulating files inside the virtual directory is just like any other directory on disk. Both UNIX and MacOS applications see nothing different about the file.

$ echo Hello World > ~/Documents/Secure/hello.txt

$ ls ~/Documents/Secure
hello.txt

$ cat ~/Documents/Secure/hello.txt
Hello World

However, if we examine the actual files stored on disk inside the Dropbox folder, we will see that both the name of the file and its contents are encrypted securely.

$ ls ~/Dropbox/Secure
CUUNdUhk0bp-k-eswFVtxG6D

$ cat ~/Dropbox/Secure/CUUNdUhk0bp-k-eswFVtxG6D
ѹ?7????3?V{1}=????|??D?x

A screenshot from my Dropbox folder

Caveats and Notes

An EncFS volume does not mount on boot, so you will need to either manually re-run the encfs command above, or create a startup script of some kind (possibly with AppleScript) to do it for you automatically.  Personally, I like this behavior — especially on my laptop. If my Macbook Pro is ever stolen, the thief would need to reboot to get past the screensaver. All of my sensitive data is securely encrypted inside the EncFS backing store and would be completely inaccessible unless the virtual filesystem is mounted (which would require my passphrase).

Probably the biggest downside to the EncFS solution (or any client-side encryption scheme) is that it breaks access to your files from the Dropbox web client, from mobile devices, or from any computer that doesn’t have EncFS installed. You’ll want to carefully consider the risk and perhaps only use your encrypted folder for especially-sensitive files that you’re not likely to need to access from a non-EncFS-enabled computer.

Amazingly, there actually does appear to be initial EncFS support for Windows. See here for details. It appears to at least work on Windows XP, but there seem to be issues with Vista and Windows 7.  Better than nothing, and quite an impressive hack given the major differences between the Windows OS kernel and Linux. At least MacOS X is based on Mach, so there’s a UNIX-like kernel underneath!

 

I’ve been building and rebuilding hosts in the home lab quite a bit lately.  After about the 3rd time reconfiguring the same host options (adding a NFS store, setting up my portgroups, etc.) I decided this was a good opportunity to learn some PowerCLI.  Could do something similar with host profiles, but that’s a project for another day.

Below is a basic script that will add a newly-built ESXi host to vCenter and apply the common settings for my lab.  Still requires manual configuration of the management interface IP info (although that could be automated through a scripted install).

# Connect to vCenter
Connect-VIServer vc1.packetslave.local

# Add host to vCenter and setup NTP
$host = "tc2.packetslave.local"
Add-VMHost $host -Force -Location (Get-Cluster HA) -User root -Password mypass
Add-VMHostNtpServer -VMHost $host -NtpServer 'time.apple.com'

# Enable vMotion
$vk = Get-VMHostNetworkAdapter -VMhost $host -VMKernel | where {$_.DeviceName -eq "vmk0"}
$vk | Set-VMHostNetworkAdapter -VMotionEnabled $true

# Connect to my NFS datastore
New-DataStore -VMhost $host -Name mynfs -Nfs -NfsHost mynas -Path /vm

# Rename the default port group
$vs = Get-VirtualSwitch -VMhost $host -Name vSwitch0
$pg = Get-VirtualPortGroup -VirtualSwitch $vs -Name "VM Network"
Set-VirtualPortGroup -VirtualPortGroup $pg -Name "Production Network"

# Add my additional port groups
New-VirtualPortGroup -VirtualSwitch $vs -Name "Test Network"
New-VirtualPortGroup -VirtualSwitch $vs -Name "Storage Admin"

# Tell DRS to re-balance VMs including the new host
Get-DrsRecommendation -Cluster (Get-Cluster HA) -Refresh

#