Someone asked me to explain the Price equation today...

I got an e-mail today asking for help understanding the Price equation, prompted partly by the recent RadioLab about George Price. The person who e-mailed me made it sound like he was OK with a long explanation, just so long as it explained the ugliness of the mathematics. Here is my response... (pardon the e-mail-esque formatting... I'm just pasting it rather than re-formatting it)

[ This post can also be found on my web page. ]

You shouldn't believe everything the media tells you about the complexity of the Price equation. I'm always frustrated when I hear someone on the radio read the Price equation out loud as a mathematical statement. It is not meant to be a mathematical statement. It is just a logical justification for something we all think should be true -- traits with higher differential fitness advantage should spread throughout a population (which is a critical aspect of natural selection). Price formalized that statement and then proved that the formalism is a tautology. That's all that's important.

It is a very simple idea, and it has almost nothing to do with statistics (because there are no random variables nor data in the price equation). The Price equation is a theoretical statement about the relationship between two sequential generations of a model population. You can use it to predict how the representation of a particular trait will change over time and eventually settle at some fixed distribution. However, again, numerical applications aside, it really is just a mathematical verification of something which makes intuitive sense.

Just to get comfortable with the notation, consider a trait like "height" across a population of n=100 individuals. Each individual might have a different height. Let's say that in our population, people basically have two different heights (perhaps due to sexual dimorphism). So we have two groups:

z_1 = 5 feet
z_2 = 6 feet

We represent the number of people with each height using the variables:

n_1 = 50
n_2 = 50

That is, there are an equal number of 5' tall people and 6' tall people from our 100 person population (note that n_1 + n_2 = n). Further, we find that both 5' tall and 6' tall people tend to have 1 offspring each. That is, they both have an equivalent "fitness" of 1:

w_1 = 1
w_2 = 1

Where w_i is the number of offspring an individual of group i will contribute to the next generation. Let's say we also know that offspring from 5' tall people end up also being 5' tall, and offspring of 6' tall people also end up being 6' tall. Then we have:

z'_1 = 5 feet
z'_2 = 6 feet

So the value of the trait (height) does not change from generation to generation.

Everything above is a parameter of the model. It represents what we know about "height" of individuals in this generation as well as the relationship between the height of an INDIVIDUAL and its offspring. What Price equation does is tell us about how the distribution of height in the POPULATION will change from this generation to the next. It might be helpful to think about Price equation as relating the AVERAGE value of a trait (e.g., height) in one generation to the AVERAGE value of the trait (e.g., height) in the next generation.

So now let's add-on the Price equation stuff. To account for the changes in the average value of the trait (height here), we have to worry about two effects -- "background bias [due to individuals]" (my term) and "differential fitness" (a quantity that drives natural selection):

1.) Imagine that 5' tall parents produced 5' tall offspring (so z'_1=z_1=5 feet, as above), but 6' tall parents produced 10' tall offspring (so z'_2=10 feet in this hypothetical scenario). Then even without worrying about "differential fitness", we might expect an upward shift in AVERAGE height from the parent generation to the offspring generation. This "background bias [due to individuals]" is related to the "E(w_i \delta z_i)" term in the Price equation. It represents the change in a trait at the individual level. I'll give more info about the math later.

2.) Now, instead, assume that z'_1=z_1 and z'_2=z_2 (so offspring height is the same as parent height) as above. It may still be the case that the average height in the offspring generation changes from the parent generation. This would occur if one height had a higher fitness than the other height. Here, we see that w_1=w_2=1. They both have the same fitness, and so we don't expect any differences IN REPRESENTATION from one generation to the other. Note that if w_1=w_2=5, then each individual would produce 5 offspring. Consequently, the TOTAL population would grow, but the DISTRIBUTION of height would stay the same. To make things more interesting, imagine that w_1=1 and w_2=2. Now each 5' tall person produces one 5' tall offspring, but a 6' tall person produces TWO 6' tall offspring. Consequently, the distribution of height would change from parent to offspring generation. The AVERAGE height would shift toward 6' tall people. The "cov(w_i, z_i)" term aggregates this change. It relates the "differential fitness" of one height to its success into growing the representation of that height in the next generation. I'll give more info about the math in a bit. [NOTE that the average fitness represents the average "background" rate of growth from population to population.]

To get ready for an explanation of the actual Price equation, let's get some terminology out of the way.

First, we define the "expectation" or "average" height in the current population with:

E(z_i) = ( n_1 * z_1 + n_2 * z_2 + ... )/n

That is, "E(z_i)" is the average value of the trait (height above). There are n_1 individuals with z_1 value of the trait, and so we have to multiply n_1 * z_1 to get the total contribution of that value of the trait. We do that for each group. We can do the same for other variables too. For example, here's average fitness:

E(w_i) = ( n_1 * w_1 + n_2 * w_2 + ... )/n

The average fitness "E(w_i)" somehow represents the average rate of population growth. If every w_i is 1, then there will be 1-to-1 replacement of parent by offspring and there will be no population growth; likewise, the average "E(w_i)" will be 1 reflecting no growth. However, if every w_i is 5, then "E(w_i)" will also be 5 and the population will grow 5 fold every generation. With some simple arithmetic, it is easy to verify that the total population in the NEXT (i.e., offspring) generation is given by the product of the number of individuals in this generation (n) and the average fitness (E(w_i)).

We can also find the average value of the trait in the NEXT (i.e., offspring) generation. To do so, we have to scale each value of the trait in the next generation (z'_i) by the number of individuals with that trait in the next generation (n_i w_i), and then we have to divide by the total number of individuals in the next generation (n*E(w_i)). So the average value of the trait in the NEXT (i.e., offspring) generation is:

E(z'_i) = ( n_1 * w_1 * z'_1 + n_2 * w_2 * z'_2 + ... )/(n * E(w_i))

For simplicity, let's use symbols "z", "w", and "z'" as a shorthand for those three quantities above. That is:

z = E(z_i)
w = E(w_i)
z' = E(z'_i)

Penultimately, let's define "delta" which gives the difference in a variable from the this generation to the next. The difference in the average value of the trait is:

delta(z) = E(z') - E(z)

that difference may be due either to differential fitness (i.e., when w_i is not the same as w) or to intrinsic height changes at the individual level. Those intrinsic height changes at the individual level are:

delta(z_1) = z'_1 - z_1
delta(z_2) = z'_2 - z_2
...

Finally, let's define this "covariance" formula. For each group i, let's say we have variables A_i and B_i (e.g., z_i and w_i). Let A be the average value of A_i across the population:

A = ( n_1 A_1 + n_2 A_2 + ... )/n

and B be the similarly defined average value of B_i across the population. Then we can define the covariance across the POPULATION in a similar way as we defined average. That is:

cov( A_i, B_i )
=
E( (A_i-A)*(B_i-B) )
=
( n_1*(A_i - A)*(B_i - B) + n_2*(A_2 - A)*(B_2 - B) + ... )/n

That is, cov(A_i,B_i) is the AVERAGE value of the product of the difference between each A_i and its average A and the difference between each B_i and its average B. We call this the "covariance" because:

* If A_i doesn't vary across values of i, then A_i=A (no "variance" in A) so there is no "covariance"

* If B_i doesn't vary, then there is similarly no covariance

* If whenever A_i is far from its average B_i is close to its average, then there is LOW (i.e., near zero) covariance. That is, both A_i and B_i vary across the population, but they don't vary in the same way.

* If whenever A_i is far from its average B_i is also far from its average, then there is HIGH (i.e., far from zero) covariance. Both A_i and B_i vary across the population, and they vary in the same way.

Note that HIGH covariance could be very positive or very negative. In the positive case, A_i and B_i have a similar pattern across values of i. In the negative case, A_i and B_i have mirrored patterns across values of i (i.e., A_i is very positive when B_i is very negative and vice versa). LOW covariance is specifically when the cov() formula is near zero. That indicates that the pattern of A_i has little relationship to the pattern of B_i.

Now, let's look at the Price equation more closely. The left-hand side:

w*delta(z)

is roughly the amount of new trait ADDED to each "average" individual. So if the average trait shifts (e.g., from 5.5' tall to 6.5' tall, corresponding to a delta(z) of 1'), but the population has GROWN as well (i.e., "w>1"), then amount of height "added" to the parent population to get the offspring population is more than just 1' per person. We scale the 1' per person by the "w" growth rate. Thus, "w delta(z)" captures effects of population growth (which naturally adds trait to a population) and mean change in representation. Note that if the AVERAGE trait did not change ("delta(z)=0") but the population did grow ("w>1"), then we interpret "w delta(z)=0" to mean that even though the "total amount" of trait increased due to population increase, there was no marginal change in each individual's trait (i.e., individuals aren't getting taller; the population is just getting larger).

Now let's look at the right-hand side:

cov(w_i, z_i) + E(w_i*delta(z_i))

This implies that the amount of new trait added to each average individual is the combination of two components.

To parallel the discussion above, let's consider the E() part first:

E(w_i * delta(z_i))

we can expand this average to be:

( n_1*w_1*(z'_1 - z_1) + n_2*w_2*(z'_2 - z_2) + ... )/n

That is, delta(z_i) gives us the average change from AN INDIVIDUAL to A SINGLE OFFSPRING from z_i to z_i'. The w_i part ACCUMULATES those changes to EACH offspring. For example, if w_1=2, then group 1 parents have 2 offspring. So the total increase in the trait from group 1 is not delta(z_1) but is 2*delta(z_1). So you can see how this is the "BACKGROUND BIAS" representing the "w*delta(z)" component that we get even without worrying about differential fitness. This represents the change in "w*delta(z)" just due to INDIVIDUALS and POPULATION GROWTH.

Next, look at the covariance:

cov(w_i, z_i)

The covariance of w_i and z_i is a measure of how much the DIFFERENTIAL FITNESS contributes to added trait. Recall the formula for cov(w_i,z_i):

E( (w_i-w)*(z_i-z) )

which is equivalent to:

( n_1*(w_1-w)*(z_1-z) + n_2*(w_2-w)*(z_2-z) + ... )/n

Here, the quantity (w_i-w) is the "differential fitness" of group i, and the quantity (z_i-z) represents the location of the trait with respect to the average trait. So:

* if the fitness varies in a similar way as the level of trait across values of i, then the average value of the trait will tend to increase from population to population

* if the fitness varies in exactly the opposite way as the level of the trait across values of i, then the average value of the trait will tend to decrease from population to population

* if the fitness varies differently than the level of the trait, then there will be little change in the average trait from population to population

* if there is no variance in either fitness nor level of the trait, there will be little change in the average trait

Put in other words:

* if high differential fitness always comes with high values of the trait and low differential fitness always comes with low values of the trait, then there will be selection toward MORE trait

* if high differential fitness always comes with to low values of the trait and low differential fitness always comes with high values of the trait, then there will be selection toward LESS trait

* if differential fitness variation has no relationship to trait level variation, then selection will not change the average value of the trait

* if there is no variation in the trait or no variation in the fitness, then selection will not change the average value of the trait

Put in MORE words at a more individual group level:

If a group i has both a high "differential fitness" (w_i-w) AND a high (z_i-z), then its FITNESS w_i is far above the average fitness w and its level of the trait z_i is far above the average value of the trait z. Either one of those alone would be enough to cause the "total amount" of trait to shift upward. On the other hand, if BOTH (w_i-w) and (z_i-z) are NEGATIVE, then the average population is already far away from this trait value AND has a much higher fitness. Consequently, the motion of the average trait will still be upward, but here upward is AWAY from the trait z_i (because z_i is under the average z). Finally, if (w_i-w) and (z_i-z) have opposite signs, the motion of the average trait z will be negative, which will either be heading toward z_i if w_i>w or away from z_i if w_i<w. The covariance formula takes the average value of (w_i-w)(z_i-z). That average represents the contribution to the amount of trait "added" to each individual due to DIFFERENTIAL FITNESS.

So there you have it. Assuming that "w" (average fitness -- which is a growth rate) is not zero (which just assumes that the population does not die out in one generation), then we can divide everything by "w" to get a less complicated (but equivalent) Price equation:

delta(z) = ( cov(w_i,z_i) + E(w_i*delta(z_i)) )/w

So now we have an equation representing the average change from parent to offspring population. If you expand all the formulas, you can verify that this statement is equivalent to:

delta(z) = cov(w_i/w, z_i) + E( (w_i/w)*delta(z_i) )

The quotient "w_i/w" is a "fractional fitness." It is a measure comparing the fitness of group i with the average fitness, where high differential fitness corresponds to w_i/w > 1 and low differential fitness corresponds to w_i/w < 1. So let's create a new variable

v_i = w_i/w

to be the fractional fitness. Then we can rewrite Price's equation to be:

delta(z) = cov( v_i, z_i ) + E( v_i*delta(z_i) )

This version gets rid of the need to worry about scaling for population growth. If you think about it, v_i is just a normalized version of w_i where you have "factored out" the background growth rate of the population. So now we basically have:

AVERAGE_CHANGE
=
POPULATION_CHANGE_DUE_TO_DIFFERENTIAL_FITNESS
+
POPULATION_CHANGE_DUE_TO_INDIVIDUAL_CHANGES

In other words:

"the change in the average value of the trait is due to two parts:

1. The differential fitness of each value represented in the population

2. The individual change from parent trait level to offspring trait level"

So if you wish to go back to the "height" example...

"The average height increases when:
1. Natural selection favors increases in height
OR
2. Tall people have taller offspring"

You could create other variations that work as well:

"The average height DEcreases when:
1. Natural selection favors DEcreases in height
OR
2. Short people have shorter offspring"

====

"The average height stays the same when:
1. Natural selection has no preference for height
AND
2. Short people have short offspring and tall people have tall offspring"

====

"The average height DEcreases when:
1. Natural selection has no preference for height
AND
2. Short people have short offspring and tall people have short offspring"

====

"The average height INcreases when:
1. Natural selection has no preference for height
AND
2. Short people have tall offspring and tall people have tall offspring"

CS Education: C# in Mono / Java

At least three times this year, I have found myself suggesting that perhaps it would be better to teach component-based software enginereing to young undergraduates using C# (via Mono, of course) as opposed to the more-restrictive Java. I feel a little like Nixon going to China when I say this...

Having said that, it's probably better to use Java for production code. But I almost think that you need flexibility to experiment at the educational level, and Java has no flexibility (because it is so tightly crafted for security, portability, and stability). C++ has a lot of flexibility, but no one wants students learning C++ anymore (a comment on that in a bit). C# (under Mono, of course) has more flexibility than Java but less than C++. It's relevant, as it is relatively easy to move from C# development to Java develpoment (and there are plenty of companies (or at lest one gigantic one) that are looking for C# developers too). It has preprocessor directives (even though it has no preprocessor), which gives it a customizable presentation for the classroom environment. So it feels like it would be easier to start with C# and then "graduate" to Java (and go to graduate school for Lisp? I dunno. And where does Python fit in? Maybe it takes the place of Logo in 4th grade).

Regarding the note above about no one wanting students to learn C++ anymore, it is interesting that software interviewers are typically of the age and rank where most of their experience is in C or C++. I was at an interview recently where the interviewer asked questions contrasting addressing individual bytes of a 4-byte unsigned integer using pointers or bitmasks; such a question does not belong in any discussion of Java. Moreover, I heard someone in a nearby cubicle on a phone interview being asked about static variables in C and C++ programs. The graduate student was more familiar with "static" in the context of Java, and so I think he worked htings out sufficiently well, but most students have never heard of this term (likewise, they don't even think to look up "persistent" in their MATLAB documentation). So it seems unfair that undergraduates are being switched over to Java... and will be asked about pointers in some of their first technical interviews. Oh well...

Installing Adobe Digital Editions on Linux with Wine

UPDATE: I have not had time to address the download issues some people have had when trying to install ADE. However, see the comments on this post for some additional tips on where to find an ADE installer download that you can run inside Wine.

Unfortunately, Adobe does not provide a simple download link for the Windows version of Digital Editions, and so if you want to purchase a PDF or ePub e-book with Adobe ADEPT DRM protection, it is very difficult to get up and running on Linux even if you have Wine installed. Here's how I did it.

1. Install Wine on your system. Most Linux distributions provide Wine as an optional install (e.g., sudo yum install wine from the command line on Fedora).
2. You will need Gecko extensions available to run the iexplore.exe web browser that comes with Wine. Visit the Wine Wiki's Gecko page for information on how to install Gecko. Your Linux distribution may have a "wine-gecko" package. If it doesn't, then follow the simple instructions on the Wine Wiki.
3. You will also need Adobe Flash installed on your Wine system, and so you should visit the Wine Wiki's Flash page to get the install_flash_player.exe download link. Download the exe and run it in Wine (e.g., wine install_flash_player.exe from the command line).
4. Now, start the iexplore.exe browser bundled with Wine. After I ran Wine for the first time, it setup a .wine directory in my home directory (i.e., ~/.wine). I found the iexplore.exe in ~/.wine/drive_c/Program Files/Internet Explorer. Change to that directory and run iexplore.exe (e.g, wine iexplore.exe from the command line).
5. Now you can visit the Digital Editions installer (which is written in Flash). From the Wine Internet Explorer, enter in the URL:

   http://www.adobe.com/products/digitaleditions/ade_web_library.swf

   That should startup a little installer. Follow the prompts to download and install Digital Editions. You will even be able to launch it from there.
   • The installer will install the single Digital Editions executable to ~/.wine/drive_c/Program Files (x86)/Adobe/Adobe Digital Editions as digitaleditions.exe.
   • You can run it again but executing that file through wine (e.g., wine digitaleditions.exe from the command line in that directory)
   • Alternatively, you can use the "Digital Editions" shortcuts that the installer probably placed on your desktop.
6. Once Digital Editions is started, you should be able to drag-and-drop ACSM files you've downloaded from e-Book commerce sites (like Google Books), and it should download the Adept-DRM'd version of your media.
   • DRM'd books are stored in the My Digital Editions folder that will be created in your home directory.
   • If you want to put them elsewhere, you need to edit the symbolic links in ~/.wine/drive_c/users/USERNAME where USERNAME is your Linux username. The My Documents folder was symlinked to my home directory by default. I got rid of the symlink and created a real directory there instead, but you could point the symlink somewhere else if you'd rather. Regardless, the My Digital Editions folder will reside inside the My Documents folder (alongside a My Books folder, but I don't know what gets put in there... maybe non-DRM stuff you put into ADE?).

Once you have your DRM'd books, you should be able to transfer them onto devices that have also been authorized with your Adept key (e.g., a Sony Reader that has been authorized with the same account you used inside Digital Editions). I suppose it's possible to get the Digital Editions running inside Wine to notice when your reader is connected, but I haven't tried it (that might involve installing the Sony Reader Library software within Wine, and I received some ugly errors when I tried to do that (but YMMV)).

MATLAB script to prove to you that the perfect squares are the switches that are toggled

Evidently, this problem exists in both "switch" and "locker" forms. Assume that there are 100 switches are initially off and numbered from 1 to 100.

• First, you toggle multiples of 1 (i.e., every switch gets turned on),
• and then you toggle multiples of 2 (i.e., every even-numbered switch gets turned off),
• and then you toggle multiples of 3 (i.e., switches 3, 6, 9, 12, ... are toggled),
• ...
• and then you toggle multiples of 99 (i.e., switch 99 is toggled),
• and then you toggle multiples of 100 (i.e., switch 100 is toggled).

After this process, which switches are "on"?

Examining the problem, you notice that switches are toggled the same number of times as they have unique factors. For example, switch 6 is toggled 4 times, which corresponds to its unique factors, 1, 2, 3, and 6. However, switch 4 is toggled only 3 times, which corresponds to its unique factors, 1, 2, and 4 (i.e., even though 4 = 2 × 2, the factor 2 is only represented in the list once because it is duplicated). Moreover, it is only the perfect squares (1, 4, 9, 16, 25, 36, 49, 64, 81, 100) that will have repeated factors (i.e., their whole square roots), and so it is only those switches that will be flipped and odd number of times. Furthermore, it is only those switches that will be "on" after this game.

To verify this to yourself, use a MATLAB script:

% All switches initially off
switches = zeros(1,100);

% Toggle multiples of the current switch, including the current switch
for i = 1:100
% These are the switches to toggle
ivec = i:i:100;

% xor'ing them with 1's toggles them
switches(ivec) = bitxor(switches(ivec), 1);
end

% Show the indexes of the switches that are "on" after this process
% (expect perfect squares: 1 4 9 16 25 36 49 64 81 100)
find( switches )

Running the script results in:

ans =

1     4     9    16    25    36    49    64    81   100

which is what we expected.

Of course, you could also keep track of how many times you made a flip. XOR can help with this too.

% All switches initially off
switches = zeros(1,100);
new_switches = zeros(1,100);
num_flips = zeros(1,100);

% Toggle multiples of the current switch, including the current switch
% (gratuitous use of bitxor as a demo)
for i = 1:100
% These are the switches to toggle
ivec = i:i:100;

% xor'ing them with 1's toggles them
new_switches(ivec) = bitxor(switches(ivec), 1);

% count the flips
% ( num_flips(ivec) = num_flips(ivec)+1  works too)
num_flips = num_flips + bitxor( new_switches, switches );

% commit new_switches to switches to maintain invariant
% (can omit new_switches by not using second bitxor above)
switches = new_switches;
end

% Show the indexes of the switches that are "on" after this process
% (expect perfect squares: 1 4 9 16 25 36 49 64 81 100)
find( switches )

Then, from the console, we can issue commands like:

num_flips( find(switches) )     % shows number of flips for switches ending "on"
num_flips( find(switches==0) )  % shows number of flips for switches ending "off"
find( mod(num_flips,2)==1 )     % verifies odd-count flips have perfect-square indexes

So that's fun.

Another rant on the supremacy of rolling release cycles for OS distributions (why Arch beats Fedora and Ubuntu)

As I sit here right now, I notice that my Arch Linux package manager ("pacman") is downloading KDE 4.6.1. KDE 4.6.0 was released at the end of January, and it was automatically installed on my system shortly after it was released. I don't notice a lot of new features, but there is one feature that I now use all the time. For a long time, dragging a KDE window to one of three edges of the screen would cause it to either be maximized (top) or resized to fill either the left half (left) or right half (right) of the screen for easy side-by-side tiling. This feature exists in Windows Vista, and I know a certain fiancée who uses it all the time. In KDE 4.6.0, they've added more granularity – the left and right edges of the screen are now partitioned into thirds. Dragging a window to the top or bottom third causes it be resized for a quarter of the screen as opposed to a half of the screen. The middle third resizes for half of the screen as before. So now it is easy to do even more sophisticated tiling by dragging, and I use it all the time.

But I can't use that feature on my machine at the office (Fedora 14) nor the HTPC connected to my TV (Ubuntu 10.10). That's because both Fedora and Ubuntu are on 6 month release cycles, and KDE is also on a 6 month release cycle. Consequently, my Fedora office machine is stuck back on some KDE 4.5.5 components, some 4.4.10 components, and some KDE 3.5.10 components, and it seems arbitrary which components are which KDE version. More importantly, it may be another year before I can ever expect to get quarter-window tiling on these machines (enterprise distributions, like RedHat EL, are even worse; you may wait a decade for features to trickle down the official release path). There are ways in which these distribution gatekeepers try to get around this delay by supporting auxiliary repositories of updated software, but then that is a whole other mess I would have to cover in another rant.

It's frustrating to me that these Linux "distributions" take on a release cycle that elevates them to a level on par with the software that supposedly is the sole driver of their value. It is one thing to wait for the next release of Windows to come out because Windows is a product itself; however, Ubuntu or Fedora is just a distribution – a glorified collection of independent software components that are possibly rapidly evolving and have some dependency structure on each other. Arch Linux recognizes this. It recognizes that a good package manager trumps a great "distribution" any time, and pacman is a great package manager (and it doesn't have a stupid name, like "yum" or "apt-verb", and isn't just a wrapper for yet another silly-named package manager (like "rpm" or "dpkg")). In doing so, it allows for everything from minimal Linux installations to maximally bloated Linux distributions, with the distinction that every package is truly up to date (and not just up to date relative to some arbitrary collection of version numbers). Arch Linux is not the only distribution with this nice feature. Gentoo, for example, also uses a rolling release cycle... and Gentoo is also well-known for its excellent package manager "emerge".

[ ...and on the subject of package managers, the pacman-wrapper yaourt augments the terrific pacman with the ability to automate building packages from source, even when you want to both customize your build environment and automatically build updates when they are available... so yaourt brings some of the nice features of Gentoo's rolling release cycle directly to Arch Linux's rolling release cycle ]

Frustration is certainly not the only feeling I can associate with the Fedora and Ubuntu distributions forced on me (by my employer in the former case and Boxee in the latter case); there's also fear. Every six months with Fedora or Ubuntu, you have to go through the horror of upgrading the entire distribution at once. I'd rather tackle upgrading problems with one package at a time as opposed to the entire OS twice a year. Not to mention the psychological trauma that upgrades cause – regardless of how nice the system works, I always have in the back of my head that maybe it would be better if I just formatted and installed fresh right now (but then the fear of the next upgrade prevents me from doing that). An operating system should not be a major cause of distress, but I find nothing but distress in Ubuntu, Fedora, and the other deleterious distributions in that pernicious pack that do not adopt rolling releases.

So there you go. I just quarter-window tiled this window, and it saddens me that when I go back into the office on Monday, I will lose half my day to half-window tiling and resizing... and then I'll lose a day or two later this year when FC15 comes out. Stupid six-month distribution release cycles...

These continuous enrollment policies remind you that schools are in the business of making money

This memo below was sent out to all department chairs, graduate studies chairs, and graduate program administrative support staff at OSU. I don't think it was intended to be read by students or anyone else, but it got forwarded to the student list, and they haven't taken me off of that list yet. For brevity, I'm omitting the PDF that was attached to the memo.

I have always thought these continuous enrollment policies (and limits on funding based on a credit hour ceiling) were slimy. It's even slimier when you read this memo which initially says it is in the benefit of the student to finish their doctorate quickly but then at the end says that it ensures the college is making money.

Patrick S. Osmer, Vice Provost and Dean
February 28, 2011

TO:
Department Chairs
Graduate Studies Committee Chairs
Graduate Program Administrative Support Staff


Colleagues:

I am writing to remind you of the Continuous Enrollment Policy that is in effect for all students who were admitted to the Graduate School autumn quarter 2008 and after. We are receiving petitions for exemption from the policy, which we need to have approved by the college before we can consider them, as I ex­plain below.

Simply put, this policy requires all post-candidacy students to be enrolled for a minimum of three credit hours every quarter (excluding summer quarter) until graduation. I am including a copy of the policy for your convenience. Recall that the policy is an outcome of the process that led to the reduction of full-time enrollment for post-candidacy students to three credit hours. One specific goal of continuous enrollment is to reduce time to degree for doctoral students by having them formally engaged with their program and the university. Going away to teach somewhere else with the intent to finish the dissertation off campus, for example, is not in the student’s best interests or the university’s.

A student may apply for a leave of absence due to extenuating circumstances such as the birth or adoption of a child or a serious medical condition, but a leave will not be granted with the sole reason of financial hardship.

If a student is requesting an exemption from continuous enrollment due to circumstances not covered by the leave of absence as stated in the Graduate School Handbook, I am asking that he or she direct that petition to the dean’s office for your review. The college may choose to cover the cost of the post-candidacy enrollment for an individual student and we will work with the college to manage that process. I remind you that according to the current budget model, the net marginal revenue to the college will be positive for a student who enrolls for three hours and for whom the college covers the (standard) fee au­thorization.

We continue to communicate this policy to students and to graduate programs and appreciate your assis­tance with that process. Please let me know if you have further questions.

Sincerely,

Pat

250 University Hall
230 N. Oval Mall
Columbus, OH 43210-1366
PH:(614) 247-7413
FAX:(614) 292-3656

Currently, students get into their post-candidacy period as soon as possible and must keep 3-hour enrollment per quarter. Before that, students were urged to do their candidacy a year before defending their dissertation (so the candidacy started the "writing year"). In general, students were asked to maintain 15-hour enrollment (because it was good for individual department budgets). But then the state got involved and started to crank down on the maximum number of hours a doctoral student could accumulate... and things went down hill from there. So now we have early candidacies and lots of mandatory 3-hour enrollment. By the way, if you didn't get the hint, students don't actually take any classes during these 3- or 15-hour enrollment periods; they take (and someone pays for) "research hours". "Research hours" are usually not given letter grades (although some universities give them letter grades, which helps graduate students to buffer their GPA's against some classes that they actually do have to take), and they rarely require any deliverables to pass them. They are throw away classes invented by bean counters.

Dr. Bernoulli gets a job: Mathematics of the Job Search – Faculty Version

I recently found out that the Duke Computer Science department had 404 applicants for the open position in their department. I mentioned that to a CS professor from a different university, and he didn't seem surprised by that number. Moreover, when you think about how many "faculty candidate" lectures there usually are within a CS-like department each hiring season, and you consider that those interviewees are likely a small selection of the total applicants, then 404 starts sounding reasonable.

When there are 404 applicants who each have PhD degrees, publications, and possible post-doctoral or existing faculty appointments, let's also assume that the objective function that each department is maximizing is pretty flat. If you don't like that assumption, then assume we have no prior information, and so we will maximize entropy and assume that each applicant has a 1/404 chance of being picked for the job (in reality, this probability is itself conditioned on whether the state steps in and has a hiring freeze... so the real probability might be closer to 1/1000). So that is a very low number. Can we fight low probability with high volume of applications?

Assume we apply to N schools where the probability of getting an offer is

p = 1/404

at each of them. Then the probability of not getting an offer from each of them is

1 - p = 403/404,

and so the probability of not getting an offer from all of them is

(1 - p)^N = (403/404)^N.

So finally we arrive at the probability of getting an offer from at least one of them, which is

1 - (1 - p)^N = 1 - (403/404)^N.

Hypothetically speaking, let's say you apply to N = 50 such positions. Then you have a

1 - (403/404)^N = 1 - (403/404)⁵⁰ ≈ 11.65%

probability of getting the offer. Of course, if you were paying attention, you remember that p (1/404) is very small in this example. Consequently, the (1 - (1 - p)^N) curve looks linear for a wide region around the origin. So even though you remember your fourth-grade math teacher teaching you that you cannot additively accumulate probabilities (i.e., your probability of getting a job is not (N × p)), in this small-p case, it is a pretty decent approximation. In particular, even with our ostensibly large N, it is the case that

N × p = (50)(1/404) ≈ 12.38%,

which is pretty close to our slightly more dismal 11.65%.

In December, I ran into a woman who just got finished submitting all of her faculty positions. She said she applying to just 10 of them because she was exhausted and figured she was just practicing this round. Setting N = 10 reduces your chances to 2.45%. Having said that, the distribution across the applicant pool is certainly not flat. Her home institution, research, adviser, and other factors make her a very attractive candidate who will likely do well with such a low N... In fact, she was recently interviewed at a university near me (that, again, may have to deal with hiring freezes, etc., in the near future).

Now, in my case... Maybe I should burn my CV and dust off my résumé... I hope I'm not too old and outdated.

Microsoft's version of Google Scholar?

I accidentally ran into this today. Microsoft has their own version of Google Scholar:

• Microsoft Academic Search (aka Journalogy(.com)?):
  • http://academic.research.microsoft.com/
  • http://journalogy.com/

Among other things, it will even generate aggregate statistics on a particular author you're interested in (number of publications, number of citations, etc.). In fact, it can even generate similar aggregates for journals and conferences (which augments the metrics you can already get from ISI databases).

Unfortunately, some of the candy tools (co-author graph and path – tools that have little functionality but lots of coolness) require Silverlight. I tried running them with Moonlight, which crashed Firefox but seemed to work in Chrome. I say "seemed" because the Silverlight/Moonlight applet loaded fine but was populated with no information. Moreover, doing searches within the applet also returned no information. However, I haven't tried it on a Windows (nor Wine) machine for comparison, and so maybe co-author graphs/paths just aren't ready for production yet. I realized yesterday that it might be wrong to interpret MAS as a product for research so much as a product still being developed within Microsoft Research.

LIBRARY ACCESS UPDATE: As of March 11, 2011, it is very possible that your university's library proxy is not yet configured to allow access through Microsoft Academic Search. If you try to access the search engine through your library proxy and it fails at the MAS address, try it again at the Journalogy address. Strangely, these two names resolve to the same address, but neither is a CNAME. Moreover, neither uses a HTTP redirect to the other. Regardless, many library proxies search their database of allowable hosts by name, and so trying either name may help. If neither name works, contact your library and have them add MAS.

natbib-compatible BibTeX style (BST) file for Springer LNCS publications

UPDATE: Additionally, if you prefer "References" instead of "Bibliography", you have to re-define the \bibname macro in your document (e.g., just after you include natbib). Just add a

\rewewcommand\bibname{\refname}

or a

\renewcommand\bibname{References}

and you should be up and running with your desired bibliography name.

For some reason, the folks over at Springer do not like to make their BibTeX style (BST) files natbib compatible. This omission seems egregious when considering the kind of authors that would submit to journals and conferences that use LNCS-style formatting. Springer does provide LaTeX support files for LNCS, but the included BST file is not natbib compatible (because it was not built with natbib and author–year extensions turned on, which are needed in natbib even when numbered references are used). What makes things more difficult is that it is produced by hacking another BST (titto.bst) that was originally generated properly using makebst, and many of the hacks were already implemented natively in merlin.mbs. So much more of the postfix BST language was hacked directly rather than would have been necessary if the correct docstrip driver options were picked in the first place. So that makes it more challenging to reproduce a refactored version with additional natbib functionality without worrying about introducing regressions.

Nevertheless, I've done my best, and I've made the the natbib-compatible result splncsnat.bst available for download. I was able to remove the need for some of the manual editing by a smarter choice of docstrip options, but I still ended up having to create a patch on top of a stock docstrip-generated BST (the docstrip driver splncsnat-unpatched.dbj and patch splncsnat.patch are also available). Hopefully that helps someone out there.

• splncsnat.bst: natbib-compatible BST file for Springer LNCS-type publications

  Download and place splncsnat.bst in same directory as document's TeX source code. In the TeX preamble of the document, use

  \usepackage[numbers]{natbib}
  \bibliographystyle{splncsnat}

  and then in the text use macros like

  \cite{smith77}       % to get a "[1]" in the text
  \citep{smith77}      % to get a "[1]" in the text
  \citet{smith77}      % to get a "Smith [1]" in the text
  \citeauthor{smith77} % to get a "Smith" in the text

  The normal \bibliography{FILENAME} can be used at the end of the text where the BBL will be inserted by BibTeX.

• splncsnat-unpatched.dbj: docstrip driver used to generate splncsnat-unpatched.bst
• splncsnat.patch: patch used to generate splncsnat.bst from splncsnat-unpatched.bst (assuming version 4.20 [2007/04/24 (PWD, AO, DPC)] of merlin.mbs)

Of course, you need only splncsnat.bst to get up and running.

More Quick Reference Cards

Remember that time when I was quick reference card happy [1, 2, 3, 4, 5]? Today, I accidentally found another good quickref card blog post by someone else (Refcards by Michael Goerz; see the original post for the source code and other versions of these cards):

• Fortran 90 (f90) Quick Reference Card
• Vim Quick Reference Card
• Python 2.5 Quick Reference Card
• Vim-LaTeX Suite Quick Reference Card
• Perl Quick Reference Card
• ASCII Quick Reference Card

He also includes a few quick reference cards not written by him (Subversion (SVN), GDB, GNU Emacs, MySQL).