Come to Kendall at 7:30 on Tuesday night, March 2nd, to watch the MHC basketball team square off against the MHC staff and faculty! It's the best and worst of basketball, all rolled into one. We try to organize a game every year to benefit a charitable organization. This year, Cause will be collecting voluntary contributions at the door (suggested donation: $2, additional generosity much appreciated) to benefit Haiti. Hope you can make it.
I've wrestled with kpartx before. This time I thought I might lose what hair I have left. This post is a little wonkish, but as before, I hope posting about my experience will help some other sap someday.
For the uninitiated, kpartx is the linux utility called by udev to assign device names to disk partitions when using multipath. Kpartx has a -p option which is used to assign the string you'd like to use as the partition delimiter in these device names. So say you have a disk called 'foo', with three partitions, and you give kpartx the string 'bar' as the argument to -p. Then you should end up with device names under /dev/mapper that look like foobar1, foobar2, foobar3. Fair enough.
In the process of setting up a new server, we elected to ensure we knew how to configure this by setting the delimiter to 'quack'. It worked fine. Of course 'quack' is not what we really wanted in production, so a few weeks later we set this back to plain old 'p'. After a reboot, however, the device names came up 'quack' again. (!) (@$%#$%). If we cleared the device names (multipath -F) and recreated them (multipath -v2) they came back the way we expected. We searched high and low for where 'quack' could be hiding, but couldn't find it.
Finally, I sent a query to the dm-devel list. These are the linux programmer gurus who write the device mapper utilities. Mike Anderson from IBM sent this helpful reply: "It sounds like there is a copy of the old rules in your initrd." Bingo! Between our original test and when we tried to revert, I had compiled a later version of the kernel for this server, which of course updated my initrd. Another compile with kpartx configured correctly, and the machine now boots with the correct names.
There's a nice excerpt on blogging from a new book by Scott Rosenberg on Salon. In it, he deconstructs the oft-heard criticism of blogging that "Self-publishing by someone of average talent is not very interesting" - a common lament of traditional publishing venues feeling besieged by the outpouring of mediocre commentary. The problem, according to Scott, is that people have their metaphors all mixed up. Instead of comparing blogging to traditional one-way communication paradigms like television, it should really be considered a new type of telephone. People value active two-way interactive conversation at least as much as passive one sided reception of broadcast style messages, he argues. In that light, criticizing a blog because it has a tiny audience is akin to criticizing a telephone conversation because the only person who cares what you are saying is Mom. Complaining about "too many blogs" is akin to complaining about too many telephone conversations. Of course, blogs have the potential to cross the public/private line, so the metaphor isn't exact, but it's nevertheless illustrative. This blog, in keeping with the on-line conversation metaphor Scott invokes, would direct you to Scott's article or book for further discussion.
At a five college network meeting yesterday, Maria Toyofuku told us about a unique music performance planned for this Saturday, May 9th. A special topics course at Amherst this semester, led by Jason Robinson, has been working on using some new network technology that enables real-time musical collaboration between distant participants.
People have used the internet to facilitate their musical collaborations for quite some time. You send me an mp3, I compose and record my response, then send it back. This is different, because the participants are collaborating in real time. One of the key concerns with real time collaboration is that there be no delay or jitter in the audio signal. If you are having a phone conversation, and the audio signal is delayed by 1/4 second, you probably won't notice. Imagine the same thing happening with a music ensemble, however. If I hear your drum beat 1/4 too late, when you hear my instrument's sound return, it will be delayed by the original 1/4 second, plus whatever delay is introduced on the return journey. Playing music this way would not be fun.
It will be very interesting to see how well this technology works. The network protocols upon which the internet is built do not support quality of service traffic prioritization. In other words, you can't say "this is an audio stream, delay and jitter is unacceptable" vs. "this is an email message, delays and even out-of-order packet transmission are perfectly fine." All internet packets receive the same prioritization (well, there are exceptions, but nothing worth going into right now.) It seems to me that the only way this can work well is if you have fat enough pipes. It's like making a boat go fast by strapping in a really big engine, vs. building something really streamlined. This collaboration, therefore, may only be possible because of the five college fiber network. It would be interesting to see how this technology works between more remote locations.
If you are interested in hearing how this all works out, you can listen to the performance here. Because setup is complicated, and this is new, they are not sure exactly what time the performance will start, but it will be sometime between 5pm and 7pm on May 9th. The exact starting time will be announced on the website at least 30 minutes prior to the performance.
We're about to enter a new phase of our blogging initiative. Some of the preparations may temporarily affect the availability and appearance of this site. Over the next several weeks, we'll be making some minor adjustments to the back end software, installing a new front page theme, making some new and upgraded themes available, and cleaning out some of the temporary testing blogs we created. Stay tuned for more details; and thanks for your patience while we make these modifications.
A few weeks ago, Dennis Bowen, Christopher Woods and I attended a NERCOMP event in Portsmouth, NH titled Directory Services Best Practices. There were two morning sessions about how to customize LDAP schemas to accomodate local directory requirements. It was a good introduction, but personally, I found it a bit boring, as this was familiar territory. In fact, one of the presentations bore an uncanny resemblance to an article on customizing LDAP schemas I had written a couple of years ago. After breaking for a rather meager lunch, I was beginning to wonder why I had signed up for such a long car ride.
However, the afternoon session was well worth the price of admission. Stephen Carmody is the project director of the Shibboleth project, an important Internet2 initiative which aims to provide a "web single sign-on across or within organizational boundaries. It allows sites to make informed authorization decisions for individual access of protected online resources in a privacy-preserving manner". Stephen's presentation gave us an overview of the project's current status, and provided a lot of good information about a number of SAML enabled services that are coming online which are of interest to the higher educational community.
We have had many informal internal discussions about Shibboleth for quite some time. While it has many advantages over traditional authentication and ad-hoc authorization mechanisms, it has always suffered from the usual chicken-and-egg impediment to adoption: no-one wants to invest in implementing an access solution until it is supported by the applications we use, and the application providers don't want to develop an access solution which no-one has the infrastructure to support. What Stephen made clear was that a number of commercial organizations and federal agencies are breaking the logjam, by provisioning SAML-mediated access to their content and services. (Shibboleth is basically an open reference implementation of the SAML protocol.)
On the commercial side, we have Google providing SAML Single Sign-On (SSO) to Google Apps. Apple is piloting federated access to iTunes U. Microsoft is making its DreamSpark suite of software development tools available to students available to students at no charge, and supports SAML assertions as proof of their status. Federal agencies, propelled by the E-Government Act of 2002, are also getting into the game. Both NIS and NSF will, in about sixteen month's time, use SAML assertions to provide access to grant proposals. The gorilla in the room, for higher ed, is FAFSA, which is looking to SAML to control access to student financial aid information.
I don't have time right now to write up all of the details of how this all works, but will do so in the near future. In the meantime, the important point is that we can no longer consider Shibboleth a solution in search of a problem. Quite the opposite - it appears imperative that we implement Shibboleth (or another compatible SAML implementation) if we want to provide our institution access to any number of important on-line electronic resources.
Interesting dittodot today. Reports from Brian Krebs' Security Fix blog convinced the ISPs providing traffic to major US-based hosting provider McColo to shut them down. Apparently McColo hosted a lot of big time operators. You can see this by looking at a recent graph of reported spam activity.
I sure hope someone gives Brian Krebs a big Christmas bonus!
We've completed the first phase of our new blogging initiative - selecting a blogging tool. We'll be implementing Roller as a campus wide blogging solution starting this fall. You are reading a Roller blog entry on our pilot server right now. We still have to develop a more detailed implementation plan before this goes live. This includes things like developing some good templates, deciding how we want to phase our implementation, and discussing procedural and policy issues.
We will be able to support both individual and group blogs. A number of people can be given permission to create and edit entries on behalf of an entire department, for example. This is a great way to "get the word out" about what is going on around campus. Watch for more news about this initiative soon.
We launched our upgraded version of Sakai, a.k.a. 'ella' on Thursday, July 31st. If we were going to pull this off, it had to be by this date, in order to give faculty enough time to prepare their courses for the new semester. We were previously running Sakai version 2.3, and are now running version 2.5. Some of the noteworthy improvements include a workable testing/quizzing module and a citations helper component that has been added to the Resources tool. You can also insert mathematical equations into the wiki editor using TeX. I've already entered a few journal entries about the installation process. I plan to add more as time permits. I have a few other pressing items to attend to first, including taking some time off.
I'm not going to cover Apache HTTP server configuration in excruciating detail; I'm only going to review how to configure Apache to proxy the Tomcat AJP connector we configured.
If you are investigating how to proxy AJP via the Apache HTTP server for the first time, you'll discover that there are several modules to choose from: mod_jk, mod_jk2, and mod_proxy_ajp. I use mod_proxy_ajp. You might consider using mod_jk, but don't waste your time with mod_jk2, as it is deprecated and development has been discontinued. The mod_proxy_ajp module is distributed with the Apache HTTP 2.x server and is much simpler to configure than mod_jk.
You have to load mod_proxy and mod_proxy_ajp. I include a few extras.
LoadModule proxy_module /path/to/modules/mod_proxy.so LoadModule proxy_connect_module /path/to/modules/mod_proxy_connect.so LoadModule proxy_balancer_module /path/to/modules/mod_proxy_balancer.so LoadModule proxy_ajp_module /path/to/modules/mod_proxy_ajp.so LoadModule proxy_http_module /path/to/modules/mod_proxy_http.so LoadFile /usr/lib/libxml2.so
Our Sakai service is configured as a virtual host where the root location is configured to proxy everything to the Tomcat AJP connector as follows:
<Location /> ProxyPass ajp://localhost:7089/ ProxyPassReverse ajp://localhost:7089/ ProxyPassReverse / ... </Location>
I also disable forward proxying, and I pass the original host header to the backend server.
ProxyRequests off ProxyPreserveHost on <Proxy *> Order deny,allow Allow from all </Proxy>
Sakai is designed to run as an Apache Tomcat web application. The version of Sakai we are installing, 2.5.1, specifically requires Apache Tomcat 5.5.26. It also requires that you use Java 5, aka 1.5. Java 6, aka 1.6 will not work. I'll often have multiple Java versions installed on the same machine, so I use a couple of simple bash scripts to set my Java environment appropriately for each context. I have one script to set my environment for building Sakai, and another script that my Tomcat startup scripts use to to set the environment for running Sakai.
I source the build environment script to be sure I'm set up to use the proper Java installation when building Sakai. The Tomcat runtime environment is sourced from Tomcat's startup.sh script, which I'll describe below. Of course you'll have to adjust these scripts to match your local setup. In particular, the Tomcat JAVA_OPTS variable setting for -Dsakai.home should point to the location of the sakai.properties file you use to configure Sakai.
I modify Tomcat's startup.sh and shutdown.sh to source the environment script I mentioned above. I also modify these scripts to run Tomcat as user 'sakai'. I do this like so:
# replace this line # exec "$PRGDIR"/"$EXECUTABLE" start "$@" su - sakai --shell=/bin/bash -c ' \ source /local/etc/sakai.stella/mhc/environment.sh && \ exec /local/apps/tomcat-sakai.stella/bin/catalina.sh start'
Similarly in my shutdown.sh script, I do:
# replace this line # exec "$PRGDIR"/"$EXECUTABLE" stop "$@" source /local/etc/sakai.stella/mhc/environment.sh && \ exec /local/apps/tomcat-sakai.stella/bin/catalina.sh stop
My Tomcat init script is nothing fancy, but it works.
I don't serve http directly from Tomcat. Instead, I configure a Tomcat ajp connector, which I then proxy through Apache's HTTP server. I do this because (1) It's my understanding that the Apache HTTP server is written better, and (2) using the Apache HTTP server mod_proxy module is how you load balance a cluster of Tomcat servers. I have not yet had to cluster my Tomcat setup to handle our load, but I want to be ready to do so should the need arise. (3) Also, I find https certificate management in the HTTP server to be much easier than dealing with certificates in Tomcat.
It is also important to learn about the various Tomcat connector configuration options, and to set them appropriately for your installation. My server.xml connector configuration looks like this:
<Connector protocol="AJP/1.3" scheme="http" address="127.0.0.1" port="7089" redirectPort="7443" maxThreads="160" minSpareThreads="20" maxSpareThreads="40" minProcessors="25" maxProcessors="500" backlog="200" enableLookups="false" disableUploadTimeout="false" connectionTimeout="60000" URIEncoding="UTF-8" />
I'm using Oracle, so of course it's necessary to install the jdbc driver, which you can download on Oracle's site. I put ojdbc14.jar into Tomcat's /common/lib directory.
Another thing I do is compile and configure Tomcat to use the Apache Portable Runtime (APR). If you do this on Linux on Tomcat 5.5.26 there are a few autoconf bugs you need to squash before you can successfully run the compile. Go to your Tomcat installation's /bin directory, and unroll tomcat-native.tar.gz. Change your working directory to tomcat-native-1.1.12-src/jni/native. If you are on Linux, edit 'configure' and 'build/tcnative.m4' and change --Wl to -Wl. In the same two files, also replace
IFS=.; set $sapr_version; IFS=' ' with tc_save_IFS=$IFS; IFS=.; set $sapr_version; IFS=$tc_save_IFS
Make sure you have OpenSSL development libraries installed, and also that you have Apache's APR available. Then do the usual configure & make dance:
./configure --with-apr=/local/bin/apr-1-config && \ make && \ make install
Finally, I like to put a simple index.html file in Tomcat's webapp/ROOT directory that will automatically redirect anyone going to the default location to the Sakai portal page.
Sakai can use either MySQL or Oracle as its database repository. Our first production installation of Sakai 2.1 used MySQL, but we have since upgraded to Oracle 10g. I wrote a set of scripts to assist with the MySQL to Oracle migration when we moved from 2.1 to 2.3. Let me know if you'd like a copy.
We have two separate Oracle servers, one production and one for backup/testing, both Dell 2950's w/ 16GB of RAM and a couple of quad core Xeon processors. The database files themselves live on our 4Gb SAN with lots of spindles, but would also easily fit on local drives. Our servers are running Redhat AS4.
I maintain different schemas for testing, backup, and production; each in their own tablespace. I use one script to set up my table spaces and to create an application owner role which is assigned to each user. I maintain separate setup and drop scripts for each user, i.e. I have similar create and drop scripts for my test and backup users.
Sakai's database connection is configured in the primary Sakai configuration file, sakai.properties. You also need to be sure to put the oracle JDBC driver somewhere Tomcat can find it. I'll cover this in more detail when I discuss Tomcat configuration.
I use Oracle's data pump to do a nightly dump of my Sakai database on my production server. This dump is then rsync'd to my test/backup Oracle server.
My import script and an example parameter file:
I use the parameter file above to import data from schema 'SAKAI' (my production data) into schema 'STELLA' (testing).
To upgrade from 2.3 to 2.5 I first migrated my data from my production server to my test schema on my test server. I then ran the migration scripts provided with the Sakai 2.5 source (located in reference/docs/conversion), like:
sqlplus stella/<password> @sakai_2_3_0-2_3_1_oracle_conversion.sql sqlplus stella/<password> @sakai_2_3_1-2_4_0_oracle_conversion.sql sqlplus stella/<password> @sakai_2_4_0-2_5_0_oracle_conversion.sql
The assignments tool needs some special attention during the upgrade. See the readme file in the Sakai source directory for assignments for more details. The basic steps are:
Edit upgradeschema_oracle.config to match your installation:
:test dbUser=myuser dbPass=mypassword
Make sure your CLASSPATH includes your oracle JDBC driver.
Run conversion script:
time ./runconversion.sh upgradeschema_oracle.config > upgrade-output.txt
It is also advisable to upgrade your Content Hosting to convert existing XML blobs to Sakai's new format, which helps performance. The scripts and readme for this procedure are in the Sakai source tree's /content directory.
Edit upgradeschema-oracle.config as appropriate (same as assignments above)
Also edit ../db/db-util/conversion/runconversion.sh, and make sure lines that edit CLASSPATH point to actual locations of jar files in maven repository (i.e. ~/.m2/repository) I had to change SNAPSHOT to M2, for example. The details will probably change w/ subsequent upgrades.
nohup ./content-runconversion.sh \ -p /local/etc/sakai.stella/sakai/sakai.properties \ upgradeschema-oracle.config &> content-runconversion.log &
If you use any third party tools, you'll have to migrate that data also. We use melete. See melete/readme/readme_upgrade.txt, specifically section 7, for upgrade details.
After testing that the data was migrated successfully, I then dump the data from our test server and restore to our production server. I'm using a new schema name for the upgraded 2.5 production data than for the 2.3 production data. I.e. - I'm keeping my old production data available in case something truly catastrophic happens with our upgrade and we need to revert to our previous installation.
I am currently upgrading Mount Holyoke College's Sakai installation, which we call 'ella', from version 2.3 to version 2.5. While the Sakai project provides installation instructions, these instructions omit many of the details required to implement a functional installation from source. During the process of performing our current upgrade, I've tried to make note of the particular details required to get things running. I'm going to try to assemble these notes into a legible series of posts about what it takes to get a full-blown Sakai installation up and running. If nothing else, this will help me later when I have to re-remember what I did. Hopefully these notes will also help others navigate their way through what can be a somewhat daunting setup.
We use Oracle for our database; MySQL is another option, but I won't describe that here. The Sakai application itself is deployed to Apache Tomcat, which we then proxy via the Apache HTTP server. I maintain our modified copy of the Sakai codebase in our own local Subversion repository. I also use Subversion to store the various configuration files and scripts we use. There's a lot of ground to cover, so I'll break the whole thing down into a series of individual posts targeting specific aspects of our setup.
Mount Holyoke College is an all women's liberal arts college located in western Massachusetts. We have about 4500 active accounts on campus. Our Sakai installation hosts about 700 courses per semester. We also use Sakai to host a number of non-course related project sites. These include student organizations, facilities projects, and various administrative workspaces. We run this on two Dell 2950's each having two quad core Xeon processors and 16GB RAM. One server hosts our Oracle installation, which hosts its database files on a 4Gb fiber channel SAN. The other server runs an Apache 2.2 HTTP server, which serves as an AJP proxy for a single Apache Tomcat instance running on the same server. We have an identical parallel setup for backup and testing. Our production instance is called 'ella', and our testing instance is called 'stella'. I thought I'd mention that, because you may see reference to either name in the posts that follow.
Getting Debian Etch to play nice with partitioned drives on a fiber channel setup requires making some modifications to the supplied udev scripts which handle multipath setups.
My first naive attempt to write the udev rules to handle creating device nodes for our fiber channel disk partitions looked like this:
PROGRAM="/sbin/devmap_name %M %m", \
RUN+="/sbin/kpartx -a /dev/mapper/%c"
This doesn't work. Hannes Reinecke kindly explained the problem to me on the dm-devel list:
Problem being the 'interesting' device-mapper design. Whenever a 'table create' operation is triggered from user-space, it's actually being broken down by the device-mapper library into 4 distinct operations:
1. create device dm-X
2. suspend device
3. load table definition
4. resume device
and you'll be getting an 'add' event for the first operation, and a 'change' event for the second and the forth operation. But clearly the 'add' event is pretty much pointless, as there is not table loaded at that time, so you wouldn't know what to do with it. And the device is locked between the second and the fourth operation, so kpartx would refuse to touch it.
So, in essence: 'add' is evil for device-mapper devices. Use the 'change' event and check if the dm-device is not suspended when you get the event.
The easiest way to accomplish this is to obtain a good kpartx.rules udev script from a recent distribution. IIRC, I got mine from Debian unstable. It won't work out of the box on Etch, though, because it calls scripts and functions that don't exist on Etch, so you have to get those too. I downloaded and installed the latest device-mapper source, and also retrieved the required /lib/udev/kpartx_id and /lib/udev/dmsetup_env scripts. Now kpartx reliably creates disk partition device nodes when udev receives kernel notifications about disk change events.
Hopefully google will help someone else find this note, and save them the trouble I went through.