I've wrestled with kpartx before. This time I thought I might lose what hair I have left. This post is a little wonkish, but as before, I hope posting about my experience will help some other sap someday.
For the uninitiated, kpartx is the linux utility called by udev to assign device names to disk partitions when using multipath. Kpartx has a -p option which is used to assign the string you'd like to use as the partition delimiter in these device names. So say you have a disk called 'foo', with three partitions, and you give kpartx the string 'bar' as the argument to -p. Then you should end up with device names under /dev/mapper that look like foobar1, foobar2, foobar3. Fair enough.
In the process of setting up a new server, we elected to ensure we knew how to configure this by setting the delimiter to 'quack'. It worked fine. Of course 'quack' is not what we really wanted in production, so a few weeks later we set this back to plain old 'p'. After a reboot, however, the device names came up 'quack' again. (!) (@$%#$%). If we cleared the device names (multipath -F) and recreated them (multipath -v2) they came back the way we expected. We searched high and low for where 'quack' could be hiding, but couldn't find it.
Finally, I sent a query to the dm-devel list. These are the linux programmer gurus who write the device mapper utilities. Mike Anderson from IBM sent this helpful reply: "It sounds like there is a copy of the old rules in your initrd." Bingo! Between our original test and when we tried to revert, I had compiled a later version of the kernel for this server, which of course updated my initrd. Another compile with kpartx configured correctly, and the machine now boots with the correct names.