Archive for the ‘Hacking’ Category

PyDev Status (1.1.0)

Wednesday, June 21st, 2006

In the past few weeks I spent some time preparing patches that backport PyDev (the Eclipse Python plugin) from Java 1.5 to 1.4 so that it can be shipped in Fedora, since for some time now PyDev has been developed on Java 1.5. Last week I updated my patches to PyDev 1.1.0, which is the latest version that will support Eclipse 3.1.2, and so the latest version that can go into FC5. Last week version 1.2.0 was also released, which is the first version that will be targeted for Eclipse 3.2, which is what will get into FC6.

I spent some time last week trying to make PyDev has sane defaults out of the box for settings like the path of the Python and Jython interpreters, I got something sort of working, but it wasn’t quite there yet. I’m planning to push PyDev into Fedora Extras in FC6, since it will now have an optional dependency on Jython. But I’m wondering whether I should release 1.1.0 as a Core 5 update, because it really is a *huge* update in stability and functionality over the version we currently we have in. I’ll probably try to make a test RPM that has the Jython bits ripped out tomorrow and push it into updates-testing, and see what kind of reaction I get.

Over the weekend or maybe next week I’ll probably prepare the PyDev 1.2.0 patch and a corresponding SRPM, that I’ll submit to extras pending on my other submissions (Jython and friends).

PyDev 1.0.6 backporting

Thursday, June 1st, 2006

As I mentioned 2 weeks ago, I decided to backport PyDev 1.0.6 to Java 1.4 so that we could include it in Fedora. I’m glad to say that I’m done, and the patches are available here . Fabio, the author of PyDev, has released versions 1.0.7 and 1.0.8 in quick succession last week, which are bug fix releases, so I’ll be updating my patches for that shortly.

Right now I’m working on getting Jython into Fedora Extras because PyDev has Jython support. I could disable that support, and push the PyDev update into Core. But I decided it would be better to get Jython in extras, it’s a pretty important package I think. What I’ll probably end up doing once that’s done is splitting the PyDev SRPM in to several binary RPMS and make the eclipse-pydev-jython one depend on Jython itself.

If anybody is willing to be the shepherd of any of the 5 packages that include Jython and it’s dependencies in Extras let me know, as I probably won’t have time to maintain them.

Migrating MovableType to WordPress 2.0.2 (or The Good, The Bad and The Ugly)

Thursday, June 1st, 2006

Over the last weekend and the week before that I spent a lot of time trying to migrate the blog at Pyre to use WordPress instead of the ancient MovableType installation it was using before. Unfortunately it wasn’t a very smooth ride, so I’ll write my conclusions here so that they could be useful for others.

Exporting from MovableType
The first step in the process was to export the post information from MovableType. This wasn’t really a hard step and the WordPress Importing from MovableType tutorial does a very good job of explaining this step. Unfortunately this didn’t quite work for me because MovableType doesn’t export post id #’s into this file. Why would you want that you ask?

Well…

Apparently MovableType hosts multiple blogs in the same database and uses the same tables for all the blogs. It actually goes even further and has global post id numbers so that the sequence of id’s for any one blog is not consecutive, you’ll have breaks in the numbering wherever someone posted to one of the other blogs. Huh! So maybe I should export all of the blogs at once? Well it turns out that you can’t. No single user, including the administrator can export all the blogs, they can only export the ones they “own”, which I guess means they started.

*sigh*

Importing into WordPress
This means that when I import entries into WordPress the post id #s will be completely different and maintaining links will be close to impossible.

At this point I went through the exercise of inserting my foot into my mouth from 10 different angles, after having said that the importing step “should be a breeze”. I tried writing a program to fetch all post numbers and titles from the old blog, trying to compare to how they import into WordPress and then making a file with all the correspoinding ids. In the end it was decided that maybe if we only preserve internal linking it would be OK. At that point I went and wrote a program that would go through the exported MT entries and substitute the correct addresses from my previously created massive file. This had it’s own problems.

In the end everything ended up working out quite nicely. Here’s how.

The Final Solution

First I found a very useful hint on how to export post ids from MT, which worked like a charm.

I then proceeded to to hack up WordPress to import the entries correctly. The hints at the website above gave me a good idea of what needs to be done but it was a bit old, using WordPress 1.0 instead of 2.0. Here is a patch for my WordPress hack. This also inserts print statements that show you that everything is indeed going according to plan. I’m going to explain the main points of the patch below, but these are copy/pasted parts of the patch that won’t apply if you use them by themselves, please use the full patch.

The first part of the patch is for the wp-includes/functions-post.php file, which defines functions for inserting a post into the database. The first chunk of the patch makes WordPress think that it’s creating a new post, instead of updating, by commenting out the update code.

@@ -16,11 +16,13 @@

// Are we updating or creating?
$update = false;
-   if ( !empty($ID) ) {
-       $update = true;
-       $post = & get_post($ID);
-       $previous_status = $post->post_status;
-   }
+   // igorfoox
+   printf(__('In wp_insert_post, ID is %d'), $ID);
+// if ( !empty($ID) ) {
+//     $update = true;
+//     $post = & get_post($ID);
+//     $previous_status = $post->post_status;
+// }

The second chunk makes WordPress always set the post id to the id we extracted from the comment.

@@ -45,8 +47,8 @@
$post_status = 'draft';

// Get the post ID.
-   if ( $update )
-       $post_ID = $ID;
+   //if ( $update )
+   $post_ID = $ID;

The third chunk is just a strategically placed print statement. The fourth chunk adds the post id to the fields that are inserted into the database.

@@ -142,12 +145,14 @@
menu_order = '$menu_order'
WHERE ID = $post_ID");
} else {
+       printf(__(' Running insert query'));
$wpdb->query(
"INSERT IGNORE INTO $wpdb->posts
-           (post_author, post_date, post_date_gmt, post_content, post_content_filtered, post_title, post_excerpt,  post_status, comment_status, ping_status, post_password, post_name, to_ping, pinged, post_modified, post_modified_gmt, post_parent, menu_order, post_mime_type)
+           (post_author, post_date, post_date_gmt, post_content, post_content_filtered, post_title, post_excerpt,  post_status, comment_status, ping_status, post_password, post_name, to_ping, pinged, post_modified, post_modified_gmt, post_parent, menu_order, post_mime_type, ID)
VALUES
-           ('$post_author', '$post_date', '$post_date_gmt', '$post_content', '$post_content_filtered', '$post_title', '$post_excerpt', '$post_status', '$comment_status', '$ping_status', '$post_password', '$post_name', '$to_ping', '$pinged', '$post_date', '$post_date_gmt', '$post_parent', '$menu_order', '$post_mime_type')");
+           ('$post_author', '$post_date', '$post_date_gmt', '$post_content', '$post_content_filtered', '$post_title', '$post_excerpt', '$post_status', '$comment_status', '$ping_status', '$post_password', '$post_name', '$to_ping', '$pinged', '$post_date', '$post_date_gmt', '$post_parent', '$menu_order', '$post_mime_type', '$post_ID')");
$post_ID = $wpdb->insert_id;
+           printf(__('||| end of insertionn with id %d'), $post_id);
}

The second part of the patch makes WordPress extract the post ids from the exported MovableType data. The first chunk ads the ‘ID’ field as a parseable token

@@ -222,6 +222,9 @@
case 'AUTHOR' :
$post_author = $value;
break;
+                       case 'ID':
+                           $ID = $value;
+                           break;
case 'TITLE' :
$post_title = $wpdb->escape($value);
break;

Then we also send this information to the wp_insert_post function in the second chunk

@@ -281,8 +284,9 @@

$post_author = $this->checkauthor($post_author); //just so that if a post already exists, new users are not created by checkauthor

-                   $postdata = compact('post_author', 'post_date', 'post_date_gmt', 'post_content', 'post_title', 'post_excerpt', 'post_status', 'comment_status', 'ping_status', 'post_modified', 'post_modified_gmt');
+                   $postdata = compact('post_author', 'post_date', 'post_date_gmt', 'post_content', 'post_title', 'post_excerpt', 'post_status', 'comment_status', 'ping_status', 'post_modified', 'post_modified_gmt', 'ID');
$post_id = wp_insert_post($postdata);
+                   printf(__('Id was %d, wanted %d'), $post_id, $ID);
// Add categories.
if (0 != count($post_categories)) {
wp_create_categories($post_categories, $post_id);

These things will probably slightly change with new versions of WordPress, but hopefully it stays similar enough so that people can figure out how to apply it.

Finally I also had to make sure that the old blog feed remained available, because WordPress uses a different address for its feed. I accomplished this with the following .htaccess rule

RewriteRule ^index.rdf$ /blog/rdf [R,L]

I know, it’s not so subtle, doing a full redirect to get the feed right, but I couldn’t figure out how to do it in any other way. What seems to be happening is that WordPress’ internal redirect system interferes with this rule unless I do a full redirect. If anybody has a more elegant solution, I’ll be glad to hear about it.

I think this needs to be put up on the WordPress wiki, because the information there is sooo out of date. Unfortunately I spent way too much time working on this, and so I don’t have that much time to do it, but hopefully after writing it all down, I’ll be able to find the time to stick it there as well in the near future.

We need a newer PyDev

Thursday, May 18th, 2006

Red Hat’s Toronto office recently got 6 new interns. It’s going to be a fun summer with 13 interns sitting together in one cube area :D .

Other than yelling very loudly for the whole day (I swear we were quieter last year!!) the interns are working on some improvements to yum. They’ve all used Eclipse before so they naturally wanted to use it. Very quickly they learned that we ship PyDev and so they tried that. Unfortunately we can only ship versions < 0.9.6, and currently we have 0.9.3. Even more unfortunately 0.9.3 sucks. Mostly because there were tons of additions and fixes since then, possibly also because of the way we’re running it, or because of GCJ. In any case, I decided that we *really* need to have a newer version, because this one is pretty close to useless as it is.

In the next few days I’m going to look to backporting PyDev 1.0.6 (latest released) to Java 1.4 and then pushing that as an update to FC5. We don’t have to do this often, but if it only takes 2 days, I can repeat the procedure before the end of the summer (when I’ll be going back to school). And then we could either repeat it again a few months after or just wait until we have real 1.5 language feature support (I don’t think PyDev uses any new library features, except Genericsized collections).

How can Python not have HTML unescaping?

Friday, May 12th, 2006

I’m in the process of trying to migrate a MovableType blog to WordPress. The process itself isn’t that bad, but I’ve found a whole slew of problems with preserving links. In the end it was decided to preserve internal link integrity but not worry about other people’s links. OK.

Due to weirdnesses in how both MT and WP handle post id’s I had to write a bunch of scripts that would extract post ID’s and titles from the old MT blog and a test installation of the WP blog with the imported entries, then compare the titles, and figure out the new URLs of every post that was internally linked to.

In the process of doing that I found out that Python doesn’t have sane handling of HTML unescaping!

Huh? What? You may ask.

Python has nice methods for escaping and unescaping URLs in urllib. But guess what? There’s no one function to do the same for HTML code. Lets see. Anything in htmllib? Nah, that’d be too easy. Anything in urllib, or urllib2, maybe? Just as a favour? Nah…

I’m still shocked that there’s no function for this. In the end I used xml.sax.saxutils.unescape() which does some of the unescaping but doesn’t handle all the HTML entities, who knows why, so I had to add some of the entities that I encountered in the titles manually.

Wow…

Fedora on the Mac Book Pro

Wednesday, April 5th, 2006

Having had a Mac Book Pro for about 2 weeks, I’ve been trying to get Fedora on it. Initially the reasoning was to get a live cd working (using Kadischi) and then hack anaconda to be able to install on to the Intel Macs. The biggest problem we had was to compile the Fedora kernel with the patches from the mactel-linux project.

Now that’s no longer needed. Today Apple released a firmware update that emulates BIOS in order to support their Windows install helper, Boot Camp. My first reaction was: OMG!

Booting the Fedora kernel

Right after that I tried to stick the Fedora Core 5 i386 install DVD and rebooted. The DVD got recognized as ‘Windows’ in the graphical boot menu, it then proceeded to boot the kernel, and start up anaconda. I had to quit it when it wanted to erase my hard drive because it was partitioned with GPT (GUID Partition Table). But now we have something much easier to work with.

Anaconda

Hurray!

New Eclipse Fedora Update Site

Wednesday, March 8th, 2006

Yesterday I uploaded a new update site, made by Jeff Pound, for the Eclipse plugins we ship in Fedora. We now finally have both the ChangeLog and Bugzilla plugins in the same update site, which should be much more convenient. The main problem that we’re seeing with this is that we currently do not support the 3.2 builds, only 3.1.x, this is because the plugin depends on a patched up org.eclipse.compare plugin. The patched version exposes API that is internal in the official Eclipse.org version, and this allows us to apply patches from bug reports to projects in the workspace. Unfortunately these patches by Jeff were largely ignored by the core Eclipse folk, and are not committed to the 3.2 branch. On top of that the code that the patch applies to is still changing, since it’s internal API that the patch exposes and is not covered by the API freeze. So for now we are only supporting Eclipse 3.1 in the Bugzilla plugin. In the near future we may branch it so that we have a cut down version without the patch functionality for 3.2 (which is not that huge of a deal anyway). But we still need the internal API to stop changing for this. I guess we’ll wait and see.

Screencasts of DrProject

Wednesday, March 8th, 2006

Yesterday Pat and I made screencasts for DrProject, these are available here. We used Wink for the screencasts. Unfortunately version 1.0 of Wink doesn’t support sound throughout the cast, and you can clearly hear my lovely voice in those screencasts :) . That’s because we used the 2.0 beta of Wink, which is currently only available for Windows. I have to say that from doing a few of these I get the impression that Wink really rocks! The cruft of doing a screencast took only a tiny proportion of the time that we spent on the whole deal. 90% was spent on figuring out what to say, and trying to do it without hysterical laughter. Right on Wink. I’ll be waiting patiently for this tool to come out of beta so I can use it on Linux.

Jars in Eclipse Plugins

Wednesday, February 15th, 2006

In the past 2 days I’ve been trying to update the Bugzilla plugin before Fedora Core 5 test 3. I’ve been trying to make it play nice with the Red Hat bugzilla through the XMLRPC interface instead of html-scraping. The functionality was already there but I discovered that Eclipse couldn’t find the jar file that contained the Java xmlrpc library , which was located in /usr/share/java. I decided to stick the jar in the plugin and be done with it. Boy was I wrong. It took me a whole day of head-banging on Monday to *not* get it working. Seems like a pretty simple issue, getting a plugin to recognize a jar *inside the plugin*. But apparently getting a jar file onto the runtime classpath is a fundamentally difficult problem. Yesterday morning after talking to some eclipse people on IRC I got a link to this life saving post, that lists 6 steps to ultimate happiness. Half an hour of checking, testing, rebuilding and more testing and I was done.
Granted I should have gone and asked a bit earlier (how about a day?) but the fact that I was tweaking the same settings for a whole day and managed to not stumble on the correct set is also telling.

Blogs, PHP, and Apache

Sunday, February 12th, 2006

So this weekend I finally decided to get my hands dirty and try to learn how to administer Apache, understand PHP, and (hopefully by virtue of the previous things), figure out how WordPress actually works. Phew, it was a long weekend.

I started out by redoing the Fedora Eclipse website, because it badly needed an overhaul. It’s not completely finished yet, but I moved it to pure XHTML/CSS from a table, I also seperated it out into sections so that we can actuallly add meaningful content. Here, take a look at my current permutation. You might notice that it looks awfully similar to the old one. That’s because my CSS skills aren’t that amazing yet to come up with a new design.

After that I read/skimmed about half an Apache book. Most of it obviously went over my head :) . But I finally figured out what and how mod_rewrite works. Yay for obfuscation! After that I read the PHP manual, which was pretty informative. I gotta say that I’m very impressed with PHP, it’s super simple, it’s expressive, and it’s got a lot of very nice modules. Impressive, to say the least. On a side note, looking at OO in PHP5 was almost indistinguishable from looking at Java, except for a few very minor details. I guess all programming languages are slowly but surely converging to The One True Java.

My head was kinda spinning towards midday Sunday from information overdose. Especially given the fact I also have a cold. But I managed to dig through WordPress documentation and figure out why my about me page displayed weird PHP errors when accessed. Even better, I fixed it. So there, I accomplished something useful.
And now, to sleep.

*zzzzz*