Ed's Gregarius » persistent.info

persistent.info

Porting Twitter Digest to Google App Engine

Posted: May 11th, 2008, 8:26pm PDT by Mihai Parparita

I've been meaning to play around with Google App Engine for a while, and as a quick project, I decided to port Twitter Digest to it (not as exciting as Kushal's Millidunst Calculator). This looked to be pretty straightforward: the original version was already in Python, and wasn't very complicated (just a single CGI script). It did indeed end up pretty easy; the whole process took a couple of hours.

The first step was to port the script from CGI-style invocation to the App Engine webapp framework. Then I looked into what it would take to get Python Twitter (the library I used for fetching data from Twitter) running. Switching it from urllib2 to urlfetch was pretty painless (though I don't use the posting parts of the API, so I didn't check if those work too). The other part of the library that I was relying on was its caching mechanism (since the digests are daily, there's no point in querying Twitter more often). DeWitt (the library's author) had thoughtfully put the caching functionality into a separate class, so it was easy to replace it with another one that implemented the same interface but was backed by App Engine's datastore.

The result (complete with App Gallery entry) is not that exciting, in the sense that it functions identically to the original. The only issue that I've run into so far is that when there are several cache misses, the URL fetches can take long enough that the request hits App Engine's deadline. However, since the successful fetches are cached, repeating the request will eventually succeed (so if consuming the digest via a feed, this shouldn't be a big deal). Ideally the urlfetch functionality would also support asynchronous fetches, since it would be easy to adapt the code to fetch all user timelines in parallel.
Intern on the Google Reader team

Posted: April 24th, 2008, 8:38am PDT by Mihai Parparita

Having interns has worked out well for the Reader team. Following my blog post, we were very pleased to get Nitin Shantharam and Jason Hall to help us out with Reader development. Their stints on the team resulted in a a bunch of features, and Jason is now back at Google working full-time (Nitin wasn't a slacker, he's just still in school).

We're looking for another intern or two this year. Internships generally last a couple of months to twelve weeks, are for full-time students, and would be in Google's Mountain View, California office. You can work on either Reader's backend (a C++ system for crawling millions of feeds, handling lots of items being read, shared, starred or tagged per second) or frontend (Java servers and JavaScript/AJAX-y craziness) depending on your interests and experience.

If you or anyone you know is interested in this internship, contact me at mihaip at google dot com. This page also has more general information about interning at Google.
persistent.coffee

Posted: April 3rd, 2008, 2:46pm PDT by Mihai Parparita

I've been trying my hand at latte art. Though I have a very long way to go, I've been documenting my efforts, with a hope of learning from my mistakes. Blogger's mobile support makes it pretty easy to collect pictures, and I've finally gotten around to making a decent template for the "blog."

coffee.persistent.info is the result. Technically, this isn't a Blogger template, since I just have some static HTML as the content. Instead, it uses the JSON output that Blogger's GData API supports. Rendering the page in JavaScript allows for more flexibility. I wanted to make pictures that I liked take up 4 slots (a layout inspired by TwitterPoster). This imposed additional constraints (in order to prevent overlap between sequential large pictures). The display is generally reverse-chronological starting from the top left, but images are occasionally shuffled around to prevent such overlaps. There is also a bit of interactivity, the pictures are clickable to display larger versions. To help with all this, I've been experimenting with jQuery (also on Mail Trends), and am liking it quite a bit.
Mail Trends

Posted: March 25th, 2008, 10:32am PDT by Mihai Parparita

I get a lot of email (especially at work). I'm trying a Inbox Zero-like approach in order to keep up with it. Though that's helping me to stay on top of things, I had the nagging feeling that I was probably on too many mailing lists, and that some of them were probably not worth it from a signal-to-noise ratio perspective.

Ideally something like the Reader Trends or Search History Trends page would exist for Gmail. I thought I could perhaps built it myself, but the absence of an official Gmail API deterred me. However, it occurred to me that the recently added IMAP support could act as an API of sorts. It should be easy to get just the message headers and slice and dice them to extract the stats that I was interested in.

Thus was born Mail Trends, an IMAP-based email analysis project. It can generate a bunch of tables, graphs and distributions based on time of day, senders, recipients, mailing lists, etc. To get a feel for what it can output, see the results of running it on a piece of the Enron Email Dataset. To run it over your own email, see the getting started page. As a caveat, the program currently loads everything into memory, so my run on 200,000 messages resulted in 1.6 gigabytes being used. You may want to use the --max_messages= flag to limit the dataset, at list for initial runs.

The project is still in its early stages, so patches and suggestions are definitely welcome (my email address is at the footer). You can also subscribe to the feed of check-ins to see changes as they are made. The plan wiki page has a very brief outline of what I'm planning on working on next.
Two Safari 3.1 Tips

Posted: March 22nd, 2008, 2:23pm PDT by Mihai Parparita
Safari 3.1 is out, and I've upgraded my Mac to it. Besides some issues with arrow keys in Reader (we're on it), it's working out well. Here are two hidden prefs that you may find useful:
```
defaults write com.apple.Safari IncludeInternalDebugMenu -bool true
```
The Develop menu that 3.1 includes is nice, but it seems to supplant the old "Debug" menu (i.e. the preference key that used to toggle it - IncludeDebugMenu - now toggles the "Develop" menu). The old menu had functionality that isn't present in the official one, most notably the "Caches" window that displayed the number of live JavaScript objects and made tracking down memory leaks much easier. If you'd like to bring back the old menu, you can use the new IncludeInternalDebugMenu key shown above
```
defaults write com.apple.Safari TargetedClicksCreateTabs -bool true
```
First spotted on Twitter, this forces new windows to open in tabs, one feature that I missed from my Firefox days.
Decommissioning Mscape Software

Posted: February 18th, 2008, 9:50pm PST by Mihai Parparita

Before I became enamored with web development, I used to be a Mac software developer under the Mscape Software moniker. My first public release (clip2cicn, a helper tool for making the icon resources necessary for Kaleidoscope schemes) was almost 10 years ago, on June 26, 1998. My flagship product was Iconographer, an icon editing tool.

I haven't had time for Mscape Software for about 4 years (the last Iconographer release was in July of 2003, and I last touched the code in early 2004). The site was still up, and I kept receiving registrations (Iconographer was a $15 shareware product). As time went on, I began to feel more and more guilty about not providing any support for (paying) customers. To this end, I have finally gotten around to dismantling Mscape Software, replacing the site with a placeholder with download links for all products. I've also put up a registration code for Iconographer so that the (annoying) registration reminder can be shut off.

I've gotten some requests to open-source Iconographer, but I'm not sure I'll have time even for that. I'm not 100% sure I can build the product given software on hand (it was built with CodeWarrior). Even then, the codebase shows its age (it was my first large-ish project and still uses many Classic constructs like a WaitNextEvent event loop).
plusplusbot: Karma for anything and anyone

Posted: November 16th, 2007, 10:14am PST by Mihai Parparita

The recent release of Foamee reminded me that I had never blogged here about plusplusbot, a little toy of a site that I started to work on this past summer. The idea behind plusplusbot is that you can express your (dis)pleasure with something or someone over Twitter, and the site will keep track of the target's score over time. All of this is done over Twitter, in a barnacle-like fashion. The thinking being that Twitter handles message delivery over a variety of mediums (Jabber, SMS, Web) and provides a ready-made social network to piggyback on (you can view the activity of you and those you follow, for example, here's mine). This concept should be familiar to those that have enountered "karmabots" in chatrooms.

The targets can be anything. Companies, food, people, even Twitter itself. For whatever reason, the site has caught on most with Japanese Twitter users, so the homepage is often incomprehensible.

Technically, the site is not too exciting. Like Twitter Digest, it also uses python-twitter to talk to Twitter and templet for simple templates. The site itself is not dynamic at all, instead a script running on one of my machines polls Twitter every two minutes (since it has to fetch both friend updates and direct messages, fetching both every minute would go over Twitter's 70 requests/hour API rate limit). If it determines that a new plusplus or minusminus has been sent, it re-generates those pages (the user's and the target's) and uploads them. The simple design means that the site can be hosted nearly anywhere.
Macros for the new version of Gmail

Posted: November 5th, 2007, 9:09pm PST by Mihai Parparita
As those of you on the new version of Gmail might have noticed, nearly all Greasemonkey scripts that used to work on the old version no longer do. Even though it looks pretty similar, the new Gmail is entirely different from a JavaScript, HTML and CSS perspective, so this isn't surprising. Some of the scripts are no longer necessary. For example, saved searches aren't really needed, since searches now get their own URLs and can be bookmarked.

However, my Macros script is still needed; while the new version of Gmail does have additional keyboard shortcuts, it's still not possible to do everything from the keyboard. I've therefore ported it for the new version of Gmail, to install it, click below.

Install Gmail Macros

More specifically, the following keyboard shorcuts have been ported over:
- g: Go to label
- l: Apply label
- b: Remove label
- e: Archive (regardless of view, unlike "y")
- d: Discard (mark as read and archive)
There is also a new keyboard shortcut, which the old script didn't have:
- f: "Focus" the current view (only show unread, starred or inbox messages)
You may remember the script supporting other keyboard shorcuts. Since the new version of Gmail supports additional shortcuts, those have obsoleted. The new ones are:
- shift + i: Mark as read
- shift + u: Mark as unread
- shift + 3: Move to trash (not actually new, but not many people seem to know this one)
- shift + 8 followed by a, n, r, u, s, or t: Select all, none, read, unread, starred, or unread
For those of you adventurous enough to look at the script source, you'll notice that it uses a gmonkey object that is present on the window, which in turn gets you a gmail object with methods like getNavPaneElement() and getActiveViewType(). What this means is that the version of Gmail, in addition to being faster, also has semi-official support for Greasemonkey scripts. I'm pretty sure docs for this API will be out soon, but in the meantime, feel free to look at the script and use a tool like Firebug to investigate the properties of the gmonkey and gmail objects and play around.

Update on 11/06/2007: And here are the semi-official docs.
Feed Proxy: View Raw RSS/Atom Feed Data in Firefox 2.0

Posted: October 1st, 2007, 8:24pm PDT by Mihai Parparita

Firefox 2.0 has a nice RSS/Atom feed preview feature, which pretty-prints the feed and allows you easily subscribe to it in a feed reader. While this is nice from a user's standpoint, it can get in the way when trying to debug or otherwise see a raw feed as a developer. There is no option to turn this mode off, which seems to be making some people unhappy.

To help with this use-case, I made Feed Proxy, a simple service that gives you back the raw XML output (by adding a 512 byte dummy header that makes Firefox not invoke its feed preview code). Its page has more details on how to invoke it and what its caveats are.
Valleywag, brought to you by....Google?

Posted: September 26th, 2007, 6:11pm PDT by Mihai Parparita
Gawker Media's use of virtual hosting for its sites was being discussed in PartyChat, so to verify this, I ran host valleywag.com to compare it with host fleshbot.com. Imagine my surprise when I saw this output:
```
$ host valleywag.com
valleywag.com has address 69.60.7.199
valleywag.com mail is handled by 1 aspmx.l.google.com.
valleywag.com mail is handled by 5 alt1.aspmx.l.google.com.
valleywag.com mail is handled by 5 alt2.aspmx.l.google.com.
valleywag.com mail is handled by 10 aspmx2.googlemail.com.
```
If I am interpreting this correctly, Valleywag's email is being handled by Gmail/Google Apps, which is slightly ironic.