New Year’s Python Meme

I love these blog memes, so I give you my version of Tarek Ziade's New Year's Python Meme.

1. What’s the coolest Python application, framework or library you have discovered in 2011?

For python libraries, that would have to be Kenneth Reitz' python-requests library. I've used it for an amazing amount of stuff and blogged about it. It took the grunge out of doing HTTP actions with Python. The API is clean and elegant, getting out of your way. It embodies the State of the art for API design, which closely matches the Zen of Python.

For applications, djangolint.com is awesome. It has helped me out so much on several projects. I would love to see something like this implemented and maintained for modern Python.

All the Python friendly PaaS efforts that have emerged are changing the landscape for those of us who want to launch projects but don't want to become full time system administrators in the process. Heroku, DjangoZoom, DotCloud, ep.io, gondor.io, and others are making it possible for developers to focus on development not server tooling. Google App Engine paved the way and it is wonderful to see the rest of the universe catch up with material that more closely follow core.

2. What new programming technique did you learn in 2011?

Event based programming! I've touched on it for years, but this year I really got a lot more more into it thanks to Aurynn Shaw kickstarting me and Audrey Roy expanding my knowledge ever since.

3. What’s the name of the open source project you contributed the most in 2011? What did you do?

I participated mostly as co-lead in the Open Comparison project, which amongst other things involved running the largest sprint at PyCon 2011. We maintained Django Packages and launched Pyramid and Plone versions of the project. We hope to launch a Python implementation in 2012.

I took a lot of notes this year at pydanny-event-notes - enough to make a book.

4. What was the Python blog or website you read the most in 2011?

Like Nick Coghlan, that would be http://planet.python.org.

5. What are the three top things you want to learn in 2012?

1. How to use whatever consistently maintained project that replaces PIL that works in Python 2.7.x and Python 3.x.
2. Really advanced Python as taught by Raymond Hettiger.
3. backbone.js

6. What are the top software, app or lib you wish someone would write in 2012?

A tool python-requests, but for shell access. Something like Unipath, but kept up-to-date and with nicely written documentation on Read the Docs.

A PIL replacement that is maintained, works for all modern Pythons, and is close enough to the PIL API to not cause too much confusion.

Something like Django Lint but for Python 2.7.x/3.x.

An open source project that tracks test coverages across PyPI and publishes reports of the results via an API.

Want to do your own list? here’s how:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

Quick conferences report: Presentations

My lovely Fiancée, Audrey Roy, was invited to be the opening keynote speaker at both PyCon Australia on Diversity in Python (video) and PyCon New Zealand on Python on the Web.

As for me, I managed to get talks into both of those conferences AND DjangoCon US. I co-presented on three of them, and I share all credit for success with my cohorts. The talks I gave at the conferences were (I'll post videos when they get up):

Confessions of Joe Developer (PyCon Australia, DjangoCon US)
The genesis of this talk was as a lightning talk at I gave at the Hollywood Hackathon. It is a talk about admitting that us mere mortals need to ask questions, take notes, and follow good practices in general. I gave it again at LA Django this summer, extending it to a full length talk complete with lots of technical content. At PyCon Australia I toned down the technical content because I was nervous, and while the response was positive, it  could have been much better. So for DjangoCon I ramped up the tech-talk and it worked much better. I've now given the talk 4 times, and I'm leaning towards retiring it.

 

Python Worst Practices (PyCon New Zealand)
This talk grew out of a SoCal Piggies lightning talk which I gave for the purpose of humor. Often we as Python developers are smug in the clarity of the language that we don't realize just how easily we can obfuscate code. In fact, I contend that Python is fully capable of a code obfuscation contest. This talk rejects a lot of crazy practices I've either done myself or had to debug from other people's work. For New Zealand I added a ton of content and tested things pretty diligently. The variable naming pages stumped some people I really respect and I was quite happy with that result.

 

Django Packages Thunderdome (co-presented with Audrey Roy, DjangoCon US)
Audrey did most of the work for this presentation. In this talk I helped review a horde of Django Packages across 7 different categories. It was nerve wracking because every part of our talk would get judged - but Audrey kept things really positive and made it clear we were providing constructive criticism. I think she got her message across to most people, and more importantly, it got a lot of people thinking about what ought to be normal community standards. I'll probably blog on those community thoughts and statements later, but I think Audrey (with help from me) accomplished what she aimed to do.

View more presentations from Audrey Roy

Advanced Django Form Usage (co-presented with Miguel Araujo)
Some time ago Miguel befriended me and helped resurrect the django-uni-form project. He graciously agreed to help me present on Django Forms and we decided to make the talk as sophisticated as possible. Previous Django form talks have been good, but focused on the fundamentals and we wanted to do something really different. This talk was hard because Miguel and I were on opposite sides of the planet, so we did a lot of github pull/pushes. In both doing research and presenting Miguel did an unbelievably good job and I hope he does more of this in the future. The response was extremely positive and I'm certain that our plan of getting our notes/work/transcript into Django core is well on it's way.

 

Ultimate Django Tutorial Workshop (DjangoCon US)
I got about 10 professional Django experts in a room, including Django core developers, and had them help me coach nearly 20 people through a modified version of the Django tutorial. Students seemed to learn tons, lots of socializing happened thanks to some happy accidents, and the experts got a chance to really see where the Django tutorial needs work. PyLadies organizer Esther Nam spent her sprint days working on something that ties the slides into the Django Tutorial - and for now I'm holding off on sharing my work until she says her work is done.

Summary
These were amazing opportunities to speak and will hopefully make a difference. I wouldn't have traded all of this for the world. It was a lot of work, and I doubt I'll ever go quite at this pace again. My plan is to do fewer talks and make them much better.

Python and Django class/hackathon!

The Los Angeles Python community (LA Django and LA PyLadies) is meeting in Santa Monica on July 23rd to teach Django and hack on all things Python on Saturday, July 23rd. The day will start with a Django class based on the official Django tutorial, then turn into a general hackathon, and finish up with lighting talks.

Leading the event is noted Pythonista Katharine Jarmul. As Katharine is giving the talk on web scraping at DjangoCon US, I'm hoping we can get her to give a lightning talk on the subject.

Learning Django

Sandy Strong will lead the effort to  teach people the fundamentals of Django. Besides all things Django and devops, Sandy is presenting the testing talk at DjangoCon US. And if that isn't good enough for you, she won't be alone teaching - there will be a bunch of us developers experienced with Django there to to provide her with support.

Even if you already know Django, please come and hang out for the first half! You can either help out others or work on your own project.

Hacking Python and Django

The second half of the day will be about working on whatever you want. If you are new to Django and want to finish the tutorial, go right ahead. Or you can work on your own pet Django or Python project. In fact, I know that there will be work on the nascent Pyramid project intended to represent the entire Los Angeles Python community.

Lightning Talks

We'll finish with lightning talks. Several people who attended the day will get the chance to talk for 5 minutes or so about a project, tool, or cause they wanted to share. If they go too long we start applauding until they step down.

Social Hour

After another awesome day of Python in LA, everyone will cool down by hanging and chatting over drinks. If you're lucky, maybe you'll get to see me do a drunken one-handed cartwheel where I don't spill a drop of what I'm holding.

My role

I'll be there in my normal role of setting up tables and chairs, helping during the class portion, and hacking on some Packaginator stuff in preparation for the forthcoming August/September Packaginator sprints at PyCon AU, Kiwi Pycon, and DjangoCon US.

Sponsors

This is all possible thanks to the sponsorship of Mahalo, Cars.com, and the Python Software Foundation

Sign up!

Tickets are selling out really fast! Sign up now!

Python Packages sprint on Sunday 4/10/2011

Want to help Python? Want to encourage PyPI to focus on being the best package index system possible and not a catalog/ratings/documentation engine? Then sprint with us on Packaginator this weekend because...

We need your help!

In my last blog post I mentioned Packaginator which will power the forthcoming Python Packages site. The purpose of this site is to do for Python what Django Packages does for Django. We aren't quite ready for launch, and the purpose of this post is to list what tasks remains. There is an enormous amount of outstanding work, and we want to launch as soon as possible so...

Before I begin, Python Packages will have some dramatic differences from Django Packages:

• The 'Add Package' controls won't exist. The only way to get a package into the site is to put your package onto PyPI.
• At launch only approved moderators will be able to create new grids and grid features.
• Only approved moderators and package owners will be able to edit the target repo URLs and add/edit/remove packages on grids.
• Users will still be able to click the "I use this" button. 

Please join us!

We invite you to sprint with us on Sunday, April 10, 2011. We are sprinting on Python Packages / Packaginator from the Los Angles OS Hackathon this weekend but also invite remote sprinters to join us from across the globe.

How to participate

How to contribute to Packaginator is really well documented and I beg you to read that before commencing any sort of work. We don't use IRC to communicate, instead relying on our convore channel at http://convore.com/packaginator.

Because we can, have, and will reject pull requests, I need to reiterate a few things:

• Run the test suite before you submit your pull request
• Keep your pull requests small and focused
• Be familiar with PEP-8 and keep your code simple

Important: Because we are "this close to launch", for Sunday we aren't the least bit interested in dramatic architecture revisions, model changes, obfuscating our settings, new design, different layouts, or adding Haystack. We'll consider those afterwards.

Tickets

Priority tickets are the real choke points that we want to overcome. We could use help on all of them:

• Low hanging fruit tasks
• Challenging tasks
• Priority tasks

When you start a ticket, please let us know in http://convore.com/packaginator so we can mark it as assigned in the Github issue tracking system or warn you if it is already being worked.

Launch

We hope to launch this sunday or in the immediate days afterwards, and for that Jacob Kaplan-Moss, Jacob Burch, and Noah Kantrowitz  have graciously volunteered their time to do the engineering work.

PyCon 2011 Sprint Report

I love sprints. I've yet to participate in a sprint where I didn't learn something that made a difference in my programming career. Off the top of my head some of the things I've learned include distributed version control, picking the right Python tool, JQuery, SQLAlchemy, Bazaar, Git, Mercurial, the true importance of unittests, and Python's built-in zip function. And the PyCon 2011 sprints were no different.

PyCon 2011 was different in that this time I was going to co-lead a project, specifically Django Packages and the hopeful launch of Python Packages.

Note: The goal of Python Packages is not to replace PyPI, but rather serve as a resource to find, evaluate, and compare packages used in the every day life of a Python. In my opinion, PyPI should be dedicated to listing and serving packages - anything else (comments, ratings, documentation, etc) just adds complexity to the project and diffuses the focus of their team.

It all started with us being the second-to-last in line at the PyCon sprint announcements. At the microphone I forgot to mention a few things so I was worried that our attendance would suck. I tried to take it in good humor, but doubt worried at my gut. My co-lead, Audrey Roy, was confident that if no one showed, then we would have fun with just the two of us hacking away.

To our delight and surprise, turnout was good with about ten (10) people showing up. And thanks to lessons learned at DjangoCon 2010 and our tricks to getting more sprinters and helping sprinters deliver code, the number of participants kept growing. In the end we had twenty-four (24) new contributors to the project, which was simply amazing.

Test coverage had been mediocre on Django Packages, but after a show stopping bug got into production (someone changed a commonly used function to a property), we did a huge amount of work to not only increase test coverage but also refactor tests to be simpler and actually test rather than just increase coverage numbers. The improved quality and quantity of test coverage gave contributors the confidence to refactor and simplify some of the 'brilliant code' that I had written during the first month of the project.

We also got a bit draconian about accepting pull requests but documented how to get pull requests accepted. That might sound mean but if stopping one person's bug allowed ten (10) other people to maintain productivity then everyone is happier. Also, it allowed us to coach some of the new Python developers coming from other languages on their work. Which was awesome because we saw people's work evolve in a day from rank beginner material to competent Pythonista submissions. To think we had some part in helping people improve is one of the best things we got out of the sprint.

And the results?

The biggest thing, which we got into place on the second evening of the sprint, is that Django Packages is now an instance of Packaginator. Packaginator is a framework for launching package comparison sites for Python based tools. After a bit more work to happen this coming Sunday (4/10/2011), we'll be able to trivially deploy Python Packages, Pyramid Packages, Plone Packages, and Flask Packages - all of them able to support patches from Packaginator without causing the maintainers of those sites to hate our guts. We should also will have the hooks to support things like Vim Packages, Ubuntu Packages, Fedora Packages and more quite shortly.

Packaginator handles repos much better and now supports Bitbucket, Github, and Launchpad out of the box. SourceForge may happen very soon. Google Project Hosting, when Google implements Project Hosting API (cause we refuse to screen scrape pages for MetaData) will be handled shortly thereafter. Thanks to the work of the 'repo men' adding a new repo is now much easier, and we hope people submit new ones to handle things like Trac and other repo systems.

Our documentation went from passable to incredible, and our installation story is awesome. You want people to participate in your project? Then learn you some RestructuredText and Sphinx and host your documentation on Read the Docs. Read the Docs is awesome and I need to blog about why all Python docs should move there.

There was a huge amount of template cleanup - and the grid X-Y access can be rotated. We haven't turned on that feature yet, but you'll see it shortly.

Conclusion

The sprints were awesome. I learned a lot about running projects and managed to get into some new coding tricks (zip() comes to mind) into my brain. That this project is helping the open source world made it even better. And the best thing of all is I got to make over twenty new friends - all of whom worked with us towards a single common goal.

PyCon 2011 Contributors

• Aaron Kavlie
• Adam Saegebarth
• Alex Robbins
• Andrii Kurinny
• Audrey Roy
• Brian Ball
• Bryan Weingarten
• Chris Adams
• Daniel Greenfeld
• Eric Spunagle
• Evgeny Fadeev
• Flaviu Simihaian
• Gisle Aas (Repo Man)
• Jacob Burch
• James Pacileo
• Jeff Schenck
• Jim Allman
• John M. Camara
• Jonas Obrist
• jrothenbuhler
• Nate Aune
• Nolan Brubaker
• Preston Holmes
• Stuart Powers
• Szilveszter Farkas (Repo Man)
• Tom Brander
• Vasja Volin

Release classifiers in distutils/pypi

Thanks to Doug Napoleone I'm now aware there is already a convention followed for the python and framework versions, but it appears that not enough people are aware of it. This post is pretty much a reposting of the second comment of the post immediately preceding this one and Doug gets full credit for this post. I'm just repeating his message:

The release classifiers in this post should be included in the standard distutils documentation. For the moment, you can see the full list of classifiers here:
http://pypi.python.org/pypi?%3Aaction=list_classifiers

For the python language version the classifier is:

Programming Language :: Python :: x.y.z

With each version on it's own line. That way you can browse the repository by python version (see the bottom of the page):

http://pypi.python.org/pypi?:action=browse&c=214

There is also support for frameworks which you can see on that page as well. There it is done with:

Framework :: Django :: x.y.z

There is Zope, Plone, and a number of other frameworks already there.

In the example you gave the proper, and supported way of writing the metadata is:

Programming Language :: Python
Programming Language :: Python :: 2.4
Programming Language :: Python :: 2.5
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Framework :: Django
Framework :: Django :: 0.96
Framework :: Django :: 1.0
Framework :: Django :: 1.1
Framework :: Django :: 1.2.1
Framework :: Django :: 1.3

Now it becomes a matter of education and illumination. This should be in the standard distutils documentation and arguably the home page of PyPI (or easily found there). And Django Packages will be supporting this functionality in the near future.

A request for new pypi classifiers

This request is to help enhance Django Packages, PyPM Index, and other projects. This would also help the Python community at large.

Would it be possible that a standard be established for listing in PyPI classifiers which versions of a package is known to operate? Using James Bennett's django-registration at http://pypi.python.org/pypi/django-registration as an example (see my bolded, last two lines to understand what I'm trying to demonstrate):

Development Status :: 5 - Production/Stable
Environment :: Web Environment
Framework :: Django
Intended Audience :: Developers
License :: OSI Approved :: BSD License
Operating System :: OS Independent
Programming Language :: Python
Topic :: Utilities
Python Versions :: 2.4, 2.5, 2.6, 2.7
Django Versions :: 0.96, 1.0, 1.1, 1.2.1, 1.3

The metadata system I'm writing about in this blog post is specified on the distutils documentation page.

I picked a Django package but this could be for Zope, Pyramid, PyQT, or anything.

If we had something like this in place then people could quickly identify on PyPI and other resources if a tool can be of use to them or if it needs to be updated to the latest code base. If this already exists, then can someone point me at the existing specification so I can promote it?

Edit: Noah Kantrowitz suggested I take a look at the 'requires' keyword which is part of the distutils spec. However, this does not show up in the PyPI API (http://wiki.python.org/moin/PyPiXmlRpc) and so doesn't fit our needs.

DjangoCon 2010 report I

The conference was like a family gathering except without any oddly weird uncles. To my utmost embarrassment I got overwhelmed a few times and forgot names of people I respect and admire.

If I went through people I got to touch base with, I would have to list three digits worth of people and play favorites. That just ain't me, so I'm just going to say everyone there was awesome and I hope to see you again at Pycon!

Alright then, some reporting:

Django Software Foundation Panel

Russel Keith-Magee, the new DSF president hosted a panel with Ben Slavin, Sean O'Conner, Jeremy Dunck, and myself. Russ went over plans for DSF finances and the hopeful creation of a DSF ecosphere of applications.

My own contribution was talking a little about Django Packages, and bit more about "Why Django".

Why Django, or maybe one day "Enterprise Djangoproject" is an advocacy site for Django targeted not to developers but to decision makers. It is a work in progress, we are collecting case studies and articles, and building out the site.

Ben Slavin then challenged me about the code and data. Other Django community projects like djangoapps and djangosites are closed source efforts without an API and this causes problems for the community. If a site goes down or is unmaintained then the community loses.

I agree.

So I promised that for these Django community projects the code is public, APIs were being made to support people fetching data from them at any time, and I'm trying to figure out how to do data dumps without handing out even salted passwords. Furthermore, I would give full access to these sites and servers behind them to those that Russell appointing to the position of being watchdog.

And I'm working on that promise. Django Packages now has an API and a BDFL has server access to Django Packages. Why Django is still a work in progress, so things are still in flux. I'll post updates to my promise of openness.

Furthermore, I challenge anyone who puts together a site useful to the community, be it a new version of Django Apps or Django Sites to follow my own promise. By all means maintain and work your project, but be willing to publish all data and keep your code under an open source license. Also provide access to whom the DSF president appoints so that others can provide maintenance for your site.

Eric Florenzano's Keynote

Eric Florenzano gave the keynote on What Sucks about Django (and how we can fix it). He did a very good job of it. Watch his talk. It has some amazingly telling points.

Anyways, I'm challenging for Eric to eat his own dogfood. The only people I saw Eric talk to were the same people he is always around and conferences. How many times did he go up to a person he didn't recognize and start talking to them? How many new people did he meet at DjangoCon 2010?

Portland Views, Food, and Beer

On Saturday, as a break, I went with Eric Holscher, Ben Firschman, and Andrew Godwin to the park around the Portland zoo. Below you can see where a lady pointed out that we could see mountains 50 and 150 miles away. That left three geeks speechless while
Ben Firschman captured the moment. This picture is Ben's and all rights belong to him.



And then I had my action shot taken by Ben. Again, all rights to him.



I just have to say that Ben is an amazing photographer and I'm honored to be in a few of his pictures of DjangoCon. In addition to that he presented on class based views in Django, which I plan to blog about as well as incorporate into Django Packages in the near future.

Edit: Blogspot is acting really oddly which is why I took out the linking and other things.

Edit II: Eric ate his own dog food. Awesome!

New features for Django Packages

Since the Django Dash ended, the Django Packages team has been working to add new features and close out bugs.

"I Use This" added

Since we only want hard metrics on this site, we incorporated an "I Use This" button on the packages. This is so you can identify which packages you use. Please don't press this button for packages that you like, only the ones that are part of your coding efforts.

Added BitBucket support

We are still working out some of the kinks for coming up with stats from BitBucket. Most of the data we collect is fetched via the API, but a little is scraped off individual project pages.

Cache the commits

Originally the commit history was  fetched live. But Github only provides the last 35 commits and BitBucket limits you to the last 50 commits. So now we store the commit history and update it nightly. Which means that the sooner you post your packages the better your commit history will look on Django Packages.

Rebuilt the package updater

Limitations on how many API calls you can make against Github (60 a minute) meant that we had to write some fun code to get around that problem. I think the problem is solved now, but I'm worried I might get to eat my words.

Added a help section

As much as we wanted a completely intuitive site, this will hopefully make it easier for people to figure out how to participate on the site.

Package Add/Edit form refactor

We completely rebuilt the Package add/edit form to make it easier to add packages. So far the response has been entirely positive.

Page cleanup and CSS Reset

We've been slowly cleaning up the HTML and resetting the CSS. Everything is looking prettier. Our goal is to make things more readable, so a lot of the changes are subtle.

Email verification works

It works, and now you get an email to confirm your account.

Django Dash Lessons Learned

Our experience with Django Dash 2010 was that it was an wonderful exercise in classic Django development, cowboy/cowgirl coding, and drinking copious amounts of caffeinated beverages. The result, Django Packages, is something we are happy with, are continuing to improve, and hope will improve the community.

Lesson: Fixtures are a must

Django gives you this amazing Admin control panel. As soon as you get your models in place and are entering test data, start creating fixtures. For the dash we named them initial_data.json and loaded them into the individual app directories. This meant that every time we blew away the database we got a reload with records in place. Sometimes this means you have to hand-edit JSON (or YAML if you swing that way), but the alternative is to waste time re-entering the same data again and again. Don't forget to change the names of your test fixtures before you launch!

From the command-line how to save a fixture pydanny-style:

./manage.py dumpdata package > apps/packages/fixtures/initial_data.json

One nice thing about fixtures is that when you do have the time/need, you can use them to help you write tests. And it makes development easier for contributors.

Lesson: Research ahead of time

In the days before the contest, we researched to see if our target repos (Github, ;BitBucket, and Google Project Hosting) each had an API and a python library to speak to that API. Github has both an API and python library, Bitbucket has an API but no library. And as far as we can tell, Google Project Hosting lacks both API and library (someone please tell me I'm wrong about Google Project Hosting lacking an API).

This meant that when we commenced coding we knew which code base to work with - we weren't trying to look up this or that random package.

We did the same thing for rendering charts.

Lesson: Get it working then optimize

Looking at some of the code makes us wince a bit. But we got it working. Now we can go back and do some code cleanup, maybe use an XML parser instead of regex to try to scrape content from PyPI, and generally feel better about ourselves.

Lesson: Plan out system architecture in advance

In retrospect it was really amusing, but the night of launch the site was serving via the Django runserver command. We were so dead tired and neither of us are crack system administrators that we did what we had to do to score the contest launch point. The next day Audrey got the site running underApache, and next week we'll be giving someone else system access to increase reliability. Next year for the contest we'll probably use something like this or get continuous integration running in the first hour.

Lesson: Don't be afraid to chat with others after the contest starts

Share your ideas, selected packages and frameworks with your competitors. The break from coding helps clear the mind and they might counter with a better idea/package/framework you can use.

Announcing Django Packages!

I'm part of a two person team that just launched that BETA site for http://djangopackages.com, a site that is designed to list all the Django Applications, Frameworks, and Packages created by the Django community. While there are already a few places to look for these things, it is quite easy to argue that they are challenging to navigate, don't give any hard metrics, or are woefully incomplete/unstable/closed. Our goal was to provide an attractive, easy-to-navigate, easy-to-add-data, stable site and share it with the world.

Also, this was our entry into Django Dash 2010, and was the culmination of a few days of brainstorming over paper, a lot of research, and two days of feverish coding/designing. The project was feature complete to our specifications at 5pm the second day, and the rest of the time was spent adding in UI tweaks, usability enhancements, and trying to deploy our creation.

Since then, we've cleaned up a the UI, improved the design, and got the site stable. The code is open source and on github, so fork and contribute!

Design Consideration: No 'Like' button or 'Rate my app' rating systems

We wanted hard metrics. So the package numbers are pulled from the repo sites such as Github, Bitbucket, and Google Code. Otherwise things get weighted funny. Sure, this system can be monkeyed with, but its a good metric for now. We've had suggestions from Django core developers of coming up with a quality check system, things like pypants and/or a formalized approval system.

Design Consideration: Grids

Early on we wanted to duplicate and improve upon the Django CMS Comparison page. There is also a version for Forums, but it would be nice to have a current one for blogs! In addition, recently I heard that 'tag clouds are the mullets of web 2.0'. This really struck a chord in my soul. Since we had metrics on packages, why not compare those metrics, and use those comparisons, which we call 'grids', instead of tags? In fact, we extended our idea and instead of traditional tabs we use grids in the top navigation area as seen below:

Design Consideration: Categories

The site groups packages into three categories, 'Apps' which are individual django applications. 'Frameworks' which are aggregates of apps and python modules. And 'Projects' which are implementations of Apps and Frameworks. We've thought about adding 'Tools' but weren't sure if there was anything that fit that concept, and we are leery about allowing regular Python efforts into the fold.

Design Consideration: Regex vs XML

Slurping data out of Github was easy, especially since I used python-github2. Bitbucket has a RESTful API as well that serves out JSON. I think Google Code does as well. PyPI does not and DOAP on PyPI seems to give little that is useful, so I was forced to do screen scrapes of version numbers and downloads. I'm much faster with Regex and string methods than XML juggling, and speed was of the issue this weekend. I'm not sure what benefit there is to redoing it in HTML5lib or lxml, since what I have works and appears to be stable.

Design Consideration: Leave caching and optimization for later

Besides a tiny bit of memory based template caching on the home page, there is/was no optimization. In time I plan to cache many things using a proper key/value store like redis or memcached. Perhaps not before more design and usability work is done.

Why scared of rabbits?

You wouldn't understand unless you live on the Kansas prairie.

Note: if you have any suggestions, issues, problems with Django Packages please use our issue tracker!