Getting Bicho Running as a process on Heroku with a Scheduler

By FĂ©licien Victor Joseph Rops (Belgium, Namur, 1833-1898) [Public domain], via Wikimedia Commons
“Ou la lecture du grimoire”
For our almost complete MoFo Interim Dashboard, I’m planning to use an issue tracker parsing tool called Bicho to work out how many people are involved in the Webmaker project in Bugzilla. Bicho is part of a suite of tools called Metrics Grimoire which I’ll explore in more detail in near future. When combined with vizGrimoire, you can generate interesting things like this which are very closely related to (but not exactly solving the same challenge) as our own contribution tracking efforts.

I recently installed a local copy of Bicho, and ran this against some products on Bugzilla to test it out. It generates a nicely structured relational database including the things I want to count and feed into our contributor numbers.

This morning I got this running on Heroku, which means it can run periodically and update a hosted DB, which can then feed numbers into our dashboard.

This was a bit trial and error for me as all the work I’ve done with Python was within Google App Engine’s setup, and my use of Heroku has been for Node apps, so these notes are to help me out some time in the future when I look back to this.

Getting this working on Heroku

$ pip freeze

generates a list of the requirements from your working localenv e.g.

BeautifulSoup==3.2.1
MySQL-python==1.2.5
feedparser==5.1.3
python-dateutil==2.2
six==1.6.1
storm==0.20
wsgiref==0.1.2

Copy this into a requirements.txt file in the root of your project

But remove the line: Bicho==0.9 (or it tries to install this via pip, which fails)

Heroku’s notes on specifying dependencies.

You can now push this to Heroku.

Then, I ran:

$ heroku run python setup.py

But I’m actually not sure if that was required.

Then you can run Bicho remotely via heroku run commands

$ heroku run python bin/bicho --db-user-out=yourdbusername --db-password-out=yourdbuserpassword --db-database-out=yourdbdatabase --db-hostname-out=yourdbhostname -d 5 -b bg --backend-user 'abugzilla@exampleuser.com' --backend-password 'bugzillapasswordexample' -u 'https://bugzillaurl.com?etc'

As a general precaution for anything like this, don’t use a user account that has any special privileges. I create duplicate logins that have the same level of access available to any member of the public.

Once you’ve got a command that works here, cancel the running script as it might have thousands of issues left to process.

Then setup a scheduler https://devcenter.heroku.com/articles/scheduler

$ heroku addons:add scheduler:standard
$ heroku addons:open scheduler

copy your working command into the scheduler just without the ‘heroku run’ part

python bin/bicho --db-user-out=yourdbusername --db-password-out=yourdbuserpassword --db-database-out=yourdbdatabase --db-hostname-out=yourdbhostname -d 5 -b bg --backend-user 'abugzilla@exampleuser.com' --backend-password 'bugzillapasswordexample' -u 'https://bugzillaurl.com?etc'

If you set this to run every 10 mins, the process will cycle and get killed periodically but in the logs this usefully shows you how the import is progressing.

I’m generally happy with this as a solution for counting contributors in Webmaker’s issue tracking history, but would need to work on some speed issues if this was of interest across Mozilla projects.

Currently, this is importing about 400 issues an hour, which would be problematic to process 1,000,000+ bugs in bugzilla.mozilla.org. But that’s not a problem to solve right now. And not necessarily the way you’d want to do that either.

Messing with Processing and a black hole

I put together a Sketch in Processing this evening. It generates a canvas at whatever size you want, adds a black hole and a couple of thousand pixels that get sucked into the black hole with some simulated gravity. The final result is a bit like the image I had in my head when I started, so I’m happy with that. I would like it to be more awesome, but this will do for the time I have to play with right now.

I quite like the scratchy ‘pixely’ quality at ‘actual’ size, but that might be my nostalgia for older games and my taste in scrappy painting.

The code’s available over at http://www.openprocessing.org/sketch/119499 where you can hit the page a few times and generate variations of the image.

blackhole

blackhole

blackhole2

Anyway, enough with this post as I should be doing some real studying now (which involves reading a play).

Concept Game – Simple Evolutionary Model

I’ve spent enough time on this now to submit it, even if it’s still a bit rough around the edges. I’ve included a bit of a write up below. This demo will run best in Chrome or Opera. Click to play.

I’ve built a simple ‘game’ called Digital Husbandry. It’s more of a time killer as it doesn’t have any serious game mechanics, but there is a visual reward to keep the user engaged.

It’s based on the idea of simulating progressive evolution through selective breeding. Much as generations of farmers have done with livestock. The player brings together critters on the screen based on visual qualities that appeal to them, and produces offspring that drive the overall appearance of the group closer to those qualities selected by the player. The ‘critters’ die when they reach the ‘deadzone’ at the bottom of the screen, freeing up space for new critters in the population. So choosing which critters to sacrifice is as important as choosing which ones to breed.

The critters are recursively drawn from a simplified ‘genetic’ code. This allows the game to have millions of possible variations of critters, and the longer you play, the more varied the critters will appear.

I ‘composed’ some music in http://www.beepbox.co (which is a hugely fun distraction from fixing bugs in code). The music gets more layered and complex in line with the number of critters in the population. The audio tracks aren’t perfectly synced, but I’m happy enough with the effect for now.

NOTE: I ran into problems publishing the sketch with audio and I’ve run out of time to do any more work on this, so I’ve had to submit this version without sound.

A quick review of the Coursera Creative Programming Course, and using Processing for this kind work:

  • It was nice to write some code that isn’t about capturing web form data or sanitizing user input!
  • The format of the course, and the challenge I set myself were a good way to revise some of the classic programming concepts I don’t actually have to use much these days
  • I was jumping between Java and JavaScript quite a bit – which is a good study exercise, especially when porting code from one to the other. My basic Java knowledge is rustier than I thought, but I got there in the end.
  • Processing isn’t quite right (yet) to take a project like this and polish it into a publishable product (which is partly why I haven’t worried too much about the finer details of this ‘game’)
  • Processing is excellent for teaching creative programming
  • It was a shame to loose the sound, I had a slightly mental retro game tune going by the end. This was the base tune, which built in complexity with each additional critter.
  • I wouldn’t want to put this project through a code review! But it does the job for this assignment.

 

Critters Sketch in Processing

If your browser is up to scratch, here’s a little JavaScript based sketch from a current personal project…


This is some early code for a simple game I’m working on for the Coursera Creative Programming course (it’s my first time building a game rather than regular software).

These shapes are generated from a limited range of numbers, which can later be turned into a simple genetic code to define these critters.

I’ve hosted this on OpenProcessing.org, so you can get to the source-code etc.

http://www.openprocessing.org/sketch/103410

Evening coding

With lots of interesting client work on at the moment, I’ve decided to spend some evening time moving along the next version of Done by When. This is nothing too stressful, but the project is getting really interesting now. I think I’m over the initial conceptual learning curve and now I’m making proper progress.

Where the launch version of Done by When was primarily a working proof of concept, this next version is about attention to detail and responsiveness (that’s the speed of interactions as opposed to the adaptive layout stuff that’s already in place).

I feel like I’m properly upgrading something when I’m spending as much time removing code as I am writing it new.

More updates soon.

jQuery UI Sortable slow in FireFox

This is just a note for other developers searching on the same issue, as I didn’t find anything online when I was looking for ideas.

jQuery UI Sortable is a great way to work with drag and drop web interfaces but I found it was running painfully slow in Firefox. It was snappy in Chrome, and surprisingly responsive in IE, but Firefox was lagging.

As the official Sortable demo ran fine in Firefox it was easy enough to work out that it was something in my own code causing the problem. After cutting out items and returning them one by one it seems that having too much CSS3-niceness within the draggable elements was the culprit.

Each item I was dragging included a set of mini buttons styled with the lovely Bootstrap effects (gradients, rounded corners, shadows etc.) and although Chrome wasn’t phased by this it seems to knockout Firefox.

Valuing responsiveness more than prettiness, my solution was to cut back on the styling of these items and I’m happy with the results.

Hopefully this is useful to you too.

Loading a new version of jQuery without breaking an old version

Sometimes you’re working on a website that already uses an old version of jQuery and upgrading is not an option at that moment in time; if for example the jQuery library is bundled with a version of Drupal and works with a set of existing plugins.

The following code will allow you to load in a newer version of jQuery and still leave the $ variable assigned to the old version…

The code:

<script type='text/javascript' src='https://www.google.com/jsapi'></script>
<script type='text/javascript'> 
//<![CDATA[
google.load('jquery', '1.7.1'); 
//]]> 
</script>
<script type="text/javascript"> 
// save the new version of jquery to a variable and revert $ to the existing version on the page
var jQuery_1_7_1 = $.noConflict(false); 
</script>

The same code with console logging for testing:

<script type='text/javascript' src='https://www.google.com/jsapi'></script>
<script type='text/javascript'>
// outputs jquery version to Firebug/chrome console to test
console.log("$=" + $().jquery);
//<![CDATA[
google.load('jquery', '1.7.1');
//]]>
</script>
<script type="text/javascript">
console.log("$=" + $().jquery);
// save the new version of jquery to a variable and revert $ to the existing version on the page
var jQuery_1_7_1 = $.noConflict(false);
console.log("$=" + $().jquery);
console.log("jQuery_1_7_1=" + jQuery_1_7_1().jquery);
</script>

After that, you can use jQuery_1_7_1 in place of $ in your newer code.

Hat-tip to the following helpful articles: