2013 Area of Focus: Efficient Code

Over the last week, I thought about what I might want to focus on over the next year. Much of my focus on what I do in 2013 will be determined for me because I work for the central IT unit of a large university. However, that does not mean I cannot commit to improvements in how I accomplish my given tasks. Code efficiency is one area of focus I can control myself.


The Big Picture

For me, efficient code does not just mean using the least amount of hardware and software resources available to complete the given task. Efficient code also means taking into account your own time to author and maintain, the time of others who may have to understand it, the code's flexibility for enhancement and updates, and other code that interacts with it.

I may have written the most computing-efficient piece of code ever written. But if does not take into account all of these external variables, it is worthless and might as well be thrown away.


Personal Efficiency

Authoring perfectly computer resource-efficient code is hard. I always use the 15% rule: The last 15% of efficiency may take the majority of the total development time. I ask myself, it is worth it?

More often than not, my answer is it is not necessary. I work in an environment where I can generally save the efficiency lessons learned for the next update cycle, or project. If I only save a few dollars or microseconds in making a low traffic site more efficient, but it takes me a week of work to do that, I am not using my time or my employer's dollars wisely.

In a high traffic web environment, every microsecond, transmitted bit, and extra CPU process adds up quick. An extra hour, day, week, or even month of your time to find the most efficient path may save thousands of dollars in software and hardware costs over the life of the code.

Deciding how much human effort to invest to squeeze out more machine efficiency is a judgement call you and your team have to make for yourselves. Accurate environment analytics and load testing should help you make that decision.


Team Efficiency

This is an area I think about a lot. Unless you are authoring your own personal web environment, from the hardware on up, you work in a team environment. Someone else will have to look at your code to understand it at some point. Even if you are the sole maintainer, you will be on vacation at some point. You will be sick. It is likely you will hand off the project to others in the future as you move on to bigger and better things.

If you write code in a way that is incomprehensible to others, it is not efficient code. You wasted the time of others as they reverse engineered their way through it. Using an agreed upon common language and code style is essential. This may mean that even though Python does something more computationally efficient, you end up coding in PHP because that is the common tongue of the team. It may also mean that even though you prefer one JavaScript framework, you use another because the team bench is deeper for the latter.

Another way to become more efficient is to use third party tools such as code frameworks or content management systems. We use both where I work. These do introduce overhead in your your code and are not necessarily the most computationally efficient code structures. However, they may significantly reduce the amount of custom code authoring needed over time, as well as introduce a common code style your whole team can mimic and recognize.

Code style is essential. I inherit code that has passed through multiple maintainers all the time. Something as simple as space versus tab code indentation renders code difficult to read. Make sure to code in a style your team agrees upon. Our team went so far as voting upon various style standards. Future maintainers will think better of your abilities if you've followed style standards. Why needlessly burn collegial bridges to soothe your own ego?

The more each team member's code acts and looks the same, the easier it is to back each other up or hand off to make room for new challenges.

Another effort not wasted is documenting your code. This can be code comments all the way to formal write-ups. Don't skimp. I guarantee you will not remember all the code you authored after a year. Do yourself (and others!) a favor and spend a few moments to write down what the code does and if necessary, why.


Code Flexibility

I never worked in a web environment where business rules did not change and code was stable. We, as coding professionals, need to constantly adjust our code to the whims and desires of the rule makers. Our code needs to be flexible and future friendly.

Writing flexible code may not be perfectly efficient. It may even take extra time to build in the necessary hooks and safety valves. But you and your team will thank you in the next update cycle. I constantly run across code where I end up praising the the code gods that it is flexible enough to add in a new enhancement or delete an old process without a major rewrite.

I also run across the opposite kind of code. Those are not happy days.


Code Interaction

The majority of the code I write code has multiple layers. Most of the time, I do not author every layer. For example; a web service. It may be most efficient for me to write JSON output, but the application someone is trying to implement using your data needs XML. Opps! I may also write the web service to output data in a certain highly efficient way. However, if it takes a frequent user of the service three AJAX requests and a collation routine to assemble data into a usable form, is that efficient?

These are very simple examples, but begin to get to the point: what is efficient for one layer of code, may in no way be efficient for another. We need to take this macroscopic view if we are to measure efficiency.


Denouement

Code efficiency is not "My code only takes 3 CPU cycles to run!" or "I can code that in two lines!" It is the entire body of work, machine and human, that make a system of code truly efficient.

Javascript Security Tip

The following is probably obvious to seasoned web veterans, but it could be a reminder or new information to others. I thought I'd share this "Huh. I should probably pay more attention to my code" moment.

In my "How the heck do I do (more) proper Javascript?" research, I ran across a security-related issue. It's not the language itself, but how Javascript, particularly jQuery, might be used.

Let's say you have a variable with untrusted data as its value. This can occur in two ways, and maybe more that I can't think of:

1. Form input.
2. AJAX call to get data (from a data provider that is out of your control).

Now, let's say you want to display this variable's value to the browser. There are several ways to do that with Javascript. I will show you two safe ways to do it, and one unsafe way that you should never use.

SAFE WAY #1:
Use jQuery's .text() function:

var untrustedTextVariable = "<script>alert('This uses jQuery .text() function to display dynamic text.');</script><i>Kewl!</i>";
$("#someID").text(untrustedTextVariable);

This will not interpret the apparent HTML tags, nor will it execute what is inside the script tag. It will replace the HTML entities with their equivalent in html-speak (&lt; &gt; and so forth) and display it as such.

SAFE WAY #2:
Use native Javascript:

var untrustedTextVariable = "<script>alert('Another safe way to display dynamic text using native Javascript createTextNode().');</script><i>Kewl!</i>";
var parentNode = document.getElementById("someID");
parentNode.appendChild(document.createTextNode(untrustedTextVariable));

This does the same as #1 above. Actually, jQuery executes .createTextNode() behind the scenes.

UNSAFE WAY:
Use jQuery's .html() function:

var untrustedTextVariable = "<script>alert('I juss injektd XIQtibul Javaskrip N UR d0M. rawR!');</script><i>Oh Noz!</i>";
$("#someID").html(untrustedTextVariable);

Congratulations, you just executed arbitrary Javascript on your web site. jQuery's .html() function makes the browser interpret its contents. And if the contents happen to be executable Javascript, awesome. In my example, it just does a pop-up alert. It could as easily have been a keylogger sending your credit card information back to a nefarious server.

Partnering with the City of Chicago

It's been a long time and no post. I've been a little busy, but that's not a good excuse. Well, let's see if I can remember how to do this...

I created a little volunteer web application for the City of Chicago's Department of Public Health. This is the first volunteer-build web application the city has integrated into their site.

http://flushots.311services.org/ 

BACKGROUND
I went to a weekly meeting of civic coders where several city agencies presented. They asked for our help in building interesting applications around their available data. I was interested in the free flu shot clinic locations since flu season was coming up fast, and I had attended a free flu shot event in the past.

I built a prototype application and advertised it via Twitter. CDPH, the mayor's office, and a few other civic coders, mainly Juan-Pablo Velez, noticed the tweet. Before I knew it, Juan-Pablo brokered a meeting between CDPH, the Mayor's office, and ourselves. The meeting was positive and I began in ernest to polish up the application.

BITS AND BYTES 
These events were published by the CDPH via their web site (subsequently published on the Chicago Data Portal). Other events were given directly by CDPH. These events data are published to a Google Fusion Table, where the code uses the Google Maps API to retrive those events. 

 The Fusion Table: https://www.google.com/fusiontables/DataSource?snapid=S705126nZTU 

MADD PROPZ
This web application could not have been as well executed without the sage feedback and assistance of Juan-Pablo Velez and Derek Eder. Thanks also to Smart Chicago Collaborative for hosting this web application. 

ERRATA
Source Code: https://github.com/tkompare/chicagoflushots
 
'flu_logo_720.png' image file is for the City of Chicago's own use and does not fall under the license found on LICENSE.TXT. You must replace it in your own implementation. It is left here to help you use this code. This code should be fairly easy to take and use by other government agencies offering flu shot clinic events. If you would like any advice on implementing this code, drop me a line.

PRESS
http://www.cityofchicago.org/city/en/depts/mayor/press_room/press_releases/2012/october_2012/city_teams_with_localwebdeveloperstoconnectchicagoanstoflushotlo.html

http://blacklinereview.com/need-a-free-flu-shot-chicago-has-an-app-for-that/

http://codeforamerica.org/2012/11/06/city-of-chicago-adopts-flu-shot-app-built-by-civic-hackers/

http://www.dailyherald.com/article/20121029/news/710299773/
 

Package Liquor Licenses in the 49th Ward, Analysis Part 2

 Sorry folks, this is a little bit off the web programmer beaten path. I needed a place to put this analysis for others to see. I'll be back with some news about a new biking web application I have in the works.


----


Here begins the post


I've stayed up late to do a more detailed analysis of the relationship, if any, between total crime and package liquor licenses.

Stay with me, this is a somewhat long involved explanation. As much as some people like simple answers, I'm sorry, there are no simple answers. Nor are there simple questions, for that matter.

First, I did not do by police beat. I did something better: by address block, as fine-grained as the publicly available Chicago crime data gets. For example "14XX W Morse".

Second, I only compared address blocks of streets that are predominantly lined with businesses: Howard, Clark, Sheridan, parts of Morse, parts of Glenwood, and a block of Jarvis. 38 total address blocks. It is unfair to compare business blocks to non-business blocks, as the zoning, their use if you will, is totally different.

Third, certain blocks of Howard border Evanston. So only half of the address block is in Chicago. In my study I doubled the total crimes that were Chicago-reported on these blocks. This is a speculative assumption, but I have no Evanston crime data, and nothing better to go on. It's probably a better assumption that leaving these data points alone.

All of the data for this study is freely available in the City of Chicago Data Portal, http://data.cityofchicago.org/

On to the analyses...

https://docs.google.com/spreadsheet/ccc?key=0Ak5nccUbvw6YdFVqTHp5Q0pfUlNKQXIxRG5XamRhSkE

1. Null Hypothesis: "There is no difference in population of total crimes on blocks with Package Liquor Licenses and blocks that do not have Package Liquor Licenses."

When comparing the populations of blocks that have at least one package liquor store to blocks that have none, the result is probably significant, as you see in the second tab of the above spreadsheet labeled "PkLqAnalysis". The result has suggestive evidence against the null hypothesis. Generally in social science, a p value of less than 0.05 is considered significant. This has a p value of 0.05871. That's pretty close.

----

I decided to do a second analysis for comparison, after I saw a hint of a pattern in these data.

2. Null Hypothesis: "There is no difference in population of total crimes on blocks with L Stops on the street and blocks that do not have L Stops."

When comparing blocks that have L stops an blocks that do not, the result is significant as you see in the fourth tab of the spreadsheet labels "LStopAnalysis". The result has moderate evidence against the null hypothesis. This has a p value of 0.049405. That's in the threshold (just barely!) for real social scientific evidence.

----

So that got me thinking, are L Stop block total crimes much different than the total crimes on package liquor blocks? I ran the numbers:

3. Null Hypothesis: "There is no difference in population of total crimes on blocks with L Stops on the street and blocks that have Package Liquor Licenses."

When comparing blocks with L Stops with Blocks with package liquor licenses, the was no significant result. The p value is 0.31596, well outside the generally accepted range. this means that L Stop blocks and package liquor license blocks are not really different from one another.

----

Further Studies...

There is speculative (on my part) evidence in the data that suggest package liquor licenses position themselves near L stops, probably due to the foot traffic. So it could be that the L Stop numbers are influenced by the presence of package liquor stores. In order to back up that hypothesis, I'd have to find L Stops in other neighborhoods where there are no package liquor stores nearby and compare those blocks with others in their neighborhoods.

It is also possible That the mere fact there is an L Stop on the block that makes it an attractant for crime. Think easy exit from the neighborhood. It possible that the crime numbers found on package liquor blocks are influenced by nearby L Stops. In order to test that hypothesis, I'd have to look at more neighborhoods with package liquor stores not near L stops and compare them with blocks in those neighborhoods. There is just not enough data in Ward 49 to make that case. Maybe a few more wards similar to ours.

It is also possible, as heavily speculated on EveryBlock, the we have at least two classes of package liquor stores, a desirable kind and a not so desirable kind, and that makes a difference in crime levels. It would need to rope in more data to make a determination about these two classes. The 13 available data points in Ward 49 is just not enough. I do not have access to data that might help determine the good from the bad. That does not mean it can't be done. I just don't have the time to do it, or do it correctly.

making access to government data easy

I've written a simple Javascript object file to simplify access and use of Socrata-driven data warehouses such as http://data.cityofchicago.org/ and http://explore.data.gov/. This object makes it a bit easier to pull these data into a web application without relying on your web site's server-side code. The data access uses Socrata's SODA 1.0 API. It is said SODA 2.0 is "coming soon", so I'll have to update this access file accordingly when the time comes.

Demo: http://jslibrary.311servic.es/tksocrataview.html

Code: https://github.com/tkompare/jslibrary/blob/master/htdocs/js/tksocrataview/tksocrataview.1.0.0.js

Docs: http://jslibrary.311servic.es/docs/tksocrataview.html

Learn about Socrata: http://socrata.com/

Comments and questions are always welcome.

Javascript Libraries for Google Maps

I've created a couple of Javascript libraries (objects, really) to help simplify putting a Google Maps map, with Google Fusion Tables geographic data into your web application. Caveat: I've designed
'tkmapfusionlayer.js' for Fusion Tables geographic point data. Fuller support for line and polygon data will come shortly.

I'm hoping between the POC demo and the code comments, it'll be enough for you to implement without too much trouble.

The POC demo is here: http://jslibrary.311servic.es/treetrim.html
The code is here: https://github.com/tkompare/jslibrary

I have plans to add features to this. So consider it a 1.0 release.

Cheers and happy hacking.

Chicago Potholes

Today, I let people know about potholes.311servic.es, a tool to explore pothole repair requests made to the City of Chicago. Here is my letter to the Open Gov Chicago(land) group:

Open Gov Chicago,

I humbly present 'Chicago Potholes', a web application to display and
search all Chicago 311 reported pothole patch requests that are
awaiting service. Included, is a simple address search for all
yet-to-be-serviced pothole repair requests within 1/2 mile of the
given address. Chicago Potholes is friendly to most modern desktop,
tablet, and smart phone browsers. I've tested on every modern desktop
and mobile device browser I could get my hands on.

http://potholes.311servic.es/

Technical notes:
-- All pothole data used for this site is pulled directly from the
City of Chicago Data Portal, using its publicly exposed Socrata Open
Data API 1.0.
-- There is no server-side processing code such as PHP, Ruby, or
Python. All business logic and glue code is written in Javascript.
-- The Ward boundary layer is stored in a Google Fusion Data Table.

Code libraries used:
-- Twitter Bootstrap v2.0.2
-- jQuery v1.7.1
-- Google Maps Javascript API v3.0

Lessons learned:
-- Socrata Open Data API (SODA) version 1.0's JSON data format is very
bulky. Slow on DSL and unbearable on 3G-LTE networks. Because of this,
I had to limit the scope of this web application to only unserviced
pothole requests. This was due to the combination of size of the JSON
payload and the sheer number of pothole records. I'll revisit this
when SODA 2.0 is available (scheduled for April 2012) and then
implemented by the City of Chicago.
-- Twitter Bootstrap is a great design framework for a non-designer or
non-CSS ninja like me. I only know enough CSS to be dangerous, and am
far being able to create a really great looking site from thin air.
-- The object oriented approach I used to author the Javascript for
this site is not well minified by automated tools like YUI Compressor.
I'll have to take a different approach if I want a more compact master
Javascript file.

Future enhancements:
-- Better styling on the Ward layer Info Window (it's terrible)
-- Automatic address/location detection via a device's GPS
-- Inclusion of historical pothole data
-- Map cave-ins versus pot hole patchings
-- Data visualizations. Yet to be determined
-- Basic statistical analysis of pothole requests:
---- What is the median wait time to request completion city-wide?
---- Within 1/2 of an address?
---- Is the difference statistically significant?
---- What about compared to another address' surrounding area?

That's about it. If you have any questions, let me know and I'll try
to answer the best I can. Comments and Criticisms are welcome too.

Cheers.

--
Tom Kompare
tom@kompare.us
@tomkompare

saving users from their wireless mobile data plans

The mobile landscape constantly changes. Devices are faster, screens have more pixels, and browsers are more capable. The wireless networks are getting faster so that downloading all those extra bits for larger/faster devices does not take any longer. For those of us on a limited data plan, this advance in technology is not all good news. Faster and better devices means more data for the same information as content providers take advantage of these new capabilities.

Good web design puts the user first. If designs feed users web content so data rich that it forces them curtail use at the end of a billing cycle because they are nearing their data limit, this is not putting the user first. The only winner in that game is the wireless providers.  Content that is too data rich is less accessible.

What can we do?

• Don't assume mobile devices have lower screen resolutions than desktop. There are smart phone models that have as many pixels as my computer monitor at work. Don't treat them the same. You are making designs that eat data plans for lunch otherwise.
• Accept that users have data limits. There is only one data plan from one major carrier in the United States that still offers unlimited data. More than 90% of us do not have that plan.
• Think of ways to modify designs to limit data download. Most conscientious developers already try to send images that are appropriately sized for a particular user's device. But does that image really need to extend the entire width of the screen for all devices? Image layout is a good place to start looking for savings.

If we want to call ourselves a profession that places the user at the center of our work, we must take into consideration the thickness of their wallet. I, for one, believe users already pay to much for internet access. Just because users can download the 4 times the data in the same amount of time than they could 18 months ago, this does not mean their data plan allows them to download 4 times the data for the same price. Let's design web experiences for users and not their wireless providers.

There is no magic bullet to limit data. I do not have all the answers. So let's start talking.

311 Services

The first thing you learn about users is they like things simple. This was reenforced in first minute of user testing of my new web application, 311 Services. Everyone had a hard time not only remembering the domain name "311servic.es" but also typing it. Anticipating this exact issue, I had also reserved the domain "311services.org". The issue instantly cleared itself up.

I decided that I would offer this web application to the City of Chicago's alderman for free. One reason is that I do not anticipate much upkeep, other than the price of hosting, which is minimal for such a small application. Also, it'll help advertise my work... to whatever end. I'm very comfortable and find my current job challenging and fun. But it never hurts to make yourself known. Another reason I made this free is that I didn't want to figure out a price point, or deal with the tax forms. We all have our shortcomings.

Also, this web application is only temporary in its usefulness. The City of Chicago is scheduled to completely redo their 311 system technology next year; part of that is moving to Open311. This application will be obsolete as soon as that system is in place.

So anyway, here is a short promotional video for the service. Enjoy.

Three things that are obvious right away are:

1. I'm terrible at marketing.
2. I'm not a good web designer or UX specialist.
3. I used jQuery Mobile for presentation. I found it to be very helpful for people like me who have barely adequate front end skills.

Other code libraries uses are jQuery and Google Maps API. The back-end is object oriented PHP. I also separated the PHP presentation templates from the PHP business logic. Things are a mess when the business logic code is all up in the HTML. It's really hard to read and maintain. I've been handed enough junky code to know what's maintainable and what is difficult. The funny thing; its really not all that hard to separate business logic from presentation. Just don't be lazy.

The most interesting and frightening thing I learned was how easy it would be for an application to spy or stalk a user. Giving over location data can be a dangerous thing. Tie that with names, email addresses, and the date/time information in the logs and you have the data for tracking. Smart phone technology really does open up a brand new frontier to unscrupulous persons.

Let me be very clear, this application does not keep any information users enter into the request form or as Javascript/Google Maps API locates their device. I do not want the extra security overhead or liability. The only thing I have are basic Apache web logs. And those don't tell me any personally identifiable.

311Servic.es launch is near

I am working on a civic coding project called 311Servic.es. It is a web application for iPhones and Androids. I have yet to fully test on other devices for lack of access to those devices. A user can request City of Chicago services from their device.


I have worked with my ward alderman (Alderman Joe Moore, 49th Ward) to make sure the application is collection the required information. Another Alderman is interested in activating this 311Servic.es for his ward. 311Servic.es should be ready to go by the end of this week. Just in time for pothole season.

Here is a short video introducing the web application. http://www.youtube.com/watch?v=7LtMHwUu8Sk

I'll write more on the code and techniques used in this application soon. Check back periodically.