This weekend, Saturday Feb 23, is International Open Data Day, sponsored by the Open Knowledge Foundation, and it is also Code Across America data, sponsored by Code for America. San Diego is participating, hosted by Open San Diego, and the Library staff will be there to talk about the Library and our Crime Mapping Project. If you’d like to participate in the conference sessions or hackathon, visit the EventBrite page to sign up. Hope to see you there.

I shouldn’t be surprised that for any interesting technical idea, XKCD covered it first, and better. Here is another view of the issue we covered regarding crime mapping: crime is most common where (a) there are a lot of people and (b) there is alcohol, and (b) is often the reason for (a).     To go ahead and ruin the joke with over analysis, the maps in the cartoon aren’t exactly the same. So, it might be interesting to subtract them from each other and explore the differences.

In preparing to formally launch the San Diego Regional Data Library, we’ve been spending a lot of time talking to nonprofits, community planing groups, city staff and city council members about their data needs. The staff for District 9 mentioned that Council member Marti Emerald is interested in infrastructure issues, particularly how crime is related to infrastructure deficits. Her staff specifically mentioned street lamps. Fortunately, we have a fantastic resource in SanGIS, and they have published a shapefile with the locations of San Diego Street Lamps. So, we went to work extracting and processing street lamp data. Based on this data, we’re creating density maps of street lights, which we will correlate with crime incidents. The map shown here is one of the example maps we’ve produced, to experiment with how best to represent the data. For this map, I wrote a Data Bundle that converted the Shapefile of street lights provided by SanGIS into a Sqlite file, which was then loaded into an image. ( You can get the code for my bundle on GitHub. ) The processing code bins the location of the street lights to 1/4000 of a degree, about 18 meters at out latitude, then for each street lamp, adds a matrix onto the output image. The matrix spreads out the point, a bit like the light from the lamp would spread out. Here is a high resolution view of the output, showing Uptown, Downtown , the Park and Golden Hill. ( Click to expand ) This is, of course, a synthetic view of light in San Diego, but NASA has generated the real view, by photographing the earth at night. Here is our map laid over NASA’s image: Our data covers only San Diego, so there are many areas that are lit where we don’t have street lamp records, and it also appears that from space, light spreads much more than we estimate with our spreading matrix. Regardless, the maps show a good correspondence. This is a simplistic analysis, since it doesn’t consider the different types of lamps, their power, or the occlusion of buildings, but it does illustrate how to make and present a density map. This is the first step in our crime analysis project; the next steps will study how the locations of street lamps relate to the locations of crime. For the full project, we will have to consider that the correlation between crime and street lights isn’t only one way. There may be more crime in dark areas, but the city may also put more lights in high crime areas, breaking a simple correlation. So, we will probably focus on finding areas that are both dark and have high crime. If you like this sort of analysis, please join our Meetup group or connect though Facebook, Twitter or email to get the announcement for the start of our crime analysis project. We’d love to have your help!

The Corporation for National and Community Service is a federal agency that encourages US citizens to volunteer, particularly for federal programs like AmeriCorps. They also have an excellent data site that provides statistics on civic engagement. We used their data extraction tool to pull a few datasets, which are posted on the Library’s data repository. Here is a bar chart of the percentage of the population of large US cities which, in the previous 12 months, volunteered for a formal organization. The chart shows the averages for the years 2004 to 2011, including the standard deviation from year to year for a single city as an error bar. The numbers on the right indicate rank.

Animated map of Bikeshare station utilization. Cities thrive when citizens get directly involved in working for their city. Not only do engaged residents solve problems and guide development, but the fact they got involved makes them love their city more, and encourages the people they socialize with to also care more about their city. Because loving something and watching it grow just feels good, creating opportunities for city residents to improve their communities has benefits both for the place and for the person. Both Boston and Washing DC have popular bikeshare systems. Massachusetts’ is called Hubway, and in DC it is called Capital Bikeshare. Both are using a system operated the company Alta Bike Share, and both are very open with their data. Here is the download page for the trip data for Capital Bikeshare, where you can get detailed information about where each bicycle was rented from, and where it was returned to. Because the data is easily available many people have used the data to create their own analyses and visualizations. Some of the visualizations include an animation of riders leaving a sports stadium after a game or interactive exploration of trips between stations. Releasing the data has also resulted in useful statistics and crowdsourced suggestions for improvements. Boston embarked on a grander project, a data challenge contest that invited residents to create the best visualizations of the Bikeshare data. The Hubway Data Challenge garnered 67 entries from citizens. The contest resulted in some new insights for the people who manage the Bikeshare system, but it also gave the community a deeper understanding of how the Bikeshare system works and why it is valuable, creating a more informed electorate that will make beter future decisions. Part of the mission of the San Diego Regional Data Library is to link volunteers with city planning groups and social non profits to perform data analysis and create new visualizations, so we very interested in these sorts of events in other cities. Since the major way that nonprofits use data is to tell stories, visualization contests can generate a lot of new ways to organizations to explain what they do or help well they are doing it, both of which are critical components to getting funding and support from the community. Visualization contests are not a substitute for the deeper commitments that people make to town councils, planning groups or non profit social organizations, but they can be an important step in a pipeline that leads from people just living in a city to actively participating in its development.

Last week I showed a wonderful heat map of crime from Trulia, that made it more clear where crimes were actually occurring. Unfortunately, the map does not really have much information; anyone who lives in the area already knows where the crime is. To illustrate, compare the crime map: Heat map of Pacific Beach, from Trulia.com To this map: Alcohol Permits in Pacific Beach

This is, unfortunately, how data sharing actually works.

US Map Simulated in Excel. UXBlog has a wonderful guide to making fake maps in Excel. The technique involves putting values from a dataset into bins by rounding latitude and longitude, so that each cell in the spreadsheet represents one degree of lat/lon. If the color of the cell is computed from the number of datapoints that fall into each cell, the data will naturally represent a map.

As part of working with the City of San Diego to develop an Open Data Policy, I and others have proposed that the city create a position for a Chief Innovation Officer, a person dedicated to identifying and developing new ways for the City to accomplish it’s job. To get a better idea of what a Chief Innovation Officer would do, see the job description for the position from the city of Austin. It’s by far the most polished job description I’ve ever seen, and demonstrates how significant the role will be to Austin’s future. It is important to note that this is not a technical position; the most specific skill requirement is ‘people skills.’ Innovation involves technical problem solving, but the most important skill is linking people who have problems with the people who can solve them, and ensuring the two are speaking the same language. If you are interested in the position, you’re too late. Austin hired Kerry O’Connor, previously a program director for the State Department.