Sponsor a Project in Our Student Data Analysis Contest

Give your nonprofit, agency or news organization valuable data-driven insights by sponsoring a project at our student data analysis contest. The San Diego Regional Data Library, the SDSU Society for Statisticians and Actuaries and Teradata are organizing a data analysis contest to aid nonprofits, journalists and government agencies in making better use of data, develop a broader regional capacity for data analysis, and introduce students interested in data analysis to future employers. Contestant teams will have one week in early March 2015 to answer a set of data-driven questions and visualize the results for one of four projects. Each project will be provided by a nonprofit, government agency or news organization. The contest will be announced in early January, primarily to college students. For a month before the contest begins, the San Diego Regional Data Library will run a special session of its Practical Data Program to train contestants on using data, Python, R, and IPython to analyze data. If you are interested in being involved in the contest, as a project sponsor, mentor or contestant, visit the overview page or contact Eric Busboom at eric@sandiegodata.org, or (619) 363-2607.

Data.gov is the top level data search system for the US, with references to over 130,000 datasets from federal and state agencies. And yet, I’ve never successfully used it for finding data. Here is an example search for “Diabetes Rates”: Search for “Diabetes Rates” on Data.gov   So, we look for diabetes, and get births, 22 year old mortality data from the US Geological Survey, and quality of service data as the first three hits. The first link at least points to the right agency, but you still have to click three times to get there. Here is the same search on Google:   Search for Diabetes Rates on Google.   Not only do I get links to real primary organizations, the fourth hit is the original source of the data, and Google helpfully gives us quick stats before the hits. Even better, if you search for “diabetes rates data” you are one click away from the primary data source for US diabetes rates. The really poor quality of search results on Data.gov has been a problem for its entire existence; I’ve never done a data search on Data.gov that returned what I wanted. I’ve always had better results with Google or browsing the website of the agency that produces the data. Data.gov has been around for about 5 years, and despite human curation is still isn’t as useful as Google’s automatic index. At some point, I’d like to stop being excited by its possibilities, and start being excited by it utility.

Here is the Tableau workbook that we’ll be using for the SPJ Hacks/Hackers Data Show event tonight. You can also get links to all of the documentation from our data warehouse.

Voice of San Diego has published a story we supported, a fact-check about the density of veterans in San Diego County. You can get all of the data behind the story – including schemas and SQL queries ! – in our data warehouse.

Once you’ve got the basic skills in programming and statistics, the best way to learn data analysis is to do it. So, we’re developing a practical experience program for aspiring data analysts. The program is in the pilot phase with a small set of students, but you can read about how it works on the Internship Program’s wiki page. The goal of the program is to develop more experience with answering questions with data in San Diego and to make that experience available to non profits, government agencies, journalists and other organizations that have questions that could be answered with data, but don’t have the time or skills to do it. We’re currently recruiting participants, mentors and projects. So, if you’d like to develop data skills, teach data skills, or have a data project, let us know.

Here are the files for my presentation to the San Diego Crime Analysts Association:

• PDF File
• PPT File

Last year, the U.S. Department of Housing and Urban Development commissioned a report to study the feasibility of creating a nationwide database of parcel information. This is a difficult task because the parcels are usually maintained by the counties, and the US has about 3200 counties. The resulting report is remarkably thorough, including a description of the data collection process and the effort required to get the data, and the information contained in each county’s dataset. The effort required varied greatly, with 13% of the counties requiring more than 6 phone calls just to identify a primary contact for the data, sometimes taking 2 hours more of total time. These difficulties are a major reason that we need more Open Data policies, and why the Open Data movement should eventually move from the municipal level where it is now to push more counties and states to adopt large-scale Open Data policies.

The Voice of San Diego is running a Q&A regarding Open Data, which just happens to involve an interview with the Director of the Library.

Today we are releasing an update to our data repository, at /dataset/. This release is based on CKAN 2.1, the same software used to run Data.Gov, one of the largest data sites in the world. The software is more capable than the older version, and it is much more attractive, with better visual design and organization.

Use your hacking skills for a better San Diego! Open San Diego, San Diego Code For America Brigade, Cali IT2, and OpenGeo are sponsoring a Civic Day of Hacking. The San Diego Regional Data Library will be presenting many of our datasets and running a crime analysis project, so join us if you’d like to learn how to use Python and QGIS to work with crime, census and infrastructure data. See the Code for San Diego website for more information.