One of the difficult parts of working on a Data Library project is setting up all of the tools, including Python, Jupyter, Metatab, Pandas and several others. To make it easier to get started, I’ve set up a hosted instance of JupyterHub, which will allow analysts to run Jupyter notebooks in the cloud. And, the instance uses Redmine for authentication, so you won’t need to create yet another account.

Once you have joined a project and gotten a Redmine account, you can use your Redmine account username and password to log into jupyter.civicknowledge.com. You’ll get a new JupyterLab server. JupyterLab is a lot like Jupyter Notebook, but has a spiffier interface.

For its data program for Summer 2018, The San Diego Regional Data Library proposes to build a data repository for water quality data, to make it easier to find water quality data, link it together and to create maps, reports and analyses. The project will be run with a small paid staff and a group of volunteers, and if the project is successful, we will pursue funding for continued maintenance. This project is over, but see theproject website for archived results. Data we expect to collect, process and store will include:

For the Spring and Summer of 2018, The Data Library is recruiting a part time intern to support the Summer projects. This position will involve a range of activities that includes community engagement, outreach and project management.

• Project Management. Work with project teams to recruit participants, track progress and set up meetings.
• Outreach. Identify and contact nonprofits, journalists, and government officials and departments who may be interested in sponsoring data projects or using the results of existing data projects.
• Community Engagement. Attend technical events to talk about the Data Library and recruit analysts.

The position will pay $15 per hour for up to 80 hours per month. To apply, email eric@sandiegodata.org with: a URL to your LinkedIn page, a description of which San Diego area social and civic organizations are most likely to be interested in pro bono data analysis projects, how you would contact those organizations, and a brief description of your pitch to them.

For Spring and Summer 2018, the Data Library will be hosting summer analysts and data scientists to work on special projects.

Special Social Analysis Projects

Pursue special projects in social analysis with large, individual-level datasets like IPUMS, GSS and NLSY. Social Science students, or any student interested in applying data science and statistical techniques to social questions, can learn to work with some of the most important datasets for studying individual attitudes, behaviors and beliefs. Each project will have 1 or two team members and will answer questions chosen by the students under the guidance of Data Library staff and analysts.

We’re working on building a new knowledge base, a collection of example Jupyter notebooks related to projects the Library has been working on. The notebooks include detailed analyses using Jupyter, Python, Pandas and other tools, with most of the notebooks covering crime and demographics. For instance, the Demographics and Business Loans notebooks shows how to fetch American Community Survey data and create a radius around the highest minority area in San Diego, displaying it a Leaflet Map: Or, from the San Diego Police Calls For Service, a rhythm map of reports of loud parties, by neighborhood.   Loud Party Rhythm Map Each of the notebooks includes a link to our example notebooks github repository, so you can also get the full source and run them yourself.

Tonight at Downtown Works, SCALE San Diego and Open San Diego are hosting a workshop on analyzing crime data with Python, Pandas and Matplotlib. Unlike past analysis we done at the Library on San Diego Crime, this analysis uses data from the San Diego Police, rather than the whole county data from ARJIS, so it is more focused, and more detailed. If you’d like to go, visit the signup page to register. It is an interesting dataset because, while the location addresses are often missing, and it doesn’t have UCR crime codes, it does have detailed call types and is linked to SDP beats. The 2.5 years of data that are published have more than 1.1M records. The San Diego Open Data Portal also has shapefiles for the beats, districts and neighborhoods, so we can make maps. For instance, here is a choropleth that is colored according to the counts of calls for “LOUD PARTY” by beat. It should surprise no one where the hotspot is: The dataset also has very nicely formated date/times so Pandas has an easy time extracting time parts, allowing us to build Rhythm Maps. This heat map displays the count of LOUD PARTY incidents, over the whole dataset, organized by hour and month" You can clearly see two significant patterns: Loud Party calls are, as expected, primarily made in the late night and early morning, and the calls are more frequent in the summer than in the winter. There are certainly a lot of other interesting patterns to find in this dataset, and if you are interesting in finding them, I hope to see you at tonight’s meeting. BTW, if you’d like to see how these charts were generated, the Jupyter Notebook is in Github. eric.

Tonight we’ll be meeting to talk about two important collections of survey data, The General Social Survey (GSS) and a broad collection of integrated survey datasets, IPUMS.org. Additionally, we’ll be meeting with David Lynn, founder of Mission Driven Finance, to talk about his volunteer data project to analyze business lending to minorities in San Diego county. In the main meeting, I’ll show how to use the web-analysis tools for the GSS and IPUMS. You can visit those sites now to explore a bit, and it would be worthwhile for you to create an account before the meeting. Here are the important URLs:

In this Hands-On presentation, we’ll be exploring social issues using two sources of microdata: The General Social Survey , a 40 year survey program that asks a representative sample of Americans questions on a huge number of social issues, and IPUMS, a curated, processed collection of international survey and census data. Using these two sources you can study a wide range of social questions.

For instance, here is an analysis of the relationship between Occupational Prestige and Personal Income, using the IPUMS online analysis tool, with data from the ACS 2010-2015:

We’re excited to announce the first of our next round of social data projects. Mission Driven Finance is a San Diego impact investment fund that provides loans to nonprofits, social enterprises, and businesses that benefit the San Diego community. MDF is interested in helping businesses understand the regional lending landscape and advocating for improvements to make it easier for businesses that have a hard time getting financing to be able to get the money they need to grow. This project will explore:

link Disadvantaged Student Performance For Summer of 2017, The San Diego Regional Data Library will be running a data internship program, working on pro-bono data analysis projects for nonprofits, governments and journalists in San Diego County. If your organization has questions that can be answered with data, you can have an undergraduate data analyst work on your project, with professional guidance, for up to 12 weeks. Project sponsors must be able to describe their needs as a set of questions that can be answered with available data, and must be able to meet with the interns at their site for 1 hour per week. Projects must have a social goal or benefit. We’re currently running two projects, and expect to be running three or four more this summer. If your organization needs data analysis this summer, please contact the Data Library Director, Eric Busboom at (619) 363-2607 or eric@sandiegodata.org.