In preparation for the Data Science for Non Profits meeting tomorrow night (Wednesday, 9 Oct), I’ve created a Google Colab notebook demonstrating how to access the homeless sleeper dataset and compare the distribution of homeless sleepers to crime, pedestrians and businesses. The main output is this chart ( click to expand).
[ To run this notebook, click on the “Open In Playground Mode”, or make a copy of it to your account. ]
After 8 months of hard work form many volunteers and staff, we’re releasing the first version of the Downtown Homelessness Dataset. This dataset has about 41,900 observations of homeless sleepers on 287 blocks in Downtown San Diego, over 62 months.
The dataset is part of our Homelessness Collection. If you’d like to learn how to work on this dataset and the others in the collection, join us for a working session at the next meetup for Data Science for Non Profits, Wednesday 9 October.
San Diego has the world’s largest municipal Internet-of-things deployment, which I got access to recently and have been pulling data from. I’ve now got about 30 million parking records – events when a car enters or leaves a parking zone – and have been processing and analyzing the data for the last few weeks. The raw data is a bit difficult to use, because there are a lot of spurious events, but it isn’t hard to clean them out. The result is records, at 15 minute intervals, of the number of cars parked in each of the 6,600 parking zones across the city. Here are the two processed datasets.
We’ve updated our crime data collection with files for 2016 and 2017. Our crime data collection now runs from 2006 to 2017, with two major files:
I’m in training this afternoon for using the API for San Diego’s CityIQ Smart street lamps. These street lamps have cameras and computers that can detect and report:
I’ll be working on extracting some of this data into an analysis package, so let me know if you’d like to work on a data project using the data.
Tonight at Downtown Works, SCALE San Diego and Open San Diego are hosting a workshop on analyzing crime data with Python, Pandas and Matplotlib. Unlike past analysis we done at the Library on San Diego Crime, this analysis uses data from the San Diego Police, rather than the whole county data from ARJIS, so it is more focused, and more detailed. If you’d like to go, visit the signup page to register. It is an interesting dataset because, while the location addresses are often missing, and it doesn’t have UCR crime codes, it does have detailed call types and is linked to SDP beats. The 2.5 years of data that are published have more than 1.1M records. The San Diego Open Data Portal also has shapefiles for the beats, districts and neighborhoods, so we can make maps. For instance, here is a choropleth that is colored according to the counts of calls for “LOUD PARTY” by beat. It should surprise no one where the hotspot is: 
The dataset also has very nicely formated date/times so Pandas has an easy time extracting time parts, allowing us to build Rhythm Maps. This heat map displays the count of LOUD PARTY incidents, over the whole dataset, organized by hour and month" 
You can clearly see two significant patterns: Loud Party calls are, as expected, primarily made in the late night and early morning, and the calls are more frequent in the summer than in the winter. There are certainly a lot of other interesting patterns to find in this dataset, and if you are interesting in finding them, I hope to see you at tonight’s meeting. BTW, if you’d like to see how these charts were generated, the Jupyter Notebook is in Github. eric.
To announce the arrival of a new set of crime data, our next meetup will be a mini data contest, with a $100 prize for the best student analysis. In this meeting, we will present the new Crime Incident dataset and talk about how to link it to other social datasets. After the presentation, we’ll challenge you to do you own analysis, with a $100 prize for the best analysis from a student, undergraduate or lower. Then, for the next meeting, we’ll invite the best analysts to present their findings and techniques.
When we last requested crime data from SANDAG, 3 years ago, it took four months of negotiation to get them to admit they could produce it, and two more months to get the price down to a reasonable amount. Last week when I requested an update, I got one clarification email, then a phone call, and the files were in my inbox a few minutes later. Thanks SANDAG! As a bonus, the data is now geocoded to census block and track, making the files more immediately analyzable. You can find both the old data file and the new one in an Ambry package on our new data repository.
Last week the Library worked with Wendy Fry at NBC7 to analyze GPS records of street sweeping and parking violations. We haven’t got the data online yet – it’s about 3GB, 11M GPS records – but let us know if you are interested in accessing it. A condensed version of the data is available as a heatmap of blocks where tickets were issued on days the streets were not swept.
For the last 5 months, SANDAG has been publishing their crime incident data to the web. The file they publish only stores the last 180 days, and it is a bit hard to find, so we’re archiving the files to our data repository .