It's been a while, but we've finally released the latest version of the NYC Geodatabase! The database contains a number of updates:
- New data tables for PUMAs, ZCTAs, and census tracts from the 2014-2018 ACS (replacing the 2013-2017 iteration).
- New data tables for ZCTAs from the 2017 ZIP Code Business Patterns (replacing 2016 iteration). Because of new Census Bureau privacy rules, the number of establishments is suppressed for any place and any industrial sector that has less than 3 business establishments; we've added a new column to our industry table that counts the number of suppressed businesses (by comparing the sum of industries to the published total and taking the difference).
- New subway station, complex, and ridership data for 2018 from the MTA. There is one less subway complex in 2018, do to the construction of a passageway that connects the Cortlandt St RW (formerly complex mn092) to Chambers St / WTC / Park Place ACE23 (complex mn088). Complex mn092 was dropped from the dataset as Cortland St RW is now part of mn088, and all the ridership data for this station was retroactively added to the new, larger complex. The notes table on subway closures has also been updated.
- New PATH station ridership for 2018. For the first time, the NYC geodatabase database includes all the PATH stations in the system, not just the ones in NYC.
- Updated facilities layers for 2019 for colleges, libraries, hospitals, and schools. Unfortunately enrollment data for the schools is no longer published in the facilities database.
As always, we're providing two formats: a SQLite / Spatialite version that's intended for QGIS users, and an MS Access personal geodatabase intended for ArcGIS users. We've updated the data dictionary document, but haven't gotten around to producing new spatial metadata files yet. We'll get those posted in the next month or so.
I have a few data updates to announce before this semester comes to an end. We've updated the bus stop and route layers for the NYC Mass Transit Spatial Layers for December 2019. We decided not to issue updates to the rail and subway files as there were no changes to the underlying data from the static transit feeds. We've also generated an updated list of non-profits in NYC as part of the IRS Tax Exempt Organizations series. The plan for next month is to release an update for the NYC Geodatabase.
The GIS Lab will be closed Dec 23rd through Jan 6th. We'll re-open on Jan 7th for the winter session; see the GIS Lab page for details.
The GIS Lab is officially back in business for the fall 2019 semester! We'll be available Mon-Tue & Thu-Fri 9am-4:30pm, and Wed 1pm-5:30pm. Ryan is also back for the semester and he'll be in on Thu & Fri. Visits are by appointment. See the GIS Lab page for contact info, hours, and exceptions when we'll be closed. The fall semester runs from Aug 27 to Dec 20.
Before the summer was over I managed to post 2018 ridership stats for the NYC subway. Visit the NYC Mass Transit Spatial Layers to access the spreadsheet.
The new data.census.gov has been formally released, and it will replace the American Factfinder as the new census data portal. I've written a new tutorial for it and have updated a few others that mentioned the AFF. Check them out on Census Tutorials page. I'll also be revising my library research guides over the coming weeks.
I decided to delay the release of a new NYC Geodatabase until December or January, because the Census Bureau has delayed the release of the County and ZIP Code Business Patterns data as they're busy tabulating the 2017 Economic Census. This will give us some time to modernize the scripts we use for updating the database. So, the next iteration of the DB at the end of this year will be chock full of new data from the ACS and Business Patterns, and there will be new features for subway stations and NYC facilities like schools, hospitals, and libraries.
Before she left, Chris created two new tutorials to introduce new users to web mapping. Introduction to Carto is for students who register for the Github Student Developer Pack. Learn how to create interactive thematic and reference maps that you can embed and share with others. If you want to create full-fledged presentations that incorporate text, maps, and other multimedia, then check out our ESRI Story Maps tutorial. Create an ArcGIS Online account to get started with telling your stories. These and other tutorials are available via our Resources page.
In other news, I've posted an updated spreadsheet that lists all the non-profits in New York City as of June 2019. See the IRS Tax Exempt Organizations. A reminder that the GIS Lab will be closed for the next two weeks. We'll be back in business on July 15th.
We've just updated our NYC Mass Transit Spatial Layers series, using the MTA's static data feed. This is the most comprehensive update that's we've done in a while, updating stops and routes for the buses, trains, and subway stops. There are a few noteworthy changes. First, the Metro-North routes now actually reflect the routes the trains travel, in that the lines follow the location of tracks. In previous versions the routes were simply straight lines drawn between stations, which made the layer useful for creating metropolitan-level schematics but not much else. With this update, the Metro-North routes layer is now just as good as the subway and LIRR layers for depicting the geographic location of routes:
The second big change is that we've created QGIS style files for the Metro-North, LIRR, and subway routes. If you add the shapefile to QGIS it will read the accompanying qml file by default and assign appropriate colors and thickness to each line, representing how the routes appear on transit maps. The LIRR styles incorporate overlay ordering, so the lines are drawn on top of one another in a way that approximates the transit map. The subway and Metro-North styles incorporate offsets so you can see lines that run side by side, without one line covering up all the others. You can see an offset example in the image above for Metro-North. If you'd prefer not to apply the styles, you can either turn them off in the symbology tab or move / delete the qml file that accompanies the shapefile.
Last, while we've recently been including the color hex code in the attribute table of each of the routes file, we've modified these attributes to insert the pound symbol in front of the six-digit code so you can readily apply these colors in QGIS. For example, for the bus routes if you go into the Symbology Tab under the Properties menu and select Single line, beside the color drop down you can click the data defined button, and for field type string you can specify the color field:
After making the selection and applying it, each line is symbolized using the color stored in this attribute column:
This is a quick way for assigning colors. It won't display the colors by line in the legend; to do that you would need to create a style file. ArcGIS users can also use the colors stored in the table to create layer files, which are the equivalent of QGIS qml styles.
As always, we've moved the older transit layers to our NYC Mass Transit Spatial Layers Archive. No updates yet for subway ridership I'm afraid. We'll keep an eye on it and will post an update shortly after the data becomes available.
It's been a busy semester, and I have a few updates to share. First and foremost, Ryan has finished a new QGIS Raster Tutorial. Our previous tutorial received a lot of downloads but was now too far out of date, so he wrote a completely new one for QGIS 3.4 that covers the fundamentals of working with rasters. It uses surface temperature and land use and land cover data in NYC as examples. The tutorial and sample data are licensed under Creative Commons for anyone to use and share, so check it out!
- I've updated the GIS Practicum manual for QGIS, moving us from 2.18 Las Palmas to 3.4 Madeira.
- Chris has created updated versions of the CUNY Campus Facilities layers for CUNY campus buildings and properties; our first update for this series in several years.
- We've updated the NYC Geocoded Real Estate Sales database, with new data for 2018 sales.
- I've posted new PATH train ridership data on the NYC Mass Transit Spatial Layers page.
Next items on the list: an update of our NYC mass transit features for May and a probably a new version of the NYC Geodatabase in July. Between now and then we should also hopefully have updates for NYC subway ridership for 2018 (the MTA hasn't posted new data yet). I'm working with Andrew at NYU to get many of our updates posted in their spatial repository, as it's been a couple years since we've actively collaborated on this.
I've also posted hours for the GIS Lab this summer. We'll be open for the last week of May and practically all of June, but will shut down June 28 through July 12. We'll open again Mon-Thu for the remainder of July. August is still a toss-up at this point, so stay tuned.
We've managed to squeeze in a few data updates before the end of the year. It's been a year since our last update to the NYC Geodatabase but we finally have a new edition for January 2019. This one contains a lot of updates: new census data tables for the 2013-2017 ACS and 2016 ZBP, ridership data from 2017 for the NYC subway and PATH trains, a new subway stations layer, and all new layers for facilities (colleges, hospitals, libraries, and schools) from 2018. The data dictionary has been updated so you can read about the changes in more detail; in particular there have been some issues with the facilities layers that have forced us to modify those files from previous versions.
Some smaller updates: we have a new data file listing IRS Tax Exempt Organizations in NYC (non-profits) as of December 2018, and have updated all the map links in the NYC data guide to point to the latest version of the census ACS.
The GIS Lab will be closed for a bit until the holidays have passed. Our hours for the winter session have been posted, and midway through January we'll post hours for spring 2019. Happy New Year!
We just completed an update for the NYC Mass Transit Spatial Layers series where we've created new data for the bus and express bus stops and routes. This update is a big one, as there were significant changes to bus stops and routes in Staten Island. We also updated the subway stops layer as a new attribute column was available that indicates whether the stop is underground, above ground, or elevated. We chose not to update the subway routes or any of the rail layers; for one thing there are no changes from the current source data and the last published version. The other reason is that we make a number of manual fixes to points and lines because the data is bad (Metro North) or the MTA has failed to update it (the subway routes file in the GTFS static feed still does not include the 2nd avenue subway, two years after it opened). Rather than redo all of our work, we're keeping the same files since there were no actual changes.
We're a little behind in releasing updates for our datasets, since I was away last academic year and we're still in the process of getting the lab back in shape. We recently completed a few updates:
- NYC Geocoded Real Estate Sales: we've added sales for 2017. There's a shapefile with all 2017 sales, and an updated Spatialite database that contains sales for all years from 2003 to 2017.
- NYC Mass Transit Spatial Layers: we've updated the ridership statistics to include data for 2017 for the NYC subway and the PATH train. The data is published in spreadsheet format.
Updates for the future? We're aiming to release a new version of our NYC mass transit layers (points and lines for buses and trains) for the month of November before this semester is over. The next version of the NYC Geodatabase will be the January 2019 iteration, which will include not just the usual ACS updates but all the updates we missed this summer (schools, hospitals, libraries, subway stations and ridership, and ZBP data). Stay tuned!
GISMO (the local chapter of the NY State GIS Association) is hosting a GIS job fair on Wednesday November 14, 2018 at Hunter College CUNY. The fair runs from 1pm to 5pm and is located in the Hunter College West Building. For more information and to register visit http://www.gismonyc.org/events/upcoming/.
The library has recently purchased updated data from the China Data Center: the 2015 edition of the China City Statistical Indicators with Maps. Data is provided in point and polygon shapefiles and in a tabular Excel format for Prefecture-level and County-level cities. Cities in China differ from the North American model, in that they represent areas that contain both urban and rural components and they cover the country in it's entirety. The point shapefiles represents the center of the urbanized area, while polygon files represent the entire administrative or legal area. Provinces are the 1st-level administrative sub-divisions of China, Prefectures are the 2nd-level, and Counties are the 3rd.
The data includes several demographic and socio-economic indicators, many for the prefectures and fewer for the counties. You can view an index of the available variables and files on the China Data Center Datasets page. The data is copyrighted for educational, non-commercial use and our license permits us to share the data only with current Baruch students, faculty, and staff. Current members of the Baruch College community can email us (using your CUNY email address) to request access to specific files.
Registration is now open for the fall semester’s GIS (geographic information systems) Practicum, Introduction to GIS Using Open Source Software (featuring QGIS). The sessions will be held in the GIS Lab at Baruch College:
- Friday Oct 26th
- Friday Nov 16th
The day-long workshop runs from 9am to 4:30pm. Current CUNY graduate students, faculty, and staff, and full-time Baruch undergrads are eligible to register. Advance registration is required; the fee is $30 and includes a detailed tutorial manual and a light breakfast. Participants must bring their own laptop with QGIS 2.18 pre-installed in order to take the class. Visit the GIS Practicum page to learn more and to register: http://guides.newman.baruch.cuny.edu/gis/gisprac.
If you are using the NYC Geodatabase with QGIS 3.2 (and possibly 3.0) and are not able to view certain layers (i.e. you drag them into the map view and nothing appears), this is due to some bug with how QGIS reads Spatialite layers that have spatial indexes. The following layers are affected: a_pumas2010, a_tracts, and a_zctas. To get them to display, you can disable the spatial index. You can do this is the Spatialite GUI or the QGIS DB Manager by running this command in the SQL window:
SELECT DisableSpatialIndex('LAYER', 'geometry');
Where LAYER is the name of the layer in quotes, i.e. 'a_tracts'. Run this command on each of the three layers. Then refresh the database or remove the connection and re-establish it, and try adding the layers to the view. You should be back in business.
Alternatively, you could go back to using QGIS 2.18, which is still the long term release and is inherently more stable and hassle free.
Janine's last day working for the GIS lab today - she will be sorely missed! While she and Anastasia are off to new adventures and my academic leave continues during the summer, the lab will be closed in July and August. I'll be back the first week of fall and will be recruiting for a new lab assistant position (possibly two), so look for an announcement here sometime in August.
Janine was pretty busy before she left - here are some updates:
- The national IRS Migration Database has been updated with a new year of data. For state to state and county to county flows the newest year is 2015-16.
- The NYC Mass Transit Spatial Layers have been updated with new stop and line features for all the buses and the subway. The subway update reflects the re-opening of the South Ferry station and the shut down of the older South Ferry Loop at the southern end of the 1 Line. The source data for the regional trains hasn't changed, so we skipped updates for those.
- We have a new IRS Tax Exempt Organizations file for NYC for June 2018, listing all the non-profits in the city.
Some of the other datasets that we usually release around this time of year will be delayed until I return in the fall. This includes: 2017 ridership data for the subway and PATH, NYC real estate sales for 2017, and an updated version of the NYC Geodatabase.
As for the GIS Practicum, I won't be updating the manual this year as QGIS 2.18 will continue to be the long term release (LTR) until the end of October 2018. I plan on running a couple of workshops in the fall based on it, and you can sign up to be notified by email when registration opens. As the final version of the 2.x series 2.18 will continue to be supported for a few more years. I will eventually update the manual to the new LTR 3.4 in 2019, but it probably won't happen until mid to late spring.
We've recently updated several of our datasets:
- NYC Mass Transit Spatial Layers: Janine created updated files for the buses last semester; since there were no changes to the subway and train files we let those go.
- NYC Geodatabase: I created version jan2018 with updates to the census American Community Survey data tables for PUMAs, ZCTAs, and census tracts. The new tables are from the 2012-2016 ACS.
- IRS Tax Exempt Organizations: We added a new file for dec2017, a list of all the non-profits in NYC.
- US Presidential Election County Results: The government deleted this data from their servers, so we've saved it. Boundaries and tables for 2004, 2008, and 2012.
The GIS Lab is now open for business for the spring semester. I'm still away on leave until the end of August, but Janine continues to captain the ship and is in on Thursdays and Fridays. GIS Lab hours for the spring are posted.