Using QGIS to convert .shp file to .geojson (GADM)

by Maria Jane S. Poncardas, May 16, 2019


    At the moment of writing, there is no direct method on how to utilize raw file from Database of Global Administrative Areas (GADM) into Python script to create charts in Bokeh package. GADM provides maps and spatial data for all countries and their sub-divisions. The goal here is to extract administrative areas in Mindanao (Southern Philippines) as it will be used for spatial and temporal analyses on terrorism (with which utilizes Bokeh for visualization). Sure, there are plenty of workarounds (such as using Plotly or simply Seaborn), but this article focuses on such as it deals with maps. I hope a few portion of this article may help you.


CONVERT .SHP TO .GEOJSON USING QGIS

  1. Download shapefile from Database of Global Administrative Areas https://gadm.org/download_country_v3.html

  2. Extract downloaded file by right-clicking on the file then from the dropdown menu, select “Extract files to <filename>\ ”

    1. In the case of Philippine shapefile, extracted files consist of four levels of information, each with five (5) file types (.shp, .dbf, .prj, .cpg, .shx):

      1. filename_0 is for country’s boundary

      2. filename_1 is for province boundaries

      3. filename_2 is for cities/municipalities boundaries

      4. filename_3 is for barangay boundaries

  3. If QGIS is unavailable, you can download and install QGIS based on your machine’s bitness for free from https://qgis.org/en/site/forusers/download.html

  4. Once setting up and installation is done, open QGIS software.

  5. From the folder of extracted files, select the .shp files and drag it over to the QGIS window.

  6. From the lower left panel of QGIS, you can toggle off or on the shapefile layers. 

C:\Users\mponcardas\Desktop\Bokeh_details\Images\QGIS.png


  1. On the same panel, right-click on the necessary layer, select Export > Save Features As…

  2. A window will prompt wherein you can fill in information such as Format and File names.

  3. Click OK.


INSTALLATION OF ANACONDA, INSTALLATION OF EXTENSION PACKAGES, AND RUNNING JUPYTER NOTEBOOK

  1. Download the Anaconda Installer from https://www.anaconda.com/distribution

  2. Double click the installer to launch

  3. Click Next.

  4. Read the licensing terms and click “I Agree”

  5. Select installation method “Just Me” or for all users

  6. Select a destination folder to install Anaconda and proceed to the installation process

Installing packages in Python have dependencies on other libraries, especially the “Geopandas” and “OSMnx” library for this study, hence, it is necessary to specify the install order in python package management to satisfy its prerequisites.

  1. Installation of extension packages for geospatial analysis:

    1. From the start menu, open “Anaconda Prompt”

    2. Install packages by typing in the prompt window: pip install <package>

    3. Packages (load order):

      1. numpy

      2. GDAL

      3. pyshp

      4. Shapely

      5. Fiona

      6. geopy

      7. pyproj

      8. descartes

      9. geopandas

      10. bokeh

      11. pandas-bokeh

There are packages that are not available in their respective repositories, however, they can be downloaded from an unofficial windows binaries https://www.lfd.uci.edu/~gohlke/pythonlibs/

If you have downloaded the package from an external source, type: pip install <file location>/<file name.extension> on the Anaconda prompt


  1. To avoid importing issues with OSMnx (tool used to retrieve, construct, analyze and visualize street networks from OpenStreetMap’s APIs), install in Anaconda prompt by typing: conda install -c conda-forge osmnx

  1. After completing the installations, run the Jupyter Notebook from the Start Menu.

REMOVE LUZON AND VISAYAS INFORMATION OF THE GADM MAP USING PYTHON

  1. Run Jupyter Notebook

  2. Import libraries: geopandas (as gpd), numpy (as np), matplotlib.pyplot (as plt)

  3. Open file by using geopandas package: df = gpd.read_file(‘gadm36_PHL_2.dbf’)

  4. Set index of the file as provinces: df.set_index(‘province_column’, inplace = True), ‘inplace’ is used to apply the indexing to the data frame.

  5. Remove rows of the unnecessary provinces: df.drop([‘province_name1’, ’province_name2’, …], axis = 0, inplace = True)

MAP DATA VISUALIZATION

  1. Run Jupyter Notebook

  2. Import libraries: geopandas, numpy, pandas_bokeh

  3. Create a python environment through os.environ

  4. To show the map with the OpenStreetMap as a Geographic Layer, call .plot_bokeh() to the dataframe

  5. You can play with the data visualizations such as figure size, category, show_colorbar, colormaps, hovertool_columns, tile_provider.


PANDAS-BOKEH

  1. Take note that once you import the Geojson file of your geospatial map, you need to setup the type of projection that will show in your OpenStreetMap (live map). 

    1. First, you need to import the os module in jupyter notebook (import os)

    2. Create an environment to place your map: os.environ[‘PROJ_LIB’] = ‘Testdata’

    3. Define your shape projection

      1. If you need to show square shapes only, you need to input the corresponding code: df.crs = {‘init’:’epsg:3857’}

      2. If you need to show the actual map, use ‘epsg: 4326’

EPSG stands for European Petroleum Survey Group. They publish a database of coordinate system information, and the Projection Engine uses a modified version of the EPSG model.


RELEVANT PACKAGES

  1. numpy – efficient container of multi-dimensional array of data

  2. GDAL – Geospatial Data Abstraction Library, used for manipulating geospatial raster data

  3. pyshp – used to read and write shapefiles

  4. Shapely – manipulation and analysis of planar features/geometries

  5. Fiona – used for reading and writing geospatial data formats

  6. geopy – locate the coordinates of addresses, cities, countries, and landmarks using third-party geocoders

  7. pyproj – performs cartographic transformations between geographic (lat/lon) and map projection (x/y) coordinates.

  8. descartes – use geometric objects as matplotlib paths and patches

  9. geopandas – allow spatial operations on geometric types

  10. bokeh – interactive visualization library through web

  11. pandas-bokeh – backend visualization of Bokeh.

Comments

Post a Comment

Popular posts from this blog

LSTM pseudocode

Image
by Maria Jane Poncardas, July 19, 2019 Forecasting models are generated based on the training of historical data of six (6) <confidential hydroelectric company> plants, which is from January 1, 2011 to May 4, 2014. The author used Python Jupyter notebook to construct training codes for RNN-LSTM forecasting method. To perform such training, missing values need to be treated hence imputation methods were employed such as Kalman filtering, per plant’s maximum rated capacity and the triple exponential smoothing. The following are the steps resembling information of the code for the forecasting: PREPROCESSING : Import necessary packages and modules for the training: pandas - is used to extract power generated out from the consolidated dataset of excel file Tensorflow’s keras is utilized to build the architecture of RNN and is also used to initiate the model sklearn is for transforming features of data by scaling to a given range, in this case, between zero and one. The dataset being r...
Read more

On-the-Job training

Image
by Maria Jane Poncardas      When I was scrolling through my documents in Physics. I stumbled upon the daily journals of my on-the-job training weekly experiences as we were mandated to write our journey. I thought I would share my immersion experience in the National Institute of Physics, University of the Philippines - Diliman through this blog as it is a precious moment in my physics life.      When we were looking for a company and laboratory to have our OJTs, we were encouraged by our seniors to gain relevant research skills through the Condensed Matter Physics laboratory (CMPL) in National Institute of Physics, UP Diliman, since the physics laboratory in our university (Mindanao State University - Iligan Institute of Technology) has a solid foundation in the field of Materials Science research. We have bountiful of options to choose from the National Institute of Physics, as they are engaged in various research areas: condensed matter, instrumentation...
Read more