Reshaping arrays into 2D matrix

by Maria Jane S. Poncardas, 9/7/2019


I came up with this post since I had been brawling into understanding how this reshaping works. During the past few weeks, I had been working on with the Recurrent Neural Network's Long-Short Term Memory (LSTM) to improve forecasting on generation of National Power Corporation's six (6) hydroelectric power plants in Mindanao. This method is also well-suited to classifying, processing, and making predictions based on time series data such as stock prices, Forex and on household electric consumption. Even a few enthusiasts utilized LSTM for predicting what would be the next word on their 10-word sentence, or next few lyrics of Ed Sheeran's songs. LSTM has quite an edge over the selective remembering of patterns and is relevant in the field of deep learning.

Working with respect on building architectures and fitting up models, it is quite relevant to know how basic reshaping works. I think it helps to visualize. For the simplest example, assuming you have make a 1D array of length 10 (codes through Python),

import numpy as np
import pandas as pd

1D = np.arrange(0,10)
mat = 1D.reshape(5,2)

With this you get 0,1,2,...,9, (exclusive of 10), reshaping should result to having no missing or left out values in a matrix. In this example, variable 'mat' reshapes the 1D array into 2D matrix with five rows and two columns.

However, you cannot reshape it into (3,3), since 9 has no place in this matrix.







Simply put, you can reshape it based on the number of values (length). Then divide it by the number of rows and columns. Say, (5,2) results to 10, unlike (3,3) has 9.


LSTM

Now for the LSTM which expects 3D input layer, it is composed of samples, time steps, and features. One sequence is one sample, one time step is one point of observation in the sample and one feature is one observation at a time step.

When fitting the model and making predictions through LSTM network, it requires us to specify the number of samples provided, number of time steps (time lag) and features. A wonderful and thoroughly explained blog is written by Jason Brownlee through this link, which explains how to setup into different cases such as multiple features and samples.

Comments

Post a Comment

Popular posts from this blog

Using QGIS to convert .shp file to .geojson (GADM)

Image
by Maria Jane S. Poncardas, May 16, 2019      At the moment of writing, there is no direct method on how to utilize raw file from Database of Global Administrative Areas (GADM) into Python script to create charts in Bokeh package. GADM provides maps and spatial data for all countries and their sub-divisions. The goal here is to extract administrative areas in Mindanao (Southern Philippines) as it will be used for spatial and temporal analyses on terrorism (with which utilizes Bokeh for visualization). Sure, there are plenty of workarounds (such as using Plotly or simply Seaborn), but this article focuses on such as it deals with maps. I hope a few portion of this article may help you. CONVERT .SHP TO .GEOJSON USING QGIS Download shapefile from Database of Global Administrative Areas https://gadm.org/download_country_v3.html Extract downloaded file by right-clicking on the file then from the dropdown menu, select “Extract files to <filename>\ ” In the case of Ph...
Read more

Data Cleaning from Global Terrorism Database (GTD)

by Maria Jane Poncardas, 06/14/19 Data preprocessing: The data was collected from an open-source database from the GLOBAL TERRORISM DATABASE website ( https://www.start.umd.edu/gtd/ ) which has annual updates of worldwide terrorist events from 1970 to 2017. The data file was obtained in the website by hovering over “USING GTD” option and then select the “Download GTD”. On the webpage, the “Action” field has a drop-down menu which contains general inquiries and acquisition of GTD file. It contains comprehensive information, news sources and among others. The researchers only selected the most relevant information for the study such as: Date City and Province Latitude and longitude Attack Type: Bombing/Explosion Armed Assault Hostage Taking (Kidnapping) Assassination Facility/Infrastructure Attack Hijacking Hostage Taking (Barricade Incident) Unarmed Assault Target type: Military Private Citizens & Property Government Business Police Transportation Utilities Educational Institution R...
Read more

LSTM pseudocode

Image
by Maria Jane Poncardas, July 19, 2019 Forecasting models are generated based on the training of historical data of six (6) <confidential hydroelectric company> plants, which is from January 1, 2011 to May 4, 2014. The author used Python Jupyter notebook to construct training codes for RNN-LSTM forecasting method. To perform such training, missing values need to be treated hence imputation methods were employed such as Kalman filtering, per plant’s maximum rated capacity and the triple exponential smoothing. The following are the steps resembling information of the code for the forecasting: PREPROCESSING : Import necessary packages and modules for the training: pandas - is used to extract power generated out from the consolidated dataset of excel file Tensorflow’s keras is utilized to build the architecture of RNN and is also used to initiate the model sklearn is for transforming features of data by scaling to a given range, in this case, between zero and one. The dataset being r...
Read more