Research Guides: Data Science: Data Finding &amp; Processing

Data Sources: QC Databases

Audit Analytics This link opens in a new window
Users can register for an account by following the instructions provided.
Bloomberg Tax This link opens in a new window
Users are required to create an account and MUST use their Queens College email to Register!
Access is available to 100 concurrent users.

Gartner This link opens in a new window
ICPSR This link opens in a new window
Infoshare Online This link opens in a new window
New data is not added to the system. It is relatively inactive.
Gale Primary Sources: Nineteenth Century Collections Online This link opens in a new window
Social Explorer This link opens in a new window
To use Social Explorer remotely, save, and share projects, please create a personal account.
Statistical Abstract of the U.S. This link opens in a new window
Child & Family Data Archive This link opens in a new window

Healthcare Administration Database This link opens in a new window
MathSciNet: Mathematical Reviews This link opens in a new window

Search Library A-Z databases to find more data in contexts.
CUNY Academic Works

Open Data Sources, General

Research Data Journals

Data in Brief

-- Policies and Guidelines
-- Public repositories to store and find data

Research Data Repositories

https://dataverse.harvard.edu/

The Harvard Dataverse Repository is open to all researchers, both inside and outside of the Harvard community.
It contains over 100,000+ multidisciplinary datasets from published studies by researchers within and outside the Harvard community.

datadryad.org

Dryad is a non-profit initiative organized by several academic institutions, research societies, and publishers to archive and share data.
Covers a broad range of research topics in natural and life sciences.

ieee-dataport.org

IEEE DataPort provides datasets from studies published in their many scholarly journals.

Directory of Data Repositories

Open data directories curate and organize lists of high-quality scholarly/research data repositories by subject area.

re3data.org

Contains over 2,000 research data repositories indexed.
Provides multidisciplinary data on Humanities & Social Sciences, Natural Sciences, Life Sciences, and Engineering Sciences.

oad.simmons.edu/oadwiki/Data_repositories

Open Access Directory (OAD) is maintained by a community of researchers and scholars.
Data repositories are categorized by research area.

	DATA.GOV Contains over 200,000+ datasets collected from its many federal agencies and departments.
	census.gov/data From the U.S. Census Bureau. Provides quality demographic data about U.S. population and economy.
	healthdata.gov From the US Department of Health & Human Services. Provides data on environmental health, medical devices, Medicare & Medicaid, social services, community health, mental health, and substance abuse.
	data.transportation.gov From the US Department of Transportation Provides data on driving to public transit, bicycling and walking. etc.

Major Organizations/Associations

	fred.stlouisfed.org From the Federal Reserve Provides over 500,000 financial and economic data series, including datasets on economic indicators, banking & finance, labor markets, employment, and national and international accounts.
	pewresearch.org From the Pew Research Center Provides data on politics, media, culture, religion, and internet/tech. [list of topics] [list of dataset categories]
	databank.worldbank.org From the World Bank Provides data on major world development indicators (i.e., GDP, population, life expectancy, and education levels).
	who.int/gho/data From WHO's Global Health Observatory (GHO) project Provides data on major global health indicators (i.e., disease spread, vaccinations, and rates of other illnesses/conditions).

Dataset Aggregators

datasetsearch.research.google.com

Google's dataset search engine.
It can be used to search for many of the government and association-provided datasets.

github.com/awesomedata/awesome-public-datasets

AwesomeData community publishes open data resources on GitHub.
Data resources are categorized neatly by resarch area.

Data Science Communities
Data science communities upload and share interesting datasets for the purposes of model building and analysis.

kaggle.com/datasets

A popular data science competition site.
Contains datasets for building predictive models.
The data the quality may vary as any user in the Kaggle community can submit datasets,

archive.ics.uci.edu/ml/datasets

From the University of California--Irvine, the UCI Machine Learning Repository
Provides datasets cleaned and preprocessed.

data.fivethirtyeight.com

Pprovides the data on politics, economics, and sports.

NYS & NYC

opendata.cityofnewyork.us

The City of New York's open data is collected by the Open Data team, which is consisted of the Mayor’s Office of Data Analytics (MODA) and the Department of Information Technology and Telecommunications (DoITT).

It provides datasets on almost all aspects of New Yorkers' life.

Business
City government
Education
Evironment
Health

data.ny.gov

New York State's Open Data Portal provides a powerful tool that provides centralized access to high-value government data to search, explore, download, and share.
Users can track visualized data on coronavirus testing and infection rates in NYS.

Global

• data.world is a platform where the world’s problem solvers can find and use a vast array of high-quality open data.

• World Bank Open Data

• Open data site finder (Tableau Public viz)

• Kaggle Open Datasets find open datasets on everything from government, health, and science to popular games and dating trends.

• Global Health Data Exchange the world’s most comprehensive catalog of surveys, censuses, vital statistics, and other health-related data.

• UNICEF Statistics

• World Health Organization Data

• The Guardian Data Blog

United States

• Data.gov is the Federal Government’s open data initiative. Hundreds of data sets.

• The Pew Research Center has a number of data sets on different social and technology topics.

• Socrata is an open data repository with data, mostly from government sources.

• US Dept. of Agriculture

• US Census Bureau

• USAspending.gov

• Centers for Disease Control and Prevention

• US Dept. of Education College Scorecard Data

• NASA's Open Data Portal

Canada

• Canada Open Data

United Kingdom

• Data.gov.uk

Bureau of Economic Analysis national and regional economic data
Bureau of Labor Statistics - Contains important economic indicators for the United States such as unemployment and inflation
CDC Data and Statistics - The data can be segmented by demographics.
College ScoreCard Data - Data on college student completion, debt and repayment, earnings, and more.
Data.gov - U.S. government data
Data Hub Collection
Data.World - SQL queries can be used within their interface to explore data and join multiple data sets.
Dryad - An international repository of data focusing on the fields of scientific and medical publications.
FBI Crime Data - Contains national crime statistics
Harvard Dataverse
HealthData.gov - Provides US healthcare data
International Monetary Fund Data - Publishes data on international finances, debt rates, foreign exchange reserves, commodity prices, and more
National Cancer Institute - Contains data about cancer incidence
National Health and Nutrition Examination Survey NHANES
NYC Open Data
UN Data
Unicef data - Provides statistics related to women and children worldwide.
United Nations Commodity Trade Statistics Database (UN Comtrade)
United Nations Data (UNdata)
United States Census - Provides demographic data at the state, city, and even zip code level.
World Bank Databank - Provides datasets covering population demographics and economic and development indicators across the world.
World Health Organization - Offers data and statistics on world health, disease and more.
Walmart historical sales data - Covers historical sales data for 45 stores located in different regions across the United States.
World Trade Organization - Provides International Trade Statistics
World Wealth & Income Database
Google Dataset Search

Data Preparation, Presentation, and Citation

Data cleaning, transformation, and integration stage will take 60-80% of your time. Data visualization can be a part of the analyze or present stages.

Data finding & extraction example:

The Censuses of Religious Bodies, 1906 - 1936

Some key objectives to think about when presenting data analysis:

Visual communication
Audience and context
Charts, graphs, and images
Focus on important points
Design principles
Storytelling
Persuasiveness
Dashboards

Example:

Library Assessment Dashboard

Some resources to help properly cite data for research:

Data Science