{ "cells": [ { "cell_type": "markdown", "id": "standing-connection", "metadata": {}, "source": [ "# Data Preparation" ] }, { "cell_type": "markdown", "id": "liberal-chain", "metadata": {}, "source": [ "OSMI has all the links of the data on their [website](https://osmihelp.org) under the [research](https://osmihelp.org/research) section for all the years starting from year 2014. The organization has hosted their datasets on kaggle. I took the data from the year 2017 onwards because the question asked haven't changed from 2017 and hence it will be easier to combine the data.\n", "\n", "All the datasets are downloaded in csv format and renamed to with respect to corresponding years in the format \"osmi_\\.csv\". Here \\ is the placeholder for the year. For example, data for year 2017 is present in file named \"osmi_2017.csv\"" ] }, { "cell_type": "code", "execution_count": 1, "id": "turkish-declaration", "metadata": {}, "outputs": [], "source": [ "# Import necessary packages\n", "import pandas as pd\n", "import re" ] }, { "cell_type": "markdown", "id": "detected-voluntary", "metadata": {}, "source": [ "## Data cleaning - 2017" ] }, { "cell_type": "code", "execution_count": 2, "id": "different-piece", "metadata": {}, "outputs": [], "source": [ "# Load 2017 data\n", "df = pd.read_csv(\"./../../../datasets/osmi_2017.csv\")" ] }, { "cell_type": "code", "execution_count": 3, "id": "important-child", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#<strong>Are you self-employed?</strong>How many employees does your company or organization have?Is your employer primarily a tech company/organization?Is your primary role within your company related to tech/IT?Does your employer provide mental health benefits as part of healthcare coverage?Do you know the options for mental health care available under your employer-provided health coverage?Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)?Does your employer offer resources to learn more about mental health disorders and options for seeking help?Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer?...What is your gender?What country do you <strong>live</strong> in?What US state or territory do you <strong>live</strong> in?What is your race?OtherWhat country do you <strong>work</strong> in?What US state or territory do you <strong>work</strong> in?Start Date (UTC)Submit Date (UTC)Network ID
0e49fe87572831232dcfa51b376b220390100-5001.01.0NoYesNoI don't knowI don't know...FemaleUnited KingdomNaNNaNNaNUnited KingdomNaN16/05/18 12:3216/05/18 12:42464b7a12f1
1a1eede444ac024928d247a8372d549310100-5001.01.0YesYesNoNoI don't know...maleUnited KingdomNaNNaNNaNUnited KingdomNaN16/05/18 12:3116/05/18 12:40464b7a12f1
237d3fd67f62bd1e0a2dea4f9cd440d980Jun-251.01.0I don't knowNoI don't knowNoYes...maleUnited States of AmericaMissouriWhiteNaNUnited States of AmericaMissouri09/05/18 5:3409/05/18 5:461eb7e0cb94
3519b759442c1cab0e9b5a8a1acb1b2160More than 10001.01.0YesYesI don't knowI don't knowYes...MaleUnited States of AmericaWashingtonWhiteNaNUnited States of AmericaWashington04/05/18 23:1904/05/18 23:2363852edbc4
4ef0af4927b575b1a3e607c11ca37870e1NaNNaNNaNNaNNaNNaNNaNNaN...femaleUnited States of AmericaIllinoisMore than one of the aboveNaNUnited States of AmericaIllinois03/05/18 0:4003/05/18 0:5343237889f1
\n", "

5 rows × 87 columns

\n", "
" ], "text/plain": [ " # Are you self-employed? \\\n", "0 e49fe87572831232dcfa51b376b22039 0 \n", "1 a1eede444ac024928d247a8372d54931 0 \n", "2 37d3fd67f62bd1e0a2dea4f9cd440d98 0 \n", "3 519b759442c1cab0e9b5a8a1acb1b216 0 \n", "4 ef0af4927b575b1a3e607c11ca37870e 1 \n", "\n", " How many employees does your company or organization have? \\\n", "0 100-500 \n", "1 100-500 \n", "2 Jun-25 \n", "3 More than 1000 \n", "4 NaN \n", "\n", " Is your employer primarily a tech company/organization? \\\n", "0 1.0 \n", "1 1.0 \n", "2 1.0 \n", "3 1.0 \n", "4 NaN \n", "\n", " Is your primary role within your company related to tech/IT? \\\n", "0 1.0 \n", "1 1.0 \n", "2 1.0 \n", "3 1.0 \n", "4 NaN \n", "\n", " Does your employer provide mental health benefits as part of healthcare coverage? \\\n", "0 No \n", "1 Yes \n", "2 I don't know \n", "3 Yes \n", "4 NaN \n", "\n", " Do you know the options for mental health care available under your employer-provided health coverage? \\\n", "0 Yes \n", "1 Yes \n", "2 No \n", "3 Yes \n", "4 NaN \n", "\n", " Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? \\\n", "0 No \n", "1 No \n", "2 I don't know \n", "3 I don't know \n", "4 NaN \n", "\n", " Does your employer offer resources to learn more about mental health disorders and options for seeking help? \\\n", "0 I don't know \n", "1 No \n", "2 No \n", "3 I don't know \n", "4 NaN \n", "\n", " Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? \\\n", "0 I don't know \n", "1 I don't know \n", "2 Yes \n", "3 Yes \n", "4 NaN \n", "\n", " ... What is your gender? What country do you live in? \\\n", "0 ... Female United Kingdom \n", "1 ... male United Kingdom \n", "2 ... male United States of America \n", "3 ... Male United States of America \n", "4 ... female United States of America \n", "\n", " What US state or territory do you live in? \\\n", "0 NaN \n", "1 NaN \n", "2 Missouri \n", "3 Washington \n", "4 Illinois \n", "\n", " What is your race? Other \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 White NaN \n", "3 White NaN \n", "4 More than one of the above NaN \n", "\n", " What country do you work in? \\\n", "0 United Kingdom \n", "1 United Kingdom \n", "2 United States of America \n", "3 United States of America \n", "4 United States of America \n", "\n", " What US state or territory do you work in? \\\n", "0 NaN \n", "1 NaN \n", "2 Missouri \n", "3 Washington \n", "4 Illinois \n", "\n", " Start Date (UTC) Submit Date (UTC) Network ID \n", "0 16/05/18 12:32 16/05/18 12:42 464b7a12f1 \n", "1 16/05/18 12:31 16/05/18 12:40 464b7a12f1 \n", "2 09/05/18 5:34 09/05/18 5:46 1eb7e0cb94 \n", "3 04/05/18 23:19 04/05/18 23:23 63852edbc4 \n", "4 03/05/18 0:40 03/05/18 0:53 43237889f1 \n", "\n", "[5 rows x 87 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Data Preview\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 4, "id": "increased-rider", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 756 entries, 0 to 755\n", "Data columns (total 87 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 # 756 non-null object \n", " 1 Are you self-employed? 756 non-null int64 \n", " 2 How many employees does your company or organization have? 643 non-null object \n", " 3 Is your employer primarily a tech company/organization? 643 non-null float64\n", " 4 Is your primary role within your company related to tech/IT? 643 non-null float64\n", " 5 Does your employer provide mental health benefits as part of healthcare coverage? 643 non-null object \n", " 6 Do you know the options for mental health care available under your employer-provided health coverage? 576 non-null object \n", " 7 Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? 643 non-null object \n", " 8 Does your employer offer resources to learn more about mental health disorders and options for seeking help? 643 non-null object \n", " 9 Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? 643 non-null object \n", " 10 If a mental health issue prompted you to request a medical leave from work, how easy or difficult would it be to ask for that leave? 643 non-null object \n", " 11 Would you feel more comfortable talking to your coworkers about your physical health or your mental health? 643 non-null object \n", " 12 Would you feel comfortable discussing a mental health issue with your direct supervisor(s)? 643 non-null object \n", " 13 Have you ever discussed your mental health with your employer? 643 non-null float64\n", " 14 Describe the conversation you had with your employer about your mental health, including their reactions and what actions were taken to address your mental health issue/questions. 172 non-null object \n", " 15 Would you feel comfortable discussing a mental health issue with your coworkers? 643 non-null object \n", " 16 Have you ever discussed your mental health with coworkers? 643 non-null float64\n", " 17 Describe the conversation with coworkers you had about your mental health including their reactions. 226 non-null object \n", " 18 Have you ever had a coworker discuss their or another coworker's mental health with you? 638 non-null float64\n", " 19 Describe the conversation your coworker had with you about their mental health (please do not use names). 230 non-null object \n", " 20 Overall, how much importance does your employer place on physical health? 643 non-null float64\n", " 21 Overall, how much importance does your employer place on mental health? 643 non-null float64\n", " 22 Do you have medical coverage (private insurance or state-provided) that includes treatment of mental health disorders? 113 non-null float64\n", " 23 Do you know local or online resources to seek help for a mental health issue? 113 non-null object \n", " 24 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to clients or business contacts? 113 non-null object \n", " 25 If you have revealed a mental health disorder to a client or business contact, how has this affected you or the relationship? 51 non-null object \n", " 26 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to coworkers or employees? 113 non-null object \n", " 27 If you have revealed a mental health disorder to a coworker or employee, how has this impacted you or the relationship? 113 non-null object \n", " 28 Do you believe your productivity is ever affected by a mental health issue? 113 non-null object \n", " 29 If yes, what percentage of your work time (time performing primary or secondary job functions) is affected by a mental health issue? 85 non-null object \n", " 30 Do you have previous employers? 756 non-null int64 \n", " 31 Was your employer primarily a tech company/organization? 667 non-null float64\n", " 32 Have your previous employers provided mental health benefits? 667 non-null object \n", " 33 Were you aware of the options for mental health care provided by your previous employers? 667 non-null object \n", " 34 Did your previous employers ever formally discuss mental health (as part of a wellness campaign or other official communication)? 667 non-null object \n", " 35 Did your previous employers provide resources to learn more about mental health disorders and how to seek help? 667 non-null object \n", " 36 Was your anonymity protected if you chose to take advantage of mental health or substance abuse treatment resources with previous employers? 667 non-null object \n", " 37 Would you have felt more comfortable talking to your previous employer about your physical health or your mental health? 667 non-null object \n", " 38 Would you have been willing to discuss your mental health with your direct supervisor(s)? 667 non-null object \n", " 39 Did you ever discuss your mental health with your previous employer? 667 non-null float64\n", " 40 Describe the conversation you had with your previous employer about your mental health, including their reactions and actions taken to address your mental health issue/questions. 105 non-null object \n", " 41 Would you have been willing to discuss your mental health with your coworkers at previous employers? 667 non-null object \n", " 42 Did you ever discuss your mental health with a previous coworker(s)? 666 non-null float64\n", " 43 Describe the conversation you had with your previous coworkers about your mental health including their reactions. 130 non-null object \n", " 44 Did you ever have a previous coworker discuss their or another coworker's mental health with you? 661 non-null float64\n", " 45 Describe the conversation your coworker had with you about their mental health (please do not use names)..1 126 non-null object \n", " 46 Overall, how much importance did your previous employer place on physical health? 667 non-null float64\n", " 47 Overall, how much importance did your previous employer place on mental health? 667 non-null float64\n", " 48 Do you currently have a mental health disorder? 756 non-null object \n", " 49 Have you ever been diagnosed with a mental health disorder? 324 non-null object \n", " 50 What disorder(s) have you been diagnosed with? 0 non-null float64\n", " 51 If possibly, what disorder(s) do you believe you have? 142 non-null object \n", " 52 If so, what disorder(s) were you diagnosed with? 313 non-null object \n", " 53 Have you had a mental health disorder in the past? 746 non-null object \n", " 54 Have you ever sought treatment for a mental health disorder from a mental health professional? 756 non-null int64 \n", " 55 Do you have a family history of mental illness? 756 non-null object \n", " 56 If you have a mental health disorder, how often do you feel that it interferes with your work when being treated effectively? 756 non-null object \n", " 57 If you have a mental health disorder, how often do you feel that it interferes with your work when NOT being treated effectively (i.e., when you are experiencing symptoms)? 756 non-null object \n", " 58 Have your observations of how another individual who discussed a mental health issue made you less likely to reveal a mental health issue yourself in your current workplace? 596 non-null object \n", " 59 How willing would you be to share with friends and family that you have a mental illness? 756 non-null int64 \n", " 60 Would you be willing to bring up a physical health issue with a potential employer in an interview? 756 non-null object \n", " 61 Why or why not? 570 non-null object \n", " 62 Would you bring up your mental health with a potential employer in an interview? 756 non-null object \n", " 63 Why or why not?.1 580 non-null object \n", " 64 Are you openly identified at work as a person with a mental health issue? 754 non-null float64\n", " 65 Has being identified as a person with a mental health issue affected your career? 84 non-null float64\n", " 66 How has it affected your career? 26 non-null float64\n", " 67 If they knew you suffered from a mental health disorder, how do you think that team members/co-workers would react? 754 non-null float64\n", " 68 Have you observed or experienced an unsupportive or badly handled response to a mental health issue in your current or previous workplace? 754 non-null object \n", " 69 Describe the circumstances of the badly handled or unsupportive response. 157 non-null object \n", " 70 Have you observed or experienced supportive or well handled response to a mental health issue in your current or previous workplace? 754 non-null object \n", " 71 Describe the circumstances of the supportive or well handled response. 0 non-null float64\n", " 72 Overall, how well do you think the tech industry supports employees with mental health issues? 754 non-null float64\n", " 73 Briefly describe what you think the industry as a whole and/or employers could do to improve mental health support for employees. 500 non-null object \n", " 74 If there is anything else you would like to tell us that has not been covered by the survey questions, please use this space to do so. 183 non-null object \n", " 75 Would you be willing to talk to one of us more extensively about your experiences with mental health issues in the tech industry? (Note that all interview responses would be used anonymously and only with your permission.) 754 non-null float64\n", " 76 What is your age? 754 non-null float64\n", " 77 What is your gender? 743 non-null object \n", " 78 What country do you live in? 754 non-null object \n", " 79 What US state or territory do you live in? 497 non-null object \n", " 80 What is your race? 482 non-null object \n", " 81 Other 15 non-null object \n", " 82 What country do you work in? 754 non-null object \n", " 83 What US state or territory do you work in? 503 non-null object \n", " 84 Start Date (UTC) 756 non-null object \n", " 85 Submit Date (UTC) 756 non-null object \n", " 86 Network ID 756 non-null object \n", "dtypes: float64(23), int64(4), object(60)\n", "memory usage: 514.0+ KB\n" ] } ], "source": [ "# Data info\n", "df.info()" ] }, { "cell_type": "code", "execution_count": 5, "id": "attached-graduation", "metadata": {}, "outputs": [], "source": [ "# Lower the column names\n", "df.columns = df.columns.str.lower()" ] }, { "cell_type": "code", "execution_count": 6, "id": "noticed-story", "metadata": {}, "outputs": [], "source": [ "columns = list(df.columns)\n", "\n", "# Remove html tags from the column names\n", "columns = [re.sub('<.*?>', '', column) for column in columns]\n", "\n", "# Remove special character \n", "columns = [re.sub('\\xa0', ' ', column) for column in columns]" ] }, { "cell_type": "code", "execution_count": 7, "id": "black-tradition", "metadata": {}, "outputs": [], "source": [ "# Reassign the columns to the respective dataframes\n", "df.columns = columns" ] }, { "cell_type": "code", "execution_count": 8, "id": "considered-quality", "metadata": {}, "outputs": [], "source": [ "# Drop unuseful columns\n", "df.drop(['#', 'start date (utc)', 'network id', 'other', 'submit date (utc)'], axis=1, inplace=True)" ] }, { "cell_type": "code", "execution_count": 9, "id": "personalized-canal", "metadata": {}, "outputs": [], "source": [ "# Add year identifying column\n", "df['year'] = '2017'" ] }, { "cell_type": "code", "execution_count": 10, "id": "honest-glenn", "metadata": {}, "outputs": [], "source": [ "# Write the cleaned file back\n", "df.to_csv(\"./../../../datasets/cleaned_osmi_2017.csv\", index=False)" ] }, { "cell_type": "markdown", "id": "positive-driver", "metadata": {}, "source": [ "## Data cleaning - 2018" ] }, { "cell_type": "code", "execution_count": 11, "id": "measured-theme", "metadata": {}, "outputs": [], "source": [ "# Load 2018 data\n", "df = pd.read_csv(\"./../../../datasets/osmi_2018.csv\")" ] }, { "cell_type": "code", "execution_count": 12, "id": "veterinary-purple", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#<strong>Are you self-employed?</strong>How many employees does your company or organization have?Is your employer primarily a tech company/organization?Is your primary role within your company related to tech/IT?Does your employer provide mental health benefits as part of healthcare coverage?Do you know the options for mental health care available under your employer-provided health coverage?Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)?Does your employer offer resources to learn more about mental health disorders and options for seeking help?Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer?...What is your gender?What country do you <strong>live</strong> in?What US state or territory do you <strong>live</strong> in?What is your race?OtherWhat country do you <strong>work</strong> in?What US state or territory do you <strong>work</strong> in?Start Date (UTC)Submit Date (UTC)Network ID
0e44a0a34f2465940beda2a1537e9b99e0More than 10001.00.0YesYesYesYesYes...FemaleCanadaNaNNaNNaNCanadaNaN29/12/18 23:4630/12/18 0:004bbb884ccc
10d698e3beca20fb75f19b9d528e36d730More than 10001.01.0YesYesNoI don't knowI don't know...maleUnited States of AmericaMassachusettsWhiteNaNUnited States of AmericaMassachusetts27/12/18 21:4027/12/18 21:45275e7543bd
261a40c9071eb36fa9caa254d31500c410Jun-250.01.0YesYesNoNoI don't know...MaleUnited States of AmericaFloridaWhiteNaNUnited States of AmericaFlorida21/12/18 17:3721/12/18 18:0843994c3dba
3f8624340bead7deb08abb766704ddf6b0Jun-251.01.0NoNoNoNoI don't know...maleNorwayNaNNaNNaNNorwayNaN21/12/18 16:3721/12/18 16:44907b3a3faa
431d3ae93b68d79e504a0a643601b6b1e026-1001.01.0YesYesYesYesYes...Ostensibly MaleUnited States of AmericaTennesseeWhiteNaNUnited States of AmericaTennessee20/12/18 19:3920/12/18 20:5826df20fea3
\n", "

5 rows × 87 columns

\n", "
" ], "text/plain": [ " # Are you self-employed? \\\n", "0 e44a0a34f2465940beda2a1537e9b99e 0 \n", "1 0d698e3beca20fb75f19b9d528e36d73 0 \n", "2 61a40c9071eb36fa9caa254d31500c41 0 \n", "3 f8624340bead7deb08abb766704ddf6b 0 \n", "4 31d3ae93b68d79e504a0a643601b6b1e 0 \n", "\n", " How many employees does your company or organization have? \\\n", "0 More than 1000 \n", "1 More than 1000 \n", "2 Jun-25 \n", "3 Jun-25 \n", "4 26-100 \n", "\n", " Is your employer primarily a tech company/organization? \\\n", "0 1.0 \n", "1 1.0 \n", "2 0.0 \n", "3 1.0 \n", "4 1.0 \n", "\n", " Is your primary role within your company related to tech/IT? \\\n", "0 0.0 \n", "1 1.0 \n", "2 1.0 \n", "3 1.0 \n", "4 1.0 \n", "\n", " Does your employer provide mental health benefits as part of healthcare coverage? \\\n", "0 Yes \n", "1 Yes \n", "2 Yes \n", "3 No \n", "4 Yes \n", "\n", " Do you know the options for mental health care available under your employer-provided health coverage? \\\n", "0 Yes \n", "1 Yes \n", "2 Yes \n", "3 No \n", "4 Yes \n", "\n", " Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? \\\n", "0 Yes \n", "1 No \n", "2 No \n", "3 No \n", "4 Yes \n", "\n", " Does your employer offer resources to learn more about mental health disorders and options for seeking help? \\\n", "0 Yes \n", "1 I don't know \n", "2 No \n", "3 No \n", "4 Yes \n", "\n", " Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? \\\n", "0 Yes \n", "1 I don't know \n", "2 I don't know \n", "3 I don't know \n", "4 Yes \n", "\n", " ... What is your gender? What country do you live in? \\\n", "0 ... Female Canada \n", "1 ... male United States of America \n", "2 ... Male United States of America \n", "3 ... male Norway \n", "4 ... Ostensibly Male United States of America \n", "\n", " What US state or territory do you live in? \\\n", "0 NaN \n", "1 Massachusetts \n", "2 Florida \n", "3 NaN \n", "4 Tennessee \n", "\n", " What is your race? Other What country do you work in? \\\n", "0 NaN NaN Canada \n", "1 White NaN United States of America \n", "2 White NaN United States of America \n", "3 NaN NaN Norway \n", "4 White NaN United States of America \n", "\n", " What US state or territory do you work in? \\\n", "0 NaN \n", "1 Massachusetts \n", "2 Florida \n", "3 NaN \n", "4 Tennessee \n", "\n", " Start Date (UTC) Submit Date (UTC) Network ID \n", "0 29/12/18 23:46 30/12/18 0:00 4bbb884ccc \n", "1 27/12/18 21:40 27/12/18 21:45 275e7543bd \n", "2 21/12/18 17:37 21/12/18 18:08 43994c3dba \n", "3 21/12/18 16:37 21/12/18 16:44 907b3a3faa \n", "4 20/12/18 19:39 20/12/18 20:58 26df20fea3 \n", "\n", "[5 rows x 87 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Data Preview\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 13, "id": "opposite-underwear", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 417 entries, 0 to 416\n", "Data columns (total 87 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 # 417 non-null object \n", " 1 Are you self-employed? 417 non-null int64 \n", " 2 How many employees does your company or organization have? 361 non-null object \n", " 3 Is your employer primarily a tech company/organization? 361 non-null float64\n", " 4 Is your primary role within your company related to tech/IT? 361 non-null float64\n", " 5 Does your employer provide mental health benefits as part of healthcare coverage? 361 non-null object \n", " 6 Do you know the options for mental health care available under your employer-provided health coverage? 324 non-null object \n", " 7 Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? 361 non-null object \n", " 8 Does your employer offer resources to learn more about mental health disorders and options for seeking help? 361 non-null object \n", " 9 Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? 361 non-null object \n", " 10 If a mental health issue prompted you to request a medical leave from work, how easy or difficult would it be to ask for that leave? 361 non-null object \n", " 11 Would you feel more comfortable talking to your coworkers about your physical health or your mental health? 361 non-null object \n", " 12 Would you feel comfortable discussing a mental health issue with your direct supervisor(s)? 361 non-null object \n", " 13 Have you ever discussed your mental health with your employer? 361 non-null float64\n", " 14 Describe the conversation you had with your employer about your mental health, including their reactions and what actions were taken to address your mental health issue/questions. 96 non-null object \n", " 15 Would you feel comfortable discussing a mental health issue with your coworkers? 361 non-null object \n", " 16 Have you ever discussed your mental health with coworkers? 360 non-null float64\n", " 17 Describe the conversation with coworkers you had about your mental health including their reactions. 106 non-null object \n", " 18 Have you ever had a coworker discuss their or another coworker's mental health with you? 360 non-null float64\n", " 19 Describe the conversation your coworker had with you about their mental health (please do not use names). 124 non-null object \n", " 20 Overall, how much importance does your employer place on physical health? 361 non-null float64\n", " 21 Overall, how much importance does your employer place on mental health? 361 non-null float64\n", " 22 Do you have medical coverage (private insurance or state-provided) that includes treatment of mental health disorders? 56 non-null float64\n", " 23 Do you know local or online resources to seek help for a mental health issue? 56 non-null object \n", " 24 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to clients or business contacts? 56 non-null object \n", " 25 If you have revealed a mental health disorder to a client or business contact, how has this affected you or the relationship? 21 non-null object \n", " 26 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to coworkers or employees? 56 non-null object \n", " 27 If you have revealed a mental health disorder to a coworker or employee, how has this impacted you or the relationship? 56 non-null object \n", " 28 Do you believe your productivity is ever affected by a mental health issue? 56 non-null object \n", " 29 If yes, what percentage of your work time (time performing primary or secondary job functions) is affected by a mental health issue? 41 non-null object \n", " 30 Do you have previous employers? 417 non-null int64 \n", " 31 Was your employer primarily a tech company/organization? 363 non-null float64\n", " 32 Have your previous employers provided mental health benefits? 363 non-null object \n", " 33 Were you aware of the options for mental health care provided by your previous employers? 363 non-null object \n", " 34 Did your previous employers ever formally discuss mental health (as part of a wellness campaign or other official communication)? 363 non-null object \n", " 35 Did your previous employers provide resources to learn more about mental health disorders and how to seek help? 363 non-null object \n", " 36 Was your anonymity protected if you chose to take advantage of mental health or substance abuse treatment resources with previous employers? 363 non-null object \n", " 37 Would you have felt more comfortable talking to your previous employer about your physical health or your mental health? 363 non-null object \n", " 38 Would you have been willing to discuss your mental health with your direct supervisor(s)? 363 non-null object \n", " 39 Did you ever discuss your mental health with your previous employer? 363 non-null float64\n", " 40 Describe the conversation you had with your previous employer about your mental health, including their reactions and actions taken to address your mental health issue/questions. 60 non-null object \n", " 41 Would you have been willing to discuss your mental health with your coworkers at previous employers? 363 non-null object \n", " 42 Did you ever discuss your mental health with a previous coworker(s)? 362 non-null float64\n", " 43 Describe the conversation you had with your previous coworkers about your mental health including their reactions. 92 non-null object \n", " 44 Did you ever have a previous coworker discuss their or another coworker's mental health with you? 363 non-null float64\n", " 45 Describe the conversation your coworker had with you about their mental health (please do not use names)..1 79 non-null object \n", " 46 Overall, how much importance did your previous employer place on physical health? 363 non-null float64\n", " 47 Overall, how much importance did your previous employer place on mental health? 363 non-null float64\n", " 48 Do you currently have a mental health disorder? 417 non-null object \n", " 49 Have you ever been diagnosed with a mental health disorder? 191 non-null object \n", " 50 What disorder(s) have you been diagnosed with? 0 non-null float64\n", " 51 If possibly, what disorder(s) do you believe you have? 81 non-null object \n", " 52 If so, what disorder(s) were you diagnosed with? 188 non-null object \n", " 53 Have you had a mental health disorder in the past? 415 non-null object \n", " 54 Have you ever sought treatment for a mental health disorder from a mental health professional? 417 non-null int64 \n", " 55 Do you have a family history of mental illness? 417 non-null object \n", " 56 If you have a mental health disorder, how often do you feel that it interferes with your work when being treated effectively? 417 non-null object \n", " 57 If you have a mental health disorder, how often do you feel that it interferes with your work when NOT being treated effectively (i.e., when you are experiencing symptoms)? 417 non-null object \n", " 58 Have your observations of how another individual who discussed a mental health issue made you less likely to reveal a mental health issue yourself in your current workplace? 320 non-null object \n", " 59 How willing would you be to share with friends and family that you have a mental illness? 417 non-null int64 \n", " 60 Would you be willing to bring up a physical health issue with a potential employer in an interview? 417 non-null object \n", " 61 Why or why not? 326 non-null object \n", " 62 Would you bring up your mental health with a potential employer in an interview? 417 non-null object \n", " 63 Why or why not?.1 333 non-null object \n", " 64 Are you openly identified at work as a person with a mental health issue? 417 non-null int64 \n", " 65 Has being identified as a person with a mental health issue affected your career? 51 non-null float64\n", " 66 How has it affected your career? 16 non-null float64\n", " 67 If they knew you suffered from a mental health disorder, how do you think that team members/co-workers would react? 417 non-null int64 \n", " 68 Have you observed or experienced an unsupportive or badly handled response to a mental health issue in your current or previous workplace? 417 non-null object \n", " 69 Describe the circumstances of the badly handled or unsupportive response. 91 non-null object \n", " 70 Have you observed or experienced supportive or well handled response to a mental health issue in your current or previous workplace? 417 non-null object \n", " 71 Describe the circumstances of the supportive or well handled response. 0 non-null float64\n", " 72 Overall, how well do you think the tech industry supports employees with mental health issues? 417 non-null int64 \n", " 73 Briefly describe what you think the industry as a whole and/or employers could do to improve mental health support for employees. 289 non-null object \n", " 74 If there is anything else you would like to tell us that has not been covered by the survey questions, please use this space to do so. 98 non-null object \n", " 75 Would you be willing to talk to one of us more extensively about your experiences with mental health issues in the tech industry? (Note that all interview responses would be used anonymously and only with your permission.) 417 non-null int64 \n", " 76 What is your age? 417 non-null int64 \n", " 77 What is your gender? 414 non-null object \n", " 78 What country do you live in? 417 non-null object \n", " 79 What US state or territory do you live in? 311 non-null object \n", " 80 What is your race? 302 non-null object \n", " 81 Other 9 non-null object \n", " 82 What country do you work in? 417 non-null object \n", " 83 What US state or territory do you work in? 314 non-null object \n", " 84 Start Date (UTC) 417 non-null object \n", " 85 Submit Date (UTC) 417 non-null object \n", " 86 Network ID 417 non-null object \n", "dtypes: float64(18), int64(9), object(60)\n", "memory usage: 283.6+ KB\n" ] } ], "source": [ "# Data info\n", "df.info()" ] }, { "cell_type": "code", "execution_count": 14, "id": "incorrect-study", "metadata": {}, "outputs": [], "source": [ "# Lower the column names\n", "df.columns = df.columns.str.lower()" ] }, { "cell_type": "code", "execution_count": 15, "id": "disabled-boost", "metadata": {}, "outputs": [], "source": [ "columns = list(df.columns)\n", "\n", "# Remove html tags from the column names\n", "columns = [re.sub('<.*?>', '', column) for column in columns]" ] }, { "cell_type": "code", "execution_count": 16, "id": "boolean-terminology", "metadata": {}, "outputs": [], "source": [ "# Reassign the columns to the respective dataframes\n", "df.columns = columns" ] }, { "cell_type": "code", "execution_count": 17, "id": "equal-trainer", "metadata": {}, "outputs": [], "source": [ "# Drop unuseful columns\n", "df.drop(['#', 'start date (utc)', 'network id', 'other', 'submit date (utc)'], axis=1, inplace=True)" ] }, { "cell_type": "code", "execution_count": 18, "id": "incoming-pierre", "metadata": {}, "outputs": [], "source": [ "# Add year identifying column\n", "df['year'] = '2018'" ] }, { "cell_type": "code", "execution_count": 19, "id": "surface-treatment", "metadata": {}, "outputs": [], "source": [ "# Write the cleaned file back\n", "df.to_csv(\"./../../../datasets/cleaned_osmi_2018.csv\", index=False)" ] }, { "cell_type": "markdown", "id": "atmospheric-chaos", "metadata": {}, "source": [ "## Data cleaning - 2019" ] }, { "cell_type": "code", "execution_count": 20, "id": "geological-database", "metadata": {}, "outputs": [], "source": [ "# Load 2019 data\n", "df = pd.read_csv(\"./../../../datasets/osmi_2019.csv\")" ] }, { "cell_type": "code", "execution_count": 21, "id": "working-antique", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
*Are you self-employed?*How many employees does your company or organization have?Is your employer primarily a tech company/organization?Is your primary role within your company related to tech/IT?Does your employer provide mental health benefits as part of healthcare coverage?Do you know the options for mental health care available under your employer-provided health coverage?Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)?Does your employer offer resources to learn more about mental health disorders and options for seeking help?Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer?If a mental health issue prompted you to request a medical leave from work, how easy or difficult would it be to ask for that leave?...Briefly describe what you think the industry as a whole and/or employers could do to improve mental health support for employees.If there is anything else you would like to tell us that has not been covered by the survey questions, please use this space to do so.Would you be willing to talk to one of us more extensively about your experiences with mental health issues in the tech industry? (Note that all interview responses would be used _anonymously_ and only with your permission.)What is your age?What is your gender?What country do you *live* in?What US state or territory do you *live* in?What is your race?What country do you *work* in?What US state or territory do you *work* in?
0False26-100TrueTrueI don't knowNoYesYesI don't knowVery easy...NaNNaNFalse25MaleUnited States of AmericaNebraskaWhiteUnited States of AmericaNebraska
1False26-100TrueTrueYesNoNoYesYesI don't know...NaNNaNFalse51maleUnited States of AmericaNebraskaWhiteUnited States of AmericaNebraska
2False26-100TrueTrueI don't knowNoNoI don't knowI don't knowSomewhat difficult...I think opening up more conversation around th...Thank youTrue27MaleUnited States of AmericaIllinoisWhiteUnited States of AmericaIllinois
3False100-500TrueTrueI don't knowNoYesYesYesVery easy...NaNNaNFalse37maleUnited States of AmericaNebraskaWhiteUnited States of AmericaNebraska
4False26-100TrueTrueI don't knowNoI don't knowI don't knowI don't knowI don't know...NaNNaNFalse46mUnited States of AmericaNebraskaWhiteUnited States of AmericaNebraska
\n", "

5 rows × 82 columns

\n", "
" ], "text/plain": [ " *Are you self-employed?* \\\n", "0 False \n", "1 False \n", "2 False \n", "3 False \n", "4 False \n", "\n", " How many employees does your company or organization have? \\\n", "0 26-100 \n", "1 26-100 \n", "2 26-100 \n", "3 100-500 \n", "4 26-100 \n", "\n", " Is your employer primarily a tech company/organization? \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "\n", " Is your primary role within your company related to tech/IT? \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "\n", " Does your employer provide mental health benefits as part of healthcare coverage? \\\n", "0 I don't know \n", "1 Yes \n", "2 I don't know \n", "3 I don't know \n", "4 I don't know \n", "\n", " Do you know the options for mental health care available under your employer-provided health coverage? \\\n", "0 No \n", "1 No \n", "2 No \n", "3 No \n", "4 No \n", "\n", " Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? \\\n", "0 Yes \n", "1 No \n", "2 No \n", "3 Yes \n", "4 I don't know \n", "\n", " Does your employer offer resources to learn more about mental health disorders and options for seeking help? \\\n", "0 Yes \n", "1 Yes \n", "2 I don't know \n", "3 Yes \n", "4 I don't know \n", "\n", " Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? \\\n", "0 I don't know \n", "1 Yes \n", "2 I don't know \n", "3 Yes \n", "4 I don't know \n", "\n", " If a mental health issue prompted you to request a medical leave from work, how easy or difficult would it be to ask for that leave? \\\n", "0 Very easy \n", "1 I don't know \n", "2 Somewhat difficult \n", "3 Very easy \n", "4 I don't know \n", "\n", " ... \\\n", "0 ... \n", "1 ... \n", "2 ... \n", "3 ... \n", "4 ... \n", "\n", " Briefly describe what you think the industry as a whole and/or employers could do to improve mental health support for employees. \\\n", "0 NaN \n", "1 NaN \n", "2 I think opening up more conversation around th... \n", "3 NaN \n", "4 NaN \n", "\n", " If there is anything else you would like to tell us that has not been covered by the survey questions, please use this space to do so. \\\n", "0 NaN \n", "1 NaN \n", "2 Thank you \n", "3 NaN \n", "4 NaN \n", "\n", " Would you be willing to talk to one of us more extensively about your experiences with mental health issues in the tech industry? (Note that all interview responses would be used _anonymously_ and only with your permission.) \\\n", "0 False \n", "1 False \n", "2 True \n", "3 False \n", "4 False \n", "\n", " What is your age? What is your gender? What country do you *live* in? \\\n", "0 25 Male United States of America \n", "1 51 male United States of America \n", "2 27 Male United States of America \n", "3 37 male United States of America \n", "4 46 m United States of America \n", "\n", " What US state or territory do you *live* in? What is your race? \\\n", "0 Nebraska White \n", "1 Nebraska White \n", "2 Illinois White \n", "3 Nebraska White \n", "4 Nebraska White \n", "\n", " What country do you *work* in? What US state or territory do you *work* in? \n", "0 United States of America Nebraska \n", "1 United States of America Nebraska \n", "2 United States of America Illinois \n", "3 United States of America Nebraska \n", "4 United States of America Nebraska \n", "\n", "[5 rows x 82 columns]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Data preview\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 22, "id": "moral-operations", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 352 entries, 0 to 351\n", "Data columns (total 82 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 *Are you self-employed?* 352 non-null bool \n", " 1 How many employees does your company or organization have? 304 non-null object \n", " 2 Is your employer primarily a tech company/organization? 304 non-null object \n", " 3 Is your primary role within your company related to tech/IT? 304 non-null object \n", " 4 Does your employer provide mental health benefits as part of healthcare coverage? 304 non-null object \n", " 5 Do you know the options for mental health care available under your employer-provided health coverage? 278 non-null object \n", " 6 Has your employer ever formally discussed mental health (for example, as part of a wellness campaign or other official communication)? 304 non-null object \n", " 7 Does your employer offer resources to learn more about mental health disorders and options for seeking help? 304 non-null object \n", " 8 Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources provided by your employer? 304 non-null object \n", " 9 If a mental health issue prompted you to request a medical leave from work, how easy or difficult would it be to ask for that leave? 304 non-null object \n", " 10 Would you feel more comfortable talking to your coworkers about your physical health or your mental health? 304 non-null object \n", " 11 Would you feel comfortable discussing a mental health issue with your direct supervisor(s)? 304 non-null object \n", " 12 Have you ever discussed your mental health with your employer? 304 non-null object \n", " 13 Describe the conversation you had with your employer about your mental health, including their reactions and what actions were taken to address your mental health issue/questions. 84 non-null object \n", " 14 Would you feel comfortable discussing a mental health issue with your coworkers? 304 non-null object \n", " 15 Have you ever discussed your mental health with coworkers? 301 non-null object \n", " 16 Describe the conversation with coworkers you had about your mental health including their reactions. 119 non-null object \n", " 17 Have you ever had a coworker discuss their or another coworker's mental health with you? 304 non-null object \n", " 18 Describe the conversation your coworker had with you about their mental health (please do not use names). 110 non-null object \n", " 19 Overall, how much importance does your employer place on physical health? 304 non-null float64\n", " 20 Overall, how much importance does your employer place on mental health? 304 non-null float64\n", " 21 Do you have medical coverage (private insurance or state-provided) that includes treatment of mental health disorders? 48 non-null object \n", " 22 Do you know local or online resources to seek help for a mental health issue? 48 non-null object \n", " 23 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to clients or business contacts? 48 non-null object \n", " 24 If you have revealed a mental health disorder to a client or business contact, how has this affected you or the relationship? 20 non-null object \n", " 25 If you have been diagnosed or treated for a mental health disorder, do you ever reveal this to coworkers or employees? 48 non-null object \n", " 26 If you have revealed a mental health disorder to a coworker or employee, how has this impacted you or the relationship? 48 non-null object \n", " 27 Do you believe your productivity is ever affected by a mental health issue? 48 non-null object \n", " 28 If yes, what percentage of your work time (time performing primary or secondary job functions) is affected by a mental health issue? 37 non-null object \n", " 29 *Do you have previous employers?* 352 non-null bool \n", " 30 Was your employer primarily a tech company/organization? 296 non-null object \n", " 31 Have your previous employers provided mental health benefits? 296 non-null object \n", " 32 Were you aware of the options for mental health care provided by your previous employers? 296 non-null object \n", " 33 Did your previous employers ever formally discuss mental health (as part of a wellness campaign or other official communication)? 296 non-null object \n", " 34 Did your previous employers provide resources to learn more about mental health disorders and how to seek help? 296 non-null object \n", " 35 Was your anonymity protected if you chose to take advantage of mental health or substance abuse treatment resources with previous employers? 296 non-null object \n", " 36 Would you have felt more comfortable talking to your previous employer about your physical health or your mental health? 296 non-null object \n", " 37 Would you have been willing to discuss your mental health with your direct supervisor(s)? 296 non-null object \n", " 38 Did you ever discuss your mental health with your previous employer? 296 non-null object \n", " 39 Describe the conversation you had with your previous employer about your mental health, including their reactions and actions taken to address your mental health issue/questions. 63 non-null object \n", " 40 Would you have been willing to discuss your mental health with your coworkers at previous employers? 296 non-null object \n", " 41 Did you ever discuss your mental health with a previous coworker(s)? 294 non-null object \n", " 42 Describe the conversation you had with your previous coworkers about your mental health including their reactions. 85 non-null object \n", " 43 Did you ever have a previous coworker discuss their or another coworker's mental health with you? 295 non-null object \n", " 44 Describe the conversation your coworker had with you about their mental health (please do not use names)..1 66 non-null object \n", " 45 Overall, how much importance did your previous employer place on physical health? 296 non-null float64\n", " 46 Overall, how much importance did your previous employer place on mental health? 296 non-null float64\n", " 47 Do you *currently* have a mental health disorder? 352 non-null object \n", " 48 Have you ever been *diagnosed* with a mental health disorder? 147 non-null object \n", " 49 *What disorder(s) have you been diagnosed with?* 0 non-null float64\n", " 50 *If possibly, what disorder(s) do you believe you have?* 72 non-null object \n", " 51 *If so, what disorder(s) were you diagnosed with?* 144 non-null object \n", " 52 Have you had a mental health disorder in the past? 349 non-null object \n", " 53 Have you ever sought treatment for a mental health disorder from a mental health professional? 352 non-null bool \n", " 54 Do you have a family history of mental illness? 352 non-null object \n", " 55 If you have a mental health disorder, how often do you feel that it interferes with your work *when being treated effectively?* 352 non-null object \n", " 56 If you have a mental health disorder, how often do you feel that it interferes with your work *when* _*NOT*_* being treated effectively (i.e., when you are experiencing symptoms)?* 352 non-null object \n", " 57 Have your observations of how another individual who discussed a mental health issue made you less likely to reveal a mental health issue yourself in your current workplace? 282 non-null object \n", " 58 How willing would you be to share with friends and family that you have a mental illness? 352 non-null int64 \n", " 59 Would you be willing to bring up a physical health issue with a potential employer in an interview? 352 non-null object \n", " 60 Why or why not? 273 non-null object \n", " 61 Would you bring up your *mental* health with a potential employer in an interview? 352 non-null object \n", " 62 Why or why not?.1 285 non-null object \n", " 63 Are you openly identified at work as a person with a mental health issue? 352 non-null bool \n", " 64 Has being identified as a person with a mental health issue affected your career? 45 non-null object \n", " 65 How has it affected your career? 18 non-null float64\n", " 66 If they knew you suffered from a mental health disorder, how do you think that your team members/co-workers would react? 352 non-null int64 \n", " 67 Have you observed or experienced an *unsupportive or badly handled response* to a mental health issue in your current or previous workplace? 352 non-null object \n", " 68 Describe the circumstances of the badly handled or unsupportive response. 89 non-null object \n", " 69 Have you observed or experienced a *supportive or well handled response* to a mental health issue in your current or previous workplace? 352 non-null object \n", " 70 Describe the circumstances of the supportive or well handled response. 0 non-null float64\n", " 71 Overall, how well do you think the tech industry supports employees with mental health issues? 352 non-null int64 \n", " 72 Briefly describe what you think the industry as a whole and/or employers could do to improve mental health support for employees. 254 non-null object \n", " 73 If there is anything else you would like to tell us that has not been covered by the survey questions, please use this space to do so. 100 non-null object \n", " 74 Would you be willing to talk to one of us more extensively about your experiences with mental health issues in the tech industry? (Note that all interview responses would be used _anonymously_ and only with your permission.) 352 non-null bool \n", " 75 What is your age? 352 non-null int64 \n", " 76 What is your gender? 347 non-null object \n", " 77 What country do you *live* in? 352 non-null object \n", " 78 What US state or territory do you *live* in? 204 non-null object \n", " 79 What is your race? 204 non-null object \n", " 80 What country do you *work* in? 352 non-null object \n", " 81 What US state or territory do you *work* in? 204 non-null object \n", "dtypes: bool(5), float64(7), int64(4), object(66)\n", "memory usage: 213.6+ KB\n" ] } ], "source": [ "# Data info\n", "df.info()" ] }, { "cell_type": "code", "execution_count": 23, "id": "controlling-locking", "metadata": {}, "outputs": [], "source": [ "# Lower the column names\n", "df.columns = df.columns.str.lower()" ] }, { "cell_type": "code", "execution_count": 24, "id": "laden-bunny", "metadata": {}, "outputs": [], "source": [ "# Remove special characters from the columns of year 2019\n", "columns = list(df.columns)\n", "\n", "columns = [column.replace('*', '') for column in columns]\n", "columns = [column.replace('_', '') for column in columns]" ] }, { "cell_type": "code", "execution_count": 25, "id": "needed-church", "metadata": {}, "outputs": [], "source": [ "# Assign the column names back to the dataframe\n", "df.columns = columns" ] }, { "cell_type": "code", "execution_count": 26, "id": "personalized-discrimination", "metadata": {}, "outputs": [], "source": [ "# Rename mismatched column names\n", "df.rename(columns = {'if they knew you suffered from a mental health disorder, how do you think that your team members/co-workers would react?':\n", " 'if they knew you suffered from a mental health disorder, how do you think that team members/co-workers would react?'}, inplace=True)\n", "\n", "df.rename(columns = {'have you observed or experienced a supportive or well handled response to a mental health issue in your current or previous workplace?':\n", " 'have you observed or experienced supportive or well handled response to a mental health issue in your current or previous workplace?'}, inplace=True)" ] }, { "cell_type": "code", "execution_count": 27, "id": "coupled-radar", "metadata": {}, "outputs": [], "source": [ "df['year'] = '2019'" ] }, { "cell_type": "code", "execution_count": 28, "id": "aerial-penetration", "metadata": {}, "outputs": [], "source": [ "# Write the cleaned file back\n", "df.to_csv(\"./../../../datasets/cleaned_osmi_2019.csv\", index=False)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 5 }