|
1 | 1 | # Project Proposal |
2 | 2 |
|
3 | | -## Finding Insights from Stackoverflow Developer Survey |
| 3 | +## Finding Insights from Stack Overflow Developer Survey |
4 | 4 |
|
5 | | -Stack overflow is a professional community for developers, Stackoverflow conducts a survey every year the collected data from 2011 has been available for open source on the web with the latest dataset 2020 released on March 5th, 2021. If the dataset analysed professionally using modern tools, would enable us to answer real-world questions effectively. The dataset has covered 275 questions in total. |
| 5 | +Stack Overflow is a professional community for developers, conducting an annual survey. The collected data from 2011 onwards has been available for open source on the web, with the latest dataset released in 2020. Analyzing this dataset professionally using modern tools would enable us to answer real-world questions effectively. The dataset includes responses to 275 questions. |
6 | 6 |
|
7 | 7 | ### Project Goal: |
8 | 8 |
|
9 | | -1. To perform Analysis on 3 years Stackoverflow Dataset and get insights. |
10 | | -2. To perform Data Analysis and answer the below questions. |
11 | | - + Impact of igher education on salary of the surveyed developers. |
12 | | - + Impact of education/experience/responsibilities on gender inequalities. |
13 | | - + Impact on participation rate due to different ethnicity. |
14 | | - + To find whether there is any difference between men and women's income. |
15 | | - + Impact on the increase in popularity of a language in the current year due to developer’s interest in the previous year. |
| 9 | +1. **Perform Analysis on 3 years of Stack Overflow Dataset:** Extract insights from the data. |
| 10 | +2. **Data Analysis Goals:** Answer the following questions: |
| 11 | + - What is the impact of higher education on the salary of surveyed developers? |
| 12 | + - How do education, experience, and responsibilities affect gender inequalities? |
| 13 | + - How does ethnicity impact participation rates? |
| 14 | + - Is there a difference in income between men and women? |
| 15 | + - How does the previous year's interest in a language affect its popularity in the current year? |
| 16 | +3. **Data Visualization Goals:** |
| 17 | + - Identify the most commonly used language. |
| 18 | + - Analyze the distribution of surveyors based on their developer roles. |
| 19 | + - Explore factors affecting job satisfaction. |
| 20 | + - Predict the growth of languages for upcoming years based on survey answers. |
| 21 | + - Provide insights for IT environment, hiring employees, job seekers, and building a solid résumé. |
16 | 22 |
|
17 | | -3. To perform data visualization on |
| 23 | +### Data Source and Background |
18 | 24 |
|
19 | | - - The most commonly used language. |
| 25 | +The dataset is sourced from the annual Stack Overflow developer survey, covering responses from developers in 180 countries. The data range from 2011 to 2020, with the focus being on the last 3 years. Respondents primarily come from the US, India, and EMEA regions, with a background in developer/coding experience. The dataset includes survey data gathered from 180 countries, with responses ranging from "Not at all important" to "Very important" and "Not at all satisfied" to "Very satisfied." |
20 | 26 |
|
21 | | - - Distribution of surveyors based on their developer role. |
| 27 | +### Data Format |
22 | 28 |
|
23 | | - - Factors affecting Job satisfaction. |
| 29 | +The data is in CSV format, consisting of 252,199 observations and 62 variables. |
24 | 30 |
|
25 | | - - Predicting the growth of languages for upcoming years based on the survey answers. |
| 31 | +### Projected Work for Insights |
26 | 32 |
|
27 | | - ###### The Insights can be used to provide information regarding IT environment, hiring employees and job seekers and build a solid résumé. |
| 33 | +#### Data Wrangling |
28 | 34 |
|
29 | | -### Data Source and Background |
| 35 | +- **Dealing with Null Values:** Handle unanswered questions marked as ‘NA’ or ‘Not Applicable’ to ensure precise analysis. |
| 36 | +- **Data Conversion/Manipulation:** Convert data for analysis, considering that respondents answered the survey through radio buttons rather than yes or no patterns (Univariate analysis). |
30 | 37 |
|
31 | | -The dataset is very diverse and came from a [Stackoverflow developer survey](https://insights.stackoverflow.com/survey/?_ga=2.208907280.304952146.1616422967-1864686930.1616422967) with 275 questions answered from 180 countries. Stackoverflow has data collected through surveys from 2011 to 2020, but for the project, the purpose is to analyze the data of the last 3 years. The people who completed the survey mostly from the US, India, and EMEA regions. The majority of the survey respondents had the background of developer/ coding experience. The data are available in the CSV format ranging from 40 to 150 MB with data of 1.5 Lakh survey participants.The dataset includes survey data gathered from 180 countries, the response ranges from Not at all important to very important/ Not at all satisfied to very satisfied. |
| 38 | +#### Techniques Expected to Use in the Project |
32 | 39 |
|
33 | | -### Data Format |
| 40 | +- ML Algorithms: Utilize algorithms like Random Forest, KNN, AUC for classification problems, logistic regression, and linear regression. |
| 41 | +- Data Visualization: Employ data visualization techniques for better understanding and presentation of insights. |
| 42 | +- Parameter Analysis: Analyze parameters to fine-tune models and improve accuracy. |
34 | 43 |
|
35 | | -The data is in a schema CSV file that consists of 252,199 observations and 62 variables. |
| 44 | +#### Project Plan |
36 | 45 |
|
37 | | -### Projected work needs to be done for Insights. |
| 46 | +**Week 8:** Project Base Setup |
| 47 | +- Source control setup on [GitHub](https://github.com/Sanjayviswa/Stackoverflow_survey_Analysis) |
| 48 | +- Project Management using tools like MS Project |
| 49 | +- Complete Data Wrangling & Basic Analysis |
38 | 50 |
|
39 | | -###### Data Wrangling |
| 51 | +**Week 10:** Baseline Model Building |
| 52 | +- Implement algorithms and build baseline models |
40 | 53 |
|
41 | | -**Dealing Null Values**: As this is a developer survey and few questions left unanswered by the respondents as ‘*NA*’ or ‘*Not Applicable*’ so dealing with null values is important to get precise information. Data conversion/ manipulation is also required, as the developer responded to the survey through radio buttons rather than yes or no pattern(Univariate analysis). |
| 54 | +**Week 11:** Model Evaluation |
| 55 | +- Run tests and evaluate the performance of models |
42 | 56 |
|
43 | | -###### Techniques expect to use in the project |
| 57 | +**Week 12:** Finalization |
| 58 | +- Prepare video presentation summarizing the analysis and insights |
44 | 59 |
|
45 | | -Planning to use ML Algorithms like Random, may include, KNN, AUC for classification problems, training model, logistic regression,data visualization, parameter analysis, Linear Regreesion, Root Mean square. |
| 60 | +#### Additional Technical Details |
46 | 61 |
|
47 | 62 | > Linear regression(RFE techniques) |
48 | 63 |
|
|
53 | 68 | > Root Mean Squared Error Calculations |
54 | 69 |
|
55 | 70 | $$ |
56 | | -rmse = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(y_{i} - x_{i})^{2}} |
| 71 | +rmse = \sqrt{\left(\frac{1}{n}\right)\sum_{i=1}^{n}(y_{i} - x_{i})^{2}} |
57 | 72 | $$ |
58 | 73 |
|
59 | 74 |
|
| 75 | +## Potential Impact and Benefits |
60 | 76 |
|
61 | | -#### Project plan |
62 | | - |
63 | | -**Week 8:** Creating Project base, Source control([GitHub](https://github.com/Sanjayviswa/Stackoverflow_survey_Analysis)), Project Management(MS Project). |
64 | | - |
65 | | -- Complete Data Wrangling & basic Analysis. |
66 | | - |
67 | | -**Week 10**: Complete baseline Model building with algorithms. |
| 77 | +- **Inform Decision-Making:** The insights derived from the analysis of the Stack Overflow Developer Survey data can inform decision-making processes in various domains, including education, recruitment, and workforce development. |
| 78 | +- **Address Gender and Ethnic Inequalities:** By analyzing the impact of education, experience, and responsibilities on gender and ethnic inequalities, this project can contribute to raising awareness and identifying strategies to address these disparities in the tech industry. |
| 79 | +- **Support Career Development:** The findings from the analysis can provide valuable insights for developers seeking to advance their careers, make informed decisions about their education and training, and enhance their job satisfaction. |
| 80 | +- **Contribute to Open Source Community:** By contributing to this project, developers have the opportunity to collaborate with others, share their expertise, and contribute to open-source initiatives aimed at improving data analysis techniques and tools. |
| 81 | +- **Empower Data-Driven Decision Making:** The project's focus on ML algorithms, data visualization, and predictive analytics empowers stakeholders to make data-driven decisions, enabling them to stay ahead in a rapidly evolving technology landscape. |
68 | 82 |
|
69 | | -**Week 11:** Run tests and evaluate model. |
| 83 | +## 👨💻 Contributing |
70 | 84 |
|
71 | | -**Week 12:** Prepare video presentation. |
| 85 | +- **Contributions Welcome:** We welcome contributions from the community to help enhance this project. Whether you're a seasoned developer or just starting out, there are various ways you can contribute: |
| 86 | + - **Code Contributions:** Help improve the analysis code, implement new features, or optimize existing algorithms. |
| 87 | + - **Data Wrangling:** Assist in cleaning and preprocessing the dataset to ensure accurate analysis. |
| 88 | + - **Documentation:** Enhance project documentation, including the README file, to make it more comprehensive and user-friendly. |
| 89 | + - **Bug Fixes:** Identify and fix any bugs or issues encountered during the analysis. |
| 90 | + - **Feature Requests:** Suggest new features or improvements to further enhance the project. |
| 91 | +- **How to Contribute:** To contribute, simply fork the repository, make your changes, and submit a pull request. Be sure to follow the contribution guidelines outlined in the repository. |
| 92 | +- **Contributors Recognition:** We greatly appreciate all contributions to this project and will acknowledge contributors in the README file. |
| 93 | +- **Join the Discussion:** Feel free to join the discussion on our [GitHub repository](https://github.com/Sanjayviswa/Stackoverflow_survey_Analysis) to share your ideas, ask questions, or collaborate with other contributors. |
72 | 94 |
|
| 95 | +Crafted by @Sanjayviswa. |
0 commit comments