Making the Whole Project/Repo Menu-Driven

[avinashkranjan]

[UPDATE]: Check this comment for progress on this issue: #831 (comment)

Is your feature request related to a problem? Please describe.
Changing directories looking for scripts to use one by one will be quite a headache.

Describe the solution you'd like
Making it a menu-driven script, where a user can select the option and the specific script will run (can take arguments too if needed at runtime) in sort like Lazy Script if you are aware of.

Let me brief you about what you have to do on this issue:

1. Make a menu-driven script which will perform the task as per selected option.
2. There are 100+ scripts currently so we need to provide all in the menu.
   Which can be done in the following ways
   • Scripts can be further categorized into sub categories
     • Automation
     • AI/ML
     • Calculators
     • Games
     • Fun
     • Scrappers
     • Bots
       and More
   • It'll help in searching through the numerous script easy
3. Give all the folder 777 permission for the execution of python files
4. Do provide comments while doing scripting
5. Be aware of file path as per linux system ie. /root/your_filename while working on github
6. Follow all the guidelines provided

Marking Guidelines

• Since This is a big Issues, Multiple contributors are welcome to contribute to the script, Depending on the Contribution PR level would be marked..the Issue would be a Level3, Each Linked PR would be of Minimum Level2.

@kaustubhgupta @antrikshmisri @santushtisharma10 Kindly Helpout the Contributors working on this particular Issue..As this one is a complex issue..If Any Issues, DM me on Discord..

Happy Coding..👨🏻‍💻

[AshuKV]

Can I work on this issue under GSSOC21? Kindly assign it to me.

[XZANATOL]

I've a suggestion. We can start by categorizing each script into a dictionary.. something like:
{"Scrappers": ["Project1", "Project2"], "Automation":["Project3", "Project4"]}
Each list can contain records of scripts that are in another category. This will assure that effiency of sorting and tag searching, and put each script into its suitable place making it organized. This dictionary can be saved in a CSV file for future adds/edits.

We can also make a function to act as an API which adds a new script with the suitable tags, and help contributors to automate the process of adding their scripts to this script instead of manually adding them. (Let's leave that for after we settle on how we will save the data of the existing scripts.)

[Ajay-Singh-Rana]

Hey..! What if we create a script that takes folder names as arguments and lets the user decide which folder thier script is to be moved in to and also an option to add a newer category.
script.py folder_name category
or
script.py -n new_category folder_name [folder_name [folder_name]

[kaustubhgupta]

Hi, @Ajay-Singh-Rana @XZANATOL @AshuKV,

As I have just returned home, I need some time to plan out the overall structure of the script, how it will handle new scripts and how we can have a GUI for that.

I will update you on this day after tomorrow

[Ajay-Singh-Rana]

Any updates..?

[kaustubhgupta]

Any updates..?

Will update by EOD

[avinashkranjan]

Hey @kaustubhgupta If Need any help or any doubt kindly text me on discord..Participants are asking about this Issue..

[kaustubhgupta]

Hi, @AshuKV @XZANATOL @avinashkranjan @Ajay-Singh-Rana (Apologies for updating late, was busy with important work),

I liked the @XZANATOL implementation to create the database of scripts based on categories. I think a big JSON would do the work. Why JSON? Because it can be loaded into the dictionary in Python. Creating this JSON would be a manual task initially as we can't trace back to every merged pull request and check if it was a scrapper or ML script as neither the participants have filled the PR template properly nor there was an option to add a category. @Ajay-Singh-Rana Your idea has a problem that a new user would not be aware of the folder where the script lies.

Also, one major problem apart from these ideas is that all participants have different entry points for their script. Some of them have main.py, scrapper.py, or script-specific names.

Here is my workaround for this issue in 4 tasks (Please read till the end before assuming why I am asking these):

• Creating the manual JSON: This JSON file, as suggested by @XZANATOL will have the root element as Category and its value as another JSON with the following information:

  • Script Name
  • Script folder relative path
  • Entry point of the script (Which file runs the program)
  • arguments it takes (if any). Now here we need to define that if the arguments will be passed using argument parser or direct CLI.
  • requirements_path: None if no requirements.txt file present else the relative path of this file
  • Contributor Name
  • Small Description

• Creating the demo Script for testing: After this database is prepared, we can start working on the implementation part. This demo script will present the user with categories of the script. Then as per user input, a list of available scripts will be presented. and then the user input script will be run. The script can be run by the subprocess module. Why not OS? Because in the subprocess module we can define the script entry point and arguments more easily. Refer this article for implementation. Now as soon as the user gives the script choice, our script should use the database to see if it has requirements.txt. If yes then first run: pip install -r $requirements_path/requirements.txt and then find the entry point of the script to call: python entrypoint.py. This command can be called via subprocess too with arguments.

• Porting this demo to GUI: If this demo script works, then we can port this to a suitable GUI with a menu to select the category, sub-script, and provide arguments if needed. We can also have a checkbox for installing requirements.txt! The GUI can be made in any framework depending upon which contributor takes this sub-issue.

• Automating JSON updation: The manual information adding is a one-time task. To add new script information to this JSON, we will use the PR template and GitHub Actions to add update this information. The PR template needs to be updated with the new parameters mentioned in the manual JSON work and every participant will have to compulsorily fill in this information. The GitHub action, on every PR merged, will push this information to this JSON and hence an application will be converted out of this repo! When this task will be performed. all other PRs will be stopped for a while and they will need to update the PR template accordingly.

• BONUS task, website updation: The website currently has hardcoded information about the scripts. The JSON generated can be used by web designers to display script information with contributor name, small description, and link to the script. That's why I asked in the PR template this information in the first place! This will help both projects!

As these tasks are themselves way too big, every task will be rewarded level3. Also, no separate issue will be created for this. All 4-5 PRs will be merged tagging this issue only. The subtasks will be assigned and ticked off in the order mentioned below in this comment only:

• Creating Manual JSON - @XZANATOL (Added 'Master Script' base JSON DB #963)
• Creating demo Script - @devRawnie
• Automating JSON updation - @XZANATOL (Action db update #1025)
• BONUS task, Website updation - @Ayushjain2205 (Updating Scripts contributors on Website with implementation of search bar #1030)
• Porting demo to GUI

Let me know what should be modified, any new ideas except this, or anything else. Also, which task you are interested in.

VERY IMPORTANT INFO: THIS WHOLE ISSUE IS NOT FIRST COME FIRST SERVE. IF YOU CAN GENUINELY HELP AND HAVE THE SKILLS TO CODE THIS THEN ONLY COMMENT AND DISCUSS.

I have created a separate discussion thread for this. Join here: #882

[Ajay-Singh-Rana]

Hey...Why not make things easy by moving scripts to specific task categories in the repo itself..?This would also make the repository much more accessible based on specificity.

[XZANATOL]

@Ajay-Singh-Rana many scripts can belong to many different categories, not only one.. so doing this will make it even harder to keep track of the scripts when we try to display them according to specificity 🤔

[kaustubhgupta]

Hey...Why not make things easy by moving scripts to specific task categories in the repo itself..?This would also make the repository much more accessible based on specificity.

We are not aiming to restructure the repo but create a master script to initiate any script of this repo

[XZANATOL]

Finally got free from my commitments.. xd
@kaustubhgupta I would like to start with the base of the project of creating the JSON file. Gonna read the each project of the repo and collect the necessary information. I can also add the code snippet of the function which allows the developer to add his/her script to the repo as I will automate this part on my local machine. xd

[kaustubhgupta]

@XZANATOL Cool, you can start working on this sub-task.

[Ajay-Singh-Rana]

@kaustubhgupta @XZANATOL do you guys mean that the repository would remain as it is and the JSON has to be manually created at first and lateron we can give an option to add the script under a particular section by passing the arguments..? Is it..?

[XZANATOL]

@kaustubhgupta Hi there, I've finsihed most of the JSON file, here is a small part of how it is structured.

The JSON file follows this structure: {category: {name: [path, entry, arguments, requirments_path, contributor, description]}}
Tell me what you think about it, and whether should I change anything about it.

[kaustubhgupta]

@kaustubhgupta Hi there, I've finsihed most of the JSON file, here is a small part of how it is structured.

The JSON file follows this structure: {category: {name: [path, entry, arguments, requirments_path, contributor, description]}}
Tell me what you think about it, and whether should I change anything about it.

Perfect, 💯

I am working on the Pull request automation locally and maybe I will create the pull request for that. Let's see how this works and then we can move towards making the demo script!

[kaustubhgupta]

@kaustubhgupta @XZANATOL do you guys mean that the repository would remain as it is and the JSON has to be manually created at first and lateron we can give an option to add the script under a particular section by passing the arguments..? Is it..?

Yes @Ajay-Singh-Rana. If you want, you can work on the pull request automation.

[Ayushjain2205]

@kaustubhgupta I would like to work the 2nd task - Creating the demo Script for testing
Just have a small question regarding how to deal with scripts that require arguments, should the user first be prompted what argument is required and then take the argument from the user as a command line argument?

[kaustubhgupta]

Hi @Ayushjain2205, Let @XZANATOL update the database and then I can explain you better about actual implementation

[XZANATOL]

@kaustubhgupta I've a doubt, There are some scripts that needs a requirments.txt file but it does not exist, however the pip commands present in the Readme.md file. Shall I make a seperate requirments.txt file and add the needed for each script (if necessary) or shall I give the path to the Readme.md file instead? tho doing the second option will prevent adding a step to prepare the script libraries using our menu-driven script. 🤔

[XZANATOL]

@kaustubhgupta ok the PR is ready, double check my changes and update me if I'm missing something.

For the added projects.. These are the projects that haven't been added till now, becuase of the issues mention in this comment

• AWS Management Scripts
• Convert2jpg
• Data-Visualization
• Internet-Speed-Test
• Restoring-Divider
• Spelling-Checker

These are the projects which I've updated in the PR.

• Coderforces_Problem_Scrapper
• Cryptography
• Instagram Follow- NotFollow
• LeetCode-Scrapper
• Malaria
• NSE Stocks GUI
• Script to check Sentiment

I've a doubt. How are we going to add these projects which are still at hold? or is there more manual work awaiting me? xD

[XZANATOL]

BTW, We need to make another PR for gh-pages branch. The update_file method doesn't work as create_file method in PyGithub library.. update_file retreives the file first then appends the changes into it. Running the action this way will throw an error. check this.

so we need to push the DB first. I'm going to wait your review about the currently openned PR and if it is good to go then I'll make one for gh-pages.

[kaustubhgupta]

I've a doubt. How are we going to add these projects which are still at hold? or is there more manual work awaiting me? xD

Haha, nope, that's where I will assign @devRawnie the task of checking the scripts for adding/modifying the requirements.txt file for all the scripts. I will add copy-paste this comment in that issue.

BTW, We need to make another PR for gh-pages branch. The update_file method doesn't work as create_file method in PyGithub library.. update_file retreives the file first then appends the changes into it. Running the action this way will throw an error. check this.

so we need to push the DB first. I'm going to wait your review about the currently openned PR and if it is good to go then I'll make one for gh-pages.

Hmm, let me research a bit about this, I think I saw a workaround for this in one of the projects. What I am expecting is that the updated JSON file needs to be pushed to both branches via the same action. Did you try getting the contents of the main branch and gh-pages branch separately and then updating the file?

[XZANATOL]

Oh that's not what I meant. the action will update the file on both branches but the thing is on the first run of the action. currently on the PR merge the DB will exist on the main branch only not gh-pages. So on the first run of the action exactly on line 45
repo.update_file("./datastore.json", "Updated datastore.json", data_store, datastore_file.sha, branch="gh-pages")
The update_file method will try to retrieve ./datastore.json which doesn't exist and will throw an error.

[kaustubhgupta]

Oh that's not what I meant. the action will update the file on both branches but the thing is on the first run of the action. currently on the PR merge the DB will exist on the main branch only not gh-pages. So on the first run of the action exactly on line 45
repo.update_file("./datastore.json", "Updated datastore.json", data_store, datastore_file.sha, branch="gh-pages")
The update_file method will try to retrieve ./datastore.json which doesn't exist and will throw an error.

Oh, my bad. Let me add the database manually on that branch,

[kaustubhgupta]

Oh that's not what I meant. the action will update the file on both branches but the thing is on the first run of the action. currently on the PR merge the DB will exist on the main branch only not gh-pages. So on the first run of the action exactly on line 45
repo.update_file("./datastore.json", "Updated datastore.json", data_store, datastore_file.sha, branch="gh-pages")
The update_file method will try to retrieve ./datastore.json which doesn't exist and will throw an error.

Oh, my bad. Let me add the database manually on that branch,

I have added the file

[devRawnie]

@kaustubhgupta I had a doubt about implementing the argument taking part in this script.

1. As you proposed a solution for this, by displaying the description. Where would that particular description exist in the codebase? And how would I extract that from the script?
2. We could take an approach by analyzing the .md file for statements such as pip3 install or pip install and install them manually.

1. As mentioned by @XZANATOL, the description is part of the database.
   
2. As I mentioned earlier also, the requirements path should be used for installing requirements.txt

@kaustubhgupta So for now, after the user selects a particular script, the description will he shown to him and the packages will be installed from requirements.txt file. Based on the description. The user would enter command line arguments if required and finally the script will run taking in those arguments, fron the user. Tell me if I am missing something. I will be finalizing this by tonight.

[kaustubhgupta]

@kaustubhgupta I had a doubt about implementing the argument taking part in this script.

1. As you proposed a solution for this, by displaying the description. Where would that particular description exist in the codebase? And how would I extract that from the script?
2. We could take an approach by analyzing the .md file for statements such as pip3 install or pip install and install them manually.

1. As mentioned by @XZANATOL, the description is part of the database.
   
2. As I mentioned earlier also, the requirements path should be used for installing requirements.txt

@kaustubhgupta So for now, after the user selects a particular script, the description will he shown to him and the packages will be installed from requirements.txt file. Based on the description. The user would enter command line arguments if required and finally the script will run taking in those arguments, fron the user. Tell me if I am missing something. I will be finalizing this by tonight.

Correct, this is the flow for this script

[devRawnie]

@kaustubhgupta I have completed the PR for the main.py script. It is throwing errors for some scripts because of the issue we discussed regarding the requirements.txt file. Kindly check

[kaustubhgupta]

@kaustubhgupta I have completed the PR for the main.py script. It is throwing errors for some scripts because of the issue we discussed regarding the requirements.txt file. Kindly check

okay I will check

[XZANATOL]

@kaustubhgupta I think there is something wrong with Personal Access Token -PAT- in the repo.. is it updated? There are 2 logs in the action tab where the script ran successfully till the part where it logins using the PAT.

There is another log that failed which was weird a bit, but looking a bit closely.. the contributor changed a letter in a line which prevented the Regex detection and threw an error.

Looking at the PR body.. turns out he changed the begging of the line of the requirments path from Requirments to Requirements (he added an 'e')

[kaustubhgupta]

@kaustubhgupta I think there is something wrong with Personal Access Token -PAT- in the repo.. is it updated? There are 2 logs in the action tab where the script ran successfully till the part where it logins using the PAT.

There is another log that failed which was weird a bit, but looking a bit closely.. the contributor changed a letter in a line which prevented the Regex detection and threw an error.

Looking at the PR body.. turns out he changed the begging of the line of the requirments path from Requirments to Requirements (he added an 'e')

The PR token thing is conveyed to the PA and we will create a new token for this. Regarding the PR body thing, well it was a mistake from your side and mine too. The spelling of requirements was wrong and the same was updated in the PR. I had to fix that in the template and the code. Also, as the action failed for 3 PRs, I had to add that data manually. 🤣🤣

[kaustubhgupta]

@XZANATOL Also, try to run this PR locally: #1030

How the database is useful for website too! Great work done by @Ayushjain2205

[XZANATOL]

can't believe that slipt down of our eyes 😂
For the PR.. you mean to test it?

[kaustubhgupta]

can't believe that slipt down of our eyes 😂
For the PR.. you mean to test it?

Yes, you can clone it locally and open the live server. The search for contributors and scripts is done using this json

[XZANATOL]

I'm not sure if I'm doing it correctly, but I've cloned the PR and nothing happens on the scripts.html page, like it doesn't load the json.. not sure 🤔 even when trying to conduct a search using a contributor name or the script title, it doesn't reflect a thing.

[kaustubhgupta]

I'm not sure if I'm doing it correctly, but I've cloned the PR and nothing happens on the scripts.html page, like it doesn't load the json.. not sure 🤔 even when trying to conduct a search using a contributor name or the script title, it doesn't reflect a thing.

It should load the scripts. Maybe do a hard refresh (pressing cntrl while pressing refresh button)

[XZANATOL]

Tried it, and no results

[kaustubhgupta]

Tried it, and no results

It works, check out the deployed website: https://avinashkranjan.github.io/Amazing-Python-Scripts/

[Sumitwarrior7]

which folder is responsible for the actual website i.e., the folder which consisi for homepage, contactpage, scriptspage
There are so many folders, i am not able to find it!

[avinashkranjan]

There are so many folders, i am not able to find it!

Switch to branch gh-pages

[Sumitwarrior7]

there is no branch named gh-pages

Because my issue is regarding improving the Contact-Us page of the website, so where can i find the code for that page?
right now i am not able to find it.

[Ayushjain2205]

https://github.com/avinashkranjan/Amazing-Python-Scripts/tree/gh-pages You can check this out