Team Name : Code Breakers
Theme : Theme 4 (Audio dubbing for YouTube videos)
Piyush Garg
Discord : infinity_star#4499
Mayank Goswami
Discord : Vegeta#9780
Shashikant Rajput
Discord : t33#9261
Video Demo : https://youtu.be/qO7u4n5Bg8o
Backend API Docs : https://ilcoder.biz/docs
Backend API test URL : https://ilcoder.biz
Test Video URL (Dubbing already available) : https://www.youtube.com/watch?v=9em32dDnTck&ab_channel=Valkeir
Step 1. Download the chrome extension folder (either by cloning or direct download)
Step 2. Go to Extensions in Chrome and enable developer mode.
Step 3. Click on Load Unpacked and select the downloaded chrome extension folder.
Make sure that the extension has been loaded in the extensions list.
You may use see this tutorial here : https://webkul.com/blog/how-to-install-the-unpacked-extension-in-chrome/
Step 4. After following the above steps, the extension is ready to use.
Open any youtube video and click on extension icon. The available audio dubbings will be listed there. Click on any audio dubbing to listen to that one.
Step 5. (Optional) You can also record and upload your own dubbing for a video. To do so, click on extension icon and fill in language. After that just click on "Mic" icon and it will ask for permission and once done, it will start recording. Click on "Stop" once done
Now you can listen, save or discard the recording.
Note :- Don't click anywhere else otherwise the recording will be lost.
- HTML
- CSS
- JavaScript
- Chrome Extension
- Python 3.9
- FastAPI
- Websockets
- Cockroach DB
- Deta Drive SDK
- VS Code
- Twilio API
- Created basic API for storage of audio files on Deta Drive ✅
- Created streaming API for sendng audio files in part ✅
- Created basic chrome extension ✅
- Created API and connected with Cockroach DB ✅
- Linked Custom Domain (https://ilcoder.biz) to backend hosting service ✅
- Hosted on Heroku for Websocket Support ✅
- Adding functionalities to the extension ✅
- Notification for new dubbing added ✅
- Beautification of extension UI ✅
- Automatic conversion of audio using Speech Recognition
- Dubbing in part of the video
We will create an extension, using which a user will be able to
- Give it's own translation for any particular video for any part of it in any language of his/her choice which will be stored on the server
- Choose a translation/dubbed audio to play instead of original audio out of available ones which is retrieved from the server
Once a user uploads the dubbed audio, the server will store the audio and save the metadata like language, date recorded, username of the recording user, etc. in the database. And retrieve the same when needed
Still one question remains - How to identify genuine dubbing?
We can ask the user for rating once the audio dubbing has ended and based on that we can rank the audio dubbings.
We will aim to create a demo which will include
- A browser extension
- A server with database and cloud storage for saving audio dubbings
The user will open a YouTube video and extension will become active. When user clicks on extension prompt, option for recording a new dub and the available dubs for that particular video will be displayed and user can choose a dub in the required language which will be played over the original audio.
- We can add automatic translation using below process :
-
We can add incentives & support to the person doing dubbing task
-
Right now, only whole video dub / dub from starting is available. As a future scope, dubbing in parts and auto-syncing will be made possible using timestamps at which audio are recorded.
-
We can also provide automatic translations using auto generated translations for captions by YouTube (though not all videos have captions, so not all videos will be translated using it)
Get timestamp at beginning of recording & store it along with recording file and duration
Sync Time :
If new audio is available at current timestamp, prompt user to continue current or select a new one
Also make an option, to always continue current audio dubbing, if it is available for current time
