brooke/whisperbenchmarks
1
0
Fork 0
Code Issues Wiki Activity

Update README.md

Browse source 
Add a how-to section and remove example data
This commit is contained in:
brooke 2023-12-02 04:06:55 +00:00
parent 4ac8d7fd2c
commit 22c7aec8db
1 changed files with 22 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
README.md
View file

@ -10,28 +10,35 @@ This repository provides easy-to-use benchmarks using audio and video content fr
| Thick accents | Moonshine for Medicine Popcorn Sutton | [Internet Archive](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) |
| Low-quality audio | 1994 90210 Melrose Place Promos Commercial | [Internet Archive](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) |
| Artifacts in audio | 2002 007 Movie Trailer Commercial Bad Video | [Internet Archive](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) |
| Ideal audio (one speaker) | 8 Bit Bookclub | [Internet Archive](https://archive.org/details/8-bit-bookclub/36+-+ANNOUNCEMENT++SUMMER+HIATUS.mp3) |
## How to Run Whisper Benchmarks
### 1. Download Video from the Internet Archive
Visit the links provided and download the relevant item listed under MPEG or H264 in the download options. MP3 will also work.
### 2. Visit Whisper WebUI Repository
https://gitlab.com/aadnk/whisper-webui
Run whisper in docker or natively by following the provided instructions, make sure you set the appropriate options for the test you want to do (ie. omit "--gpus=all" if you would like to run a CPU benchmark)
Note: Occasionally the docker image provided does not work, if you run into any errors then try the gitlab registry. Also, the docker method is generally recommended as it's a lot easier to get up and running.
## Results
Links are embeded for each category
### CPU Benchmarks
| Video | Time (s) | CPU Model |
|------------------------------------------------------------|----------|-----------------------------------|
| [Body camera footage from July 10 traffic stop](https://archive.org/details/cobmn-Body_camera_footage_from_July_10_traffic_stop) | 15.25 | Intel Core i9-9900K |
| [Moonshine for Medicine Popcorn Sutton ](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) | 22.50 | AMD Ryzen 7 5800X |
| [1994 90210 Melrose Place Promos Commercial](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) | 10.75 | Intel Core i7-10700K |
| [2002 007 Movie Trailer Commercial Bad Video](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) | 18.30 | AMD Ryzen 9 5900X |
| CPU Model | Poor mic placement (s) | Thick accents (s) | Low-quality audio (s) | Artifacts in audio (s) | Ideal audio (one speaker) | (Docker/Native) |
|-|-|-|-|-|-|-|
### GPU Benchmarks
| Video | Time (s) | GPU Model |
|------------------------------------------------------------|----------|-----------------------------------|
| [Body camera footage from July 10 traffic stop](https://archive.org/details/cobmn-Body_camera_footage_from_July_10_traffic_stop) | 8.50 | NVIDIA GeForce RTX 3080 |
| [Moonshine for Medicine Popcorn Sutton ](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) | 15.75 | NVIDIA Quadro P5000 |
| [1994 90210 Melrose Place Promos Commercial](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) | 8.20 | AMD Radeon RX 6800 XT |
| [2002 007 Movie Trailer Commercial Bad Video](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) | 12.45 | NVIDIA Tesla V100 |
| GPU Model | Poor mic placement (s) | Thick accents (s) | Low-quality audio (s) | Artifacts in audio (s) | Ideal audio (one speaker) | (Docker/Native) |
|-|-|-|-|-|-|-|
## Todo:
- [ ] Write easy bash scripts for running a set of benchmarks
- [ ] Create a standard format for exporting the data into a spreadsheet
- [ ] Write easy bash scripts for running a set of benchmarks with an easy cleanup
- [ ] Finalize a standard format for exporting the data into a spreadsheet