diff --git a/README.md b/README.md index 874916c..0d57626 100644 --- a/README.md +++ b/README.md @@ -10,28 +10,35 @@ This repository provides easy-to-use benchmarks using audio and video content fr | Thick accents | Moonshine for Medicine Popcorn Sutton | [Internet Archive](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) | | Low-quality audio | 1994 90210 Melrose Place Promos Commercial | [Internet Archive](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) | | Artifacts in audio | 2002 007 Movie Trailer Commercial Bad Video | [Internet Archive](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) | +| Ideal audio (one speaker) | 8 Bit Bookclub | [Internet Archive](https://archive.org/details/8-bit-bookclub/36+-+ANNOUNCEMENT++SUMMER+HIATUS.mp3) | + +## How to Run Whisper Benchmarks + +### 1. Download Video from the Internet Archive + +Visit the links provided and download the relevant item listed under MPEG or H264 in the download options. MP3 will also work. + +### 2. Visit Whisper WebUI Repository +https://gitlab.com/aadnk/whisper-webui + +Run whisper in docker or natively by following the provided instructions, make sure you set the appropriate options for the test you want to do (ie. omit "--gpus=all" if you would like to run a CPU benchmark) + +Note: Occasionally the docker image provided does not work, if you run into any errors then try the gitlab registry. Also, the docker method is generally recommended as it's a lot easier to get up and running. ## Results +Links are embeded for each category + ### CPU Benchmarks -| Video | Time (s) | CPU Model | -|------------------------------------------------------------|----------|-----------------------------------| -| [Body camera footage from July 10 traffic stop](https://archive.org/details/cobmn-Body_camera_footage_from_July_10_traffic_stop) | 15.25 | Intel Core i9-9900K | -| [Moonshine for Medicine Popcorn Sutton ](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) | 22.50 | AMD Ryzen 7 5800X | -| [1994 90210 Melrose Place Promos Commercial](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) | 10.75 | Intel Core i7-10700K | -| [2002 007 Movie Trailer Commercial Bad Video](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) | 18.30 | AMD Ryzen 9 5900X | +| CPU Model | Poor mic placement (s) | Thick accents (s) | Low-quality audio (s) | Artifacts in audio (s) | Ideal audio (one speaker) | (Docker/Native) | +|-|-|-|-|-|-|-| ### GPU Benchmarks -| Video | Time (s) | GPU Model | -|------------------------------------------------------------|----------|-----------------------------------| -| [Body camera footage from July 10 traffic stop](https://archive.org/details/cobmn-Body_camera_footage_from_July_10_traffic_stop) | 8.50 | NVIDIA GeForce RTX 3080 | -| [Moonshine for Medicine Popcorn Sutton ](https://archive.org/details/this-is-the-last-dam-run-of-likker-ill-ever-make-full-movie/+Moonshine+for+Medicine++++Popcorn+Sutton.mp4) | 15.75 | NVIDIA Quadro P5000 | -| [1994 90210 Melrose Place Promos Commercial](https://archive.org/details/1994variouscommercials/1994+90210+Melrose+Place+Promos+Commercial.mkv) | 8.20 | AMD Radeon RX 6800 XT | -| [2002 007 Movie Trailer Commercial Bad Video](https://archive.org/details/2002variouscommercials/2002+007+Movie+Trailer+Commercial+Bad+Video.mp4) | 12.45 | NVIDIA Tesla V100 | - +| GPU Model | Poor mic placement (s) | Thick accents (s) | Low-quality audio (s) | Artifacts in audio (s) | Ideal audio (one speaker) | (Docker/Native) | +|-|-|-|-|-|-|-| ## Todo: -- [ ] Write easy bash scripts for running a set of benchmarks -- [ ] Create a standard format for exporting the data into a spreadsheet +- [ ] Write easy bash scripts for running a set of benchmarks with an easy cleanup +- [ ] Finalize a standard format for exporting the data into a spreadsheet