rewrite, cleanup, remove video replace with comparison images
Some checks failed
Hugo / build (push) Failing after 41s

This commit is contained in:
brooke 2024-06-03 03:16:01 -04:00
parent d50059d751
commit d7a7ecd7db
9 changed files with 50 additions and 27 deletions

View file

@ -2,21 +2,12 @@
title: 3D Scanning
---
{{< callout type="info" >}}
**Hey!** This page is a work in progress. If you'd like to assist in the process of writing, take a look at the [git repository](https://git.myco.systems/mycosystems/midtowndrafting.com)
{{< /callout >}}
In the prototype lab, we provide many methods for students to learn 3D scanning through practice. This page serves as the starting point to briefly outline what we can do.
# Hardware
{{< cards >}}
{{< card link="./nerf" title="NeRF and 3D Gaussians" icon="nerf" >}}
{{< /cards >}}
The prototype lab comes equipped with top-of-the-line hardware to facilitate the processing of large 3D models. The main components of the hardware setup include:
- **Computer**: The lab is equipped with a powerful computer, featuring:
- 2x **NVIDIA A5000** GPUs: These high-performance GPUs provide the necessary computing power for inference with NeRF models or heavy math compute for software like Meshroom, with a total of 48 GB of GDDR6 memory.
- 1x **AMD EPYC 7763 CPU**: This processor features 64 cores and 128 threads, offering exceptional multitasking capabilities.
- **8x 64GB DDR4** DIMMs: A total of 512 GB of RAM ensures smooth operation of large image datasets.
- **Camera**: The lab maintains all of the equipment to do photogrammetry, including:
- 2x **GVM LED Ring Lights**: Ring lights with 6 removeable light bars, a color range of 3200 to 5600K, adjustable from 32" to 87" height.
- 1x **Sony A6500**: A 24MP mirrorless camera capable of 4K video, has in-body stabilization, a decent 107 RAW image buffer, and an 11 fps still photo mode.
{{< callout type="info" >}}
**Hey!** This page is a work in progress. If you'd like to assist in the process of writing, take a look at the [git repository](https://git.myco.systems/mycosystems/midtowndrafting.com)
{{< /callout >}}

Binary file not shown.

After

Width:  |  Height:  |  Size: 282 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 296 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 691 KiB

View file

@ -2,13 +2,34 @@
title: NeRF
---
3D scanning is a very wide field, including many, many different use-cases. I personally like to look at the ideas around using 3D scanning as another modal for allowing people to interact with the world around them, I'm talking about museums 3D scanning all of the works they hold, or capturing high quality 3D scans of monuments that are at risk of being lost forever, and I think education in general can be greatly aided by introducing more active methods of learning based on working in 3D.
<svg style="width:60%; margin-top:2rem;" viewBox="2.9938 7.971 92.0062 32.0289" xmlns="http://www.w3.org/2000/svg">
<path style="fill:#FFD600;" d="M7,10.971l-3,8c0,0,1.633,0.124,3.233,0.575c2.077,0.982,1.97,3.073,1.249,5.039&#10;&#9;C8.33,24.712,8.175,24.839,8,24.971c-2.813,2.125-4.688,9-4.688,9L11,38.252c0,0-0.834-9.281,3-9.281h0.011&#10;&#9;c1.39,1.159,2.989,2,2.989,2c0.09,0,0.342,0,1,0c2.001,0,3.739-0.781,4.823-2h15.735l0.545,1.633&#10;&#9;c0.272,0.817,1.036,1.367,1.897,1.367c1.105,0,2-0.895,2-2v-19H7z M14,25.971c0-1.657,1.791-3,4-3c2.209,0,4,1.343,4,3s-1.791,3-4,3&#10;&#9;C15.791,28.971,14,27.628,14,25.971z" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<path style="fill:#455A64;" d="M40,24.971H25v-10h15V24.971z M10,8.971v-1H8v3h13v-2H10z M9.836,39.883&#10;&#9;C10.342,40.2,11,39.835,11,39.238v-0.985l-7.688-4.281c0,0-0.123,0.467-0.282,1.128c-0.131,0.546,0.101,1.117,0.577,1.415&#10;&#9;L9.836,39.883z" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<path d="M40,18.971H28v-2h12V18.971z M40,20.971H28v2h12V20.971z" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<path style="fill:#FF6D00;" d="M43,19.971h2v-5h-2V19.971z M40,24.971h-2v-10h2V24.971z M18,25.971c0,1.103,0.508,2.096,1.321,2.819&#10;&#9;c-0.415,0.11-0.855,0.181-1.321,0.181c-2.209,0-4-1.343-4-3s1.791-3,4-3c0.465,0,0.905,0.071,1.32,0.181&#10;&#9;C18.508,23.875,18,24.868,18,25.971z" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<path style="fill:#90A4AE;" d="M22,10.971l-2,6H4.75l0.658-1.755l-1.041-0.347C3.551,14.596,3,13.832,3,12.971c0-1.105,0.895-2,2-2&#10;&#9;H22z M14,28.971c0,0-1.416-1.25-4-2c-3.371,1.908-4.225,6.279-4.547,8.192L11,38.252C11,38.252,10.583,28.971,14,28.971z" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<g transform="matrix(1, 0, 0, 1, 13.000000000000007, 0)">
<rect x="65" y="16" width="15" height="3" style="stroke: rgb(0, 0, 0); stroke-opacity: 0; fill: rgb(30, 66, 154);"/>
<path style="stroke: rgb(0, 0, 0); stroke-opacity: 0; fill: rgb(255, 109, 0);" d="M 80 16 H 81 A 1 1 0 0 1 82 17 V 18 A 1 1 0 0 1 81 19 H 80 V 16 Z"/>
</g>
<rect x="71.762" y="16.199" width="5" height="0.5" style="stroke: rgb(0, 0, 0); stroke-opacity: 0; fill: rgba(255, 255, 255, 0.23);" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<rect x="72.587" y="18.292" width="5" height="0.5" style="stroke: rgb(0, 0, 0); stroke-opacity: 0; fill: rgba(255, 255, 255, 0.23);" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
<rect x="65.294" y="17.261" width="10.001" height="0.5" style="stroke: rgb(0, 0, 0); stroke-opacity: 0; fill: rgba(255, 255, 255, 0.23);" transform="matrix(1, 0, 0, 1, 7.105427357601002e-15, 0)"/>
</svg>
In the past, it has been prohibitively expensive and has had many drawbacks. It used to be that capturing a 3D model meant expensive equipment and prep time for your model, every surface to be captured needed to be perfectly matte or even had to be covered in a partical pattern. Photogrammetry and the wider concept of neural radiance fields have introduced a more software-defined approach to 3D scanning. [NeRF](https://www.matthewtancik.com/nerf), [MIP-NeRF](https://arxiv.org/abs/2103.13415), [3D Gaussians](https://arxiv.org/abs/2308.04079), and more techniques show off the ability to use neural networks to define a complete 3D model from a 2D reference point.
## Intro
I should say that none of these techniques are anywhere near a "production ready" stage, you still cannot derive an accurate 3D mesh from these techniques, but they have brought a lot of interesting concepts forward. For one, being able to share a color-accurate model has been the focus of my research recently, using what are called 3D guassians you can use an entirely software-defined approach to create very light (the below model is 6MB) and high quality 3D models.
The field of 3D scanning encompasses a wide range of applications, [NeRF](https://www.matthewtancik.com/nerf) makes a few of those applications a lot easier. The concept of using 3D scanning as an alternative means for people to interact with their surroundings is intriguing. For instance, museums could 3D scan their entire collections, or capture high-quality scans of monuments at risk of being lost forever. Additionally, education can be enhanced by introducing more active learning methods based on working in 3D. All of this is better served by how easy it is to produce models using a *more* software-defined method like NeRF.
Below you can see an example of a 3D guassian using the method "splatfacto", created by the engineers working on the [nerfstudio](https://docs.nerf.studio/) project inspired by the SIGGRAPH paper "[3D Gaussian Splatting for Real-Time Rendering of Radiance Fields](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)".
Though techniques such as NeRF, [MIP-NeRF](https://arxiv.org/abs/2103.13415), [3D Gaussians](https://arxiv.org/abs/2308.04079), and others demonstrate a fast and simpler method of 3D scanning compared to traditional laser or projection scanning. Neural network based scanning is still not considered a perfect "production ready" method, these models are made up of very dense point clouds that can not easily be translated into CAD. Supplimentary software is getting better but laser and projection scanning is still the best for high-accuracy work.
3D guassians have been a new focus of research, you can use an entirely software-defined approach to create very light (the below model is 6MB) and high quality 3D models. With [polygon-based](https://en.wikipedia.org/wiki/Polygonal_modeling) models you would have never been able to get this amount of accuracy in a 3D model that can run on a phone.
## Showcase
### 3D Gaussian Splatting
Below you can see an example of a 3D guassian using the method "Splatfacto", created by the engineers working on the [nerfstudio](https://docs.nerf.studio/) project inspired by the SIGGRAPH paper "[3D Gaussian Splatting for Real-Time Rendering of Radiance Fields](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)".
<head>
<link rel="stylesheet" href="styles.css">
@ -29,15 +50,25 @@ Below you can see an example of a 3D guassian using the method "splatfacto", cre
</div>
<button style="margin:0.5rem; padding:0.25rem; padding-left: 1rem; padding-right: 1rem;" class="not-prose hx-font-medium hx-cursor-pointer hx-px-6 hx-py-3 hx-rounded-lg hx-text-center hx-text-white hx-inline-block hx-bg-primary-600 hover:hx-bg-primary-700 focus:hx-outline-none focus:hx-ring-4 focus:hx-ring-primary-300 dark:hx-bg-primary-600 dark:hover:hx-bg-primary-700 dark:focus:hx-ring-primary-800 hx-transition-all hx-ease-in hx-duration-200" id="load-button">Load Scene</button>
<div style="display: flex;justify-content: space-between; padding: 0.5rem; padding-top: 0;">
<a class="gsplat-js" href="https://github.com/huggingface/gsplat.js/">gsplat.js</a>
<a style="color:#fff;text-decoration: none;" class="gsplat-js"> left click <strong>rotate</strong>, right click <strong>pan</strong>
</a>
<a href="https://github.com/huggingface/gsplat.js/">gsplat.js</a>
<p style="margin:0px;text-decoration: none;"> left click <strong>rotate</strong>, right click <strong>pan</strong>
</p>
</div>
</div>
Below are some pre-rendered scenes based on the [nerfacto](https://docs.nerf.studio/nerfology/methods/nerfacto.html) method also developed by nerfstudio. Though I have found that in some context the "splatting" method produces a smaller file and is more efficient to run, NeRF still provides pretty excellent quality.
### NeRF
{{< cards cols="2" >}}
<a class="hextra-card hx-group hx-flex hx-flex-col hx-justify-start hx-overflow-hidden hx-rounded-lg hx-border hx-border-gray-200 hx-text-current hx-no-underline dark:hx-shadow-none hover:hx-shadow-gray-100 dark:hover:hx-shadow-none hx-shadow-gray-100 active:hx-shadow-sm active:hx-shadow-gray-200 hx-transition-all hx-duration-200 hover:hx-border-gray-300 hx-bg-gray-100 hx-shadow dark:hx-border-neutral-700 dark:hx-bg-neutral-800 dark:hx-text-gray-50 hover:hx-shadow-lg dark:hover:hx-border-neutral-500 dark:hover:hx-bg-neutral-700"><video controls alt="Watch Toast at Cynosport Finals 2019" src="2024-05-31-17-55-11.mp4"></video><span class="hextra-card-icon hx-flex hx-font-semibold hx-items-start hx-gap-2 hx-p-4 hx-text-gray-700 hover:hx-text-gray-900 dark:hx-text-neutral-200 dark:hover:hx-text-neutral-50"></span></a>
<a class="hextra-card hx-group hx-flex hx-flex-col hx-justify-start hx-overflow-hidden hx-rounded-lg hx-border hx-border-gray-200 hx-text-current hx-no-underline dark:hx-shadow-none hover:hx-shadow-gray-100 dark:hover:hx-shadow-none hx-shadow-gray-100 active:hx-shadow-sm active:hx-shadow-gray-200 hx-transition-all hx-duration-200 hover:hx-border-gray-300 hx-bg-gray-100 hx-shadow dark:hx-border-neutral-700 dark:hx-bg-neutral-800 dark:hx-text-gray-50 hover:hx-shadow-lg dark:hover:hx-border-neutral-500 dark:hover:hx-bg-neutral-700"><video controls alt="Watch Toast at Cynosport Finals 2019" src="2024-05-31-17-55-14.mp4"></video><span class="hextra-card-icon hx-flex hx-font-semibold hx-items-start hx-gap-2 hx-p-4 hx-text-gray-700 hover:hx-text-gray-900 dark:hx-text-neutral-200 dark:hover:hx-text-neutral-50"></span></a>
{{< /cards >}}
Below are some pre-rendered photos based on the [Nerfacto](https://docs.nerf.studio/nerfology/methods/Nerfacto.html) method also developed by nerfstudio. Though I have found that in some context the "splatting" method produces a smaller file and is more efficient to run, NeRF still provides pretty excellent quality.
{{< cards cols="2">}}
{{< card title="Generated" subtitle="Nerfacto" link="4000-001.min.webp" image="4000-001.min.webp" >}}
{{< card title="Generated" subtitle="Nerfacto" link="4000-002.min.webp" image="4000-002.min.webp" >}}
{{< card title="Reference" subtitle="Sony A6400" link="4000-003.min.webp" image="4000-003.min.webp" >}}
{{< card title="Reference" subtitle="Sony A6400" link="4000-004.min.webp" image="4000-004.min.webp" >}}
{{< /cards >}}
## How-To
{{< callout type="info" >}}
**Hey!** This page is a work in progress. If you'd like to assist in the process of writing, take a look at the [git repository](https://git.myco.systems/mycosystems/midtowndrafting.com)
{{< /callout >}}

1
data/icons.yaml Normal file
View file

@ -0,0 +1 @@
nerf: <svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 48 48" style="enable-background:new 0 0 48 48;" xml:space="preserve"><path style="fill:#FFD600;" d="M7,10.971l-3,8c0,0,1.633,0.124,3.233,0.575c2.077,0.982,1.97,3.073,1.249,5.039C8.33,24.712,8.175,24.839,8,24.971c-2.813,2.125-4.688,9-4.688,9L11,38.252c0,0-0.834-9.281,3-9.281h0.011c1.39,1.159,2.989,2,2.989,2c0.09,0,0.342,0,1,0c2.001,0,3.739-0.781,4.823-2h15.735l0.545,1.633c0.272,0.817,1.036,1.367,1.897,1.367c1.105,0,2-0.895,2-2v-19H7z M14,25.971c0-1.657,1.791-3,4-3c2.209,0,4,1.343,4,3s-1.791,3-4,3C15.791,28.971,14,27.628,14,25.971z"/><path style="fill:#455A64;" d="M40,24.971H25v-10h15V24.971z M10,8.971v-1H8v3h13v-2H10z M9.836,39.883C10.342,40.2,11,39.835,11,39.238v-0.985l-7.688-4.281c0,0-0.123,0.467-0.282,1.128c-0.131,0.546,0.101,1.117,0.577,1.415L9.836,39.883z"/><path d="M40,18.971H28v-2h12V18.971z M40,20.971H28v2h12V20.971z"/><path style="fill:#FF6D00;" d="M43,19.971h2v-5h-2V19.971z M40,24.971h-2v-10h2V24.971z M18,25.971c0,1.103,0.508,2.096,1.321,2.819c-0.415,0.11-0.855,0.181-1.321,0.181c-2.209,0-4-1.343-4-3s1.791-3,4-3C18.508,23.875,18,24.868,18,25.971z"/><path style="fill:#90A4AE;" d="M22,10.971l-2,6H4.75l0.658-1.755l-1.041-0.347C3.551,14.596,3,13.832,3,12.971c0-1.105,0.895-2,2-2H22z M14,28.971c0,0-1.416-1.25-4-2c-3.371,1.908-4.225,6.279-4.547,8.192L11,38.252C11,38.252,10.583,28.971,14,28.971z"/></svg>