Hyperspectral Neural Radiance Fields, Supplementary Videos/Images

Anonymous 3DV 2025 Submission
Paper ID #393,

3DV 2025

Supplementary

HS-NeRF extends NeRF to hyperspectral data to address the 3D reconstruction challenges inherent to hyperspectral data. We demonstrate applications such as hyperspectral super-resolution and camera sensor simulation, with potential future applications in non-destructively estimating 3D distributions of material compositions for agriculture, mining, medicine, and more.

Note: We have made our best effort to package all the most helpful/representative images&videos in the zip file, but some less critical images&videos were too large to include in the zip file so we have uploaded them to github (timestamped & version controlled) and serve the assets via github links here.
Should you wish to review without consideration of internet-downloaded images&videos, please feel free to turn off your internet and be aware that some videos may not display.

Tip: These are collapsible sections.
Please click on the headings to expand/collapse.

Raw Dataset Images

Here are some examples of raw images taken by the hyperspectral cameras.

BaySpec Camera Datasets

Anacampseros

Caladium

Pinecone

Surface Optics Camera Datasets

Rosemary

Basil

Tools

Origami

RGB Baseline Comparisons

Our HS-NeRF approach (Ours-Hyper) using 128 channels clearly outperforms NeRFs using only RGB

Sample ground truth image

nerfacto
(baseline)

Ours-RGB
(discrete 3ch radiance & transmittance)

Ours-Cont
(continuous 3ch radiance & transmittance)

Ours-Hyper
(HS-NeRF, continuous 128ch)

Depth Maps

Depth maps can reveal issues with the 3D structures of NeRFs that are not immediately apparent in the rendered images.
Qualitatively, ours is noticeably better than the rgb baselines, and is among the best of the ablations.

RGB

Comparing the depth maps against RGB NeRF baselines, we find that our hyperspectral NeRF produces much better results.

nerfacto
(baseline)

Ours-RGB
(discrete 3ch radiance & transmittance)

Ours-Cont
(continuous 3ch radiance & transmittance)

Ours-Hyper
(HS-NeRF, continuous 128ch)

Ablations

Comparing the depth maps of ablations, ours and

\begin{array}{rcl} C_{1} σ_{0} P_{0} \end{array}

are consistent and high quality, with the fewest spurious discontinuities in foreground surfaces.

\begin{array}{rcl} C_{1} σ_{1} P_{0} \end{array}

is notably strong for the Anacampseros scene, but finds a completely degenerate solution for both the Caladium and Pinecone scenes, likely due to low contrast between the foreground and background color (black).

\begin{array}{rcl} C_{1} \\ σ_{0} \\ P_{0} \end{array}

\begin{array}{rcl} C_{1} \\ σ_{1} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ_{0} \\ P_{0} \end{array}

(ours)

\begin{array}{rcl} C \\ σ \\ P_{0} \end{array}

\begin{array}{rcl} C_{2} \\ σ_{2} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ \\ P_{λ} \end{array}

Point Clouds

Finally, we can also export the NeRFs to point clouds to verify that their 3D structure is reasonable. Shown below are screenshots of the point clouds for Anacampseros (left) and Caladium (right)

Ablations

Qualitatively, our stated approach appears among the best, but slightly different architectures are not significantly worse.

Careful observation reveals that the networks using

\begin{array}{rcl} C \end{array}

(including ours) tend to be the most accurate in terms of absolute intensity (see for example that the other networks tend to over-estimate the leaf brightnesses in the middle row). Meanwhile, the network using

\begin{array}{rcl} C_{2} \end{array}

often appears fuzzy, which suggests that mixing together the spatial and wavelength information is not as efficient. Finally, although the networks uing

\begin{array}{rcl} C_{1} \end{array}

generally appear slightly sharper than ours, they do not have a continuous wavelength representation so they do not as naturally interpolate wavelengths and, as mentioned, they tend to have slightly less accurate absolute intensities (e.g. on the middle row's leaf brightness).

Fixed camera viewpoint not in training set (wavelength sweep)

This video is the easiest for comparing the performance between different methods.

In this video, t = 0s depicts the shortest wavelength (image channel 0) and we depict progressively longer wavelengths until the end (t = 5s) depicts the longest wavelength (image channel 127). This camera view was not in the training set.

GROUND TRUTH

$\begin{array}{rcl} C_{1} \\ σ_{0} \\ P_{0} \end{array}$

\begin{array}{rcl} C_{1} \\ σ_{1} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ_{0} \\ P_{0} \end{array}

(ours)

\begin{array}{rcl} C \\ σ \\ P_{0} \end{array}

\begin{array}{rcl} C_{2} \\ σ_{2} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ \\ P_{λ} \end{array}

Moving camera viewpoints (6 wavelengths)

This video shows the camera moving to evidence that the NeRF's performance is consistently good across novel viewpoints.

Select which scene to view:

Anacampseros
Caladium
Pinecone

This grid of videos shows different ablation variations as columns and different wavelengths as rows.

$\begin{array}{rcl} C_{1} \\ σ_{0} \\ P_{0} \end{array}$

\begin{array}{rcl} C_{1} \\ σ_{1} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ_{0} \\ P_{0} \end{array}

(ours)

\begin{array}{rcl} C \\ σ \\ P_{0} \end{array}

\begin{array}{rcl} C_{2} \\ σ_{2} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ \\ P_{λ} \end{array}

Ch 15 (477nm)

Ch 35 (576nm)

Ch 55 (675nm)

Ch 75 (772nm)

Ch 95 (869nm)

Ch 105 (918nm)

Moving camera viewpoints (wavelength sweep)

This video shows the camera moving simultaneously while the wavelength sweeps to evidence that the NeRF's performance is consistently good across novel viewpoints over all wavelengths.

In this video, t = 0s depicts the shortest wavelength (image channel 0) and we depict progressively longer wavelengths until the end (t = 5s) depicts the longest wavelength (image channel 127). This camera view was not in the training set.

$\begin{array}{rcl} C_{1} \\ σ_{0} \\ P_{0} \end{array}$

\begin{array}{rcl} C_{1} \\ σ_{1} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ_{0} \\ P_{0} \end{array}

(ours)

\begin{array}{rcl} C \\ σ \\ P_{0} \end{array}

\begin{array}{rcl} C_{2} \\ σ_{2} \\ P_{0} \end{array}

\begin{array}{rcl} C \\ σ \\ P_{λ} \end{array}

Super-resolution (Wavelength Interpolation)

Our NeRF loses almost no accuracy in predicting the full hyperspectral image, even when trained on only 1/8th of the wavelengths.

Videos: Sweep hyperspectral wavelengths [+]

Our NeRFs can accurately predict an unseen viewpoint consistently across all wavelengths.
Furthermore, even withholding 87.5% of wavelengths from the training set has marginal impact on accuracy.

GROUND TRUTH

Trained on every wavelength

Trained on every 2nd wavelength

Trained on every 4th wavelength

Trained on every 8th wavelength

Videos: Sweep hyperspectral wavelengths & Moving Camera [+]

Watch this rotating video to see that the wavelength interpolation performance is consistently good across novel viewpoints.

Trained on every wavelength

Trained on every 2nd wavelength

Trained on every 4th wavelength

Trained on every 8th wavelength

Image Grid: Sweep hyperspectral wavelengths [+]

Inspect the individual rendered images by wavelength and amount of interpolation.
This is the same as the video above, but in image form to allow for closer inspection.

Currently only every 11th wavelength is loaded.

Select which scene to view:

Anacampseros
Caladium
Pinecone

Key:

Wavelength in train set

Wavelength NOT in train set

Ground Truth

Trained on every wavelength

Trained on every 2nd wavelength

Trained on every 4th wavelength

Trained on every 8th wavelength

Camera Image Sensor Simulation

Hyperspectral NeRFs allow us to simulate arbitrary camera image sensors from a single reference image.

Different camera imaging sensors have different spectral responses which produce unique image qualities. Similarly, filters are often used by photographers to achieve different artistic effects. From a hyperspectral image, we theoretically have enough information to reconstruct what the image would look like if it were taken with a different camera sensor or filter.

We demonstrate results using the following procedure:

Construct a hyperspectral NeRF using HS-NeRF
Take a single photo with the target camera
Localize the target photo's camera pose in the NeRF scene
Render a hyperspectral image for that camera pose
Run a least-squares optimization to find spectral response functions which result in a simulated photo that best matches the target photo
Render the simulated photo

For each scene, we present simulated photos for an iPhone photo with various digital filters applied.
A number of different target photos (left of each pair) can be simulated by HS-NeRF (right of each pair) with excellent agreement.
Note that the agreement often appears so good/consistent that it seems as though a filter was applied to both target and simulation together, but the filters were only applied to the target photos, then the simulations were generated according to the above procedure.