support for marigold #385

affromero · 2023-12-13T10:54:18Z

Discussion: #379
Issue: #383
I continued the implementation and this is an example:

Input	Leres No Boost	Marigold No Boost

	Leres Boosted	Marigold Boosted

affromero · 2023-12-13T11:56:34Z

Input	Leres Boosted	Marigold No Boost	Marigold boosted

graemeniedermayer · 2023-12-13T20:01:53Z

src/depthmap_generation.py


    # From Marigold repository run.py
    with torch.no_grad():
+        image = (image * 255).astype(np.uint8)


We should be able to simplify by removing
image = (image * 255).astype(np.uint8)
and turning line 431

rgb = np.transpose(image, (2, 0, 1)) # [H, W, rgb] -> [rgb, H, W] rgb_norm = rgb / 255.0

into
rgb_norm = np.transpose(image, (2, 0, 1)) # [H, W, rgb] -> [rgb, H, W]

That way we aren't multiplying and dividing 255.

The rationale behind that is that in Marigold they resize the image, and the input of that function is PIL.Image, so I just thought of having the same option. But yeah, maybe then multiplying and dividing in L427?

It does look cleaner to resize with a numpy array using something like
image = cv2.resize(image, (h, w), interpolation=cv2.INTER_CUBIC).

Although I'm not sure if there's any subtle differences between cv2 resize and PIL.Image resize.

Great changes overall!

I have no strong feeling either way.

semjon00 · 2023-12-13T21:45:51Z

Hello and thank you so much for this! I will test as soon as I have time. Meanwhile I will ask for some clarifications...

semjon00 · 2023-12-13T21:47:54Z

pix2pix/models/pix2pix4depth_model.py

@@ -94,8 +94,8 @@ def set_input_train(self, input):
        self.real_A = torch.cat((self.outer, self.inner), 1)

    def set_input(self, outer, inner):
-        inner = torch.from_numpy(inner).unsqueeze(0).unsqueeze(0)
-        outer = torch.from_numpy(outer).unsqueeze(0).unsqueeze(0)
+        inner = torch.from_numpy(inner).unsqueeze(0).unsqueeze(0).float()


Are you super-duper sure this does not break anything?

Not sure whether this break something to be honest. Are there tests/unittests I can run?
I modified this part because for some reason when selecting Marigold mode was raising an error because inner and outer were double tensors and the network weights are in float. Not sure why this is a particular error of marigold mode and not for the others.

semjon00 · 2023-12-13T21:54:51Z

Overall looks good. I almost did not believe that I stopped short of only 16 lines with my branch - let alone adding boost with the same 16 lines.

semjon00 · 2023-12-13T22:00:10Z

My only concern is that the depthmaps are inverted. In this script, white=near convention is used. I think the only thing to fix that would be adding Marigold (10) to line 298 in depthmap_generation.py: raw_prediction_invert = self.depth_model_type in [0, 7, 8, 9]. However, after changing, please test if it would actually work nicely, also that it will it generate sensible stereoimages, etc.

semjon00 · 2023-12-13T22:03:38Z

Also, would you be so kind to append README with Marigold mentions and an Acknowledgement? Before pushing Marigold support to main we will also need a version bump (misc.py) and a changelog (CHANGELOG.md) with one change.

graemeniedermayer · 2023-12-13T22:03:48Z

My only concern is that the depthmaps are inverted. In this script, white=near convention is used. I think the only thing to fix that would be adding Marigold (10) to line 298 in depthmap_generation.py: raw_prediction_invert = self.depth_model_type in [0, 7, 8, 9]. However, after changing, please test if it would actually work nicely, also that it will it generate sensible stereoimages, etc.

I have tested this. It works without boost.

Maybe Marigold should be git clone into "extensions/stable-diffusion-webui-depthmap-script/Marigold" rather than "repositories/Marigold" to avoid future conflicts (this is why midas is installed there). Also I believe diffusers is now a requirement.

semjon00 · 2023-12-13T22:08:05Z

@graemeniedermayer Indeed, it needs diffusers>=0.20.1, should be added to install.py. Thankfully, it seems to be a rather lightweight dependency. About conflits - I'm not sure about it... maybe?

graemeniedermayer · 2023-12-13T22:12:19Z

@graemeniedermayer Indeed, it needs diffusers>=0.20.1, should be added to install.py. Thankfully, it seems to be a rather lightweight dependency. About conflits - I'm not sure about it... maybe?

Also marigold standalone mode doesn't function with the current imports.

semjon00 · 2023-12-13T22:12:51Z

Oh, supporting standalone is definitely needed - a change to requirements.txt.

affromero · 2023-12-14T16:33:39Z

Hey folks, I added some of your suggested changes. I am not entirely sure if there was a protocol for the requirements/install files, so let me know what should I change. In particular, I use this project as standalone, so the install.py I am not sure about it.

graemeniedermayer · 2023-12-14T18:38:27Z

I think this should be merged onto the marigold branch. There might be some small changes/extra testing to do before merging into main.

Great work!

semjon00 · 2023-12-14T21:30:24Z

Merged and did some fixes. Now it would be awesome to support automatic repository pull for standalone. Yes, we can just copy all the files into the root again... But I'd rather not have any new code linked this way.

aulerius · 2023-12-14T21:32:29Z

This is really good news!

I also want do draw some attention to equivalent implementation in ComfyUI, and that it implements floating point EXR export:

I added a remap node to see the full range better, and OpenEXR node to save the full range, works wonders compared to default png when used in VFX/3D modeling software.

Which relates to #372 and #370
Could it be a nice opportunity to try that as well?

semjon00 · 2023-12-14T22:55:39Z

Not now, I am reeeaaally busy. Should be doing my homework rn, in fact.

semjon00 · 2023-12-14T22:58:32Z

@affromero @graemeniedermayer I can't get it to work! Please help... :(
I get things like
WARNING:xformers:WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.1.1+cu121 with CUDA 1201 (you have 2.1.1+cpu) Python 3.10.11 (you have 3.10.6) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details
For whatever reason trying to have XFORMERS automatically installs the wrong torch version

GPU: 1080Ti
OS: Windows 10

graemeniedermayer · 2023-12-14T23:11:02Z

Oh I just realized I tested it with a different install.py file maybe removing all the new requirements besides diffusers would be best to avoid conflicts. The others are very likely to cause conflicts with other repos. And I think the others are required by a1111

graemeniedermayer · 2023-12-14T23:32:52Z

I also want do draw some attention to equivalent implementation in ComfyUI, and that it implements floating point EXR export:

It shouldn't be too challenging to save the numpy arrays is there a library for converting to EXR?

aulerius · 2023-12-14T23:38:07Z

I also want do draw some attention to equivalent implementation in ComfyUI, and that it implements floating point EXR export:

It shouldn't be too challenging to save the numpy arrays is there a library for converting to EXR?

Apparently so, simply called "OpenEXR".
This is how it's done in the aforementioned ComfyUI implementation.
And TIF files also support floating point 32bit precision.

support for marigold

6384f23

graemeniedermayer reviewed Dec 13, 2023

View reviewed changes

semjon00 reviewed Dec 13, 2023

View reviewed changes

affromero added 7 commits December 14, 2023 17:25

Invert depth map

dd2d907

move mult/div to resize_to_max_res to leverage resize_max_res fn

b7ee6fd

repositories.Marigold -> Marigold

889ed86

bump version

9d11eed

Marigold ref to readme

1585a50

Changelog marigold

4e8a81a

Marigold requirements

548cd73

semjon00 merged commit 128fe5b into thygate:marigold Dec 14, 2023

support for marigold #385

support for marigold #385

Uh oh!

Conversation

affromero commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

affromero commented Dec 13, 2023

Uh oh!

graemeniedermayer Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

affromero Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

graemeniedermayer Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

semjon00 Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

semjon00 commented Dec 13, 2023

Uh oh!

semjon00 Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

affromero Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

semjon00 commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semjon00 commented Dec 13, 2023

Uh oh!

semjon00 commented Dec 13, 2023

Uh oh!

graemeniedermayer commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semjon00 commented Dec 13, 2023

Uh oh!

graemeniedermayer commented Dec 13, 2023

Uh oh!

semjon00 commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

affromero commented Dec 14, 2023

Uh oh!

graemeniedermayer commented Dec 14, 2023

Uh oh!

semjon00 commented Dec 14, 2023

Uh oh!

aulerius commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semjon00 commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semjon00 commented Dec 14, 2023

Uh oh!

graemeniedermayer commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graemeniedermayer commented Dec 14, 2023

Uh oh!

aulerius commented Dec 14, 2023

Uh oh!

Uh oh!

affromero commented Dec 13, 2023 •

edited

Loading

affromero Dec 14, 2023 •

edited

Loading

semjon00 commented Dec 13, 2023 •

edited

Loading

graemeniedermayer commented Dec 13, 2023 •

edited

Loading

semjon00 commented Dec 13, 2023 •

edited

Loading

aulerius commented Dec 14, 2023 •

edited

Loading

semjon00 commented Dec 14, 2023 •

edited

Loading

graemeniedermayer commented Dec 14, 2023 •

edited

Loading