Magento 2.4.1 Media Gallery....not ready for prime time. Fails when you have large amounts of data (images)

[perfpcs]

Preconditions (*)

The new media gallery introduced in Magento 2.4.1 certainly was not vetted using a large robust shop. Maybe it was designed with 100-200 products like the using Magento sample data but I can tell you from my experience trying to use this, it does NOT work when you have a large amount images. Since we use compression software to have images ready for desktop, table, phone and WEBP use. We have generated over a half a million images for our shop. What happens is that the images will load but the directory tree fails to generate due to sessions being destroyed and memory exhaustion. Even setting php.ini to basically unlimited use of ram and setting our instance to 192 GB physical ram, the media gallery still fails to load the directory tree. This is very disappointing! What I don't understand either is why would the media galley be set with /media as the storage root? This is the main problem as 95% of the images in the media gallery are product images. Most shops don't use product images to create banners or make CMS blocks or pages. The correct root for media gallery in my opinion should have been the /wysiwyg directory. That is what is used when dealing with media and the CMS block and pages creation. If this was set as root then these issues with session destruction and memory failures would never happen. Or, the media gallery should exempt the catalog/product images like it is in the old media gallery. If you go to those catalog/product folders you will see there are no images. That is another correct way this gallery should have been setup, in my humble opinion.

How to produce error (*)

1. Install M 2.4.1
2. Move your media content from your current or old website that has 500,000 or more images
3. Go to content/media galley
4. Media gallery will open with images but directory tree is missing. You can't find folders or see any folders or even see new ones if you make any. Note the folders are created but because the tree is not there, you cannot see the folders. Also it is random whether the search, sort or other fields show up. Sometimes they do and sometimes they dont.

Steps to reproduce (*)

1. Install M 2.4.1
2. Move your media content from your current or old website that has 500,000 or more images
3. Go to content/media galley
4. Open Media gallery

Expected result (*)

1. I expect that the directory tree, search, sort and full media gallery opens, not a crippled version of the media gallery that does not even work.

see various images. It should look like this:

Actual result (*)

1. Many errors in php log:

[19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 8192 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Driver/File.php on line 962
[19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 135168 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Session/SessionManager.php on line 151
[19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php on line 793
[19-Oct-2020 04:26:51 UTC] PHP Fatal error: Uncaught RuntimeException: Unknown: Cannot call session save handler in a recursive manner in [no active file]:0[19-Oct-2020 08:29:18 UTC] PHP Warning: DOMXPath::query(): Invalid expression in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 494
[19-Oct-2020 08:29:18 UTC] PHP Warning: DOMXPath::query(): Invalid expression in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 494
[19-Oct-2020 08:29:18 UTC] PHP Warning: Invalid argument supplied for foreach() in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 503
[19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 8192 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Driver/File.php on line 962
[19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 40960 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Session/SessionManager.php on line 151
[19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php on line 793
[19-Oct-2020 09:06:41 UTC] PHP Fatal error: Uncaught RuntimeException: Unknown: Cannot call session save handler in a recursive manner in [no active file]:0[19-Oct-2020 09:54:47 UTC] PHP Fatal error: Uncaught RuntimeException: ObjectManager isn't initialized in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManager.php:36
Stack trace:
#0 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Directory/Read.php(78): Magento\Framework\App\ObjectManager::getInstance()
#1 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Directory/Read.php(212): Magento\Framework\Filesystem\Directory\Read->validatePath()
#2 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Code/GeneratedFiles.php(123): Magento\Framework\Filesystem\Directory\Read->isExist()
#3 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Code/GeneratedFiles.php(98): Magento\Framework\Code\GeneratedFiles->isCleanGeneratedFilesAllowed()
#4 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManagerFactory.php(114): Magento\Framework\Code\GeneratedFiles->cleanGeneratedFiles()
#5 /home/ in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManager.php on line 36

2. Screenshots
   
   

Things we tried and results:

1. Our Host of AWS Cloud server adjusted these settings in this order:
   a. From the error shown you have reached the limit of 768M that was set on the php config file.
   We have increased it and set to memory_limit = 1024M
   b. The php-fpm of cli had memory_limit = 4048M. We have set this to unilimited
   c. So we have increased the memory limit to :8092M on file ".user.ini" on Magento root & inside pub/ location.
   mgt@performance-pcs:/home/cloudpanel/htdocs/www.performance-pcs.com/pub$ cat .user.ini
   memory_limit = 8092M
2. How we fixed: Well we didn't fix actually but we proved to ourselves that it was the number of images that caused the problem. So we moved all our images out of media except for about 200 or so. And vola! The media gallery opened fine with the proper tree structure, search and sort fields and all functions worked.

So now what?? We need to access all our NON-PRODUCT IMAGES!. I can turn back on old media gallery and it does work but I would like to use the new one with the Adobe image features, that is handy!

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [X ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

Additional info from Engcom

To reproduce the issue

1. Execute this file to generate empty txt files in media folder https://gist.github.com/Nazar65/61c756ce76afa8c6718bf670794d548c execute in CMD bash glob.sh
2. Set memory_limit in php to 100MB
3. Open Media Galley

Actual Result: Media gallery directory tree component renders directory tree component

[m2-assistant]

Hi @perfpcs. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[m2-assistant]

Hi @engcom-Bravo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[Nazar65]

@magento I am working on this