France
Joined: Nov 7, 2005
Post Count: 9426
Status:
Offline
Re: Suddenly, huge file size...
I may have found a new track: exploring the data in two files showing this bug, I discovered that the error could come from a material that references a texture image of an other 3D model in a home file. A simple way to check this issue in your sh3d file is to search an occurence of ".jpg", ".bmp" or ".png" in its ContentDigests entry. In a clean file, no occurence shouldn't be found. In a buggy sh3d file, the folder entry that contains a duplicate home will appear at the line where ".jpg", ".bmp" or ".png" is found. For example, if that line contains:
Name: 117/110/texture.jpg
the buggy folder is 117. Having such references isn't a blocking issue, since Sweet Home 3D is able to open those files. The problem comes from the saving routine that save such an image with all the home data from which it comes, instead of saving just the texture image itself. I could try to change this routine to avoid this behavior, but I would prefer to avoid such references. Unfortunately, I didn't find any way to produce such a reference and then a buggy file yet. I tried to copy/paste objects with modified materials from a file to another one, to paste style between such objects, to export/import such objects. Every time, the saved file was correct. Hope you can now help me to reproduce the bug more easily with this track in mind.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator
France
Joined: Nov 7, 2005
Post Count: 9426
Status:
Offline
Re: Suddenly, huge file size...
No need to search further. I found how to reproduce this bug and as it's actually easy enough to provoke it, I fear it could happen often enough Here's the simplest way to provoke it: - In a new file, add a piece that contains at least a texture - Save the file and quit Sweet Home 3D - Reopen the file and edit the materials of the piece it contains - Select the material that references a texture and click on the Texture radio button - Confirm and save. The sh3d file gets buggy and will contain the file hierarchy a second time. If you want to fix the error in your sh3d file by yourself, it's quite easy once you found the piece that uses its own texture in one of its materials: edit its materials and select the Unchanged radio button for this material in the Furniture materials dialog box.
Expect soon a beta version that will avoid this bug to happen again (but I don't think it will be able to fix existing files). It shouldn't be too difficult to fix
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator
USA
Joined: Mar 25, 2015
Post Count: 153
Status:
Offline
Re: Suddenly, huge file size...
Well, that description certainly applies to my current 'double-stuffed' project: ContentsDigests entry 61 contains a reference to texture.png -- and directory /61 contains the recursively saved house plan.
Is there a way to determine which object(s) contain the offending texture? In mentally retracing the steps leading to the corrupted save, I can't recall adding any texture-containing objects to the plan. (As best I can remember, the only new objects since the previous, non-corrupt save were light sources I copy/pasted from the save level.)
The only seeming constant I've found so far has had to do with time since last save -- which is partly why I asked about autosave. In every instance I can recall, a corrupted save occurred only when I'd been working on a unsaved plan for longer than, say, 20 minutes. Not all prolonged work times resulted in a bum save, but I've not yet saved a file open less than 15 - 20 minutes and had it double in size.
Also, in an earlier reply you said
The ContentDigests entry in a .sh3d file gives the SHA-1-Digest value of every entry of the .sh3d file used by Sweet Home 3D. This value is used to quickly check whether an entry is correct or not. So if you edit an entry in the .sh3d zipped file, Sweet Home 3D will think the file isn't correct anymore. I didn't program this feature to avoid users to manually edit an .sh3d file of course, but to check if an error on the disk didn't happen and inform the user accordingly. If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.
---which suggests to me that I should be able to save-and-compress a corrupted design, open the compressed .sh3d file in an archiver, delete the spurious directory, delete ContentDigests, and load the modified .sh3d file. When I try this, SH3D starts to load the plan but then pauses with a 'remove, replace, or halt' warning. If I respond with either remove or replace, the app simply stops the load. This occurs with the offending directory removed or merely left empty and with ContentsDigests deleted or merely edited to remove any reference to the directory. Am I missing something here? Being able to hack the plan back to size and not lose the time worked since the previous save would be of great benefit -- even if I had to reimport and position an object.
USA
Joined: Mar 25, 2015
Post Count: 153
Status:
Offline
Re: Suddenly, huge file size...
Ah, the joy of simultaneous entry! Feel free to ignore my previous reply.
I can easily understand why I occasionally trigger this bug: At times I want to apply a texture to an object to match an existing object. Rather than go to the trouble of hunting through my texture list, I've gotten into the habit of opening an object with the desired texture, going into materials, reselecting the existing texture, and saving the [unchanged] object. This adds the texture thumbnail to the quick-pick array across the bottom of the texture selecter, making it easy to apply the texture elsewhere. I use the same technique with colors, as well.
This also explains why the trouble only kicked in with a vengeance for me with v 5.0: After I installed the latest version and loaded your predefined texture libraries, it still left me without a number of textures I'd used under 4.x. Although I still had the original images, in many cases I didn't recall at what resolution they'd originally been imported. Rather than go through a trial-and-error process to identify the correct size mapping,* I discovered I could make even textures not present in the current libraries available through the above steps. Unsurprisingly, that's when the trouble started.
My recent month+ without save errors most likely reflects I've been working primarily with newly created objects (and, accordingly, standard or newly loaded textures).
Given this latest information, I now recall that one of the changes I made prior to my latest bum save was an attempt to address the undefined face problem with some Sketchup models: I modified the materials in a long-existing object in hopes of correcting a photo texture that would occasionally disappear depending upon its orientation in relation to the camera. The texture in question no longer exists in the 5.0 library, so I picked it up from the existing object definition. Since the object had been present in the plan since May, it never occurred to me it could have contributed to the problem -- and, in fact, I'd completely forgotten I'd changed it.
I'm currently not at home, and for some reason I can no longer run SH3D on my system here. As soon as I can, I'll see if I can duplicate the problem. (Do I understand your last reply to mean I can open my 1/2-gig plan, either delete the offending object *or* change its material definition to remove references to the texture, and re-save the file once again in its 250-meg glory?)
USA
Joined: Mar 25, 2015
Post Count: 153
Status:
Offline
Re: Suddenly, huge file size...
Oops. The asterisk in the previous entry should be footnoted as such:
* It would be nice if the dimensions at which a texture had been imported was visible in the texture selecter. It'd be even better if the dimensions were editable and thus could be changed without having to re-import the image.
USA
Joined: Mar 25, 2015
Post Count: 153
Status:
Offline
Re: Suddenly, huge file size...
FYI, I opened the 500+ Mbyte .sh3d file with the recursive save and replaced the furniture item containing a reference to a texture used in an object (that is, as opposed to textures accessed through the texture picker window) with a corrected item, and re-saved the file -- which suddenly shrank to 253 Mb.
So, yeah -- I think you've got this one figured out.
France
Joined: Nov 7, 2005
Post Count: 9426
Status:
Offline
Re: Suddenly, huge file size...
Hi,
I recently was in contact with a user of Sweet Home 3D who uses very often the list of recent textures in the textures choice dialog box to copy a texture from a material of an object to a different one. She used this tip so often with versions of Sweet Home 3D older than 5.2 that some of her files were larger than 50 GB! This is not a typo, you correctly read GB and not MB! Beside the amazing fact that Sweet Home 3D was still able to read and write such huge files (with a lot of patience), I couldn't let her continue to lose her time on this bug. There was no problem with brand new files made with version 5.2, but as soon as she copied/pasted some objects coming from sh3d files designed with older versions, there was a risk that the final file get uselessly huge. As she wasn't able to fix the .sh3d files by herself, I finally programmed a small tool named Huge File Cleaner that removes useless entries in her .sh3d files. She successfully tested this tool on different files (which are now always smaller than 100 MB), that's why I propose this tool here (7 KB - source code included) too, in case you encounter the same problem. The tool is a Jar executable file, so once downloaded, just double click on it to launch it if Java is installed on your computer. The tool will request you to select a .sh3d file to fix, will analyze the file, and if it founds a potential issue, it will request to select another .sh3d file where the fixed file will be saved (files must be different). Then, starts the operation that copies only necessary files from the buggy .sh3d to the destination file. At the end, just check the saved file in Sweet Home 3D. Note that the destination file is always compressed, so it may be much smaller just because the original file wasn't compressed.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator
