HUGE files, that still work, but defy streamlining (corrupt?)

Help and Troubleshooting
1

So I’ve run into this issue a lot over the years, and it’s getting easier to troubleshoot but is consistent:

Working in a file, it starts to bloat, becoming bigger and bigger, sometimes 2-3GB even. These files take forever to open, and even longer to save, and often crash the entire PC when saving (e.g. screen goes black, video card freaks out and becomes VGA res from 4K, etc.) I thought it was this PC, but it happens across users as well.
Once a file has this issue, it seems to be almost like a viral infection. I mean that a file made up of its parts, when saved out as a disparate part, does not reduce in size.

Case in point, this is the nuclear option and not hyperbole: I’ll have the file open, create a new model set that’s blank, do edit>add geometry>cube. I can then delete EVERY other model set, thus obliterating every material, I can gut every single camera, every image setting, every environment. This cube will have no material from the file, I’ll do ‘Save Active Model Set as’, and/or export as .bip, - file size is still 3GB. Makes no sense.

Any export of any kind still results in the same issue. Doing Help>Open (Recovery Mode) doesn’t provide any relief.

Even trying to import the geometry from one of these files into a blank file, creates a 3GB file.

Lo and behold - exporting only the geometry (e.g. a single part) as an OBJ, suddenly it’s only a few MB in a new save.
Also, saving out an FBX or GLB, 25mb.

Exporting every piece of geometry, and saving all materials from the file to the Library, and rebuilding the file in a new file… what was 3GB is now 100mb.

This seems to be the only way to fix these huge corrupt files, but it takes a LOT of manual labor, If anyone has a better fix, I’m all ears.

2

I’ve had this happen in the past as well where a bip file will randomly get corrupted, or crash at a certain point when its opening the file. Only way to fix was to create a new fresh project and import the corrupt project into the new one. Sometimes I would ahve to go through options of not importing anything but the model, sometimes I’d have to do the cameras and envronments seperately. Didn’t really tie to file size, but when the files were huge, of which I have some the same size as you. it would slow down remarkably. Wasn’t able to really figure out what triggered it but I do know that ditching Targa files that 3dsmax seems to love from our old old old projects did help tremendously.

3

I’m always interested in these kind of puzzles so if you want I can take a look. I’m aware that you don’t want your scenes all over the internet but they are safe with me or if it helps I can sign a NDA or so. Just let me know by PM if you want.

The only thing I can think of is that there are instances in the scene which are not treated as such so they get all separate objects. Images are not saved within the file if I’m right so it’s a bit of a miracle how big it got if it’s just geometry.

4

absolutely a mystery, we had thought that because we had some PSDs in labels it may have contributed, as even backplates in environments get saved in a KSP, but it wasn’t part of it. I can actually send you a file, it’s literally got a cube in it. that’s it. but it’s 3GB. LOL I’ll try to get it up on WeTransfer XD

1 Like
5

I didn’t know backplates got saved with the file, thought they were treated as other textures. I’m sure you can compress it a bit with 7-zip or zip :D, can’t be 3GB of actual info :wink:

6

The backplate images generally aren’t really the factor to bloating the size, but I find the art that is placed into texture labels for say a controller face, except we often have like 10-20 different muti-material variations to one product in one season, which usually end up making the files huge. The texture art we have embedded are generally rather high resolution as they are used for mass production and also permits us to render out extreme close macro shots of the product while maintaining fidelity to the applied artwork.

7

Just chiming in to say that this has been the bane of our existence on recent projects.

We will often merge in old files / lighting setups to new files and all of the issues described above persist, despite a landslide of trial and error. We have yet to find a solution, short of the FBX export / re-import trick, but can confirm that all of the “obvious” tricks don’t work (i.e., importing the scene into a fresh BIP, clearing out the scene and re-importing assets, etc.). On a previous project we were dealing with very large ABC sequences and also spotted a memory/RAM leak, where we could delete literally everything from a scene and it was still using 96GB+ of RAM… That issue seems to have been fixed in more recent releases, but this “phantom” file issue described above has been a constant nuisance and always hangs around the 40% mark while saving as well…

Really hoping the Luxion folks have seen this / working on a solution, it’s a massive productivity killer and not a small issue…

2 Likes
8

I hope you’ve also sent such files to Luxion since it’s sometimes really hard to replicate pretty vague problems and a file with those issues can help a lot to resolve/fix those bugs.

9

Definitely have in the past! I believe doing so helped them resolve the RAM leak. But in other instances, it’s often been met with radio silence if they’re unable to resolve after checking the file, and honestly until now, we thought we were the only ones experiencing these issues. Kind of relieved to know we’re not alone.

3 Likes
10

Sometimes it’s really hard I think to replicate things. I’m quite busy testing latest betas since I’ve some time currently and sometimes I’ve a problem but can be so hard to replicate. Think I’ve around 4-5 things that do appear now and then but I just can’t replicate/record them which makes it also very hard to resolve.

If you want you may send me a file with a PM, I bundle all my stuff in a document together with files to replicate bugs or videos showing the issue and send them off to Luxion every few weeks. A support ticket should work as well but it sounds like a serious issue so it won’t hurt to get some more attention.

I’ve never experienced the problem myself but if your file and the one from Joey are exported as fbx from the same application or f.e., as a certain FBX version, that could narrow things down maybe. I’ll try at least if I can get KS misbehave as well importing some fbx files.

11

Hey, I really appreciate that! I will try to clean out the file and save the “phantom” / empty version and pass along! (might be a day or two)

Sometimes we’re on such a tight timeline that the easiest solution is just to rebuild the file vs. wait several days for Support. I will say, 100% of the time it occurs when importing new CAD into old scenes (or the opposite). I’m not sure where to even start with trying to replicate the issue, but just like the file above, you can completely empty the file (down to something like a cube) and the issue persists so it seems like some kind of “phantom” / corrupt data that’s stored in the file. It’s SO similar to the memory leak we were experiencing last summer it almost feels like a remnant of that somehow, only RAM use isn’t the issue now, it’s framerate, file size and load/save times, regardless of what’s deleted from the file…

2 Likes
12

I hate to say it, but I’ve had the same situation when taking the time to send these large KSPs to luxion, radio silence and then automated ‘your ticket was closed’ emails. and same here that it’s mission critical so we just need to rebuild the file from original CAD, and then forget we had the issue and move on. The most heart breaking part of it all is that despite the file being incredible unstable and slow, we usually can’t ever batch render from these files, we’ll end up often just getting a freeze “keyshot has stopped responding” and lightened screen; and occasionally getting either an out of memory error or a GPU memory error.

13

So this file just had this happen tonight, I spent 4 hours trying to salvage it, and I give up. I just deleted EVERYTHING in the file, and left a cube from ‘edit>add geometry’, and added a single studio for it. it’s 1.15gb, and took 5 tries to save. enjoy: https://we.tl/t-4MBx5TA47Q

screenshot showing everything empty:

Screenshot%202023-05-19%20032618
Screenshot 2023-05-19 032618.jpg2678×1605 533 KB

1 Like
14

Well it’s definitely a pretty messy file. I dropped it into a hex-editor to see the contents but besides some folder and file names from locations at your computer it didn’t make much sense. I also see it eats a lot of memory away, around 43GB.

What format have those CAD files you import? STL, STEP? Not working with those a lot but I want to try if I can get the same problems while taking notes about the steps I follow. Hopefully that will help to be able to recreate the issue at Luxion so they can develop a fix.

From my limited knowledge I’ve the feeling it must be some import module issue which continues to add CAD data to the file in some kind of loop while the actual objects are already imported. That data you basically don’t see gets also saved in to the file if it didn’t crash beforehand.

You could maybe prevent it by converting the CAD files you import to f.e. an OBJ/FBX which might be not ideal because you can’t re-tessellate it, and it’s an extra step, but it might save you time if it does work with the batch rendering (until this is fixed).

15

We’ve tried STP, OBJ, and mostly the Keyshot plugin from Fusion360 which is where many of our models come from. Regardless of the source file type, it keeps happening.
Makes sense that it could be something about importing causing, it. what doesn’t make sense is why/how Keyhsot would retain any of that data if the model sets are deleted.

16

Updating as this thread was mentioned in another post about this issue someone else had as well - the recurring theme seems to be importation of .bip files into each other.
This is a horrifying discovery because we’ve set up things like camera sets we like to import into other files to re-use. What seems to be happening, is that despite unchecking ‘Geometry’, this phantom data or phantom geometry persists and gets re-imported. Something also I suspect may be happening is how if you’ve ever tried to import geometry and forget to select ‘New Model Set’, Keyshot defaults to importing geometry into every active model set - so if you’re working on a scene with 7 model sets turned on, you’ll get 7 copies of the geometry (fun!) I suspect this is happening behind the scenes, and the keyshot file ends up with several copies of geometry that we can’t see or delete. #wildspeculation

Here’s the standardized fix we’ve started doing:

  1. Create a folder in the Library for the project
  2. Save each important material to this folder naming it in a way you can remember later
  3. Save important color swatches in your color picker (+)
  4. Retrieve original CAD from your source (best results from using a Keyshot plugin in the modeling app) or export your CAD as something other than .bip (e.g. obj)
  5. Create a whole new file, import your CAD
  6. Re-apply all of your materials from your library folder

Bonus points:
Keyshot has a built-in ‘Material Templates’ function that so long as your model part names remain the same, it can be used to match up materials to parts from your previous file, I haven’t gotten this to work yet because it requires every part to have a unique name either set in your CAD app - and/or matched up after the fact, and it gets fubar’d if you ever rename any geometry in keyshot itself because your source CAD will be different; so if you’re re-importing the same exported CAD, theoretically it should work. This might help some work faster: https://www.youtube.com/watch?v=ViVxWFRQV34

2 Likes
17

That’s really interesting information, thanks for sharing. Also your #wildspeculation sounds sensible since I rarely use model sets but actually import bip files quite often. That could explain why I haven’t had the issue. Currently working on a model with a few model sets so I’ll try to force it into the behaviour you mentioned (after creating some backup files).

18

@joey.lopez

Today I was trying to get the same problems as you had. And there for sure some weird things in how the file can grow but until now I also still got it down to the original size. Which, like you said, didn’t work if you update the model with the .bip but does work if I loaded the original .fbx for some reason.

I’m a bit puzzled why adding model sets actually seem to increase the file size. It doesn’t when you just add a model set, but if you show/hide different parts of that extra model set the file size gets doubled.

In the manual it says a model set is a scene tree variation which in my opinion is always the same model but in a different state with layers hidden or shown or parts moved around. No reason for a file to double the size. It’s also not that consistent and sometimes after it doubles, reimporting the fbx and updating the model does reduce the size again. Odd behaviour.

testing
testing.png1080×946 39.5 KB

1 Like
19

Interesting! You’ve gotten further in replicating the behavior than I have, it seems to be random but interesting to see that there’s some kind of possible pattern!

I recall that certain actions in Keyshot create instances, and others create entire duplicates.
According to KS documentation https://manual.keyshot.com/manual/models-tab/duplicating-models-parts/ duplicating a model or part creates an instance that corner rounding, then applies to all copies; Similarly the Pattern tool also creates instances. I wonder if at what point the link is broken when performing ‘new model set’ and what triggers the break in instancing (what is baking the model duplicate out?) Is it a bug? a predictable behavior? what should we NOT do to avoid this doubling issue? and how can we fix it after the fact if deleting doesn’t reduce the file size? very strange!

1 Like
20

I’ve the same thoughts/questions, I also requested a feature so in the item list it’s clear weather it’s an instance of an object or copy without actual link. In MODO which I use those kind of instances get their name in italic so you have more grip on what actually impacts the model/size and how it will behave.

In KeyShot you can do a lot which ‘breaks’ that connection so even though you created something using the pattern tool they are suddenly not actual instances anymore. In my opinion that’s a bit confusing especially since you can’t see the ‘role/type’ of an object so you only get to know if you for example put a material on one item and see the rest doesn’t change.

In MODO for example there is no way to change an instance of something other than to change it’s actual source. You can convert instances to meshes if you want to break that link but than it name would also changed from italic to regular.

I feel that it’s important to know more of how things get handled internally by KeyShot or just make it more clear in the interface. Model Sets can be really handy but if they create files which get extremely large it can be problematic.

Maybe at PDP there is someone who knows Python well enough because I think it’s possible to check the model for actual instances.

My findings go to KeyShot anyway but I’m still not really satisfied because until now I never ended up with a file like you had, zero geometry but 2GB of file size.

1 Like