Working with REALLY large files - a better way?

[pseudodigm]

So, every couple of months I have to update a very large (about 16gig) photoshop file. I've pared it down to about 10 layers, but I can't collapse it anymore in order to maintain working with it in the future.

As you can imagine every little file operation takes minutes. It takes about 1.5 hours to open it and save it back out (I have to use a photoshop large image format).

Someone mentioned something to me that there's a program that will dynamically load JUST the parts that you are working on. But I'll still need at least some decent photoshop-ish functionality.

Any ideas?

Thanks,

Rick

[Silence04]

what are the specs of the machine your working on?

[pseudodigm]

Dual core, 4 gigs of ram, Windows XP, nothing special about the hard drive config.

I don't have any problems running anything else, and I do some pretty large print banners. This is just a monsterous file.

(counting the seconds until someone tells me to get a Mac...)

[Craig B]

Wow! a 16 Gb photoshop file?

Why so large?

[Virgo Nightingale]

Ooo, your ps file is sooo BIG... ;)

[pseudodigm]

Awww, don't get jealous! ; )

It's for a google map type application. The full sized map file is tens of thousands of pixels wide by tens of thousands of pixels high.

[jimking]

Get a Mac! :D

Kidding. 16 gigs is a huge file regardless. PrintDriver may have an idea for you. The only thought I came up with is using a program that will open PS files but I believe it is out of business. It's called "Live Picture." This program worked somewhat like Indy or Quark does in that it opens a proxy of the image to view only. You would then edit the graphic very quickly and then at the end you'd render it. Of course this is where it would start hogging time, but you would be able to open and edit quickly.

[pseudodigm]

Yeah, that's kind of what I heard. But he couldn't remember the name of the program. My luck, it'll be what you're talking about and is defunct. Hehe. Oh well, back to staring at my progress bar!

[Silence04]

You'll need more Ram to work on those files, try 8gigs (or even 16 if your motherboard can handle it).

You should also get another hard drive and make it the primary scratch disk for photoshop if you don't have it setup like that already, that way when your OS or any other application needs to access the startup disk it doesn't interrupt photoshop from using the hard drive.

The amount of Ram you have installed is the amount of data that can be opened at once on your machine. When photoshop opens a 16gig file, it's doesn't necessarily need all 16gigs then and there, but it will need a big portion of it (mainly the layers you are currently working on). As soon as the amount of 'open' data goes over the amount of ram you have installed, the excess data will be sent to your hard drive. And from that point, it becomes an endless "tug of war" between your Ram and Hard drive. That is what causes the machine to drastically slow down.

More ram will give your computer more room to work, and it won't be sending data back and forth as much.

[pseudodigm]

Well, it's even debatable that Win XP can even recognize the 4 Gigs of RAM that it has, being a 32 bit OS. I originally had 2 Gigs in it but asked for more when working with these maps last year.

I have the scratch disk setup on my secondary drive - which isn't the drive where the OS or Photoshop is installed, BUT it is the drive that the map is sitting on. That could be something to look at.

Now that you mention it I DO need to check out the memory settings in Photoshop, to make sure that it's taking enough of it. Although, I've worked on this map on a different machine that *was* set up right, but only made a bit of difference.

I think I'm just fated to watching the progress bar until a.) I can find another application that can handle the file better, b.) figure out a way to get the file size even smaller (which isn't really possible since the map geometry just keeps getting larger) with better use of the layers, or c.)maybe use illustrator or something. I kind of don't want a clean vector look, but it's worth looking into if this game geometry keeps getting bigger and bigger.

Thanks for the tips all!!

[PrintDriver]

All I have to say about 16 gigs is :
:eek:

However, is Photoshop the correct program for you? Seems if you are in the map business that large, you might have a different method of working on the image. Different map software. I'm not familiar with the softs, only the results.What is your output format? Web or print?

[pseudodigm]

That's kind of my question: is there an alternative to photoshop in this case.

I'm a graphic designer at a game company and this map represents about .01 percent of my annual duties. I only have to update it anytime there's an addition to the game geometry (which was probably only...4 times last year). BUT it's a pain in the old arse for about 2 solid days when it's update time. It got to the point where they put an additional computer on my desk so I could hit ctrl+c on the map, then switch back to whatever else I'm working on.

The map is, above and beyond, the largest, most unique image that I have to deal with. The next biggest thing is probably...10 foot by 10 foot banners. Though once, I did the "wraps" for 4 cars...that was kind of cool and big, but very low res.

The map's final output is for the web, but not as-is. It gets split into a few thousand png files that get accessed by a Google Maps application. It's pretty neat, actually.

It could be that it's just my cross to bear for those 8 days out of the year, but I thought I'd put a feeler out to see if someone else has similar issues.

Thanks for the help!

[urstwile]

Would Illustrator be a possibility?

[Virgo Nightingale]

Would it be easier to split the file into segments so you can work on one (smaller) section at a time?

[Silence04]

is the reason for having such a high resolution map because there will be a lot of zooming in and out?

[pseudodigm]

Would it be easier to split the file into segments so you can work on one (smaller) section at a time?
It's something I've tried in the past, and it definitely solves the problem of having such a large file open at once, though once it's all done you still have to composite the image and fix the seams. It wasn't really worth it.

is the reason for having such a high resolution map because there will be a lot of zooming in and out?

Yep, exactly. There are 7 different zoom levels.

Illustrator is definitely worth looking into for the surrounding bits. The game geometry itself is a tiff, but I could just link it and not embed it until the last moment. I *kind* of wanted to stay away from that because I wanted the map to look rough and...map like, and not so clean and vectory. I could achieve that look with textures and stuff, but I guess that would put me back where I started...well, no, not necessarily. If I was smart and made the textures tile nicely, it could work...I'll try a concept of that.

[urstwile]

I don't suppose you could post a screenshot, it might help us to give you better advice in terms of the approach.

[SurfPark]

I can't help but think that you'd be better with 7 smaller files. When you zoom out, you're losing resolution. When you zoom in you need higher resolution. Would it be possible to separate the zoomed-in maps and stitch them together later on?

[Riya]

Would it be possible to put each of the layers in it's own separate file and work on each individually. Then run a script to have photoshop open each of them in one document and save when you leave for the day?

[PrintDriver]

The Library of Congress uses Jpeg2000 compression on their interactive maps.
Here is a link example:
http://memory.loc.gov/cgi-bin/map_item.pl?data=/home/www/data/gmd/gmd9/g9110/g9110/ct001242.jp2&style=rochmap&itemLink=r?ammem/gmd:@filreq(@field(NUMBER+@band(g9110+ct001242))+@ field(COLLID+rochmap))&title=Carte%20g%26eacute;n%26eacute;rale%20de%20l% 27Oc%26eacute;an%20Atlantique%20ou%20Occidental,%2 0dress%26eacute;e%20au%20D%26eacute;p%26ocirc;t%20 g%26eacute;n%26eacute;ral%20des%20cartes,%20plans, %20et%20journaux%20de%20la%20marine,%20et%20publi% 26eacute;e%20par%20ordre%20du%20Ministre%20pour%20 le%20service%20des%20vaisseaux%20fran%26ccedil;ais %20en%201786.
Or they use the MrSID technology.
example here:
http://memory.loc.gov/cgi-bin/map_item.pl?data=/home/www/data/gmd/gmd376/g3762/g3762b/ar093800.sid&style=rochmap&itemLink=r?ammem/gmd:@filreq(@field(NUMBER+@band(g3762b+ar093800))+ @field(COLLID+rochmap))&title=Plan%20d%27une%20partie%20de%20la%20rade%20d e%20Boston%20pour%20faire%20connaitre%20le%20dispo sitif%20de%20ses%20deffenses.

I don't know anything about the programs used to create them. Only that the detail is incredible.

[Riefnu]

autoCAD!


What? why are you all looking at me like that?

[Cooper]

I can't help but think that you'd be better with 7 smaller files. When you zoom out, you're losing resolution. When you zoom in you need higher resolution. Would it be possible to separate the zoomed-in maps and stitch them together later on?

This. This is how Google does it. It isn't one huge Jpeg photograph of the whole world, it's mosaics of seperate square photographs taken at different zoom levels and stitched together in-browser. When you open google maps on a certain area, that area is made up of a series of squares, probably a grid about 4 squares by 6 or something, at a fixed resolution suitable for the level of zoom you are at. When you move N, E, S or W, it dumps the squares no longer visible on the grid and loads the newly visible squares onto the screen. When you zoom in, a whole new set of photographs is loaded taken at a larger zoom, it doesn't hold the images at 600dpi and then zoom in on them when the user requests it, it just loads new photographs taken at a tighter zoom.

It sounds like the image you are editing gets chopped up into smaller squares after your done working on it anyway, so I don't understand why you don't just work on it in seperate segments to begin with. It'll be easier for you to work on and, if you provide the images at multiple, fixed levels of zoom, it'll likely be faster for the end users to browse as well.

[БεRΓ]

If you can't split up the file, see if you can put together another PC with a 64-bit version of XP. That will get you around the 4GB RAM limit and you can load it up with as much as your motherboard will handle (provided your hardware can support 64-bit OSes).

[pseudodigm]

This. This is how Google does it...

It sounds like the image you are editing gets chopped up into smaller squares after your done working on it anyway, so I don't understand why you don't just work on it in seperate segments to begin with. It'll be easier for you to work on and, if you provide the images at multiple, fixed levels of zoom, it'll likely be faster for the end users to browse as well.

Yeah, that's exactly right.

The thing is, there are elements that span tiles and if I worked on individual tiles you would totally see the seams. See, it's this...weird mix of hand drawn elements and actual game landscape. It was decided that within the context of the website, the game geometry was too oddly shaped to have no background, and they really wanted you to show its relation to the entire world.

So I created this huge "hand drawn" looking map, and then I lay a masked game geometry map ontop of it. It's not perfect, by any means, but for the most part people aren't looking at the hand drawn bit, but the game bit. It's just there to kinda look nice at the zoomed out levels...which...actually could be an idea. I could create the hand drawn versions to look good at an optimal zoom level, and just blurry when zoomed in because you're probably not looking at it anyway.

I DO like the idea of creating the different files out of the different layers. There would be areas where different layers would relate to other layers and I'd have to do it in my head or something, but that could work. Then stitch them together with a script. Nice.

Actually, you can see an earlier version in action here: lorebook.lotro.com

I really want to restyle it, make it nicer looking, but it's SO tedious to work with it's hard to get into it. It's also very difficult to make a tiled texture not look so painfully tiled when you're looking at such a huge map at such a small size. I would have to repaint it which would take a while, but maybe worth it.

[Cooper]

Yeah, that's exactly right.

The thing is, there are elements that span tiles and if I worked on individual tiles you would totally see the seams. See, it's this...weird mix of hand drawn elements and actual game landscape. It was decided that within the context of the website, the game geometry was too oddly shaped to have no background, and they really wanted you to show its relation to the entire world.

Ah, I see the problem. I think it's one that can be solved via savvy workflow though. Perhaps have tiles adjacent to the edges of the one you are working on open simulataneously so you can tie the seems together neatly, then when you shift to the edges of the next tile, close the ones no longer adjacent and open the next three along, so effectively you'd have nine windows open in Photoshop at once, working across the adjacent edges (possibly even temporarily cut and paste all nine tiles into one large file to make smoothing the seams even better).

It'd be slightly more tedious than just working on one big file, but probably a lot less tedious than having to wait 1.5 hours for that one file to open. I can't imagine how logn each stroke of the brush takes either, I remember working on big files on old Macs and it'd take ages to process any alterations.

[Riya]

You could also possibly put your seams at the rivers.

[pseudodigm]

Ah, I see the problem. I think it's one that can be solved via savvy workflow though. Perhaps have tiles adjacent to the edges of the one you are working on open simulataneously so you can tie the seems together neatly, then when you shift to the edges of the next tile, close the ones no longer adjacent and open the next three along, so effectively you'd have nine windows open in Photoshop at once, working across the adjacent edges (possibly even temporarily cut and paste all nine tiles into one large file to make smoothing the seams even better).

It'd be slightly more tedious than just working on one big file, but probably a lot less tedious than having to wait 1.5 hours for that one file to open. I can't imagine how logn each stroke of the brush takes either, I remember working on big files on old Macs and it'd take ages to process any alterations.

That is a damned good idea! I'll try to come up with a plan for that and see if I can make that work. Thanks, everyone really and truely, for helping me and letting me bounce ideas around.

Oh man...Yeah, brush strokes. When the game geometry comes in, it's blocky around the edges, so I mask it and paint in a nice soft edge with an insanely large brush with soft edges. It's like *paint* *doodle* *paint* *do another drawing*...

Unfortunately, the seams are in very specific places which are constraints of the google map system already in place. Has something to do with the coordinate system or some such. I let the programmer do all that number stuff ; )

[Silence04]

you might want to try checking out how the Google Maps application works on the iPhone. It's different than than Google Maps online.

When zoomed out it uses a low res image of the map, and it loads in higher res sections as you're zooming in.