October 25, 2020, 04:26:33 AM

News:

Due to SPAM attacks, new members must be approved before posting.  Please email jclough@warrenpinnacle.com when registering and your account will be approved.


Maximum number of cells

Started by kschmid, August 31, 2009, 02:50:04 PM

previous topic - next topic
Go Down

kschmid

I have heard that there is a limit on the number of cells that can be computed - what if any is the maximum number of cells in SLAMM at present?

Jonathan S. Clough

Several years back we moved SLAMM data management into computers' memories as opposed to their hard-drives.  The memory limitation then becomes the 32 bit windows 2GB per application.  Furthermore, SLAMM currently requires contiguous memory for processing large arrays.  We also performed some additional cell optimization in which we don't track as much information about open water and high-elevation cells.  We have found practically that the number of cells that we can model without causing a crash is roughly 50 million. 

However, we're banging our head against this memory limitation over and over again as input data is available in higher resolution.  For this reason, I'm currently writing a memory manager for 64 bit operating systems that far surpass the 2GB limitation.  The problem is, the development platform in which this software is written does not have a 64 bit implementation (Delphi 2009).  So we're either going to port the whole software to 64 bit or try to write a 64 bit memory management DLL to solve this problem.

You may also run the model using the hard-drive for memory management in which case this limitation is effectively removed. However, we have found this increases run time by about an order of magnitude.  One of my hard-drives crashed shortly after a 36 hour run as well making me think this option puts considerable stress on the drive...  Perhaps a flash drive of 10GB or so would be an option but we have not tried this yet.

-- Jonathan

kschmid

Thanks for the information - that helps us subset the work flow.

Jonathan S. Clough

Here's the latest on conversion to 64 bit...

Quote64 bit DLLs cannot be called from 32 bit applications.

This means we would be forced to port the entire piece of software to a 64 bit development platform to go forward in that manner

http://msdn.microsoft.com/en-us/magazine/cc300794.aspx

While running a fully 64-bit Windows system sounds great, the reality is that you'll very likely need to run Win32 code for a while. Towards that end, x64 versions of Windows include the WOW64 subsystem that lets Win32 and Win64 processes run side-by-side on the same system. However, loading your 32-bit DLL into a 64-bit process, or vice versa, isn't supported. (It's a good thing, trust me.) And you can finally kiss good bye to 16-bit legacy code!

In x64 versions of Windows, a process that starts from a 64-bit executable such as Explorer.exe can only load Win64 DLLs, while a process started from a 32-bit executable can only load Win32 DLLs. When a Win32 process makes a call to kernel modeĀ­to read a file, for instanceĀ­the WOW64 code intercepts the call silently and invokes the correct x64 equivalent code in its place.

Of course, processes of different lineages (32-bit versus 64-bit) need to be able to communicate with each other. Luckily, all the usual interprocess communication mechanisms that you know and love in Win32 also work in Win64, including shared memory, named pipes, and named synchronization objects.


However, I'm currently experimenting with RAM Disks in the 64 bit implementation to see if that might solve our problem with a minimum of consternation.  I doubt that accessing data in a RAM-Drive will be as fast as accessing memory registers but stay tuned...

Go Up