Issue
Cube dimension processing on a 33GB SSAS database sometimes runs extremely slowly (possibly infinitely) while processing a 9GB dimension.
Environment
- SQL Server 2008 R2 SP1 Enterprise Edition 10.50.2500 (both SSAS and SSDS) running on Windows 2008 R2 Standard Edition.
- OS hosting SSAS has 32GB, with SSAS low/total memory limits set to 24GB/28GB. I bumped it up repeatedly to allow cube to process over the past few weeks.
- Dimensions are processed in parallel with ProcessFull.
- Large dimension has 28 million rows, estimated size of query results ~9GB. Dimension is set to build ByAttribute.
- Other databases are active on the SSAS instance. Some are able to build properly while the first cube is still processing slowly.
- I can reproduce this on SQL version 10.50.4266, although in the lab on a much smaller box with less memory.
Symptoms
When the slow processing occurs, I see:
- A SELECT DISTINCT … query for a large dimension running against the relational database. Wait time is ASYNC_NETWORK_IO.
- SSAS processing row reads/sec drops dramatically from 100,000 to about 200.
- SSAS memory usage creeps over the low memory threshold when dimension processing starts, then drops below threshold.
- SSAS memory cleaner kicks in just before processing slows down.
- SSAS profiler shows repeated Progress Report Begin/Current/End combinations with EventSubclass 19- GroupDataRecord and IntegerData of 0.
- This problem is sporadic. Some times the processing runs ok, even with memory cleaner kicking in; other times it stalls out when memory cleaner kicks in.
Workaround
So far, we’ve limped along by:
- Adding more memory. Last month we increased SSAS low memory from 12GB to 20GB. Last week I increased it to 24GB, then 28GB and still ran into the problem.
- Clearing other databases out of SSAS memory before processing begins.
- Optimizing the dimension to remove unneeded attributes.
- Increasing BufferMemoryLimit to 4GB.
- Processing in steps (ProcessData followed by ProcessIndexes).
These steps have helped, but I suspect we might run into the same problem again as data grows and other databases on the shared SSAS instance take up memory.
Goal
I'd like to know why Analysis Services drops down to GroupDataRecord events by such a small rowset (1 or 0 depending on where I look). I assume it has to do with memory pressure, but why does it pick such a low amount when there is more memory available after the cleaner kicks in? Seems like it over-reacts to memory pressure and gets stuck.
Thanks for your help.