In the developer settings for Android you can use the memory profiler to find out what app is doing it. If you are running 15 you will need to enable it then give it a while to accumulate stats.
Discussion
That's a great tip! I had it enabled but it was at 3 hours so not so useful. Now I switched it to 1 day. Let's see. Android OS is at 4.5GB...