Basic Map Caching Question

Topics: Help
Dec 28, 2011 at 8:19 PM

First I just want to thank you radioman for this great control.  I work for a small development group in the FAA and since we’ve found your control earlier this year, we’ve utilized it in several of our applications.

Our current application requires us to allow the user to pre-fetch tiles while online and then use cached tiles while offline.  So they will be presented with a menu where we allow them to select an area to pre-fetch and the desired zoom levels.  I know this is possible after looking at the examples in your code.  After some experimentation I have some questions.

1. We want the tile database to be transparent to the user and to never be size limited.  If it grows to 1 TB that is perfectly fine.  So anytime we ever pre-fetch tiles we want them to go into the database.  And we never want to remove any previously fetched tiles.  Is this possible?  Are there any flags or properties I need to set for this behavior?
2. What is the best built-in method for pre-fetching tiles to achieve the above desired behavior?  I checked out GMap.NET.WindowsForms.TilePrefetcher.cs and saw you have a CacheTiles method.  Is this the best model to follow to achieve the above desired behavior?
3. Is there anyway you can recommend for threading the pre-fetch process?  Or is that going to be a bad idea considering we are writing to the database and don’t want concurrent writes?

Any help is appreciated.  Thanks!

Dec 28, 2011 at 8:20 PM

To clarify #3: I mean creating multiple download threads at once to speed the process.

Coordinator
Jan 9, 2012 at 11:48 AM
  1. it's done automatically, there is theoretical cache size limit, ~60TB
  2. tileprefetcher use one thread, if you use more, you may be blocked by tile provider, it can block you even if you requesting to fast ;}
Jan 9, 2012 at 11:59 AM

1. thats perfect for what i need
2. i am only using one thread with a sleep of 100 ms between tiles.  i haven't been blocked yet (for anyone curious that is against the Bing servers).

i have it written and working as i need it.  i did re-write your tileprefetcher.cs so that i could pass it multiple tile providers and a min/max zoom level.  so one call to begin pre-fetching grabs all the desired zoom levels from all the requested tile providers.  for anyone interested here is my code:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Windows.Forms;
using GMap.NET;
using GMap.NET.Internals;
using GMap.NET.MapProviders;

namespace FAA.Spectrum.DirectionFinder.Forms
{
    public partial class PrefetchTiles : Form
    {
        BackgroundWorker worker = new BackgroundWorker();
        public bool ShowCompleteMessage = false;
        
        public PrefetchTiles()
        {
            InitializeComponent();

            worker.WorkerReportsProgress = true;
            worker.WorkerSupportsCancellation = true;
            worker.ProgressChanged += new ProgressChangedEventHandler(worker_ProgressChanged);
            worker.DoWork += new DoWorkEventHandler(worker_DoWork);
            worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);
        }

        #region Form Loader/Unloader
        public void Start(RectLatLng area, int minZoom, int maxZoom, GMapProvider[] providers, int sleep, string cacheLocation)
        {
            if (!worker.IsBusy)
            {
                this.label1.Text = "Please wait: Creating task list...";
                this.label2.Text = "Calculating Remaining Time";
                this.label3.Text = "Saving maps to cache database located at:\n" + cacheLocation;
                this.progressBar1.Value = 0;
                
                WorkerArgs args = new WorkerArgs();
                args.MinZoom = minZoom;
                args.MaxZoom = maxZoom;
                args.Area = area;
                args.Providers = providers;
                args.TileSleeper = sleep;
                
                GMaps.Instance.UseMemoryCache = false;
                GMaps.Instance.CacheOnIdleRead = false;
                
                worker.RunWorkerAsync(args);

                this.ShowDialog();
            }
        }

        public void Stop()
        {
            if (worker.IsBusy)
                worker.CancelAsync();
        }
        #endregion

        #region Background Worker
        void worker_DoWork(object sender, DoWorkEventArgs e)
        {
            BackgroundWorker worker = (BackgroundWorker)sender;
            WorkerArgs args = (WorkerArgs)e.Argument;
            List<GPoint> list = new List<GPoint>();

            ProgressObject.Reset();
            for (int z = args.MinZoom; z <= args.MaxZoom; z++)
                foreach (GMapProvider provider in args.Providers)
                    ProgressObject.TotalTiles += (ulong)provider.Projection.GetAreaTileList(args.Area, z, 0).Count;
            
            ProgressObject.StartTimer();
            for (int z = args.MinZoom; z <= args.MaxZoom; z++)
            {
                if (worker.CancellationPending)
                    break;

                foreach (GMapProvider provider in args.Providers)
                {
                    if (worker.CancellationPending)
                        break;

                    if (list != null)
                    {
                        list.Clear();
                        list = null;
                    }

                    // Get the tiles we need to download
                    list = provider.Projection.GetAreaTileList(args.Area, z, 0);
                    Shuffle<GPoint>(list);

                    int numfiles = list.Count;

                    // Download all tiles in the list
                    int retry = 0;
                    for (int i = 0; i < numfiles; i++)
                    {
                        if (worker.CancellationPending)
                            break;

                        // Download the tile
                        // Retry if there is a failure
                        if (CacheTiles(z, list[i], provider))
                            retry = 0;
                        else
                            if (++retry <= 1)
                            {
                                i--;
                                System.Threading.Thread.Sleep(1111);
                                continue;
                            }
                            else
                                retry = 0;

                        ProgressObject.AddFileCompleted();

                        worker.ReportProgress(0, null);
                        System.Threading.Thread.Sleep(args.TileSleeper);
                    }
                }
            }

            if (worker.CancellationPending)
                e.Cancel = true;
        }

        void worker_ProgressChanged(object sender, ProgressChangedEventArgs e)
        {
            this.label1.Text = "Overall progress: " + ProgressObject.OverallCompleted + " of " + ProgressObject.TotalTiles + " files (" + ProgressObject.OverallProgress + "%)";
            
            if (ProgressObject.OverallProgress == 0)
                this.label2.Text = "Calculating Remaining Time";
            else
                this.label2.Text = ProgressObject.EstimatedCompletionTime.ToString("d\\ \\D\\a\\y\\s\\,\\ h\\ \\H\\o\\u\\r\\s\\,\\ m\\ \\M\\i\\n\\s") + " Remaining";
            
            this.progressBar1.Value = ProgressObject.OverallProgress;
        }
        
        void worker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            if (ShowCompleteMessage)
            {
                if (!e.Cancelled)
                    MessageBox.Show("Prefetch Complete!");
                else
                    MessageBox.Show("Prefetch Canceled!\n\nOverall progress: " + ProgressObject.OverallCompleted + " of " + ProgressObject.TotalTiles + " files (" + ProgressObject.OverallProgress + "%)");
            }

            GMaps.Instance.UseMemoryCache = true;
            GMaps.Instance.CacheOnIdleRead = true;

            this.Close();
        }
        #endregion

        #region Standalone Methods
        bool CacheTiles(int zoom, GPoint p, GMapProvider provider)
        {
            foreach (var pr in provider.Overlays)
            {
                Exception ex;
                PureImage img;

                img = GMaps.Instance.GetImageFrom(pr, p, zoom, out ex);

                if (img != null)
                {
                    img.Dispose();
                    img = null;
                }
                else
                {
                    return false;
                }
            }
            return true;
        }

        void Shuffle<T>(List<T> deck)
        {
            int N = deck.Count;
            Random random = new System.Random(0);

            for (int i = 0; i < N; ++i)
            {
                int r = i + (int)(random.Next(N - i));
                T t = deck[r];
                deck[r] = deck[i];
                deck[i] = t;
            }
        }
        #endregion

        #region Form Event Handlers
        private void Prefetch_PreviewKeyDown(object sender, PreviewKeyDownEventArgs e)
        {
            if (e.KeyCode == Keys.Escape)
            {
                this.Close();
            }
        }

        private void Prefetch_FormClosed(object sender, FormClosedEventArgs e)
        {
            this.Stop();
        }
        #endregion

    }

    #region Helper Classes
    internal class ProgressObject
    {
        static public ulong TotalTiles = 0;
        static public ulong OverallCompleted = 0;
        static public int OverallProgress = 0;
        static private DateTime startTime = new DateTime();
        static public TimeSpan EstimatedCompletionTime = new TimeSpan(0);

        static public void AddFileCompleted()
        {
            OverallCompleted++;
            OverallProgress = Convert.ToInt32(OverallCompleted * 100 / TotalTiles);
            if (OverallProgress != 0)
                EstimatedCompletionTime = new TimeSpan((DateTime.Now - startTime).Ticks * 100 / OverallProgress);
        }

        static public void Reset()
        {
            TotalTiles = 0;
            OverallCompleted = 0;
            OverallProgress = 0;
        }

        static public void StartTimer()
        {
            startTime = DateTime.Now;
        }
    }

    internal class WorkerArgs
    {
        public GMapProvider[] Providers;
        public int TileSleeper;
        public RectLatLng Area;
        public int MinZoom;
        public int MaxZoom;
    }
    #endregion
}
And here is my single call to begin pre-fetching:
if (MainMap.Manager.Mode == AccessMode.ServerAndCache)
{
    RectLatLng area = MainMap.CurrentViewArea;
                        
    DialogResult res = MessageBox.Show("This process will pre-fetch (download) mapping\n" +
                                        "data to your computer for use offline.  Your\n" +
                                        "current screen view area will be downloaded at\n" +
                                        "all possible zoom levels.\n\n" +
                                        "This process may take up to several hours\n" +
                                        "depending on the size of the geographical area.\n\n" +
                                        "Proceed?",
                                        "Map Data", MessageBoxButtons.YesNo);
    if (res == DialogResult.Yes)
    {
        PrefetchTiles obj = new PrefetchTiles();
        obj.ShowCompleteMessage = true;
        obj.Start(area, MainMap.MinZoom, MainMap.MaxZoom, new GMapProvider[] { GMapProviders.BingMap, GMapProviders.BingHybridMap }, 100, MainMap.CacheLocation);
    }
}
else
    MessageBox.Show("Can't perform this action while offline.", "Map Data", MessageBoxButtons.OK);
Coordinator
Jan 9, 2012 at 12:19 PM
Edited Jan 9, 2012 at 12:19 PM

p.s. to be sure you can use map.Manager.OnTileCacheComplete event to know when all tile data is flushed to the disk