• March 2015
    M T W T F S S
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    3031  
  • Latest Posts

  • Latest Comments

  • Archives

Batching Items with EF 6.0

Headline: It’s not too hard to batch items with or without async capabilities in Entity Framework (EF) 6.0.

Here I have a large number (ok… not in the sample test, but in real life) of items and I’d like to pull them in as in batches rather than wait for all of them to be returned. I’ll go ahead and use async here, even though I know I’m not using it properly…

So here is my sample. It reads 3 items at t time and adds them to the collection. With a little rearranging of the code you could have a decent solution.

Batch Sample
  1. [TestMethod]
  2. public void Batching_Items()
  3. {
  4.     // Arrange
  5.     const int expected = 7;
  6.     var take = 3;
  7.     var skip = 0;
  8.     var locations = new Collection<FieldLocation>();
  9.     using (var repository = new MyRepository())
  10.     {
  11.         repository.Context.Database.Log = Console.Write;
  12.         var count = repository.Items.Count();
  13.         var remaining = count;
  14.  
  15.         // Act
  16.         while (remaining > 0)
  17.         {
  18.             var task = repository.Items.OrderBy(f => f.ReservoirId).Skip(skip).Take(take).ToListAsync();
  19.             foreach (var result in task.Result)
  20.                 locations.Add(result);
  21.             skip += take;
  22.             remaining = count – skip;
  23.             if (take > remaining) take = remaining;
  24.         }
  25.  
  26.         // Assert
  27.         var actual = locations.Count;
  28.         Assert.AreEqual(expected, actual);
  29.     }
  30. }

You can see that the key is the use of the Skip and Take items (and Skip requires the OrderBy). If that code is isolated in a different method then you could use the async to better advantage.

And here’s what happened in the SQL Server:

Test Name:    Batching_Items
Test Outcome:    Passed
Result StandardOutput:   
Opened connection at 3/14/2015 2:36:37 PM -06:00
SELECT
    [GroupBy1].[A1] AS [C1]
    FROM ( SELECT
        COUNT(1) AS [A1]
        FROM (SELECT
    [FieldLocationView].[ReservoirCodeId] AS [ReservoirCodeId],
    [FieldLocationView].[Latitude] AS [Latitude],
    [FieldLocationView].[Longitude] AS [Longitude]
    FROM [dbo].[FieldLocationView] AS [FieldLocationView]) AS [Extent1]
    )  AS [GroupBy1]
— Executing at 3/14/2015 2:36:38 PM -06:00
— Completed in 158 ms with result: SqlDataReader

Closed connection at 3/14/2015 2:36:38 PM -06:00
Opened connection asynchronously at 3/14/2015 2:36:38 PM -06:00
SELECT
    [Extent1].[ReservoirCodeId] AS [ReservoirCodeId],
    [Extent1].[Latitude] AS [Latitude],
    [Extent1].[Longitude] AS [Longitude]
    FROM (SELECT
    [FieldLocationView].[ReservoirCodeId] AS [ReservoirCodeId],
    [FieldLocationView].[Latitude] AS [Latitude],
    [FieldLocationView].[Longitude] AS [Longitude]
    FROM [dbo].[FieldLocationView] AS [FieldLocationView]) AS [Extent1]
    ORDER BY [Extent1].[ReservoirCodeId] ASC
    OFFSET 0 ROWS FETCH NEXT 3 ROWS ONLY
— Executing asynchronously at 3/14/2015 2:36:38 PM -06:00
— Completed in 145 ms with result: SqlDataReader

Closed connection at 3/14/2015 2:36:38 PM -06:00
Opened connection asynchronously at 3/14/2015 2:36:38 PM -06:00
SELECT
    [Extent1].[ReservoirCodeId] AS [ReservoirCodeId],
    [Extent1].[Latitude] AS [Latitude],
    [Extent1].[Longitude] AS [Longitude]
    FROM (SELECT
    [FieldLocationView].[ReservoirCodeId] AS [ReservoirCodeId],
    [FieldLocationView].[Latitude] AS [Latitude],
    [FieldLocationView].[Longitude] AS [Longitude]
    FROM [dbo].[FieldLocationView] AS [FieldLocationView]) AS [Extent1]
    ORDER BY [Extent1].[ReservoirCodeId] ASC
    OFFSET 3 ROWS FETCH NEXT 3 ROWS ONLY
— Executing asynchronously at 3/14/2015 2:36:38 PM -06:00
— Completed in 76 ms with result: SqlDataReader

Closed connection at 3/14/2015 2:36:39 PM -06:00
Opened connection asynchronously at 3/14/2015 2:36:39 PM -06:00
SELECT
    [Extent1].[ReservoirCodeId] AS [ReservoirCodeId],
    [Extent1].[Latitude] AS [Latitude],
    [Extent1].[Longitude] AS [Longitude]
    FROM (SELECT
    [FieldLocationView].[ReservoirCodeId] AS [ReservoirCodeId],
    [FieldLocationView].[Latitude] AS [Latitude],
    [FieldLocationView].[Longitude] AS [Longitude]
    FROM [dbo].[FieldLocationView] AS [FieldLocationView]) AS [Extent1]
    ORDER BY [Extent1].[ReservoirCodeId] ASC
    OFFSET 6 ROWS FETCH NEXT 1 ROWS ONLY
— Executing asynchronously at 3/14/2015 2:36:39 PM -06:00
— Completed in 161 ms with result: SqlDataReader

Closed connection at 3/14/2015 2:36:39 PM -06:00

So, with this approach you could easily build an application that will progressively load your items.

Cheers,

Karl

Leave a Comment