Sign In Register

How can we help you today?

Start a new topic
Answered

Iterating through Game Data Service in Cloud Script and cursor size limit of 100

Hello,


I'm trying to iterate through every document with Cloud Script with code similar to this: 

var q = API.queryItems("WEAPONS", null);
var weaponsCounter = 0;

while(q.cursor().hasNext())
{
    var entry = q.cursor().next();
    var id = entry.getId();
    var weaponData = entry.getData();
    weaponsCounter++;

    // do stuff with weapons here
}

 The problem is the fact that it breaks when I have more than 100 documents, because it seems to return at most 100 items. Therefore, some documents are not counted nor processed by my script.


Is there a way to somehow process every document of given data type?


Regards,


-- 

Bartosz Chodorowski



Best Answer

Hi Bartosz,


GameData services can only return a maximum of 100 documents. In order to get all items from game data services you would have to do some pagination.This be can done through the following steps.


Step 1: Sort the data using Spark.getGameDataService().sort() //For example sort by id

Step 2: Perform your initial query on the database

Step 3: Store all the returned documents into an array or keep track of the count //this will be your list for all your items

Step 4: Perform your changes to the first returned 100 documents

Step 5: After the query is completed i.e 100 documents have returned. Check to see if the arrays length is divisible by 100 i.e array.length % 100 == 0 //This means that there can be more documents to be returned

Step 6: Create a new query that is greater than the last index of the array sorted field.// For example if you are sorting by id and the last index id is 100 than the query will be retrieving all id's between 101 and 200


Here is an example of how to do pagination in cloud code you will just have to add your custom updates for the documents in the query while loop. 

Just a note it would be a good idea to put in a hard limit on the amount of documents you want to return. This is because its rather expensive getting all the documents back especially if there are thousands of documents and you would be at risk that the request might timeout. The limit will depend on your use case on how many documents you actually need.


I hope this helps you out if you have any other questions don't hesitate to contact us.


Regards,

Garreth.


Answer

Hi Bartosz,


GameData services can only return a maximum of 100 documents. In order to get all items from game data services you would have to do some pagination.This be can done through the following steps.


Step 1: Sort the data using Spark.getGameDataService().sort() //For example sort by id

Step 2: Perform your initial query on the database

Step 3: Store all the returned documents into an array or keep track of the count //this will be your list for all your items

Step 4: Perform your changes to the first returned 100 documents

Step 5: After the query is completed i.e 100 documents have returned. Check to see if the arrays length is divisible by 100 i.e array.length % 100 == 0 //This means that there can be more documents to be returned

Step 6: Create a new query that is greater than the last index of the array sorted field.// For example if you are sorting by id and the last index id is 100 than the query will be retrieving all id's between 101 and 200


Here is an example of how to do pagination in cloud code you will just have to add your custom updates for the documents in the query while loop. 

Just a note it would be a good idea to put in a hard limit on the amount of documents you want to return. This is because its rather expensive getting all the documents back especially if there are thousands of documents and you would be at risk that the request might timeout. The limit will depend on your use case on how many documents you actually need.


I hope this helps you out if you have any other questions don't hesitate to contact us.


Regards,

Garreth.


1 person likes this

Hi Garreth!


I appreciate your input. The idea is clear to me and we might use it in our cases, thank you.


However, the example code you provided seems to have a bug where it spinlocks if the number of documents in the database is divisible by 100. In that case, hasMore variable is always true and the loop never stops.


Regards,


-- 

Bartosz Chodorowski

Hi.


How can I implement pagination without id's that are numerical?


We are trying to implement a league system and need to segregate all the players into sub leagues.


Thanks.


Stefan.

Hi Stefan,


Please note that IDs don't have to be numerical, they can be strings as well. You can then use Spark.getGameDataService().S("id") instead of Spark.getGameDataService().N("id"). Also make sure that you store IDs in "id" field in every document.


Hope that helps a little bit,


Regards,


-- 

Bartosz Chodorowski


Login to post a comment