Hi Adam, 

First of all what format is this data? GameSparks uses JSON. Anything stored in for example CSV or  XML will first have to be converted.

Secondly, this is quite a substantial amount of data. The usual practice for adding this data to a collection is to use our every minute script(found in cloud code-> system-> Every Minute). 

You can use this to paginate your entries i.e entering all 100,000 objects would cause a time out. However it is possible to enter 1000 a minute or there about (You can test the upper limit and see what works for you).

The cloud code it'self should use the .findAndModify() function to enter the data in the collection: https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/

Also , for purpose of pagination , use SparkCache: https://docs.gamesparks.net/documentation/cloud-code-api/utils-cloud-code-api/sparkcache 

   This will allow you to reference the index of your last entry and continue from there in the next iteration of the loop. 

Does that answer your question Adam? 

Regards Patrick. 

Yes and the timeout is to protect our infrastructure server side. If 40,000 is your upper limit then paginate using that figure.

Make sure to update me ,or if necessary feel free to ask further questions until this is resolved. But I'm pretty sure this what you are looking for. 

Regards Patrick.  

Hi Adam,
That's to be expected, as for our security, we must have controls in place when it comes to the uploading and downloading of data. Collections of this nature are either typically grown over time  Or part of some of the larger projects here on GameSparks. As you can imagine, we deal with thousands of requests etc. daily. So this is in place to ensure quality for all users, both paying and non-  paying. 
Just for reference  , here is our usage policy, which might explain things better. http://www.gamesparks.com/fair-usage-policy/
To answer your question on an alternative:

GameSparks does not like large dumps of data(security controls). However, we can deal with a large number of small requests no problem. So maybe  you could do some parallel processing. Try multiple small uploads for instance. 

Regards Patrick.  

Hi Adam,

Just to add to this, I've written a small python program that can take json documents, with arrays of objects, and insert them into a collection. I copied your process, I created 50 json files, filled them with some data, and executed the script. Within ~7 minutes I had inserted 18,000 documents into the collection.

Here's a sample of the data I inserted:

each JSON document had 360 entries in them.

Here is the python script, it's not perfect but it gets the job done:

import requests
#pip install requests
#http://docs.python-requests.org/en/latest/user/install/#install
import json
import os
from base64 import b64encode


#this is the array we'll use to store all the entries from the files
arrayOfDocuments = []

#Get a list of all files in our directory
files = os.listdir()
for jsonDocument in files:
	#We only want the json files
	if jsonDocument[-5:] == ".json":
		print(jsonDocument)
		#open the json file, load it to a variable and append it to the array of documents we want to process
		with open(jsonDocument) as data_file:
			data = json.load(data_file)
			arrayOfDocuments.append(data)


#GameSparks User and pass
userPass = b64encode(b"username:password").decode("ascii")
#The gameId we wish to insert documents into
gameId = "295886pTJ8Xm"
#The stage we wish to insert our documents into
stage = "preview"
#The collection we wish to insert these documents into
collection = "script.dictionary"
#This is the url for mo
postURL = "https://portal.gamesparks.net/rest/games/" + gameId + "/mongo/" + stage + "/" + collection + "/insert"
#Our generated auth token
auth =  {"Authorization" : "Basic " + userPass}

#We'll use this function to insert our documents
def batchInsert(documents):
	for x in documents:
		postData = requests.post(postURL, headers = auth, data = {"document" : json.dumps(x)})
		# print (postData.json())
		print("insert successful")

print("starting")
batchInsert(arrayOfDocuments)
print("finished inserting")

So lets say you run the script as I've sent it over, in one folder put X amount of JSON documents, plus the script, then execute it. While thats doing it's thing, in another folder, place another batch of files, a copy of the script and execute that too, in 10 minutes you'll have significantly increased you're output. I have personally tested this with 5 instances of the script running, processing 50 JSON files each, with a total of 18,000 documents per 50 JSON files, and in ~7 minutes I had 90,000 documents inserted into the collection, that ain't half bad!

Obviously this will vary based on the size of your documents and internet speed, but it's very doable!

I'm also attaching the exact files/folder I used for this so you can test it for yourself. 

Hope this helps,

Shane

No problem at all. Happy to help. ! 

Regards Patrick and Shane.