Key value character limitation


#1

i want to set key value pairs like so:

https://example.com?id=1 > {example:"here is data"}

But, I’m getting an error in the log:
Error: The "key" parameter may contain only the following characters: [a-zA-Z0-9!-_.'()

This character restriction is not mentioned in the documentation at https://sdk.apify.com/docs/api/keyvaluestore

Also, this really puts a big limit on characters. My only idea is to MD5 the data:

crypto = require('crypto');
urlkey = crypto.createHash('md5').update(request.url).digest('hex')
await store.setValue(urlkey, await page.content());

it probably doesn’t matter, but this makes the keys illegible to me without running the script. but it also seems like a lot of extra effort to create the key. Is there a better way to do this?


#2

You’re right, there is this restriction on key names, we’ll update the documentation. The reason for this restriction is that if we allowed e.g. “/” chars to be in the name of the key, then it would cause issues when accessing the key record using API, since “/” character has a special meaning in URLs. Of course, people could URL-encode the key name in URL, but some browser automatically change it back and the API calls wouldn’t work then.


#3

BTW, one way to handle your use case it to use a dataset to create a record for each URL stored in the key-value store. For example:

await Apify.pushData({
  url: 'https://example.com?id=1',
  key: 'some-key-1',
  some: 'Other info',
})

#4

I think I didn’t realize I could just convert to hex. The following works fine without crypto library:

Buffer.from(request.url,'utf8').toString('hex');

ex: https://example.com becomes 68747470733a2f2f6578616d706c652e636f6d


#5

That’s definitely possible, just beware the maximum length of the key is 256 characters.