Recommendations on structure of bigger Actor Projects


#1

I wonder if there is kind of architecture for more complex actor projects.
I scrap many different sites and write a specific actor for each individual site.

This is the way I do it by now:

  1. I copy another actor project
  2. rename the project folder, rename the description of the actor in the apify.json and package.json file
  3. install the apify package and all the other npm-packages I use
  4. modify the main.js file

I have coupled some of the more general functionality in a local project that I import using require.

If I want to use apify cloud and the github integration for automated builds I would have to initialize a github repo on each of my actor projects.

But this process seems like a bad project structure with a lot of equal files.

I thought I could merge all actor-projects into one. Implementing each actor in a different file (actorXY.js, actorWZ, etc.) and modify the package.json file to run multiple files on start.

Would you suggest this way or is there a more elegant way to organize complex Actor projects on apify cloud?


#2

Hi,

it is up to you. But from my experience, the best way to initiate new projects is by using Apify CLI.

In some project, I need to share some code between actors. For these use cases, I created a simple bash script, which builds local npm package. This package is shared between actors. You can check the how-to-share-code-between-apify-actors repo on github.


#3

You can build and run actors from subfolders and even branches of Git repositories, so you can keep a single repo and just structure your actors into folders. See docs.