Configuring CI Using GitLab and Nx
There are two general approaches to setting up CI with Nx - using a single job or distributing tasks across multiple jobs. For smaller repositories, a single job is faster and cheaper, but once a full CI run starts taking 10 to 15 minutes, using multiple jobs becomes the better option. Nx Cloud's distributed task execution allows you to keep the CI pipeline fast as you scale. As the repository grows, all you need to do is add more agents.
Process Only Affected Projects With One Job on GitLab
Below is an example of an GitLab setup that runs on a single job, building and testing only what is affected. This uses the nx affected command to run the tasks only for the projects that were affected by that PR.
1image: node:18
2
3stages:
4 - lint
5 - test
6 - build
7
8.distributed:
9 interruptible: true
10 only:
11 - main
12 - merge_requests
13 cache:
14 key:
15 files:
16 - package-lock.json
17 paths:
18 - .npm/
19 before_script:
20 - npm ci --cache .npm --prefer-offline
21 - NX_HEAD=$CI_COMMIT_SHA
22 - NX_BASE=${CI_MERGE_REQUEST_DIFF_BASE_SHA:-$CI_COMMIT_BEFORE_SHA}
23
24variables:
25 GIT_DEPTH: 0
26
27format-check:
28 stage: test
29 extends: .distributed
30 script:
31 - npx nx-cloud record -- nx format:check --base=$NX_BASE --head=$NX_HEAD
32
33lint:
34 stage: test
35 extends: .distributed
36 script:
37 - npx nx affected --base=$NX_BASE --head=$NX_HEAD -t lint --parallel=3
38
39test:
40 stage: test
41 extends: .distributed
42 script:
43 - npx nx affected --base=$NX_BASE --head=$NX_HEAD -t test --parallel=3 --configuration=ci
44
45build:
46 stage: build
47 extends: .distributed
48 script:
49 - npx nx affected --base=$NX_BASE --head=$NX_HEAD -t build --parallel=3
50The build and test jobs implement the CI workflow using .distributed as a template to keep the CI configuration file more readable.
Distribute Tasks Across Agents on GitLab
To set up Distributed Task Execution (DTE), you can run this generator:
❯
npx nx g ci-workflow --ci=gitlab
Or you can copy and paste the workflow below:
1image: node:18
2
3# Creating template for DTE agents
4.dte-agent:
5 interruptible: true
6 cache:
7 key:
8 files:
9 - yarn.lock
10 paths:
11 - '.yarn-cache/'
12 script:
13 - yarn install --cache-folder .yarn-cache --prefer-offline --frozen-lockfile
14 - yarn nx-cloud start-agent
15
16# Creating template for a job running DTE (orchestrator)
17.base-pipeline:
18 interruptible: true
19 only:
20 - main
21 - merge_requests
22 cache:
23 key:
24 files:
25 - yarn.lock
26 paths:
27 - '.yarn-cache/'
28 before_script:
29 - yarn install --cache-folder .yarn-cache --prefer-offline --frozen-lockfile
30 - NX_HEAD=$CI_COMMIT_SHA
31 - NX_BASE=${CI_MERGE_REQUEST_DIFF_BASE_SHA:-$CI_COMMIT_BEFORE_SHA}
32 artifacts:
33 expire_in: 5 days
34 paths:
35 - dist
36
37# Main job running DTE
38nx-dte:
39 stage: affected
40 extends: .base-pipeline
41 script:
42 - yarn nx-cloud start-ci-run --stop-agents-after=build
43 - yarn nx-cloud record -- nx format:check --base=$NX_BASE --head=$NX_HEAD
44 - yarn nx affected --base=$NX_BASE --head=$NX_HEAD -t lint,test,build --parallel=2
45
46# Create as many agents as you want
47nx-dte-agent1:
48 extends: .dte-agent
49 stage: affected
50nx-dte-agent2:
51 extends: .dte-agent
52 stage: affected
53nx-dte-agent3:
54 extends: .dte-agent
55 stage: affected
56This configuration is setting up two types of jobs - a main job and three agent jobs.
The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs.
The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks.
Two Types of ParallelizationThe agents and the --parallel flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used.