Neo4j EC2 Deployment Guide

Neo4j EC2 Deployment Guide

This guide covers deploying Neo4j on EC2 via FSD and connecting to it.

Prerequisites

1. FSD CLI installed
2. AWS credentials configured for stage-services/prod-services accounts
3. Neo4j password secret in Secrets Manager (format: neo4j/<password>)

# Create secret if it doesn't exist
aws secretsmanager create-secret \
  --name "stage-consumer-graph-neo4j-password" \
  --secret-string "neo4j/YourSecurePassword123!" \
  --region us-east-1 \
  --profile stage-services

Deploy Neo4j EC2 Instance

Via GitHub Actions (Recommended)

Trigger the “Deploy Neo4j EC2” workflow from the Actions tab. Select the environment (stage/prod) and action (deploy/destroy).

The workflow automatically:

1. Runs fsd service ec2 deploy
2. Adds security group rules for ports 7687 (Bolt) and 7474 (HTTP)
3. Updates the {env}-consumer-graph-neo4j-uri secret in Secrets Manager with the new IP

After the workflow completes, redeploy the worker to pick up the new URI from Secrets Manager.

Via CLI (Manual)

# Stage
fsd service ec2 deploy --env stage --account stage-services --region us-east-1 consumer-graph-neo4j-ec2.yml

# Production
fsd service ec2 deploy --env prod --account prod-services --region us-east-1 consumer-graph-neo4j-ec2.yml

Note: CLI deploys do not automatically configure security groups or update the URI secret. Use the GitHub Actions workflow instead, or manually run the post-deploy steps (see Troubleshooting).

Verify Deployment

1. Get the EC2 Private IP

aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=stage-consumer-graph-neo4j*" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress,State.Name]' \
  --output table \
  --region us-east-1 \
  --profile stage-services

2. Test Port Connectivity

From VPN or another service in the VPC (replace <PRIVATE_IP> with actual IP):

# Bolt protocol (used by worker)
nc -zv <PRIVATE_IP> 7687

# HTTP (used by browser)
nc -zv <PRIVATE_IP> 7474

Connecting to Neo4j

Browser Access (Neo4j Browser UI)

1. Connect to Fetch VPN (Cloudflare WARP or Fetch-Admin)
2. Open in browser (replace with actual private IP):

   http://10.2.8.53:7474

3. Login with:
   • Username: neo4j
   • Password: Value from Secrets Manager (without the neo4j/ prefix)

Consumer Graph Worker Connection

The worker reads the Neo4j URI from Secrets Manager via the NEO4J_URI environment variable (injected by FSD). This is automatically updated by the “Deploy Neo4j EC2” GitHub Actions workflow.

The config/unified-worker-{env}.yaml files contain a hardcoded URI as a local-dev fallback, but in ECS the NEO4J_URI env var takes precedence (see cmd/unified-worker/config.go).

The password is injected via the NEO4J_PASSWORD environment variable from Secrets Manager.

SSH Access (via SSM)

To connect directly to the EC2 instance:

# Get instance ID
aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=stage-consumer-graph-neo4j" \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text \
  --region us-east-1 \
  --profile stage-services

# Connect via SSM
aws ssm start-session \
  --target <instance-id> \
  --region us-east-1 \
  --profile stage-services

Once connected:

# Check Neo4j status
sudo systemctl status neo4j

# View Neo4j logs
sudo journalctl -u neo4j -f

# View user-data installation log
cat /var/log/user-data.log

Redeploy Worker

After Neo4j is running, redeploy the worker to pick up the new URI from Secrets Manager:

fsd service ecs deploy --env stage --account stage-services --region us-east-1 consumer-graph-worker.yml

No config file changes needed — the worker reads NEO4J_URI from Secrets Manager at task startup.

Cleanup Old ECS Neo4j (Optional)

Once EC2 Neo4j is verified working:

fsd service ecs delete --env stage --account stage-services consumer-graph-worker-db.yml

Manual Neo4j Installation (if user-data fails)

If the automated user-data script fails (check /var/log/user-data.log), you can manually install Neo4j via SSM.

1. Install SSM Plugin (one-time setup)

# macOS
brew install --cask session-manager-plugin

2. Connect to the EC2 Instance

# Find instance ID
aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=stage-consumer-graph-neo4j*" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress]' \
  --output table \
  --region us-east-1 \
  --profile stage-services

# Connect via SSM
aws ssm start-session \
  --target i-xxxxxxxxxx \
  --region us-east-1 \
  --profile stage-services

3. Format and Mount Data Volume

sudo mkfs -t ext4 /dev/xvdf
sudo mkdir -p /data
sudo mount /dev/xvdf /data
echo '/dev/xvdf /data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab

4. Install Java 17

sudo yum install -y java-17-amazon-corretto java-17-amazon-corretto-devel

5. Add Neo4j Repository

sudo rpm --import https://debian.neo4j.com/neotechnology.gpg.key

echo -e "[neo4j]\nname=Neo4j RPM Repository\nbaseurl=https://yum.neo4j.com/stable/5\nenabled=1\ngpgcheck=1\ngpgkey=https://debian.neo4j.com/neotechnology.gpg.key" | sudo tee /etc/yum.repos.d/neo4j.repo

6. Install Neo4j

sudo yum install -y neo4j-5.15.0

7. Create Data Directories

sudo mkdir -p /data/neo4j/data /data/neo4j/logs /data/neo4j/transactions
sudo chown -R neo4j:neo4j /data/neo4j

8. Configure Neo4j

echo -e "server.directories.data=/data/neo4j/data\nserver.directories.logs=/data/neo4j/logs\nserver.directories.transaction.logs.root=/data/neo4j/transactions\nserver.default_listen_address=0.0.0.0\nserver.bolt.listen_address=0.0.0.0:7687\nserver.http.listen_address=0.0.0.0:7474\nserver.memory.heap.initial_size=4g\nserver.memory.heap.max_size=8g\nserver.memory.pagecache.size=4g\ndbms.security.auth_enabled=true" | sudo tee /etc/neo4j/neo4j.conf

9. Set Password and Start Neo4j

# Get password from Secrets Manager (strips neo4j/ prefix)
PASSWORD=$(aws secretsmanager get-secret-value --secret-id "stage-consumer-graph-neo4j-password" --region us-east-1 --query SecretString --output text | sed 's|neo4j/||')

# Set initial password
sudo neo4j-admin dbms set-initial-password "$PASSWORD"

# Enable and start Neo4j
sudo systemctl enable neo4j
sudo systemctl start neo4j

# Verify it's running
sudo systemctl status neo4j

10. Add Security Group Rules (Automated via GitHub Actions)

The “Deploy Neo4j EC2” workflow automatically adds ports 7687 and 7474 after deploy. If you deployed via CLI instead, add them manually:

SG_ID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=stage-consumer-graph-neo4j*" \
  --query 'Reservations[].Instances[].SecurityGroups[].GroupId' \
  --output text \
  --region us-east-1 \
  --profile stage-services)

aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp \
  --port 7474 --cidr 10.0.0.0/8 --region us-east-1 --profile stage-services
aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp \
  --port 7687 --cidr 10.0.0.0/8 --region us-east-1 --profile stage-services

Verify Neo4j is Working

Via Browser

1. Connect to Fetch VPN
2. Open http://<private-ip>:7474 (e.g., http://10.2.8.53:7474)
3. Login with username neo4j and password from Secrets Manager (without neo4j/ prefix)

Get Password from Secrets Manager

aws secretsmanager get-secret-value \
  --secret-id "stage-consumer-graph-neo4j-password" \
  --region us-east-1 \
  --query SecretString \
  --output text \
  --profile stage-services

Sample Cypher Queries

Once logged into Neo4j Browser, test with:

// Check database status
SHOW DATABASES;

// Count all nodes
MATCH (n) RETURN count(n) AS nodeCount;

// Create a test node
CREATE (t:Test {name: 'connectivity-test', created: datetime()}) RETURN t;

// Verify and clean up
MATCH (t:Test {name: 'connectivity-test'}) DELETE t;

Troubleshooting

Worker can’t connect to Neo4j

1. Check DNS resolves: nslookup stage-consumer-graph-neo4j...
2. Check port is open: nc -zv ... 7687
3. Check Neo4j is running: SSH in via SSM and run systemctl status neo4j
4. Check password matches: Verify secret value matches what Neo4j was initialized with

Neo4j won’t start

SSH into the instance and check:

# Check user-data ran successfully
cat /var/log/user-data.log

# Check Neo4j logs
sudo journalctl -u neo4j --no-pager -n 100

# Check data volume is mounted
df -h /data

Need to reset Neo4j password

# Stop Neo4j
sudo systemctl stop neo4j

# Reset password
sudo neo4j-admin dbms set-initial-password "newpassword"

# Start Neo4j
sudo systemctl start neo4j

# Update Secrets Manager to match
aws secretsmanager update-secret \
  --secret-id "stage-consumer-graph-neo4j-password" \
  --secret-string "neo4j/newpassword" \
  --region us-east-1

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         VPC (10.0.0.0/8)                        │
│                                                                 │
│  ┌──────────────────┐              ┌──────────────────────────┐ │
│  │  ECS (Worker)    │──── TCP ────▶│  EC2 (Neo4j)             │ │
│  │                  │    7687      │                          │ │
│  │  consumer-graph  │              │  Ports:                  │ │
│  │  -worker         │              │  - 7687 (Bolt)           │ │
│  │                  │              │  - 7474 (HTTP)           │ │
│  └──────────────────┘              │                          │ │
│                                    │  Storage:                │ │
│  ┌──────────────────┐              │  - /data (EBS gp3)       │ │
│  │  Browser (VPN)   │──── HTTP ───▶│                          │ │
│  │                  │    7474      └──────────────────────────┘ │
│  └──────────────────┘                                           │
│                                                                 │
│  DNS: {env}-consumer-graph-neo4j.us-east-1.{account}.fetchrewards.com │
└─────────────────────────────────────────────────────────────────┘