Self-hosted Sonatype Nexus repository OSS

IT May 2, 2024

In my homelab I run "many" VM's and LXC containers with Debian and Ubuntu as operating system. Those OS's need to be kept up to date regularly. My internet connection is pretty fast, currently I have a symmetric 200 Mbit fiber connection, but I thought it still a waste to download all those Apt packages multiple times. Couldn't they be cached locally instead?

I had seen that on AWS EC2 and Lightsail instances Apt updates and packages would be downloaded from an AWS domain, which suggested they run their own Apt server. This would definitely make sense. Now, I'm not a cloud provider, so my savings will be much smaller. Still some caching would be cool.

With Apt you can run your own mirror. This means downloading the full apt repository (for your specific distirbution) and keeping it in sync with (mirrored) the upstream repository. I don't know how big this repository is, but this looked also wasteful to me. I only need some packages, not all of them.

That's when I found Sonatype Nexus Repository. As the name suggest, this is a repository manager. You can create private repositories, but it can also do "proxy" repositories. The latter is what I was looking for. You configuring your server to connect to this proxy, which in turn connects to the upstream. The proxy caches packages and pulls them from the upstream transparently. The client doesn't see the difference. Nexus can do a lot more than just Apt. I ended up configuring Apt, Docker, Npm and Nuget repositories.

Installation

I choose to run Nexus using Docker with an NFS share to my NAS to store all the repository data. I run Docker in an LXC container which made adding the NFS share a little bit more challenging. But in the end it works beautifully.

The compose file is really simple. Define the volume mapping for the NFS share with the Nexus data. And open two ports: 9081 for the WebGUI, 18444 (or any other valid port number) for a private Docker repository.

services:
  nexus:
    image: sonatype/nexus3
    volumes:
      - "/mnt/nexus-data:/nexus-data"
    ports:
      - "9081:8081"
      - "18444:18444"

Initial configuration

DNS + Reverse proxy

In an upcoming post I will go into more detail about my current setup with regards to DNS, TLS certificates, firewall and reverse proxies. I use PiHole as my DNS server in which I can enter "Local DNS records". In here I pointed nexus.joeplaa.com to the IP address of my reverse proxy.

💡

In some screenshots and examples I also use sonatype.jodibooks.com. This is the same Nexus installation, which I initially used for jodiBooks. Later I added a DNS entry and reverse proxy for joeplaa.com. But basically for me they are interchangeble. It does matter in the reverse proxy location blocks though. There the path should match the repository name.

The reverse proxy terminates the internal https connection and passes the request on to the VM running Docker on port 9081. I use Nginx as my reverse proxy of choice and below is the config I use for Nexus (the order of location blocks is important!). I plan on explaining more about all the Nginx settings I use in another post, however for now this was my main source.

upstream backend_nexus {
    zone upstreams 64K;
    keepalive 2;
    server 10.33.30.21:9081 max_fails=1 fail_timeout=2s;
}

upstream backend_nexus_docker {
    zone upstreams 64K;
    keepalive 2;
    server 10.33.30.21:18444 max_fails=1 fail_timeout=2s;
}

server {
    listen 80;
    server_name nexus.joeplaa.com sonatype.joeplaa.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name nexus.joeplaa.com sonatype.joeplaa.com;

    # Allow internal
    allow 10.33.30.0/24;
    # Deny external
    deny all;

    # https://help.sonatype.com/en/running-behind-a-reverse-proxy.html
    keepalive_timeout  5 5;
    proxy_send_timeout 120;
    proxy_read_timeout 300;
    proxy_buffering    off;
    proxy_request_buffering off;

    # allow large uploads of files
    client_max_body_size 2G;
    # optimize downloading files larger than 1G
    proxy_max_temp_file_size 2048m;

    location ~ /npm-joeplaa/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://backend_nexus;
        proxy_next_upstream error timeout http_500;
    }

    location ~ /joeplaa/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        if ($http_user_agent ~* docker) {
            proxy_pass http://backend_nexus_docker;
        }

        proxy_pass http://backend_nexus;
        proxy_next_upstream error timeout http_500;
    }

    location /v2 {
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://backend_nexus/repository/docker-group$request_uri;
        proxy_next_upstream error timeout http_500;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://backend_nexus/;
        proxy_next_upstream error timeout http_500;
    }

    # Reverse proxy settings
    real_ip_header X-Forwarded-For;
    set_real_ip_from 127.0.0.1;
    set_real_ip_from 10.33.30.0/24;

    # Logs
    access_log /var/log/nginx/nexus-joeplaa-com_access.log main;
    error_log /var/log/nginx/nexus-joeplaa-com_error.log warn;

    # SSL
    ssl_certificate /home/joeplaa/cert/joeplaa.com.fullchain;
    ssl_certificate_key /home/joeplaa/cert/joeplaa.com.all.pem;
    ssl_trusted_certificate /home/joeplaa/cert/joeplaa.com.fullchain;
}

Users + permissions + Realms

Open a browser and goto the address specified for your Nexus instance, in my case nexus.joeplaa.com. You should be greeted with a website.

To log in you need to find the admin password first. This is a randomly generated password stored in the "data directory ($data-dir, typically sonatype-work/nexus3) in a file called admin.password" (https://help.sonatype.com/en/post-install-checklist.html#1---change-the-admin-password). In our case /mnt/nexus-data.

I'm writing this after using Nexus for 2-3 years now, so I don't remember if there was an onboarding wizard. If so, nice! If not, we have to do the configuration ourselves step by step. I hope I get all the steps in the correct order. As said, I'm now copying my working configuration, which took me a long time to converge on.

Click the settings sprocket next to the search bar and go to "Security".
Anonymous Access. We don't want our servers to have to login everytime. Reading is always allowed. That's also a reason why I don't expose this service to the internet.

Roles. There should be an nx-admin and nx-anonymous role. Create a new role user.

Realms: I will be using Docker and npm, so added these too. Local Authentication is a must.

Email server

(Click the settings sprocket next to the search bar and) go to "System".
Email server. Enable this if you want Nexus to send emails. I run an smtp relay that forwards email I want to send to AWS SES, my email service. Settings will be different for Gmail or other mail services.

Blob store

(Click the settings sprocket next to the search bar and) go to "Repository".
Blob Stores. I created multiple blob stores, one for each repository type (docker-group is useless, but I can't change it now that a repository is linked). I think you can use only one, but I like to see how much data each type is consuming.

Cleanup policies. These will be used later to clean up unused packages from the proxy repositories.

Repository configuration

Finally, we're getting somewhere.

Apt proxy

Let's start with the one it all started with for me.

ℹ️

You have to create a separate repository for each version of the OS. For example, for Ubuntu I created a proxy repository for Focal, another for Jammy and another one for Noble.

⚠️

For some unexplained reason using a Nexus proxy doesn't work when using a desktop version of Ubuntu. You get File has unexpected size errors.

Click the settings sprocket next to the search bar and go to "Repository".
Click Repositories
Click the Create repository button. A long list of options is available.

We want "apt (proxy)"
Enter a name (this will be used in the url), the Distribution and the Remote storage url. In this example we use Ubuntu 24.04 noble which I want to download from my local ubuntu mirror.

As I entered an https url, I have to click the View certificate button and click Add certificate to truststore.

Scroll down an select a Blob store and Cleanup policy

Click the Create repository button.
In the overview click the copy button to get the url to the repository.

Use this address when installing Ubuntu or update /etc/apt/sources.list or /etc/apt/sources.list.d/ubuntu.sources in 24.04.

Docker group

Next on my list was a cache for docker images. The process for creating a Docker proxy is virtually the same as for Apt above, but we first need to create a Docker group repository.

Click the settings sprocket next to the search bar and go to "Repository".
Click Repositories
Click the Create repository button. A long list of options is available.

We want "docker (group)"
Enter a name, I called it docker-group. Yes, I'm very creative.
Select Allow anonymous docker pull
Select Enable Docker V1 API
Select Strict Content Type Validation
Click the Create repository button.

Docker proxy

Click the settings sprocket next to the search bar and go to "Repository".
Click Repositories
Click the Create repository button. A long list of options is available.

We want "docker (proxy)"
Enter a name (this will be used in the url). I try to use the domain of the Remote storage url (see below)
Check Allow anonymous docker pull
The Remote storage url. In this example I use https://registry-1.docker.io
Select Use the Nexus Repository truststore. As I entered an https url, I have to click the View certificate button and click Add certificate to truststore
Select "Use proxy registry" as Docker index

Also check Auto blocking enabled
Select a Blob store
Select Strict Content Type Validation and Not found cache enabled.
Add a Cleanup policy

Click the Create repository button.
Add the repository to the docker group. Open the group and add it under Member repositories
Update all docker-compose files or regular docker commands to use images from nexus.joeplaa.com. For example: nexus.joeplaa.com/nginx or nexus.joeplaa.com/guacamole/guacamole.

I added multiple docker repositories this way and added them all to my docker group. Nexus will figure out which one to use, which is super convenient.

Docker hosted

If you create custom images and want to host them locally, you can create a hosted Docker repo.

Click the settings sprocket next to the search bar and go to "Repository".
Click Repositories
Click the Create repository button. A long list of options is available.
We want "docker (hosted)"

Enter a name (this will be used in the url).
As HTTP port use the port you forwarded in the docker-compose.yml file earlier. In my case that was 18444
Check Allow anonymous docker pull
Select a Blob store
Select Strict Content Type Validation

Choose "Allow redeploy" as Deployment policy
Do not add a Cleanup policy

Click the Create repository button.
Add the repository to the docker group. Open the group and add it under Member repositories.
Update all docker-compose files or regular docker commands to use images from nexus.joeplaa.com. For example: nexus.joeplaa.com/joeplaa/nginx.
In your reverse proxy add a backend to the (private) hosted docker repo.

upstream backend_nexus_docker {
    zone upstreams 64K;
    keepalive 2;
    server 10.33.30.21:18444 max_fails=1 fail_timeout=2s;
}

Also add two location blocks (order is important, see the full config). The first to send upload requests to the hosted docker repo (see part Upload custom stuff --> Docker images). The second to rewrite the url to the docker group url.

location ~ /joeplaa/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_headers_hash_max_size 512;
    proxy_headers_hash_bucket_size 128;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host:$server_port;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;

    if ($http_user_agent ~* docker) {
        proxy_pass http://backend_nexus_docker;
    }

    proxy_pass http://backend_nexus;
    proxy_next_upstream error timeout http_500;
}

location /v2 {
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_headers_hash_max_size 512;
    proxy_headers_hash_bucket_size 128;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host:$server_port;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass http://backend_nexus/repository/docker-group$request_uri;
    proxy_next_upstream error timeout http_500;
}

Nuget proxy

This is similar to the docker proxies.

Create a group
Create proxy (or proxies)
Add proxies to group
Copy group url
In your IDE / project enter the url to the Nuget group.

And/or update NuGet.Config in the main folder of the application repository.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <config>
        <add key="repositoryPath" value="$\..\packages" />
    </config>
    <packageSources>
        <add key="jodiBooks nuget-proxy" value="https://sonatype.jodibooks.com/repository/nuget-group/index.json" />
        <!-- <add key="NuGet official package source" value="https://api.nuget.org/v3/index.json" /> -->
    </packageSources>
</configuration>

Npm + Yarn proxy

This is similar to the docker proxies.

Create a group: yarn-npm-group
Create npm proxies (https://registry.npmjs.org & https://registry.yarnpkg.com)
Add proxies to the group
Copy group url
In your yarn projects add npmRegistryServer: "https://nexus.joeplaa.com/repository/yarn-npm-group/" to your .yarnrc.yml, remove yarn.lock and run yarn install. This will recreate yarn.lock with the Nexus repository instead of the default npm one.

Npm hosted

If you create custom npm packages and want to host them locally, you can create a hosted npm repo.

Click the settings sprocket next to the search bar and go to "Repository".
Click Repositories
Click the Create repository button. A long list of options is available.
We want "npm (hosted)"
Enter a name (this will be used in the url).
Select a Blob store
Select Strict Content Type Validation
Do not add a Cleanup policy
Add the repository to the yarn-npm group. Open the group and add it under Member repositories.
In your reverse proxy add a location block (order is important, see the full config) to send upload requests to the hosted npm repo (see part Upload custom stuff --> Npm packages).

location ~ /npm-jodibooks/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_headers_hash_max_size 512;
    proxy_headers_hash_bucket_size 128;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host:$server_port;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass http://backend_nexus;
    proxy_next_upstream error timeout http_500;
}

Proprietary Repositories

I haven't exposed Nexus to the internet, so this section is basically optional. However, better be prepared.

Click the settings sprocket next to the search bar and go to "Repository".
Click Proprietary Repositories
Select the hosted docker and npm repositories.

Further configuration

Now that we have our repositories set up, we can start adding users and we need to schedule some maintenance.

Roles

Click the settings sprocket next to the search bar and go to "Security".
Click Roles
Select the user we created in the beginning
Add nx-repository-view-docker-docker-jodibooks-* and nx-repository-view-npm-npm-jodibooks-* to the Applied Privileges
Add nx-anonymous to the Applied Roles

Users

Click the settings sprocket next to the search bar and go to "Security".
Click Users
Create a user with role user

Tasks

Click the settings sprocket next to the search bar and go to "System".
Click Tasks
Click Create task
Actually trigger the cleanup policies: Admin - Cleanup repositories using their associated policies

Click Create task
Compact the blob store: Admin - Compact blob store (repeat for each blob store)

Click Create task
Cleanup docker repos part 1: Docker - Delete incomplete uploads

Click Create task
Cleanup docker repos part 2: Docker - Delete unused manifests and images

Upload custom stuff

Docker images

Create a docker file for example this Nginx container to accompany a Wordpress site.

FROM sonatype.jodibooks.com/nginx:stable-alpine
LABEL maintainer="jodiBooks <sysadmin@jodibooks.com>"

# Import our settings
COPY nginx /etc/nginx

# WordPress container uses user and group `www-data` (id 82) to read, write and modify files with build in PHP engine.
# The standard Nginx container runs Nginx as user `nginx` (id 101) to read and write files. This makes setting file persmission on the host a nightmare.
# Solution: Build our own container with `www-data` (82) user and run nginx as that user (see nginx.conf). Source: https://stackoverflow.com/a/48958748
RUN set -x ; \
    addgroup -g 82 -S www-data ; \
    adduser -u 82 -D -S -G www-data www-data && exit 0 ; exit 1

Build and tag the image

docker build -t sonatype.jodibooks.com/jodibooks/jodibeautypages-nginx .
docker tag sonatype.jodibooks.com/jodibooks/jodibeautypages-nginx:latest sonatype.jodibooks.com/jodibooks/jodibeautypages-nginx:1.0.0

docker login sonatype.jodibooks.com

Push the image

docker push sonatype.jodibooks.com/jodibooks/jodibeautypages-nginx

Npm packages

Build your package
Login to repository

npm login --registry=https://sonatype.jodibooks.com/repository/npm-jodibooks/

Publish the package

npm publish --registry=https://sonatype.jodibooks.com/repository/npm-jodibooks/

Recommended for you

Homelab

Use --fastlist to decrease Backblaze costs

2 months ago • 2 min read

Homelab

Mellanox ConnectX-3 on Proxmox 8.2

2 months ago • 4 min read

Shrink qcow2 virtual disk

6 months ago • 2 min read

Use --fastlist to decrease Backblaze costs

Mellanox ConnectX-3 on Proxmox 8.2

Shrink qcow2 virtual disk