Incident casebook · RIY-EXT-CMWS · 22 Oct 2026

Why dnf got a 403 from Red Hat

A RHEL 9 server needed a PHP update. Every dnf run died on the same line. Over about two and a half hours of a shared-screen support call, we removed suspects one at a time until only Red Hat itself was left.

server 10.10.160.133 proxy 10.10.160.90:8080 jump host 10.10.120.69 OS RHEL 9 PHP 8.2.30 (AppStream module)
The setting

Every request took four hops

The server has no direct internet. All traffic goes through one corporate proxy, and the IP in the error message is that proxy, not Red Hat. The first question was which hop was saying no.

Jump host RIY-INT-JMP01 10.10.120.69 RIY-EXT-CMWS RHEL 9 · 10.10.160.133 cert valid to 2027-08-08 Proxy 10.10.160.90:8080 tunnels TLS, no inspection cdn.redhat.com Akamai edge org 19357488 SSH ✓ HTTP CONNECT ✓ 403 Access Denied TLS ok, request refused
verified workingrefused
The suspects

Seven explanations, one survivor

Each card is a cause we considered. Select one to highlight the moments in the log that tested it.

Showing every step.
The log

How the call went

Times are from the Riyadh desktop clock in each photo. Nobody on our side could touch the server; every command was typed by the client and sent back as a phone photo of the screen.

  1. Reading the error

    dnf had a proxy set in /etc/dnf/dnf.conf, and no proxy variables in the shell. The 403 named the proxy's IP and was only 465 bytes. The proxy was clearly reachable. Something past it was saying no.

    # grep -i proxy /etc/dnf/dnf.conf
proxy=http://10.10.160.90:8080/
# dnf --setopt=proxy=http://10.10.160.90:8080 update 'php*'
  - Status code: 403 for https://cdn.redhat.com/.../repomd.xml (IP: 10.10.160.90)

    Hypothesis. The proxy decrypts HTTPS and strips the client certificate Red Hat needs. Common on government networks. This turned out to be wrong.

  2. Is the server registered?

    # subscription-manager status
Overall Status: Disabled
Content Access Mode is set to Simple Content Access.
# subscription-manager identity
system identity: cc17f676-ad58-48bb-a901-907d93d72b59
org ID: 19357488

    Ruled out. "Disabled" looks alarming but is the normal status line under Simple Content Access. The server is registered.

  3. Other repos work through the same proxy

    # subscription-manager config --list | egrep 'proxy_hostname|proxy_port'
   proxy_hostname = 10.10.160.90
   proxy_port = 8080
# dnf upgrade --refresh "php*"
GetPageSpeed packages for Enterprise Linux 9     219 kB/s | 702 kB
Extra Packages for Enterprise Linux 7             76 kB/s |  17 MB
Red Hat Enterprise Linux 9 - BaseOS             403 on every file

    Narrowed. General HTTPS through the proxy works. Only the one site that requires a client certificate fails. Side finding: an EPEL 7 repo was configured on a RHEL 9 box.

  4. The decisive curl

    We sent the same request with the server's entitlement certificate and Red Hat's CA, and read the TLS handshake.

    # curl -v -x http://10.10.160.90:8080 --cert "$C" --key "$E" \
    --cacert /etc/rhsm/ca/redhat-uep.pem https://cdn.redhat.com/.../repomd.xml
> CONNECT cdn.redhat.com:443 HTTP/1.1
< HTTP/1.1 200 Connection established
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
*  issuer: O=Red Hat, Inc.; CN=Red Hat Entitlement Operations Authority
*  SSL certificate verify ok.
< HTTP/1.1 403 Forbidden
<TITLE>Access Denied</TITLE>

    Turning point. The proxy only opened a tunnel. Inside it, the server spoke directly to Red Hat with a valid certificate. The refusal came from Red Hat's side. The SSL-inspection theory was dropped.

  5. Trying to go around it

    # dnf upgrade 'php*' --disablerepo='rhel-9-*'
Dependencies resolved.
Nothing to do.

    Dead end. Later rpm -qa showed why: every PHP package was 8.2.30-1.module+el9.7, from Red Hat AppStream. Its updates could only come from the blocked CDN.

  6. The clock

    # date
Thu Oct 22 09:38:42 AM EDT 2026

    Ruled out. 09:38 EDT is 13:38 UTC, which is 16:38 in Riyadh, matching the desktop clock.

    Slip: four commands were pasted as one line. Their line breaks were lost, so everything after the first grep became its arguments. From here on, one command per paste.

  7. Reading the certificate

    # rct cat-cert /etc/pki/entitlement/6054126864648727161.pem | grep -Ei 'start date|end date|...'
    Start Date: 2026-08-08 10:18:45+00:00
    End Date: 2027-08-08 10:18:45+00:00
    Name: Content Access

    Valid dates. list --consumed said "No consumed subscription pools", which is also expected under Simple Content Access.

    Slip: the first version of this grep included type:, which matched every repo in the certificate and flooded the screen with Type: yum.

  8. Does the certificate cover BaseOS?

    # rct cat-cert ...pem | grep -F -e '/content/dist/rhel9/$releasever/x86_64/baseos/os' \
                              -e '/content/dist/rhel9/$releasever/x86_64/appstream/os'
    URL: /content/dist/rhel9/$releasever/x86_64/appstream/os
    URL: /content/dist/rhel9/$releasever/x86_64/baseos/os

    Ruled out. The certificate allows exactly what dnf asks for. Every check on the client's side now passed.

  9. Akamai's reference number

    # curl --proxy 10.10.160.90:8080 https://cdn.redhat.com/.../repomd.xml
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
# curl --proxy 10.10.160.90:8080 https://cdn.redhat.com/.../repomd.xml -k
<H1>Access Denied</H1>
Reference #18.ecc11302.1792680174.3bbb612c
https://errors.edgesuite.net/18.ecc11302.1792680174.3bbb612c

    Red Hat side. The "self-signed" error is normal: cdn.redhat.com uses Red Hat's private CA, which is not in the system store. The Access Denied page is served by Akamai, the CDN in front of Red Hat. The reference number is what Red Hat support can trace.

  10. Plan B: PHP from a public repo

    The team drafted an EPEL 9 + Remi switch to the php:remi-8.2 stream. The review added two corrections: put --disablerepo='rhel-*' on every dnf line, not just the upgrade, because dnf aborts on the 403 otherwise; and remove the stray EPEL 7 repo first.

    Exposure. A Notepad window on the shared, recorded screen showed a password in plain text. It should be rotated.

  11. A colleague updated PHP another way

    Using dnf against a different package source, so the update no longer depended on the Red Hat CDN. Which source exactly is still to be confirmed with dnf list installed 'php*' and rpm -qi php-cli.

Where it stands

PHP is updated. The 403 is not fixed.

Verified fine

  • Proxy tunnels cdn.redhat.com without inspecting it
  • dnf and rhsm both point at 10.10.160.90:8080
  • Registered to org 19357488, Simple Content Access
  • Server clock correct
  • Entitlement certificate valid and covers BaseOS and AppStream

Still open

  • Red Hat case: org 19357488, system cc17f676-…d72b59, the 403 URL, Akamai reference, and the proxy's public IP from curl -x http://10.10.160.90:8080 https://ifconfig.me
  • Check console.redhat.com for an active RHEL subscription or an account hold
  • Kernel, OpenSSL and the rest of the OS still get no updates
  • Replace the EPEL 7 repo with EPEL 9
  • Rotate the password shown on screen
  • Record which source the PHP update came from
Lessons

What to do first next time

  1. Run the verbose curl with the entitlement cert first. One command shows the proxy's CONNECT reply, the TLS issuer and the CDN's answer. It would have split proxy from Red Hat in minutes.

  2. Under Simple Content Access, "Disabled" and "no consumed pools" are normal. A valid certificate proves registration, not that the account may download.

  3. Find where the package came from before planning the update. rpm -qa 'php*' showing module+el9 meant the fix lived behind the blocked CDN.

  4. Over photos and remote desktops, send one command per paste and avoid greps that can match hundreds of lines.

  5. Keep secrets off a shared screen. Close Notepad and password managers before sharing.